Exercises give communities, states, and regions a set of essential tools to prevent, prepare for, respond to, and recover from disasters.

The Homeland Security Exercise and Evaluation Program (HSEEP) provides a set of guiding principles for exercise programs and a common approach to exercise program management, design and development, conduct, evaluation, and improvement planning.

This course introduces the basics of emergency management exercises. It also builds a foundation for subsequent exercise courses, which provide the specifics of HSEEP.

An Introduction to Exercises is designed for personnel from all emergency management sectors to understand the basic fundamentals of exercise design, development, conduct, evaluation, and improvement planning. 

This course contains eight lessons.

The remainder of Lesson 1, Introduction and Program Fundamentals, covers HSEEP fundamentals, reasons to exercise, and the types of exercises.

• Lesson 2, Exercise Management Program, delves deeper into the functionality, goals, and outcomes from the different types of discussion- and operations-based exercises, and exercise program management.
• Lesson 3, Exercise Overview, discusses multi-year exercise planning priorities, the benefits of exercise foundation, and defines the exercise cycle.
• Lesson 4, Establishing Exercise Foundation, steps through building the foundation from the prject management timeline to planning the activities based on the capabilities to be tested, and scheduling the appropriate planning meetings.
• Lesson 5, Design and Development Phase, defines key elements of the design including,the objectives and scope of the exercise and the scenario to be used. This is where required documentation is determined and developed and needed resources are identified.
• Lesson 6, Conduct, covers the actions taken for discussion- and operations-based exercises prior to, during conduct and wrap-up activities at the end of the exercise.
• Lesson 7, Evaluation, addresses evaluator selection and required documentation and methods of observation and data collection.
• Lesson 8, Improvement Planning, provides insight into corrective action and improvements following exercise evaluation analysis.

HSEEP is a consistent approach to capabilities-based exercise program management that uses a common methodology to measure progress toward building, sustaining, and delivering core capabilities. HSEEP provides a set of guiding principles for exercise programs and a common approach to exercise program management.

Exercise:

• (n) Something performed or practiced in order to develop, improve, or display a specific power or skill.
• (v) To practice in order to train, strengthen, or develop

Source: Merriam Webster's Dictionary

Exercises improve readiness by:

• Involving the whole community.
• Providing a way to evaluate operations and validate plans and capabilities.
• Reinforcing teamwork.
• Identify both capability gaps and areas for improvement.
• Demonstrating a community's resolve to prepare for disastrous and catastrophic events.

The fundamental principles of HSEEP are:

• Guided by Elected and Appointed officials
• Capability-based, Objective Driven
• Progressive Planning Approach
• Whole Community Integration
• Informed by Risk
• Common Methodology

Capabilities can be delivered with any combination of properly planned, organized, equipped, trained, and exercised personnel to achieve an intended target. 

Exercises are built around core capabilities as identified in the National Preparedness Goal (NPG). NPG identifies 32 core capabilities that are associated with the five mission areas: Prevention, Protection, Mitigation, Response, and Recovery. When capabilities are aligned with mission areas the outcome examines current capability levels and identifies gaps, or opportunities for improvement.

Using the HSEEP methodology, organizations can use exercises to focus on assessing performance against capability-based objectives. 

A progressive planning approach employs the use of various exercise types aligned to a common set of program objectives within a cycle of exercises. The complexity of the exercises increases over time.

Senior leadership positions in state, territory, tribal, and local government, and organizations of all sizes can provide direction on the principal objectives that guide each organization's approach to planning and preparedness.

The whole community approach focuses on enabling participation of a wider range of all levels of government, emergency, private, and nonprofit sectors to foster better coordination and working relationships.

Exercise planners are encouraged to engage the whole community, where appropriate, throughout exercise program management, design and development, conduct, evaluation, and improvement planning.

Whole community integration may include the entities listed below:

• Representatives from relevant disciplines, first responders or utility companies, for example
• Senior leaders or those responsible for providing support and/or resources
• Community member representation
• Representatives with administrative responsibility
• Representatives from volunteer, nongovernmental or nonprofit sectors, and social support organizations such as advocates for children, seniors, limited access and functional needs, limited language proficiency, and racially and/or ethnically diverse neighborhoods

Identifying and assessing risks and associated impacts helps organizations identify priorities, objectives, and core capabilities to be evaluated through exercises.

Planners should first identify pertinent risks, and then assess the potential impacts associated with each risk. For example, looking at a community's Threat and Hazard Identification and Risk Assessment (THIRA) is a good starting place for exercise program priorities.

HSEEP includes a common methodology for exercises that is applicable to all mission areas—prevention, protection, mitigation, response, and recovery, as defined in the National Preparedness Goal. This methodology enables organizations of divergent sizes, geographies, and capabilities to have a shared understanding of exercise program management, design and development, conduct, evaluation, and improvement planning; and fosters exercise-related interoperability and collaboration.

The Exercise Cycle is the common planning methodology provided by HSEEP for use with all exercise types. Regardless of scope or scale of an exercise, the exercise cycle will include exercise program management, design and development, conduct, evaluation, and improvement planning.

The Presidential Policy Directive 8 (PPD-8) describes the Nation's approach to national preparedness.

The National Preparedness Goal (NPG) is the cornerstone for the implementation of PPD-8. The Nation's core capabilities are identified across five mission areas: Prevention, Protection, Mitigation, Response, and Recovery. The NPG identifies the desired achievements and the goals that have been set.  Mission areas and core capabilities will be covered in depth later in this lesson.

The National Preparedness System (NPS) is the mechanism the Nation uses to build, sustain, and deliver the core capabilities to meet the NPG and meet the goal of a secure and resilient Nation. The NPS helps define, the process communities can use to reach identified preparedness goals. The NPS includes a series of five national planning frameworks, one for each mission area. Each framework provides the following guidance:

• Clarification and assignment of roles and responsibilities
• Core capabilities for the specific mission area
• Critical tasks for each of the core capabilities
• Direction for delivering the core capabilities

Through exercises, whole community stakeholders evaluate and validate plans and capabilities, and identify capability gaps and opportunities for improvement. They bring together and strengthen the whole community in its' efforts to prevent, protect against, mitigate, respond to, and recover from all types of hazards.

HSEEP provides a common approach and consistent methodology for exercise planning. It is flexible, scalable, and available to all organizations regardless of size or scope of the organization or planned activity.

Exercises help to:

• Clarify roles and responsibilities
• Improve interagency coordination
• Find resource gaps
• Develop individual performance
• Identify opportunities for improvement

The National Preparedness System (NPS) outlines a process that can be used by communities in preparedness activities to achieve the National Preparedness Goal (NPG).

NPS consists of six components, including identify and assess risk, estimate capability requirements, build and sustain capabilities, plan to deliver capabilities, validate capabilities, and review and update risks, tools, and resources. Together, these components provide communities with a consistent and reliable approach for identifying high-priority capability targets.

Advance to the next screen for descriptions of each component.

The National Preparedness Goal presents an integrated, layered, and whole community approach to preparedness. The Goal, itself, is a result of contributions from the whole community. It recognizes that everyone can contribute to and benefit from national preparedness efforts.

Utilizing and implementing core capabilities is what we, as a Nation, require in order to deal with the risks we face.

As we begin to look at the core capabilities that fall under the five mission areas, notice that three capabilities span all mission areas: Planning, Public Information and Warning, and Operational Coordination. These help to unify mission areas and, in many ways, are necessary for the success of the remaining core capabilities. In addition, many core capabilities involve more than one mission area and are displayed in multiple mission areas, as appropriate.

For additional information on Core Capabilities, visit the Preparedness Resource Library and select the FEMA Home Page, then search for Core Capabilities.

There are three core capabilities that span all five mission areas: Planning, Public Information and Warning, and Operational Coordination. Within mission areas, there are capabilities that are unique to the specific mission area, but not unrelated.

Planning: Conduct a systematic process engaging the whole community as appropriate in the development of executable strategic, operational, and/or tactical-level approaches to meet defined objectives. 

Public Information and Warning: Deliver coordinated, prompt, reliable, and actionable information to the whole community through the use of clear, consistent, accessible, and culturally and linguistically appropriate methods to effectively relay information regarding any threat or hazard, as well as the actions being taken and the assistance being made available, as appropriate.

Operational Coordination: Establish and maintain a unified and coordinated operational structure and process that appropriately integrates all critical stakeholders and supports the execution of core capabilities.

Exercise management program is the ongoing process of overseeing and integrating a variety of exercises over time.

An effective exercise program helps organizations maximize efficiency, resources, time, and funding by ensuring that exercises are part of a coordinated, integrated approach to building, sustaining, and delivering core capabilities. This approach—called multi-year planning—begins when elected and appointed officials, working with whole community stakeholders, identify and develop a set of multi-year exercise priorities informed by existing assessments, strategies, and plans. These long-term priorities help exercise planners design and develop a progressive program of individual exercises to build, sustain, and deliver core capabilities.

Effective exercise program management promotes a multi-year approach to:

• Engaging elected and appointed officials.
• Establishing multi-year exercise program priorities
• Developing a multi-year TEP.
• Maintaining a rolling summary of exercise outcomes.
• Managing exercise program resources.

Exercising is a continual process where previous risk assessments, jurisdictional and threat changes are analyzed to establish new preparedness goals to be evaluated in the next series of exercises.

The multiyear approach is driven by these steps:

• Engaging elected and appointed officials
• Establishing multiyear exercise program priorities
• Developing a multi-year Training and Exercise Plan (TEP)
• Maintaining a rolling summary of exercise outcomes
• Managing exercise program resources

A progressive exercise program is a series of exercises tied to a set of common program priorities. Each exercise builds on previous exercises using more sophisticated simulation techniques or requiring more preparation time, personnel, and planning.

Regardless of exercise type, each exercise within the progressive series is linked to a set of common program priorities and designed to evaluate associated capabilities.

Effective planning of exercises and integration of the necessary training will reduce the waste of limited exercise resources and serve to address known shortfalls prior to the conduct of the exercise.

The different types of exercises that may be included in a multi-year plan are described in the following sections.

The Exercise Cycle is provided by HSEEP for all exercise types and scopes. Exercise program management is the coordination of multiple exercises. Exercise project management is the coordination of a single exercise.

Project managers are responsible for:

• Design and development (Design and Development)
• Execution of a specific exercise (Conduct)
• Evaluation (Evaluation)
• Planning improvements (Improvement Planning)

Good project management involves:

• Developing a project management timeline
• Establishing project milestones
• Identifying the exercise planning team
• Scheduling planning meetings

Multi-year exercise program priorities are the outcome of leadership's assessment of hazards and risks related to specific core capability targets. At the program management level, the following priorities guide the development of exercise objectives, ensuring events that will form a progressive, comprehensive, integrated exercise program:

• Identified by elected and appointed officials
• Guide program planning and resource allocation
• Determine type and range of training/exercises
• Provide a roadmap for selection and prioritizing individual exercise development

The Training and Exercise Planning Workshop (TEPW), lays the foundation for the strategy and structure of an exercise program. The TEPW purpose is to engage elected and appointed officials in identifying exercise program priorities and planning a schedule of training and exercise events to meet those priorities.

An essential component of the exercise management process, the TEPW is a collaborative environment where whole community stakeholders can engage in a forum to discuss and coordinate training and exercise activities across local organizations to maximize the use of available resources and prevent duplication of effort.

During the workshop, participants:

• Review program accomplishments to date
• Review each jurisdictions progress and accomplishments over the past year, or since the last TEPW
• Identify needs and modifications required—including changes to the midyear training and exercise schedule and planning that might need updating
• Translate needs into priorities and develop specific objectives to address through exercises
• Compare Improvement Plan (IP) actions against current capabilities, training, and exercises
• Identify and coordinate potential funding sources
• Coordinate exercise activities and scheduling—a major part of the workshop should be spent on schedule coordination. The workshop is an excellent opportunity for all jurisdictions to coordinate their exercise schedules to avoid duplication of efforts and collaborate to maximize resources

TEPWs are convened on a periodic basis, usually annually or biannually, based on the program needs.

The outcome of the TEPW is the Multi-year Training and Exercise Plan.

The multi-year training and exercise schedule should reflect a progressive program of increasingly complex exercises with each building on the capabilities evaluated and validated in previous exercises. Coordinating a program's various exercises and exercise series is a crucial part of a Multi-year Training & Exercise Plan.

The schedule should include a graphic representation of the proposed activities scheduled in the multi-year exercise plan, it should emphasize coordination through jurisdictions, and allow adequate time for progressive exercises designed to build and validate capabilities.

Program managers use the Multi-year Training and Exercise Schedule to:

• Avoid duplicating their efforts
• Combine exercises and ensure the exercises don't conflict
• Combine training and ensure training does not conflict
• Optimize and combine funding where possible
• Prevent "over" training and exercising

The Exercise Planning Team manages and is responsible for these aspect of an exercise:

• Design
• Development
• Conduct 
• Evaluation

The size of the team will vary depending on the type and scale of the exercise. Exercise planners may elect to use the Incident Command System (ICS) structure when establishing the structure and organization of the planning team (see organization chart).

A team consists of a Lead Exercise Planner and planning team members. The team lead has complete management responsibility, assigning tasks to team members and ensuring the successful execution of the exercise.

The team size should:

• Be kept manageable but represent a full range of participating organizations and other relevant stakeholders.
• Include whole community stakeholders such as:
• First responders/support agencies
• Advocacy groups
• Those with limited English language proficiency

In addition, Subject Matter Experts (SMEs) bring functional knowledge from their area of expertise.

SMEs contribute to the exercise planning team by:

• Adding expertise to the exercise planning team
• Providing functional knowledge for player-specific tasks evaluated through objectives
• Helping make the scenario realistic and plausible
• Ensuring appropriate evaluation of capabilities

Trusted agents:

• Are individuals who understand that confidentiality must be preserved to maintain the integrity of the exercise.
• All members of the exercise planning team are trusted agents.
• Team members must not reveal details or share insight in order to protect the integrity and confidentiality of the exercise and evaluation process.
• This is especially important to those team members who may also serve as controllers or evaluators during exercise conduct.

Successful Planning Teams

Regardless of the size or scope of an exercise, an exercise planning team performs best with a clear organizational structure which clearly defines roles, responsibilities, and functional requirements for each role or position on the planning team.

Tips for creating a successful planning team:

• Define roles, responsibilities, and functional requirements
• Engage elected and appointed officials and whole community leadership in exercise planning
• Use project management principles
• Follow standardized process
• Organize the team using NIMS Incident Command System (ICS) or other structure that defines support roles for each team member

Planning Team Roles and Responsibilities

Exercise planning team members are responsible for the exercise design and development, conduct of the exercise, and exercise evaluation. Based on elected and appointed officials' guidance and program priorities, the team will establish exercise objectives and determine the core capabilities to be assessed during exercise play. Next, the team is responsible for creating a realistic scenario to assess the identified capabilities and objectives. As mentioned earlier, SMEs play a big role in creating plausible scenarios. The team develops supporting documentation for the exercise as well as process documentation, control and simulation documentation, and customized evaluation packets for participants. Team members also help with building and distributing pre-exercise materials and conducting exercise planning meetings, briefings, and training sessions. Keep in mind that being part of an exercise planning team is usually a collateral duty. It is very important to ensure candidates can make the commitment to actively participate throughout the process.

Exercise planning team members are responsible for the exercise design and development, conduct of the exercise, and exercise evaluation. Based on elected and appointed officials' guidance and program priorities, the team will establish exercise objectives and determine the core capabilities to be assessed during exercise play.

Next, the team is responsible for creating a realistic scenario to assess the identified capabilities and objectives. As mentioned earlier, SMEs play a big role in creating plausible scenarios. The team develops supporting documentation for the exercise as well as process documentation, control and simulation documentation, and customized evaluation packets for participants.

Team members also help with building and distributing pre-exercise materials and conducting exercise planning meetings, briefings, and training sessions.

Keep in mind that being part of an exercise planning team is usually a collateral duty. It is very important to ensure candidates can make the commitment to actively participate throughout the process.

HSEEP doctrine defines meetings as smaller events focused on a specific topic. The exercise planning team members decide the type and number of planning activities needed to successfully plan a given exercise.

Meetings are held to discuss, review, or develop exercise content. Typically, these are face-to-face meetings, which are critical in the initial and final stages of exercise development. This forum allows for coordination and collaboration among participants. Face-to-face meetings help organizations share knowledge and build relationships.

While not all development can be completed during face-to-face meetings, they are a great medium to assess team progress, assign new responsibilities, review completed work, and establish deadlines. Once the team is fully established, online collaborative meetings may be used to facilitate the planning process.

• Concept and Objectives Meeting (C&O)
• Initial Planning Meeting (IPM)
• Midterm Planning Meeting (MPM)
• Master Scenario Events List Meeting (MSEL)
• Final Planning Meeting (FPM)

The remainder of this lesson will discuss each meeting type in further detail.

Think of design as the framework of an exercise.

• Establish exercise scope
• Set exercise objectives
• Create exercise scenario
• Develop exercise documentation
• Determine media and public relations guidance

Defining the scope of an exercise enables planners to meet objectives while staying within the resource and personnel constraints of the exercising organization(s). By definition, the scope is an indicator of the extent of an exercise.

• Type of exercise
• Participation level
• Duration
• Location
• Parameters

Some elements are discussed and/or determined through program management activities or grant requirements. However, the responsibility of the planning team is to finalize the scope based on exercise objectives.

Objectives are the distinct outcomes an organization wishes to achieve during an individual exercise. An objective is a description of the performance expected from participants. It conveys specifically how the exercise should achieve its purpose.

• Driven by Exercise Program priorities
• Establish the cornerstone of scenario design, development, exercise conduct, and evaluation
• Follow SMART guidelines for development

Generally, the number of exercise objectives should be limited to enable timely execution, facilitate design of a reasonable scenario, and promote successful completion of the exercise purpose.

• Methodical tracking of progress throughout the exercise programs and/or cycles
• Standardized collection of exercise data used to inform preparedness assessments
• Fulfillment of grant or funding-specific reporting requirements

SMART is an acronym used to identify the characteristics of good objectives. SMART objectives identify who should do what, under what conditions, according to which standards.

SMART objectives are specific, measurable, achievable, relevant, and time-bound. Use the following guide to develop good exercise objectives.

A scenario is an outline or model for a simulated sequence of events that take place during an exercise or discussion. Scenarios are usually written in a narrative format but also depicted in timeline events.

Three basic elements:

• Context or comprehensive story of an incident event (narrative)
• Conditions allowing players to exhibit proficiency in demonstrating core capabilities and satisfying objectives
• Technical details needed to depict the conditions and events of the scenario (timelines, etc.)

Based on:

• Realistic events
• Plausible threat
• Challenging, but not too complicated that it overwhelms players

Situation Manual (SitMan) - The primary reference material provided to all participants. It is a textual background for the facilitated exercise and discussion. The SitMan structure includes an overview (scope, objectives, core capabilities, rules, and exercise agenda), the scenario, broken into modules, and discussion questions at the end of each module.

Facilitator Guide - The purpose of the facilitator guide is to assist facilitators managing discussion-based exercises. It outlines facilitator instructions and key issues for discussion during the exercise. The facilitator guide also includes background information to prepare the facilitator to answer questions.

Multimedia Presentation - A multimedia presentation is frequently used to illustrate the scenario. It helps participants focus and makes the exercise more realistic. The presentation is shown at the start of the exercise (StartEx) and supports the SitMan. Presentation components include an introduction; exercise scope, objectives, and core capabilities; exercise play rules and administrative information and is segmented into modules that correlate with the SitMan.

Exercise Plan (ExPlan) - A summary of the planned exercise, including the scope, objectives, and core capabilities to be validated. The plan is published and distributed to all participating organizations. Exercise players and observers should review all elements of the ExPlan prior to exercise participation. Components of the ExPlan: Exercise scope, objectives, and core capabilities; participant roles and responsibilities; rules of conduct; safety and security policies; logistics; communication methods; maps and directions; and duration, date, and time of exercise and schedule of events.

Controller/Evaluator Handbook (C/E Handbook) - The C/E Handbook contains detailed scenario and logistics information and a communications plan which is meant for designated controllers and evaluators only.

• The controller portion of the C/E handbook, also known as the Control Staff Instructions (COSIN), provides guidelines for control and simulation support and establishes a management structure for the activities.
• The evaluator portion of the C/E handbook, sometimes known as the EvalPlan, provides evaluation staff with guidance and instruction on evaluation or observation methodology, as well as essential materials required to perform their specific functions.

Master Scenario Events List (MSEL) - A chronological listing of events, in spreadsheet format, that drive exercise play. The MSEL is used during operations-based exercises or complex discussion-based exercises. MSEL events include contextual injects, expected action events (milestones), and contingency injects.

Injects are used to drive exercise play. MSEL injects serve three primary functions: linking simulation to action, enhancing exercise experiences for players, and reflecting an incident or activity that will prompt players to implement the policy or procedure currently being evaluated.

• Designated scenario time and actual execution time
• Who delivers the inject (and by what means) and who is designated to receive the inject
• Event description (what will happen)
• Inject content (quoted text as it is to be delivered) plus any specific direction needed
• Expected action (Based on the objective being tested, identify specific tasks, conditions, and standards to accomplish the objective.
• Note section for controllers/simulators to record pertinent information

Considerations when developing the MSEL:

• Review capabilities
• Identify chronology of key events
• Anticipate player actions
• Identify information resources
• Compile all MSEL events into a single list
• Refine selected MSEL events; create a detailed long version

Once the MSEL is drafted, the exercise planning team should review it, arrange entries in sequential order, and resolve any scheduling conflicts between events for best results during the exercise.

Additional exercise documents include:

Extent of Play Agreements (XPAs) - XPAs are agreements formed between the exercise sponsor and participating organizations. Specific details of participant involvement are identified, such as, one fire company, with two ambulances for 5 hours. These agreements can be vital to exercise planning, evaluator recruitment, and development of support requirements.

Exercise Evaluation Guides (EEGs) - Exercise Evaluation Guides (EEGs) provide evaluators with a checklist of critical tasks to be completed by participants during an exercise. EEGs contain the information to be discussed by participants, space to record evaluator observations, and questions to consider after the exercise. 

Participant feedback form - The participant feedback form should be given to participants during exercise wrap-up. At a minimum, ask participants for feedback on the strengths and opportunities for improvement based on the organization's plans, policies, SOPs and their impressions on the exercise conduct and logistics.

Waiver forms - All actors should receive a waiver form prior to the exercise. Once signed, these forms waive liability for all exercise planners and participants. Volunteers under the age of 18 must be signed by parental or legal guardian.

Weapons and Safety Policy - All exercises, where applicable, should employ a written weapon and safety policy that is in accordance with all applicable State and/or local laws and regulations. Exercise sponsors should coordinate the application of this policy with the appropriate safety and/or legal department, as necessary.

Exercise development encompasses planning for the four critical elements involved in exercise conduct. These elements include:

• Logistics
• Control Structure
• Communications
• Safety and Security

Planning for logistics is crucial to the success of the exercise. Well thought out logistical details are the difference between safe and smooth exercise and one that is confusing and unsafe.

• Physical Environment: venue, facilities/rooms, supplies, food, refreshments, IDs.
• Actors (Volunteers) are used to make exercises more realistic as players administer simulated victim care.
• Prior to StartEx, actors should be provided instruction and logistical details, as well as symptomatology cards containing their signs and symptoms.
• Parking, transportation and designated areas are to be clearly identified and staffed to direct vehicles.

The framework that allows for coordination and communication between controllers at multiple sites is known as the Control Structure. In a discussion-based exercise, the control structure is usually minimal. Operations-based exercises, on the other hand, may require an extensive structure to support proper coordination between the sites, venues, and control cells.

Exercises that span multiple sites and/or venues can improve communication and coordination by establishing a master control cell to oversee and coordinate venue control cells, as identified in the graphic.

A SimCell is used to generate injects, receive player responses, and disseminate information in place of an organization that may not participate in the exercise but would be on the scene of the incident, if it were real.

• Controller communications
• Timing and content of communications
• Communications methodology

Safety and Security are important considerations, especially in exercises that involve dangerous play or classified operations. The controller's role is critical to safeguarding the integrity and security of the exercise scene.

Safety is paramount when planning any exercise. For operations-based exercises, consider appointing safety controller(s), outline safety requirements and policies, and plan for dedicated ambulance/EMT coverage for real-world emergencies.

Security of the exercise site should also be considered. Local law enforcement can be employed where appropriate. If classified information is involved, follow appropriate security protocols to ensure information is not compromised.

Participants are selected based on exercise scope, type, objectives, and specifics of the scenario being played out. Discussion-based exercises usually requires senior-level decision making personnel. Alternatively, Operations-based exercises usually select players from junior and senior staff. Selections are based on rank and experience. Participant roles include:

• Exercise Director
• Evaluator(s)
• Lead Evaluator
• Facilitator(s)
• Controller(s)
• Simulators
• Senior Controller
• Safety Controllers
• Exercise Assembly Area Controller
• Players
• Actors
• Observers

Exercise Orientation Briefings are used to educate each participant group about their particular roles and responsibilities during exercise play. Participant groups should only attend briefings designed for their specific groups. These briefings are mandatory for all participants. Specialized briefings should be presented for Elected/Appointed Officials, Controller/Evaluators, Actors, Players, and Observers.

Elected/Appointed Officials should be reengaged before exercise conduct to verify that the exercise, as designed, meets leadership expectations and intent.

Controllers should receive an overview of the exercise, specifics on their role and assigned location, the schedule of events, and direction on MSEL injects they will be delivering during the exercise.

Evaluators should receive an overview of the evaluation plan, methodology, objectives, and evaluation materials. Evaluators should receive instruction on the use of exercise materials, including the Exercise Handbook and Exercise Evaluation Guide (EEG). Evaluators should understand the exercise methodology and objectives, and know the schedule or agenda.

Controller/Evaluators are briefed on both roles with the understanding that consistency in the dual role will contribute to the accuracy of the evaluation process.

Controllers are assigned participant groups to oversee. These oversight roles include Actor Controller, Exercise Assembly Area Controller, and Observer/Media Controller. Shortly before StartEx, controllers conduct a briefing for their particular groups.

Actor Controllers deliver the actor briefing. The briefing gives actors an overview of the exercise and their roles as "victims", adding realism to the exercise.

Exercise Assembly Area Controllers deliver the player briefing, which relays information on player roles and responsibilities, safety considerations, exercise parameters, and responses to questions and concerns from players. If security credentials are needed, proper usage should be covered in the briefing.

Observer/Media Controllers deliver the observer briefing, which provides a background on the exercise program, specifics for the exercise about to commence, observer restrictions and/or limitations, and the exercise schedule of events. This briefing is also attended by VIPs and media representatives.

An operations-based exercise is developed to be a realistic representation of the capabilities to be examined.

During conduct of operations-based exercises, the Senior Controller or Exercise Director role is usually filled by the exercise planning team leader. Controllers and evaluators report their findings and concerns to the senior controller. The senior controller is responsible for officially beginning (StartEx) and ending (EndEx) exercise play.

Operations-based exercises can be staged in an actual environment or simulated if the environment is too dangerous to stage. For example, responding to an infectious virus spillage would be simulated to protect all involved.

Exercise areas for operations-based exercises need to be clearly defined, with exercises taking place within the designated areas. For instance, Functional exercises take place in a command or control center. Any additional activity is simulated by staff in SimCells. Clearly marked area help to eliminate confusion between the exercise and real-world operations.

Exercise data collection is recorded in Exercise Evaluation Guides (EEGs) by all evaluators. Positioning evaluators strategically to observe play will accommodate better data collection.

Lastly, an effective contingency plan is crucial to assure the exercise can be stopped, rescheduled, or canceled if a real-world event were to take place. All participants and stakeholders should be briefed on this plan.

Operations-based control structure identifies how controllers communicate and coordinate within themselves and how they track exercise information. Along with roles and responsibilities, communication procedures are contained in the C/E Handbook.

Control is guidance for:

• Communication and coordination
• Roles and responsibilities
• Safe and effective play
• Simulation

During functional exercises (FEs), SimCell control is especially important. FEs use a lot of simulated activity, Thus, requiring a robust, detailed MSEL and close communication between site controllers and SimCells.

During full-scale exercises (FSEs) and drills, the exercise assembly area controller plays an integral role. Thus, the exercise assembly area controller must be in close communication with controllers and personnel to ensure safe and realistic deployment of resources.

Complete and thorough wrap-up ensures that relevant data is collected so effective evaluation and improvement planning can occur.

Debriefings immediately follow exercise conduct. They are short meetings conducted with the exercise planning team to discuss levels of satisfaction with the exercise, air any issues or concerns, and propose improvements. Debriefing notes, attendance lists, and participant feedback forms should be included in the meeting outcome.

The Player Hot Wash provides an opportunity for players to discuss strengths and opportunities for improvement. Hot washes are facilitator-led. They are usually brief and constructive. Comments can be used to develop the After Action Report/Improvement Plan (AAR/IP).

The Controller/Evaluator Debriefing is a forum for controllers and evaluators to review the exercise. The exercise planning team leader facilitates this meeting. Captured discussion comments an lessons learned are captured and incorporated into the AAR/IP.

Evaluation is the link between exercise and improvement planning. Through evaluation, capabilities necessary to accomplish a mission or function or meet an objective are assessed, based on performance.

Effective exercise evaluation involves:

• Planning for exercise evaluation
• Observing the exercise and collecting exercise data during conduct
• Analyzing collected data to identify strengths and areas for improvement
• Reporting exercise outcomes in a draft After-Action Report (AAR)

In the early stages of the exercise planning process, the exercise planning team leader should appoint a lead evaluator to manage the evaluation process. The lead evaluator participates as a member of the planning team.

Together, the planning team and lead evaluator should determine which tools and documentation are needed to support the evaluation team. Data collection methods are determined by the lead evaluator.

Evaluation Team Members should:

• Be familiar with the mission areas, core capabilities, plans, policies, and procedures to be examined during the exercise
• Determine the structure of the evaluation team
• Determine the supporting tools and documentation
• Conduct a pre-exercise C/E Briefing
• Recruit, train, and assign additional evaluators

The Evaluation Plan (EvalPlan), developed under the direction of the lead evaluator, is the comprehensive plan exercise evaluators will use to observe, collect data, and evaluate player performance. The EvalPlan is the Evaluator portion of the Controller/Evaluator (C/E) Handbook.

Evaluation Plan development tasks:

• Define evaluation requirements
• Prepare a plan for evaluating the exercise
• Select or develop evaluation forms
• Finalize the evaluation plan

Exercise Evaluation Guides (EEGs) provide a consistent tool to guide exercise observation and data collection. Aligned to exercise objectives and core capabilities, EEGs list capability targets and critical tasks.

EEGs accomplish several goals:

• Streamline data collection
• Enable thorough assessments of the participant organizations’ capability targets
• Support development of the After-Action Report (AAR)
• Provide a consistent process for assessing preparedness through exercises
• Help organizations map exercise results to exercise objectives, core capabilities, capability targets, and critical tasks for further analysis and assessment

EEGs are designed to document who, what, when, where, and how tasks were completed.

• Core capabilities: Critical elements needed to achieve a specific mission area
• Capability target(s): Performance thresholds for each core capability. Typically written as quantitative or qualitative statements.
• Critical Tasks: Elements required to perform a core capability and describe how the target will be met. Generally, they include activities, resources, and responsibilities required to fulfill capability targets.
• Performance ratings: Description of performance against target levels. Includes target ratings and core capability ratings.

Before StartEx, the lead evaluator should meet with all evaluators to confirm roles, responsibilities, and assignments.

Evaluation Documentation and Tools include:

• Evaluation Plan
• SitMan or scenario
• C/E Handbook
• Evaluator Team Organization
• Assignments
• Locations
• Evaluation Instructions
• Instruction on use of tools, logs, and forms
• Evaluation Tools
• Jurisdiction-specific or organization-specific SOPs
• EEGs
• MSEL

The C/E handbook or Evaluation Plan is the primary evaluation documentation for the exercise and typically contains the following information:

• Exercise-specific details: Exercise scenario, schedule of events, and evaluation schedule
• Evaluator team organization, assignments, and locations: A list of evaluator locations, shift assignments, a map of the exercise site(s), evaluation team organizational chart, and team contact information
• Evaluator instructions: Step-by-step instructions for evaluators for activities before, during, and after the exercise
• Evaluation tools: EEGs, MSEL, or a list of venue-specific injects, electronic or manual evaluation logs or data collection forms, relevant plans and procedures, Participant Feedback Forms, and Hot Wash Templates

Observation and data collection can differ between discussion-based and operations-based exercises. Data collection forms the analytic basis for determining if critical tasks were successfully demonstrated and capability targets were met.

Discussion-based exercises focus on issues involving plans, policies, and procedures. Observations may consist of an evaluator recording data from participant discussions on the EEGs.

Operations-based exercises focus on issues affecting the operational execution of critical tasks and capabilities. Observations are recorded from participant actions. These observations form the basis used to determine if critical tasks were successfully demonstrated and capability targets met.

Observations come from many different sources. All attempts should be made by the evaluator to collect as much information as possible.

Some additional sources include:

• Event logs
• Video or audio recordings
• Evaluator notes
• Photographs

Observation notes include if and how quantitative or qualitative targets were met. For example, a capability target might state, “Within 4 hours of the incident…” Observation notes on that target should include the actual time required for exercise players to complete the critical task(s).

• Actual time required for exercise players to complete the critical task(s)
• How target was or was not met
• Decisions made and information gathered to make decision
• Requests made and how requests were handled
• Resources utilized
• Plans, policies, procedures, or legislative authorities used or implemented
• Any other factors contributed to the outcomes.

Based on their observations, evaluators assign a target rating for each capability target listed on the EEG. Evaluators then consider all target ratings for the core capability and assign an overall core capability rating.

• Performed without Challenge (P)
• Performed with Some Challenges (S)
• Performed with Major Challenges (M)
• Unable to be performed (U)

What is an AAR?

• The document that summarizes key information related to the evaluation
• Overview of performance related to each exercise objective and associated core capabilities
• Length, format, and development timeframe of the AAR depend on the exercise type and scope

Elements of the AAR include:

• Exercise overview
• Analysis of core capabilities (The main focus of the AAR)
• Appropriate appendices (Improvement plan and participant list)

AARs also include basic exercise information such as the name and type of the exercise, dates, location, participating organizations, mission areas, sponsor, and POC.

Exercised sponsor distributes the AAR draft to participating organizations and elected and appointed officials to:

• Review and determine areas for improvement
• Determine organization with responsibility for corrective actions

Corrective actions are concrete, actionable steps intended to resolve capability shortfalls identified in exercises or real-world events. Elected/appointed officials should review and revise the draft AAR, as needed, prior to the After-Action Meeting (AAM) to confirm the validity of evaluator-identified issues.

Corrective actions may include:

• Changes to plans and procedures, organizational structures, and/or management processes
• Additional training, equipment, or resources

Organizational reviewers determine which issues fall within their purview and take ownership of those issues for action. At this point, a draft improvement plan (IP) is developed.

The AAM is a forum to review the AAR and the draft IP. Prior to the meeting, the exercise sponsor is responsible for distributing the revised AAR, which includes feedback on strengths and opportunities for improvement, as well as the draft IP.

Meeting Outcomes:

• Final consensus on draft corrective actions
• Develop deadlines for implementation of corrective actions
• Identify corrective action owners and assignees

Finalizing the After-Action Report/Improvement Plan (AAR/IP) – Corrective Action Tracking and Implementation

• Distribute to exercise planners, participants, and other preparedness stakeholders, as appropriate
• Track corrective actions to completion
• Ensure a system is in place to validate previous corrective actions have been successfully implemented (Tracked continuously until implementation)

Identifying strengths, areas of improvement, and corrective actions resulting from exercises help organizations build capabilities as part of a larger continuous improvement process.

The principles of continuous improvement are:

• Important to the National Preparedness System
• Consistent approach to strengthening Whole Community preparedness
• Builds capabilities as part of a larger continuous improvement process
• Proven method of issue resolution and information sharing
• Applicable to all operational phases