The Ongoing Quest to Assess & Measure Preparedness

Since 9/11, billions of dollars and an enormous amount of effort have been directed at enhancing national preparedness efforts as they relate to human-caused and natural disasters, yet many jurisdictions and organizations still struggle to determine how prepared they are and how prepared they need to be. 

Despite the advent of the national preparedness system and associated assessment efforts, the emergency management community is still challenged to measure and articulate local, state, and national preparedness. One of the biggest challenges to measuring preparedness stems from the fact that preparedness means different things to different people. Additionally, how communities and organizations prepare greatly depends on what they are preparing for. Following is an examination of the ongoing quest to assess and measure preparedness with the goal of identifying good practices, ideas, and recommendations for the Federal Emergency Management Agency (FEMA) and other whole community stakeholders – including public sector, private sector, and nonprofit organizations – to consider.

Progress Has Been Made

Assessing and measuring preparedness are not new ideas and, over the years, FEMA and others have made progress. For example, FEMA’s capability-based model that started with Homeland Security Presidential Directive 8 (HSPD-8) and has continued with Presidential Policy Directive 8 (PPD-8) provides a common framework, to include a series of capabilities that can be assessed and measured over time. The creation of standards such as National Fire Protection Association 1600 (NFPA-1600) and Emergency Management Accreditation Program (EMAP) standards have also proven to be helpful benchmarks for agencies to measure themselves against. Technology is aiding the effort as well, as the American Red Cross and others have developed intuitive web-based tools to help organizations assess their preparedness levels. Websites like the National Health Security Preparedness Index are also helping to promote the importance of preparedness assessments and the need to track progress over time.

In addition to the NFPA 1600, which has become a common framework used to guide private sector preparedness efforts, the creation of a voluntary Private Sector Preparedness Accreditation and Certification Program (PS-Prep), has been an important advancement. Although more narrowly focused, the cybersecurity framework created by the National Institute for Standards and Technology is another good example of a mechanism that can be used to assess preparedness levels (related to cybersecurity) and has become an industry standard for both public and private sector organizations.

Room For Improvement

Despite progress, there is still a great deal of room for improvement, especially concerning the use of the Threat and Hazard Identification and Risk Assessment (THIRA) and associated State Preparedness Report (SPR) process to assess local, state, and national preparedness. Although the assessments are done differently across the country, FEMA “rolls up” the various data points to help produce the National Preparedness Report (NPR), which can lead to some potentially misleading data and conclusions. Although the SPR assessment process may be too subjective, a criticism echoed by the Government Accountability Office (GAO), the SPR’s use of the planning, organization, equipment, training, and exercises (POETE) framework to examine the capabilities is intuitive.

Other methods and tools being used to assess preparedness include: after action reports from exercises and real-world events, surveys, subject matter experts, risk assessments, strategic plans, performance indicators, and standards such as EMAP. Despite the various approaches, however, many do not have comprehensive programs in place to analyze the various data and information sources.

When it comes to preparedness, it is important to ensure the various preparedness efforts (including assessments) are grounded in risk. The various threats and hazards are simply too dynamic and it is impossible to prepare for everything equally. People, processes, and technology are constantly changing as well. Preparing for disasters is an enduring mission that requires ongoing and focused commitment, as well as some degree of ongoing financial support from the federal government to state and local governments for homeland security/emergency management purposes, particularly if there is a desire to be able to develop, sustain, and deploy specialized response capabilities (e.g., Incident Management Teams). However, no amount of money will guarantee preparedness, so risk-informed investments are important as is accountability for how the funds are used.

More effort is also needed to educate elected leaders and oversight agencies so that they better understand the ongoing nature of preparedness and appreciate that the nation will never be “done” preparing. Although it is unlikely that any one system will adequately measure national preparedness, the use of common tools and frameworks can certainly help the various stakeholders examine preparedness in a more consistent way.

Measuring What Matters

The emergency management community has struggled to develop metrics to measure preparedness. FEMA is working to develop a series of objective measures for the core capabilities, and some jurisdictions have made a lot of progress in developing their own measures for the core capabilities. Following are some examples of good practices:

• The New York State Division of Homeland Security and Emergency Services (DHSES) developed a County Emergency Preparedness Assessment (CEPA) Program that includes workshops in each county (and New York City) to assess local risk and capabilities using a POETE-based model.
• The Florida Division of Emergency Management has several innovative initiatives, including a program to assist counties with obtaining EMAP accreditation.
• The Bay Area Urban Area Security Initiative (UASI) partners worked with a consultant and their local stakeholders to develop a series of preparedness-related performance measures and associated tools to capture information from the jurisdictions within the UASI region.
• The National Preparedness Leadership Initiative (NPLI) at Harvard’s Kennedy School of Government is an example of an innovative effort to educate leaders and to better understand executive decisions and attributes that can contribute to improved levels of preparedness.
• FEMA’s National Preparedness Assessment Division (NPAD) has recently created an Evaluations and Decision Support Unit that is actively looking to identify and leverage various data and information sources to better understand preparedness.
• The American Red Cross has created the Ready Rating Program to help organizations assess their readiness and understand what steps they can take to improve preparedness.
• Of the other countries examined, New Zealand appears to have the most robust system in place to assess and measure preparedness. Like New York’s CEPA program, New Zealand’s National Capability Assessment is highly collaborative and captures data through a series of regional workshops.

Recommendations

Following are recommendations related to assessing and measuring preparedness that FEMA (and perhaps others) should consider:

• Promote POETE: FEMA should focus more on promoting its definition of preparedness and the associated POETE methodology, which is intuitive and can likely be used by other public and private sector organizations.
• Streamline and improve the THIRA/SPR process: FEMA should work with state and local stakeholders to improve the THIRA/SPR process by making it more intuitive and user-friendly.
• Trust but verify: FEMA should trust the state and local data but develop mechanisms to verify the process used to capture the data and consider becoming a more active participant in the process, rather than simply ensuring the appropriate boxes are checked.
• Invest in preparedness analysts: FEMA, states, and others should consider the use of preparedness analysts to help analyze and assess preparedness.
• Participate in executive education initiatives: Public, private sector, and nonprofit organizations should make a concerted effort to educate their leaders through programs like those offered at FEMA’s Emergency Management Institute, Center for Homeland Defense and Security (CHDS), and Harvard’s Kennedy School of Government.
• Create an Incident Command System (ICS) improvement officer position: FEMA should consider the establishment of an improvement officer position and function within the ICS command staff structure.
• Establish a community of Practice: FEMA should engage stakeholders by creating a preparedness assessment work group or community of practice.
• Consider a deliverables-based grant model: The grant guidance is currently very broad and the funds can be used to support a wide variety of activities, which is a good thing, but FEMA should consider requiring some specific deliverables as well.
• Explore new assessment frameworks: Much of the focus to date has been on assessing capabilities (ability and capacity), but other components such as competency (leadership and experience), collaboration (communication and coordination), and community (economics and demographics) warrant much further examination, to include the identification of relevant metrics and indicators for the various components (see Fig. 1).

Further Exploration

This is not the first effort to examine how the emergency management community can better assess and measure preparedness. Ideally, others will take this research even further and delve deeper into the issues identified. Much of the work to date has focused on assessing capability, but without sound leadership and effective relationships even the most capable organizations may struggle during a crisis. As such, the “Four C” framework warrants much further examination. Preparedness is a never-ending process that requires a broader and more holistic analytical perspective to be truly understood. Progress has been made, but no single system or approach will suffice. To address an enduring challenge facing the emergency management community, it is time to think differently and determine how to assess and measure preparedness.

This article is based on a research project conducted as part of the Emergency Management Executive Academy at the FEMA’s Emergency Management Institute. The project team for this effort included emergency management professionals from federal, state, local, and nongovernment agencies. Click here to read the full report and see below for more information on the team members.