Michael I. Thornton* | Alba Iannotti | Riccardo Quaranta | Colomba Russo
© 2021 IIETA. This article is published by IIETA and is licensed under the CC BY 4.0 license (http://creativecommons.org/licenses/by/4.0/).
OPEN ACCESS
The World Health Organization carried out a global survey in 2018 to ascertain the level of preparedness for pandemic influenza. It was discovered that simulation exercises to test national pandemic influenza preparedness plans were carried out in the previous 5 years, by 42 out of the 104 countries that completed the survey. The table-top exercise (TTX) being the preferred format, with 86% of the countries using them. Although no table-top exercise can convey a realistic picture of a pandemic, they can be used to assess plans, policies, and procedures, clarify roles and responsibilities, and identify resource gaps in an operational environment. However, table-top exercises are only effective if they are properly designed, carefully conducted, fully evaluated, and most importantly, the results and recommendations identified are actually implemented. TTXs used as part of preparedness for pandemics are not cost free, a failure to implement the lessons learned from them can have both human and economic consequences. To understand the value of TTXs, a sample of national and large scale TTXs are examined in an effort to identify the effectiveness of table-top exercises, as a part of improving pandemic crisis preparedness.
crisis, effectiveness, evaluation, exercise, influenza, pandemic, preparedness, table-top
It is only a matter of time before an avian flu virus - most likely H5N1 - acquires the ability to be transmitted from human to human, sparking the outbreak of human pandemic influenza.
We don't know when this will happen. But we do know that it will happen.
This is the time to build global consensus. This is the time for every country to prepare their national action plan - and act on it.
If we are unprepared, the next pandemic will cause incalculable human misery. Both directly from the loss of human life, and indirectly through its widespread impact on security. No society would be exempt. No economy would be left unscathed [1].
The words of a Former Director-General of the World Health Organization (WHO): Dr Lee Jong-wook in November 2005, when he laid down the challenge to countries to examine their pandemic flu national action plans and to test them, to ensure that governments worldwide would be prepared.
The WHO carried out a global survey in 2018 to ascertain the level of preparedness for pandemic influenza. It was discovered that simulation exercises to test national pandemic influenza preparedness plans were carried out in the previous 5 years by 42 out of the 104 countries that completed the survey. The table-top exercise being the preferred format, with 86% of the countries using them [2].
The European Centre for Disease Prevention and Control (ECDC), following discussions with EU countries, recognised that simulation exercises (local, national, EU or international level) have been of great importance for capacity building and that performing table-top public health preparedness exercises could support efforts to ensure sustainability of capacity [3].
Table-top exercises involving response staff are particularly valuable and are a useful aspect of operational training. They allow participants to work together on simulated environments, testing plans and procedures and are a cost-effective way of training. Although no table-top exercise can convey a realistic picture of a pandemic, they can be used to assess plans, policies, and procedures, clarify roles and responsibilities, and identify resource gaps in an operational environment.
Table-top exercises are only effective if they are properly designed, carefully conducted, fully evaluated, and most importantly, the results and recommendations identified are actually implemented by the entity for which the exercise was designed for, be that private or Governmental.
A Google search for, “pandemic flu table-top exercise after: 2020-03-11,” the date that the WHO declared a pandemic, gives rise to 1000’s of results. Of more interest, is what happened before the pandemic broke out, and the aim of this study is to explore the impact of table-top exercises prior to the declaration of a pandemic, as part of flu pandemic preparedness in selected countries.
The Global Health Security (GHS) Index is a comprehensive assessment and benchmarking of health security and related capabilities across 196 countries that make up the States Parties to the International Health Regulations (IHR [2005]) [4]. The index attempts to assess a country’s capability to prevent and mitigate epidemics and pandemics.
The report highlights the fact that no country is fully prepared for epidemics or pandemics, and the country that overall is the most prepared is the United States of America. As basis for this study, three countries were examined, the US, UK and New Zealand, which were ranked overall in the index in 1st, 2nd and 35th position respectively [5].
For each country, national or large-scale simulation exercises carried out prior to 11 March 2020 and where the findings were made public were considered. All the exercises examined were either based totally on table-top or a table-top was part of the exercise series.
For each of the exercises studied, a short description of the scenario is given along with the evaluation methodology and key findings.
2.1 USA
In 2004, the U.S. Department of Health and Human Services contracted the RAND Corporation to develop and pilot-test a table-top exercise that tested the relationships between local public health agencies and their local healthcare delivery and governmental partners in response to a pandemic flu emergency. Since that time many table-top exercises have been run following the RAND TTX model [6].
The National Pandemic Implementation Plan was issued in May 2006 (last publically available update dated 2017) by the President and his Homeland Security Council and requires that all states and communities should have influenza pandemic preparedness plans and conduct pandemic exercises. The simulation exercises that were subsequently carried out usually included table-top exercises.
An example of a state run TTX is given below.
2.1.1 Dallas County Avian Flu Pandemic TTX: 2006
Dallas County Health & Human Services (DCHHS) sponsored the Pandemic Avian Flu Table-Top Exercise on 21 April 2006. The exercise was designed as an opportunity for DCHHS, key agencies to include the private sector, and the participating cities (Richardson, Dallas, and Garland) to discuss how they would organize and execute their response to a pandemic. Detailed information of the exercise was contained in a draft after action review, dated 31 May 2006 [7].
Scenario. Health workers identify a new strain of avian flu that is carried unknowingly by a traveller from Vietnam who arrived in the US via Dallas Fort Worth International Airport. Some days later throughout the United States, a large number of people began presenting at hospitals with influenza-like symptoms. This pattern is subsequently repeated in large cities throughout the world, including London, Tokyo, and Hong Kong.
Evaluation methodology. Presentation at the end of each group discussion, evaluation form and hotwash, (a debrief conducted immediately after an exercise or test with the staff and participants).
Key findings. Participating agencies need help to clarify roles and responsibilities. Issues with pandemic data information collection. Communication issues with the need to provide clear direction, accurate, timely, and updated information both to the public and key response agencies. A shortage of Personal Protective Equipment (PPE) due to demand. Issues with the potential use of triage of the public and a lack of facilities for storage of dead bodies. Existing vaccines not suitable and medicines in short supply. The use of State and Federal military medical reserves. Need for criterion for sending notifications and shutting the schools down.
2.1.2 Clade X: 2018
The Johns Hopkins Center for Health Security hosted the Clade X pandemic table-top exercise on 15 May 2018, in Washington, DC. This was an unusual table-top exercise in as much as it was privately funded and conducted in front of a live audience and was streamed live on Facebook [8].
Scenario. Outbreak of novel para-influenza virus in Germany and Venezuela that is moderately contagious and moderately lethal and for which there are no effective medical countermeasures. There is no vaccine, and pressure grows as cases appear in the United States. Within a year, 150 million people die from the disease, 15 million in the United States alone.
Evaluation methodology. Hot wash.
Key findings. The need to develop a capability to produce new vaccines and drugs for novel pathogens within months, not years.
Demand for surgical masks and respirators will far exceed supply.
Hospitals in the United States will be quickly overwhelmed.
Clear need for someone to coordinate federal agencies’ responses, taking into consideration the sometimes-competing interests of health security, politics and foreign policy.
Need for officials to be proactive on social media to counter misinformation.
2.1.3 Crimson Contagion: 2019
Crimson Contagion was the largest-ever exercise in the U.S. Department of Health and Human Services’ history [9]. It included two table-top exercises, a seminar, and a functional exercise and involved twelve federal departments/ agencies, twelve states, 96 local jurisdictions, 24 native American Tribes, 87 hospitals, and more than 100 private sector partners [10].
Detailed information of the exercise was contained in a draft after action review, dated October 2019 which was not published by the US Government until the New York Times published it in March 2020 [11].
Scenario. Outbreak of the respiratory virus began in China and was quickly spread around the world by air travellers. WHO declares pandemic. 110 million Americans were expected to become ill, leading to 7.7 million hospitalized and 586,000 dead.
Evaluation methodology. Hot washes, evaluators used evaluator logs, after action report analysis forms, and exercise evaluation guides to record observations.
Key findings. Federal government lacks sufficient funding to respond to a severe influenza pandemic. Confusion between which federal agency would take the lead in the crisis. Response partners lack clarity on data sharing policies. Some states were not clear on pre-pandemic vaccine or the strategic national stockpile asset distribution in response to influenza pandemic. The current medical countermeasure supply chain and production capacity cannot meet the demands imposed by nations during a global influenza pandemic. A lack of production capacity to meet the demands for protective equipment and medical devices. The current medical countermeasure supply chain and production capacity cannot meet the demands imposed by nations during a global influenza pandemic. A lack of clarity on the roles of different federal agencies. The distributed nature of school closure decisions caused confusion among exercise participants.
2.1.4 Event 201: 2019
The Johns Hopkins Center for Health Security in partnership with the World Economic Forum and the Bill & Melinda Gates Foundation hosted event 201 in New York on 15 October 2019 [12]. The one-day table-top event with participants from global business, government, and public health was held in front of an invited audience of 130 people and was live streamed.
Scenario. The exercise simulated an outbreak of a novel zoonotic coronavirus transmitted from bats to pigs to people that eventually becomes efficiently transmissible from person to person, leading to a severe pandemic. The simulation imagined no available vaccine. The scenario ended at the 18-month point, with 65 million deaths worldwide.
Evaluation methodology. Hot wash.
Key findings. There will be issues with business continuity. Need to enhance internationally held stockpiles of medical countermeasures. Travel and trade are essential to the global economy should be maintained. Governments should provide more resources and support for the development and surge manufacturing of vaccines, therapeutics, and diagnostics that will be needed. Global business should recognize the economic burden of pandemics and fight for stronger preparedness. Governments and the private sector should assign a greater priority to developing methods to combat mis- and disinformation prior to the next pandemic response. Governments will need to partner with traditional and social media companies to counter misinformation. National public health agencies should work in close collaboration with WHO to create the capability to rapidly develop and release consistent health messages [13].
2.2 UK
Under the Civil Contingencies Act 2004, designated responders have a legal responsibility to exercise and test the effectiveness of their pandemic flu plans and procedures.
With this in mind, the Government agency responsible for protecting the nation's health and wellbeing, Public Health England (PHE), prepared Off The Shelf Exercises (OTSE) that covered, amongst other areas, pandemic influenza. These exercises, designed to be used by Public Health England, National Health Service (NHS) funded providers, and other key local partners as a resource to help review and enhance emergency preparedness, and resilience and response within health have been used by many of the NHS regions [14]. An example of this type of OTSE is Exercise Cold Play, which was used on more than 50 different occasions in 2005, and included every region across England [15]. Other table-top OTSEs include Icarus Nov. 2004, Artic Sea Jun. 2005, Aurora Sep. 2005, New Day 1 and 2 Oct 2005, Athena Jan. 2007, Phoenix Nov. 2007 and New Day 5 Dec. 2007. The latest publicly available Influenza Pandemic Plan is dated August 2014 [16].
2.2.1 Winter Willow: 2007
Over 5,000 people from a large number of UK organisations representing government, industry and the voluntary sector participated in Exercise Winter Willow. It was designed to exercise structures, plans and procedures across the whole of the UK and at all levels of emergency response. It was a combination exercise consisting of the table-top, followed by full-scale exercise. According to the “Lessons Identified” report, the evaluation of the exercise was undertaken within a structured process at local regional and national levels in England and the devolved administrations. Feedback from all levels has indicated that the exercise was helpful in checking and validating plans and communication channels although there were clearly areas for improvement” [17].
In the “Lessons Identified” report it was stated that, “the UK remains ahead of most countries in its development of pandemic preparedness plans…”.
Scenario. Outbreak of Pandemic Influenza in the UK. Stage 1 of the exercise on 30 January 2007 was set at WHO Alert Level 6 (increased and sustained transmission in general population) - UK Level 2 (virus isolated in the UK).
Evaluation methodology. The UK Department of Health, Exercise Winter Willow Lessons Identified, 2007 report was generated from following a “structured process of evaluation” [18].
Key findings. Need to improve channels of communication. Need to review pandemic data collection, collation, and use. Public messages needed to be refined and that communications from central government departments and agencies needed to be better coordinated to ensure clarity and consistency. Need clarification or further development concerning travel advice. Review the scientific evidence base for pandemic preparedness planning and clarifying the role of the Department of Health Science Advisory Group during a pandemic. Work on the management of the surge in demand for medical supplies such as masks and antibiotics. Need further detailed guidance to help local planning for the operational aspects of increased death rates. Need further work on the practical aspects of the implementation of school closures and the wider implications. The Exercise identified most clearly that throughout a pandemic, business continuity would be a significant challenge to all organisations.
In 2015 the UK Government identified pandemic influenza as the most likely significant civil emergency risk to occur within the next 5 years. They classified the outbreak as having an overall relative impact of 5, on a scale of 1 to 5, the same impact that a nuclear terror attack might have [19].
2.2.2 Cygnus and Cygnet: 2016
In 2014 as part of the UK government’s national exercise programme, the UK government planned a multiagency exercise called Cygnus [20]. It had the aim of assessing the preparedness and response to an influenza pandemic in the United Kingdom. Due to the outbreak of Ebola in Africa and the response that followed, the exercise was delayed for two years, finally taking place in October 2016 with 950 participants. Exercise Cygnus was a Command Post Exercise, with participants based, as much as practicable, where they would usually work during the pandemic response and was preceded by Cygnet, a national table-top exercise. The results of the evaluation process led to 22 lessons learned and a favourable judgement by the participants of the success of the whole exercise with between 70 - 90% of them agreeing or strongly agreeing with the statement that, “the aim of the exercise was achieved” [21].
Exercise Cygnus and Cygnet followed the typical methodology of simulation exercises, scripting the exercise, executing the exercise, performing an evaluation and writing the report. However, the final report and recommendations were not published. During the outbreak of COVID-19, questions were asked to the Government about their level of preparedness and Lord Hunt of Kings Heath asked the government in the House of Lords on the 9 June 2020, “why are the Government not prepared to be open and transparent and to publish the report and recommendations.” The Lord Speaker, on behalf of the Government replying that, “Publication of these reports is not in the national interest and we do not have plans to publish them in the future” [22].
A copy of the report was leaked to the Guardian newspaper in May 2020. The report, marked “Official – Sensitive,” was published, although names and email addresses of government officials were redacted [23].
Subsequent to the information release by the Guardian, an NHS doctor and a journalist have tried to force the Government to publish, “all the findings, lessons or recommendations arising out of Exercise Cygnus, including reports sent to and/or by participants in the exercise”, and are bringing a legal case to the High Court. As the claimants believe that the full extent of the findings in relation to the Cygnus exercise has not been disclosed [24].
Following this request, the UK Information Commissioner’s Office in October of 2020, stated that the Government should either publish the findings under the Freedom of Information Act or issue a refusal notice, which would describe why this information should not be released into the public domain. A report on exercise Cygnus was published on 20 October 2020 although there are claims that there are several subsidiary reports relating to Cygnus that have yet to be released [25].
Scenario. A new virus had emerged in Thailand, later identified as a strain of H2N2. Within a month the World Health Organization had declared a public emergency, triggering the UK’s response plans as the world mobilised to tackle an outbreak of “swan flu”.
Evaluation methodology. Evaluation consisted of participant feedback during the hot wash immediately following the exercises, debriefing notes acquired at some time after, a team led structured debrief and an evaluator questionnaire.
Key findings. Organisations should ensure that their emergency preparedness resilience and response training and exercising is consistent with best practice. Pandemic influenza planning should be considered a multi-agency responsibility. Further work is required to relating to the possible use of population-based triage. Further work is required to consider surge arrangements. All organisations should examine the issues surrounding staff absence during a pandemic to provide greater clarity for planning purposes. The Department of Health should work with partners to further develop the strategy for the use of antivirals during a pandemic. Pandemic communications plans should be developed and practised to ensure that they provide necessary reassurance and provide adequate levels of information to the public. The Department for Education should study the impact of school closures on society. Expectations of the Ministry of Defence’s capacity (the military) to assist during a worst-case scenario influenza pandemic should be considered. The process and timelines for providing and best presenting data on which responders will make strategic decisions during an influenza pandemic should be clarified. PHE to define and communicate who will receive PPE from national stockpiles and which parts of the private and voluntary sectors are expected to make their own arrangements to safeguard their workers in the event of an influenza pandemic.
2.2.3 Silver Swan (Scotland): 2015
Exercise Silver Swan was delivered during the latter part of 2015 as a series of table-top exercises across the whole of Scotland involving 662 participants [26]. Its aim was to assess the preparedness and response of Scotland’s local and national arrangements for influenza pandemic over a prolonged period. The Exercise report was leaked to the Scottish Newspaper, the Daily Record and published on 2 May 2020, even though the exercise took place four years earlier [27].
Scenario. The exercise was set around six weeks after the first pandemic cases emerged in Scotland. The scenario provided participants with the chance to consider in detail how they would deal with the large number of pandemic cases emerging and how they would prepare for the peak of the pandemic ahead. The scenario was designed with case numbers that were largely manageable, while recognising that different impacts would be specific to certain geographical locations and service areas.
Evaluation methodology. Plenary discussions and feedback from facilitators. Participants survey sent at the end of the series of exercises, in which 90% of respondents agreed that the objectives were mainly or completely met.
Key findings. Review national plans to ensure learning from the exercise is incorporated. Ensure coordination of the multi-agency response and avoid duplication in information gathering. Investigate potential staff capacity issues and redeployment. Review arrangements for public communication. Consider Business Resilience/Continuity standards in the contracts for the supply of key or essential services such as care homes. Availability of consumables such as PPE and pharmaceutical products. Develop guidance on body storage due to mass fatalities – monitor body storage and system capacity when demand outstrips supply. Conduct a national and local review of the ability to distribute and issue antivirals effectively during influenza pandemic. Ensure that the plans to distribute the stockpile of PPE, including information on prioritised key staff and groups and when PPE should be used, are well understood. Ensure fit testing procedures are in place and being followed.
2.2.4 Iris (Scotland): 2018
Exercise Iris was a table-top exercise that took place 12 March 2018 to explore the challenges that the National Health Service Scotland would face during a Middle East Respiratory Syndrome Coronavirus (MERS-COV) outbreak. Participant feedback was broadly positive and they welcomed the opportunity to explore the key issues with a wide range of colleagues and to think about similarities with readiness for other types of outbreaks such as Pandemic Flu [28]. The report was published on the Scottish Government’s website on 3 June 2020. Publication of the report followed a freedom of information request by the BBC in April 2020 [29]. Information concerning updates to health board pandemic plans following exercise Iris, was also released by the Scottish Government following a further freedom of information request on the 15 June 2020.
Scenario. This Scenario is based on a patient presenting to our next and emergency department of the hospital in January in Scotland, with typical flulike symptoms. Her condition deteriorates and she was transferred to intensive care for ventilation. Suspecting MERS-COV or similar the hospital contact the family to understand her travel history. 10 days prior to admission she returned from Dubai where she had shared a coach with other British holidaymakers, and excursions including a trip to camel trekking facility.
Evaluation methodology. Hot wash during final plenary.
Exercise Iris was well received by the participants who engaged positively and constructively in support of the aims and objectives of the day [30].
Key findings. The need for a clear command and control structure. The need for clarification of roles and responsibilities. A lack of PPE at beginning of the outbreak and PPE procurement problems including fit, training and testing issues. Communication issues to be dealt with by NHS 24 who should be engaged in the early stages to prepare scripts for call handlers and be ready to stand up a dedicated helpline. Need to consider resource impact of extensive contact tracing. Recognition of the burden that may be placed on mortuaries, burial and cremation facilities.
2.3 New Zealand
Planning for a pandemic flu began in 2002 and a series of exercises to test preparedness were developed and run - Virex (Feb 2002); Makgill (Nov 2006) and Cruickshank (May 2007). The latest publically available Influenza Pandemic Plan is dated 2017.
2.3.1 Virex: 2002
New Zealand began pandemic table-top exercises in 2002 with one of the first national exercises in the world to test readiness for a pandemic, exercise Virex. This exercise took nine months to plan and involved 400 participants. The exercise report was published in 2002 and highlighted a number of actions to be taken [31].
Scenario. New Zealand's health authorities are on alert following an outbreak of a new strain of influenza in Wellsun and Hong Kong. Wellsun, a fictitious country is imagined to be the world's most populous country. The virus spreads to the United Kingdom, Australia, Canada, Singapore, Fiji and New Zealand. Subsequently the WHO declares the onset of influenza pandemic worldwide. Mortality among hospitalised patients has reached 30% placing considerable pressure on the country’s mortuary facilities.
Evaluation methodology. Discussion during table-top.
Key findings. Need clarity on roles and responsibilities. Need guidance and advice on vaccination strategy for prioritisation and delivery, border control, closure of schools, businesses and mass gatherings and a national communication strategy. Need to consider capacity and capability to deal with large numbers of dead. Ministry of Agriculture and Forestry will work on linked surveillance between human and animal networks. There would be an increased demand for ambulance services. New Zealand’s Crown Research Institute specialising in science for communities will look at the supply and distribution of testing kits. Need to establish databases within locality identifying local providers, resources and supplies. Improve communications capability. Improve surveillance data by capturing hospitalisation rates.
2.3.2 Makgill and Cruickshank: 2006/2007
The Pandemic Exercise Programme 2006/07 consisted of two major exercises, Makgill and Cruickshank, in November 2006 and May 2007 respectively [32]. Exercise Cruickshank being the largest and was a combination of table-top, discussion exercise formats with a limited amount of operational deployment. It practised the four stages of a pandemic response across more than 40 government agencies at local, regional and national levels in New Zealand. The exercise involved 1500 people and a budget of $750,000 was allocated to conduct the exercise [33].
Fifteen international observers from the World Health Organization, the Australian Government Department of Health, Niue, Tokelau, New Caledonia and the British Government were invited to watch the exercise [34].
The Ministry of Health evaluated the overall exercise response at national and local levels.
The after action report, containing a description of what took place, as well as a full description of the 68 performance indicators, evaluation results and a complete list of recommendations, was published and made available online in October 2007 [35].
Scenario. Avian virus H5N1 was infecting and killing people in many countries and had developed into a form that could be transmitted easily between humans. Within days of the announcement of the outbreak an Asian country with which New Zealand has significant trading and tourism links announced that influenza cases had occurred in several of its cities. The outbreak was developing into a pandemic.
Evaluation methodology. Hot washes, cold debriefs, participant feedback forms and feedback from the designated evaluators.
Key findings. The need to identify clear systems and responsibilities for collecting, providing, validating and disseminating information. Adopt procedures for customising and disseminating their own pandemic-related key messages for their key communities. Identify the response role of different pandemic planning, advisory and decision- making groups. District Health Board public health services should share local solutions for providing support, in particular, food and other critical supplies for people in quarantine. The Ministry of Health, in liaison with the inter-sectorial Border Working Group, will progress its border management work programme to, for example, developing plans for exit screening and address gaps in procedures (for example, how to quarantine aircrews). Review pandemic resources and the Ministry of Education’s pandemic website. Consider providing more comprehensive guidance for implementing public health controls, Ensure information on major public health interventions, such as school closures and openings, is disseminated to inform agency and business planning during a pandemic response. The supply of PPE and pharmaceuticals was not considered an issue but the distribution in some cases was.
“Evaluation can be defined as the act of reviewing or observing and recording exercise activity or conduct, assessing behaviours or activities against exercise objectives, and noting strengths, weaknesses, deficiencies, or other observations” [36].
There is no standard form of table-top evaluation. The United States has developed HSEEP, which is the Homeland Security Exercise and Evaluation Program. This is a national standard for exercise design, conduct, evaluation and improvement planning used by many entities, mainly in the US. The latest revision of the doctrine highlights the “Whole Community” and “refers to whole community preparedness goals and not just the Core Capabilities” [37].
As good as this methodology may be, it does not provide common metrics to apply in an evaluation of an exercise and is primarily used within the United States.
Table-top evaluation is carried out via a number of methods, the hot wash and participant feedback form being the two most commonly used methods. The hot wash provides the evaluators the opportunity to clarify points, gather the impressions and reflections from participants about the exercise or to collect information from the players before they leave the exercise. The participant feedback form is usually a goal-based evaluation technique. It attempts to measure the extent to which certain predetermined objectives have been achieved and is largely concerned with the projected benefits and results of an exercise. Although effectively this is a qualitative measurement, responses can be converted to a numerical scale and hence become quantitative. The problem with this methodology is that in many cases the questions or statements posed tend to be rather broad and generic.
Sufficient time should be allocated to complete the initial feedback of the evaluation process. Some organisations use the 50:50 rule whereby 50% of the time is given to conduct the exercise and 50% of the time allotted to receive feedback [38]. Those that have delivered table-top exercises will recognise that in reality this is rarely the case. The evaluation phase often consists solely of a hot wash, which by its definition is the last part of the exercise and therefore will occur at the end of the exercise day and often has time constraints.
Evaluation tools and methodologies help determine the effectiveness of instructional interventions. Despite its importance, there is evidence that evaluations of training programs are often inconsistent or missing [39]. Possible explanations for inadequate evaluations include: insufficient budget allocated, insufficient time allocated and lack of expertise or participant fatigue.
Good exercise evaluation is dependant on good exercise objectives. If the pre-defined objectives are vague or even inappropriate, then a good exercise evaluation will not be possible.
Table-top exercises are a valuable instrument in the training toolbox of disaster preparedness that allows for plans, procedures and policies to be validated, roles and responsibilities to be clarified, resource gaps to be identified and relationships to be built.
The validation of existing plans and the issues that occurred during game play are typically discussed during the post exercise, “hot wash.” At this time the table-top will also be evaluated, from a participant’s point of view often utilising participants feedback forms, with the added input from evaluators / facilitators. Knowledge is lacking however, regarding the effectiveness of table-top exercises as a component of developing preparedness.
The delivery of any exercise contains a degree of risk and two risks can be clearly identified and should be noted.
The first risk is simple to deal with by stopping the exercise and releasing the required personnel, the greater problem is with the second risk.
Conducting an exercise is important, but if the exercise is not evaluated correctly and the lessons learned are not applied, then there is little point in performing the exercise. TTXs are not cost free, sometimes large sums of money and personnel resources are invested in conducting it. Their effectiveness has to be demonstrated or they are potentially a waste of time, money and human resources.
The most common issues identified in exercise key findings examined in this study are tabulated in Table 1. Data from the US exercises Clade X and Event 201 were not included in the table as they were not government run exercises, although they did highlight many of the same issues identified in other exercises.
Table 1. Common issues identified in exercise key findings
|
Exercise: Country (Year) |
Vaccine |
Medical & PPE |
Roles & responsibilities |
Communication |
Pandemic data |
School closures |
Death rates & triage |
|
Dallas County: US (2006) |
X |
X |
X |
X |
X |
X |
X |
|
Crimson Contagion: US (2019) |
X |
X |
X |
X |
X |
X |
|
|
Winter Willow: UK (2007) |
X |
X |
X |
X |
X |
X |
X |
|
Cygnus and Cygnet: UK (2016) |
X |
X |
X |
X |
X |
X |
X |
|
Silver Swan: Scotland (2015) |
X |
X |
X |
X |
X |
|
X |
|
Iris: Scotland (2018) |
NObj. |
X |
X |
X |
NObj. |
NObj. |
X |
|
Virex: New Zealand (2002) |
X |
X |
X |
X |
X |
X |
X |
|
Makgill and Cruickshank: New Zealand (2006/2007) |
X |
X |
X |
|
X |
X |
|
Notes: An X indicates that the issue was identified in the key findings of the exercise. NObj indicates that the issue was not part of the exercise objectives and therefore not considered.
In the UK’s exercise Winter Willow in 2007, it was identified that, public messaging needed to be refined to be better coordinated to ensure clarity and consistency. Yet 9 years later during exercises Cygnus and Cygnet it was identified again as an issue, pandemic communications plans should be developed and practised. The issue of communicating with the public was an issue once more in the UK in 2007, Public messages needed to be refined to be better coordinated to ensure clarity and consistency and in 2016 it was suggested that pandemic communications plans should be developed and practised. Winter Willow highlighted the need for a clearer understanding of the role of authorities involved in response, and 9 years later during exercises Cygnus and Cygnet, again the need to clarify roles was an issue identified. In 2007 issues with antiviral distribution was identified as a problem and in 2016 it was again identified that there was a need to develop the strategy for the use of antivirals.
This repetition of findings identified in 2007 and then subsequently in 2016 is repeated across the main issues identified during the pandemic flu exercises examined.
In 2018, Scotland, representatives of whom were present during the UK wide exercises, held a national exercise (Iris) and again issues identified included a lack of PPE at the beginning of an outbreak, which was also identified during the Scottish exercise Silver Swan in 2015. There were also PPE procurement problems including fit, training and testing issues along with the need to clarify roles and responsibilities and communication issues. This demonstrates that issues identified in exercises are not being dealt with and are therefore being identified in subsequent exercises.
Within the US a similar pattern emerges. Issues identified in the Dallas County exercise of 2006 were subsequently identified in 2019 during the exercises Crimson Contagion and Event 201. In 2006 the US Government exercise identified that there was an issue with unclear roles during the pandemic, and subsequently in the Crimson Contagion exercise it was noted that there was a lack of clarity of the roles of different federal agencies. The same repetition was found with respect to the procurement and delivery of vaccines and medical equipment, communication, pandemic data and school closures.
Compounding the situation of identifying the same issue in subsequent TTX’s is the fact that the findings of exercises are not always published. Exercise Cygnus and Cygnet are a case in point, where the UK Government initially refused to publish the data as it deemed it, not in the public interest. Only following a Freedom of Information request was the information released to the public, four years after the exercise. It was then that the media noted that some of the key points that were identified were not fully implemented into Government’s preparedness plans. Furthermore, other countries that were running similar exercises in this four-year period could have benefited from the information.
The WHO report of 2018, “A practical guide for developing and conducting simulation exercises to test and validate pandemic influenza preparedness plans,” includes New Zealand’s Cruickshank report, as an example of how to report.
New Zealand not only promptly and publicly reported the findings of the pandemic flu exercises; they also produced clear public information about what the Government was doing and what they expected the public to do [40]. The report indicated the need to update information on communication channels and procedures in the New Zealand Pandemic Action Plan.
The UK Government have recognised pandemic planning and the need to conduct regular exercises as important. The last major national one in the UK was Exercise Cygnus that simulated a flu outbreak in 2016. But as the government did not publish its findings, many stakeholders, including private care home providers, were unaware of it and so unable to learn from its lessons.
Transparency and trust are vital elements of pandemic preparedness [41].
It can be seen from the table-top exercises examined, that in many cases the lessons learnt in one exercise were not addressed, as subsequent exercises gave rise to the same or very similar issues being identified. This happened even though exercise evaluation often indicated that the exercise had been a success. For example, the UK’s exercise Cygnus report stated that the results of the evaluation process led to a favourable judgement by the participants of the success of the whole exercise with between 70 - 90% of them agreeing or strongly agreeing with the statement that, “the aim of the exercise was achieved” [42]. Similar high scores are achieved in other exercises, e.g. the Dallas Exercise of 2006 where overall the participant’s ratings were favourable in all categories, with average scores between 4.2 and 4.6 (with 5 being the highest score) [43].
Traditional methods of TTX evaluation cannot measure the effectiveness of the exercise. At best they can help identify the gaps and issues and at worst they are simply a means by of giving self-satisfaction to the exercise designers and facilitators.
For a TTX to be effective, the issues identified need to be addressed. There is currently no system, to the author’s knowledge, that can monitor the successful or even partial implementation of identified issues that is then linked to the TTX in which the issues were initially identified. An improved method of measuring the efficiency of a TTX, could be to examine the issues reported, and see if they have been successfully dealt with. This could be as an extra part of the TTX assessment. Hence there would be a measure of how well the exercise was run and or how well participants performed during the exercise, i.e. the traditional exercise evaluation, followed by a measure of the exercise’s effectiveness, by identifying for example, 8 out of 10 of the issues identified were subsequently dealt with in a reasonable time frame. Where one issue is of more importance than another, the results could be weighted accordingly. At the very least, at the beginning of any subsequent TTX, addressing the same or similar issues, there should be an examination of the issues identified in the previous TTX to see if they have been fully implemented. There is little point in running subsequent exercises that will identify the same issues.
To perform an evaluation of high quality and to give a measure of the TTX’s effectiveness is labour intensive and takes time. It should be noted that performing pandemic simulation exercises, as recommended and part of the benchmarking of health security and related capabilities, reported in the GHS Index, does not mean that a particular country will be able to confront a pandemic such as COVID-19, as can be seen in Table 2. However, performing exercises that are properly evaluated and, by promptly and publically publishing after-action reports, countries can transparently demonstrate their response capabilities and actions for improvement. Monitoring the implementation of these identified actions and linking their implementation to the exercise in which the issue was first identified, will give a measure of the exercise’s effectiveness.
Table 2. GHS Index position 2019 and COVID-19 deaths for countries examined
|
GHS Index position |
Country |
Deaths from COVID-19 |
Deaths / 100k population |
|
1 |
United States |
347 555 |
105 |
|
2 |
United Kingdom |
74 570 |
112 |
|
3 |
New Zealand |
25 |
0.5 |
Notes: COVID-19 death data taken from the WHO dashboard. [44] Data correct as of 4 January 2021.
There have been many simulations over the past two decades that have highlighted the risks of a pandemic and identified gaps in the ability of governments and organizations around the world to respond. Although no table-top exercise can convey a realistic picture of a pandemic, they are useful as a tool to help assess plans, policies, and procedures, clarify roles and responsibilities, and identify gaps in resources needed for response.
One of the problems with table-top exercise evaluation is that it is often a form of self-evaluation with exercise designers fulfilling the role of evaluators. To reduce the potential for bias and to have a more impartial assessment of an exercise, external evaluators should be employed, wherever possible.
The extent to which deficiencies identified in exercises are addressed is a measure of the table-top exercise effectiveness. If similar issues are discovered at a subsequent exercise, as was seen in this study then the exercise is not effective. This results in lessons not being learned, actions or recommendations not being implemented and is a waste of resources, both in time and money.
Following a table-top exercise or other pandemic simulation exercise, government departments should openly publish the key findings and a roadmap for correcting issues identified. Instead of repeating an exercise that is designed to identify issues, it would be more effective and perhaps less costly, to monitor the progress of implementation of the recommendations made at the end of the previous exercise.
Even though New Zealand scored badly on The Global Health Security Index, which considers amongst other issues pandemic threats, coming in 35th in the world rankings, it has managed the COVID-19 pandemic better than many countries. New Zealand not only promptly and publicly reported the findings of the pandemic flu exercises, they also produced clear public information about what the Government was doing and what they expected the public to do. In contrast, information from government exercises carried out in the US and the UK were shrouded in secrecy and only came to light following freedom of information requests. Transparency and trust are vital elements of pandemic preparedness. Although New Zealand’s success in pandemic management cannot be considered to be solely due to successful implementation of issues identified in exercises, it must have helped.
It is essential to improve the evaluation of exercises and find ways to determine the effectiveness of table-top exercises as the number of exercises will most likely increase in the future.
Following publication of the final exercise report it is necessary to track the implementation of actions. Completion of an action identified as necessary in an exercise is potentially a measure of the exercise’s effectiveness. Exercise evaluation and determining the effectiveness of exercises are part of the on-going process of improvements in preparedness.
Conducting an exercise is important, but if the exercise is not evaluated correctly and the lessons learned are not applied, then there is little point in performing the exercise. Table-top exercises are not cost free, their effectiveness has to be demonstrated or they are potentially a waste of time and resources and do not contribute significantly to pandemic preparedness.
[1] Dr Lee Jong-wook, 7 November 2005, Opening remarks from the meeting on avian influenza and pandemic human influenza, Geneva, Switzerland.
[2] Pandemic influenza preparedness in WHO Member States: report of a Member States survey (2019). Geneva: World Health Organization.
[3] European Centre for Disease Prevention and Control. Preparedness planning for respiratory viruses in EU Member States – Three case studies on MERS preparedness in the EU. Stockholm: ECDC; 2015.
[4] International Health Regulations (IHR [2005]).
[5] Global Health Security Index (2019) Nuclear Threat Initiative.
[6] Dausey, D.J., Aledort, J.E., Lurie, N. (2006). Tabletop exercises for pandemic influenza preparedness in local public health agencies. Rand Corporation.
[7] Dallas County Health & Human Services, Avian Flu Pandemic Table Top Exercise After Action Report, 31 May 2006.
[8] Johns Hopkins Center for Health Security (2018) Clade X Exercise. https://www.centerforhealthsecurity.org/our-work/events/2018_clade_x_exercise/index.html, accessed on 16 Sep. 2020.
[9] Department of Health and Human Services (HHS) Crimson Contagion 2019 Functional Exercise After-Action Report, 2020.
[10] US Department of Health and Human Services Annual Report (2019).
[11] Crimson Contagion 2019 Exercise Draft After-Action Report, US Department of Health and Human Services, Oct 2019. https://int.nyt.com/data/documenthelper/6824-2019-10-key-findings-and-after/05bd797500ea55be0724/optimized/full.pdf#page=1, accessed on 16 Sep. 2020.
[12] Johns Hopkins Center for Health Security (2019). Event 201.
[13] Johns Hopkins Center for Health Security, Event 201 recommendations. https://www.centerforhealthsecurity.org/event201/recommendations.html, accessed on 16 Sep. 2020.
[14] NHS England Emergency Preparedness Resilience and Response (EPRR) Core Standards 2016: Compliance Update. https://sandwellandwestbhamccg.nhs.uk/publications/governing-body-papers/2016/august-10/1328-13-enc-12a-agenda-item-8-1-governing-body-nhs-england-emerggency-preparedness-resilience-and-response-eprr-core-standards-2016-compliance-update/file, accessed on 16 Sep. 2020.
[15] House of Lords Science and Technology Committee, 3rd Report of Session 2008-09, Pandemic Influenza: Follow-up, Report with Evidence, 28 July 2009.
[16] Public Health England, Pandemic Influenza Response Plan 2014.
[17] UK Department of Health, Exercise Winter Willow Lessons Identified, (2007).
[18] ibid.
[19] National Risk Register of Civil Emergencies chapter 1: main types of civil emergency, 27 March 2015, https://www.gov.uk/government/publications/national-risk-register-for-civil-emergencies-2015-edition/national-risk-register-of-civil-emergencies-chapter-1-main-types-of-civil-emergency, accessed on 25 Sep 2020.
[20] Public Health England, Pandemic Influenza Response Plan 2014.
[21] Annex D – Participant feedback, Exercise Cygnus Report Tier One Command Post Exercise Pandemic Influenza 18 to 20 October 2016, Public Health England, (2017).
[22] Lord Hunt of Kings Heath, UK Parliament, Hansard, HL Deb (9 June 2020), vol.803.
[23] Pegg, D., Booth, R., Conn, D. (2020). Revealed: the secret report that gave ministers warning of care home coronavirus crisis,” The Guardian, 7 May 2020. https://www.theguardian.com/world/2020/may/07/revealed-the-secret-report-that-gave-ministers-warning-of-care-home-coronavirus-crisis, accessed on 7 May 2020.
[24] Leigh Day, Legal case issued in campaign for publication of Cygnus Report, 1 June 2020. https://www.leighday.co.uk/News/Press-releases-2020/June-2020/Legal-case-issued-in-campaign-for-publication-of-C, accessed on 26 Sep 2020.
[25] Nuki, P. (2020) The six crucial pandemic lessons the government ignored, The Telegraph. https://www.telegraph.co.uk/global-health/science-and-disease/six-crucial-pandemic-lessons-government-ignored/, accessed on 26 Nov 2020.
[26] Exercise Silver Swan Overall Exercise Report, Scottish Government, April 2016.
[27] Hutcheon P. (2020) Secret Scottish Government report warned of pandemic planning four years ago, Daily Record. https://www.dailyrecord.co.uk/news/politics/secret-scottish-government-report-warned-21959615, accessed on 29 Sep 2020.
[28] Exercise Iris Report, Scottish Government, 3 Jun 2020.
[29] Titheradge, N., Kirkland, F., Coronavirus: Outbreak exercise showed ‘clear gap’ in readiness (2020) BBC News, 5 June 2020. https://www.bbc.com/news/uk-52927462, accessed on 29 Sep. 2020.
[30] Exercise Iris Report, Scottish Government, 3 Jun 2020.
[31] Exercise Virex National Report, Ministry of Health New Zealand, March 2002.
[32] Exercise Cruickshank, Exercise Coordinating Instruction, Ministry of Health New Zealand, Version 2: 27 Feb 2007.
[33] FluTrackers.com. https://flutrackers.com/forum/forum/oceania/h5n1-tracking-ai/new-zealand/18925-nz-exercise-cruickshank-to-test-pandemic-plans, accessed on 28 Sep. 2020.
[34] NZ’s Influenza Pandemic Prepardness World Class, Scoop Health, 30 Oct 2007. https://www.scoop.co.nz/stories/GE0710/S00120/nzs-influenza-pandemic-preparedness-world-class.htm?from-mobile=bottom-link-01, accessed on 28 Sep. 2020.
[35] Report on Exercise Cruickshank, Ministry of Health New Zealand, Oct 2007.
[36] Preparation, conduct and evaluation of exercises for security of nuclear and other radioactive material in transport, IAEA, VIENNA, 2018.
[37] Homeland Security Exercise and Evaluation Program (HSEEP), Homeland Security, Jan 2020.
[38] A practical guide for developing and conducting simulation exercises to test and validate pandemic influenza preparedness plans. Geneva: World Health Organization; 2018.
[39] Deniz, E. (2002). Approaches to evaluation of training: Theory & practice. Educational Technology & Society, 5(2): 93-98.
[40] Working from the same page consistent messages for CDEM, Pandemic Influenza, Ministry of Civil Defence & Emergency Management Version 1.0 April 2010.
[41] Global Health Security Index. (2019). Nuclear Threat Initiative.
[42] Annex D – Participant feedback, Exercise Cygnus Report Tier One Command Post Exercise Pandemic Influenza 18 to 20 October 2016, Public Health England, 2017.
[43] Dallas County Health & Human Services Avian Flu Pandemic Table-Top Exercise After Action Report, 31 May 2006.
[44] WHO Coronavirus Disease (COVID-19) Dashboard, data retrieved on 4 January 2021, https://covid19.who.int/table.