Disaster Risk Reduction for Urban Sustainable Development by Training and Exercises in Transport: A Logical Framework Approach

The analysis of documents and papers highlights the main gap in the literature that this paper aims to address. This section refers to:

•LFA specified for structuring an evacuation plan;

•scientific works presenting methods and models for reducing exposure, with specific reference to the analysis of training and exercises.

2.1 Logical Framework Approach structures

Sustainability and global challenges related to DRR impose the evaluation of the potential impacts produced by evacuation planning using transportation networks. The aim is to understand how the resources (human and financial) used to develop actions produce impacts in terms of risk reduction. The primary concern is to ascertain the extent to which evacuation training and exercises contribute to risk reduction. To this end, the paper adopts the LFA approach used in various technical and scientific fields to assess in a structured manner the effects of a generic action in the context of the “theory of change” [10].

European Commission adopts the LFA (or its variants) as a standard method required by many donors for grant applications [11]. The LFA tool is recommended for assessing how resource use can achieve objectives in relation to quantitative targets. A further element to consider concerns the difference in the assessment of tangible and intangible actions aimed at risk reduction.

The effects of tangible actions affect the material/immaterial structure of a system and can be assessed using established methods (e.g. structural calculations). For example, the design of embankments or earthquake-resistant buildings can be assessed using scientific and empirical approaches. Nevertheless, the quantitative estimation of the effects of immaterial actions, including exercises, is more complex. Indeed, in the latter scenario, the consequences are contingent on heightened awareness, behavioural modification, or an augmentation in competencies and organisational elements [12].

The Hyogo Framework for Action (HFA) 2005–2015 is the action program preceding the Sendai Framework; HFA recommends adopting the LFA approach to assess the level of preparedness [13]. For this reason, the United Nations has produced guidelines for defining classes of indicators for the individual components of LFA [14]. The proposed indicators mainly refer to capacity and capability in relation to preparedness planning and readiness for response.

Russo and Rindone [15] proposed a prototypal adaptation of LFA for evacuation planning, specifying its main components:

•inputs or available resources to implement evacuation procedures, including people (for instance, police forces, volunteers, coordinators) and means for enabling mobility between dangerous and safe areas;

•exogenous activities or evacuation planning actions, including exercises to test evacuation procedures;

•outputs or products and services useful for enabling mobility in evacuation conditions;

•endogenous activities representing an increase in knowledge with respect to internal interactions in the transport system between its components (demand and supply); for instance, consider how transport services influence mode choice;

•outcomes or intermediate results in terms of the modification level of knowledge regarding the influence of the transport system on the travel users' behaviour in evacuation conditions;

•endogenous activities to evaluate the evolution from outcomes, measured in terms of knowledge of transport system performances (e.g. link times and flows; path times towards safe areas), to global capability regarding goals, measured in terms of achievements of SDG and Sendai targets;

•goals or final results in terms of the modification level of exposure and then risk reduction.

Each component can be measured with SMART (Specific, Measurable, Achievable, Relevant, Time-bound) indicators and compared with available and fixed targets.

In LFA, elements are connected by relationships that follow an “if” → “then” logic. The transition between two successive elements depends on external factors, i.e. on all the boundary conditions in which the system evolves (Figure 4).

image006.png

Figure 4. Logical Framework Approach

Table 3. Example of LogFrame for training and exercises

Note: LogFrame = Logical Framework Matrix.

The LFA is summarised in the Logical Framework Matrix (or LogFrame), translating into quantity and formal relationships the framework in the LogFrame matrix, consisting of (Table 3):

•Seven rows, one for each component (input, exogenous activities, output, endogenous activities, outcomes, endogenous activities and goals);

•Four columns corresponding to:

◦indicators, i.e. measures associated with each component;

◦baseline and targets;

◦means of verification, including models and methods for calculating indicators;

◦external factors, including elements that cannot be controlled by the planner.

Further investigation is required to determine the potential benefits that result from the implementation of one or more training and exercise actions [16]. It is necessary to distinguish between the organisational phases and the related resources (input), activities (activity) and products (output) from the immediate results (outcome) and long-term results (goal). An example of estimating the risk occurrence in the case of transporting dangerous goods is presented in the studies of Russo and Rindone [15, 17], which adapts Transport Risk Analysis (TRA) and estimates indices based on the type and quantity of hazardous substances transported.

2.2 Transport Risk Analysis

TRA provides a systematic framework for quantitatively estimating the risks associated with transport systems through the analysis of both the likelihood of catastrophic events and the consequences arising from their occurrence. TRA is applied across numerous transport sectors, contributing to the quantification of the achievements of the 2030 Agenda in various ways [18].

In the specific context of this paper, TRA enables us to assess how the proper management of transport infrastructure and services can reduce exposure, as outlined in the introduction. Quantitative models need to be developed to simulate transport demand and supply and their mutual interactions, in order to represent the progression of disaster scenarios and identify risk mitigation measures through transport planning. As highlighted [5], transport networks, interactions between users, infrastructure and regulations can either amplify or mitigate risk. The ultimate aim of TRA is to support public decision-makers in defining effective mitigation strategies, through physical and non-physical infrastructure interventions, mobility management policies and safety measures, adopting an integrated and preventive approach [19]. Recently, the integrated use of Emerging Information and Communication Technologies (e-ICT) tools alongside Transport System Models (TSMs) has made a significant contribution to TRA in the planning of transport systems under risk conditions [20], as well as to the preparedness phase through the design and implementation of emergency simulations and exercises [21]. It should be noted that TRA’s contribution to achieving the SDGs is evidenced by recent literature [22].

2.3 Transport System Models for evacuation planning

The study of risk exposure involves understanding transport systems and their components, i.e. mobility demand in terms of people's travel needs over a given period of time, transport supply, which includes the infrastructure and services that enable people to travel, and their interaction in terms of performance (e.g. flows and times) of individual elements and the entire system [23, 24]. Literature presents consolidated approaches for the modelling of transport systems and their individual components. TSMs are widely used to represent systems under normal conditions. TSMs are currently being developed to represent transport systems in emergency situations, particularly during evacuations [5].

TSMs can be employed to simulate the performance of transport systems under emergency conditions produced by a disaster. The main mathematical equations representing the TSM in a dynamic approach with ordinary conditions are presented in previous research [25, 26]. Advanced analyses regard the transport supply component aimed at quantifying the level of centrality of nodes and links in a general urban transport network. This is possible by adopting network analysis approaches [27]. In evacuation conditions, it is important to represent the temporal variations of the transport system in relation to dynamic evolutions. The entire system is represented by means of six equations that enable analysts to evaluate transport system performance during evacuation conditions. For improving these performance towards risk reduction goals and target, two methodological approaches can be adopted [28, 29]: (i) a “what-if” approach, which assesses the impacts of planned interventions aimed at modifying transport services and infrastructure to manage emergencies; and (ii) a “what-to” approach, which seeks to design an optimal transport system configuration that minimizes exposure to risk. Some comprehensive reviews of evacuation transportation modelling are provided by different authors [30]. TSM applications allow analysts to calculate outcome indicators in relation to the ex-ante evaluation of evacuation plans and experiment with these plans in a stress-free environment.

Emerging information and communication technologies (e-ICTs) can enhance TSM's ability to represent real emergency conditions. Information and data collected by e-ICT can be used to integrate the TSM building process (specification, calibration and validation). E-ICT tools can be employed to update travel utility functions [31], thereby enabling dynamic modelling of travel choices under emergency conditions [20]. ICT solutions and TSMs can be integrated within a spatial Decision Support System (DSS), which is configured as a digital twin (DT) [32], which provides emergency managers with preventive insights regarding the potential impacts of their decisions. The most advanced approach regards the construction of a serious game for representing user behaviours in an evacuation procedure [21] that, from a general point of view, converges to a DT.

Inside the general problem of evaluating goal indicators from outcomes, there is a recent problem that emerges (RISK 2022 [20, 33, 34]) concerning the method, for instance the specification of the model, as a result of the definition of the problem, and the identification of the relevant parameters and any coefficients for experimentation with the definition of the tests to be carried out in reality and any emerging ICTs to be used [35, 36]. This involves creating specific exercises [37] which are supported by the use of e-ICTs such as Digital Twins, that reproduce an emergency scenario in a virtual environment [32]. Further experimentation may be useful to explore the contribution of other e-ICTs, such as blockchain [35]. It is necessary to verify which parameters are present in the international literature to reduce the risk component related to exposure in order to plan the safe destination to be reached and the route to be followed in the urban network by the various population groups after the risk of a disastrous event that could affect the city has been communicated (or is approaching). The challenge is to assess the level of user perception of the evacuation plan instructions [20, 31, 34, 38, 39].

Table 4 summarises the literature referenced and produced as part of the PRIN RISK 2022 research project. The table classifies each reference according to:

•Research topic;

•Exercise class;

•Typology of e-ICT;

•Presence of indicators to measure some LFA component.

Table 4. Summary of scientific literature about Transport Risk Analysis (TRA) and Transport System Models (TSMs)

Note: LFA = Logical Framework Approach.

2.4 Logical Framework Approach specification for exposure reduction in urban transportation networks

The proposed framework is based on the LFA approach [12, 15]. It is necessary to specify:

•Inputs, in terms of human, financial and technical resources to perform an evacuation; it is possible to evaluate their effectiveness, cost-effectiveness and appropriateness;

•Exogenous activities and processes, related to the execution of tasks to perform an evacuation, including influencing factors;

•Outputs or immediate products and services provided by the evacuation plan;

•Endogenous activities related to travel user behaviour during an evacuation and the factors that influence travel choices;

•Outcomes in terms of impacts or transport network performances (e.g. link times, flows and evacuation times);

•Endogenous activities related to the transformation from transport performances to risk reduction;

•Goals measured as exposure reduction and therefore risk reduction.

Actions in terms of inputs and exercises, through exogenous activities, produce outputs in terms of capabilities (Figure 5) that must be measured.

The problem then becomes predicting the introduction of analytical formulations that allow the effect of the exercises on changes in the behaviour of decision-makers and end users to be estimated in advance. A concrete example concerns evacuation routes. The set of exercises has effects on two sub-processes:

•Awareness of the routes to follow in an emergency to reach a safe destination;

•Learning process on the optimal route indicated in the evacuation plan, which allows the minimum level of exposure (E_min) to be achieved.

A series of exercises helps to achieve the Emin exposure level, which corresponds to the reduction in disaster risk estimated in the plan.

From the previous analyses, it is clear that specific indicators should be identified for each of the LFA phases in each class of exercise. The different types of indicators make it possible to identify the achievement of the targets for the four components: input, output, outcome and goals.

The proposed specification, therefore, allows the entire process to be monitored, not just the achievement of the final goals. Monitoring the process allows shortcomings and deficiencies to be identified and any necessary changes to be made along the way, rather than only finding out the results by evaluating the indicators relating to the goals.

image007.png

Figure 5. Proposed framework: from prescriptive to performance-based approach

4.1 Immaterial actions of an emergency plan

The previous section highlighted some gaps in technical literature. It is therefore useful to refer to scientific literature to verify the existence of a scientific frame of reference for evacuation using transport network.

Specific attention is given to the field of research concerning training and exercises for evacuation using the transport system and the possibility of measuring the benefits in terms of risk reduction.

Training and exercises are among the immaterial measures of a plan aimed at reducing risk and, in particular, exposure.

The plan measures and planning methods are of interest to emergency managers (M) and public authorities (Pa).

The plan measures include training and exercises. Russo and Rindone [58] emphasised the role of emergency exercises and their potential contribution to emergency planning in a back path. The methodological framework for organising exercises is based on the American HSEEP Programme [7]. In particular, it is necessary to measure the potential benefits of exercises for citizens (C).

The actions in the plan concern civil protection by public authorities (Pa) and are managed by safety managers (M). There are various training courses on disaster management. One of these concerns the transport safety advisor, who is mandatory in the dangerous goods transport sector. However, emergency managers focus more on organisational aspects, the procedures to be implemented and the chain of command for managing an evacuation. Instead, the effects of a plan must be assessed both before and after implementation, and to this end, methods and models must be defined to assess the effects produced by the exercises [59]. In particular, in the past, several research projects have developed methods and models for simulating mobility in emergency conditions [19]. In more recent years, there has been a greater understanding that training and exercises are part of a broader cycle of emergency planning and management.

Table 7 summarises the reference literature developed by the authors, with all the literature cited therein, regarding methods and models for assessing risk reduction through training and exercises. The works cited emphasize the importance of training and emergency exercises, but above all, the need to adopt methods and models for simulating transport systems to assess in advance the travel choices of users in evacuation conditions using appropriate indicators.

To this end, it is necessary to have a structure in place to measure the reduction of exposure to disaster risk through training and exercises.

Table 7. Methods and models for risk reduction through training and exercises

Note: C = citizen; M = manager; Pa = public authority.

The problem that arises, as mentioned above, is to define the organisational phases in two steps:

•Mitigation, with the preparation and model simulation of the plan in order to achieve optimal evacuation times

•Preparedness, with the preparation of the exercises and assessment of the effects produced by them in order to achieve the optimal evacuation times calculated in the plan.

As seen in Chapter 3, some of the guidelines analysed allow for the assessment of capabilities as well as capacities. However, there is no link between capacities/capabilities and final goals.

In order to overcome the limitations of the literature, it is necessary to develop a structure that allows for the definition of analytical functions that measure, using appropriate indicators, the level of exposure reduction based on acquired capacities and capabilities.

International guidelines emphasise the importance of pursuing SDGs with related indicators and targets to measure risk reduction, as seen above. However, while the analytical formulations of the plan indicators have been identified, there is a lack of analytical formulations that measure the effects of the training and exercises included in emergency planning. These actions include those related to community preparedness that improve awareness and knowledge of the behaviours to follow in an emergency.

The proposed structure requires the introduction of analytical formulations for the different indicators needed to measure risk reduction. For each indicator, it may be useful to refer to a target against which to make a quantitative comparison to determine whether the objective has been achieved or can be achieved.

4.2 Building a specific exercise

This section details the LFA components required to structure a specific exercise aimed at training managers who specialise in the management of evacuation procedures. Structuring the activity in accordance with LFA makes it possible to identify the main components and highlight the logical flow that links the activity’s objectives, the resources required to achieve them, and the effects it may produce in terms of capacity and capability.

The example referred to relates to a training activity aimed at training emergency managers who can apply the knowledge and skills required to manage an evacuation procedure using a transport network.

The individual components of the training activity are specified below in accordance with LFA.

Input.

The training program must ensure verifiable improvements in the knowledge and skills acquired by managers regarding TSM and TRA methods and models.

It is assumed that a public administration body intends to train at least 10 managers who can demonstrate that they have acquired the necessary knowledge and skills.

To this end, the organization is funding a training course aimed at a potential 20 participants (n_input); in funding the project and the delivery of the course, the organization requires that the following conditions be respected:

As regards the outputs:

•A participation rate for the entire training programme ($\bar{\upsilon}_{\text {output}}$) of at least 80% of the total number of participants, expressed as:

$\bar{\upsilon}_{\text {output }}=80 \%$

As regards the outcomes:

• $\overline{\mathrm{k}}_{\text {outcome}}$: Minimum number of correct answers to acquire the capability.

•A success rate of 60% of participants in the basic interim tests on acquired knowledge and skills (TSM and TRA), expressed as:

$\bar{\upsilon}_{\text {outcomes}}=60 \%$

As regards the goals:

•f: Correct evacuation flows estimated by plan.

•A success rate of 50% in a final test on the application of the TSM, expressed as:

$\bar{\upsilon}_{\text {goals}}=50 \%$

Exogenous activities

Exogenous activities include discussion-based activities including the awarding of a certificate upon passing an exam.

The training activities are organised into modules which, in line with the objectives of the initiative, are designed to cover the following topics:

•TRA methods, with reference to the risk exposure component;

•TSM methods for estimating flows on a network under different supply and demand conditions.

Output.

Once the external activities have been completed, the main outcome is the participants’ capacity, which can generally be measured using a set of indicators. In the case of the test analysed, the number of participants who completed the planned sub-activities is quantified (n_output).

Output verification

The indicator $\upsilon_{\text {output}}$ is used to verify the outputs’ achievement; the values of $\upsilon_{\text {output}}$ must be greater than or equal to the threshold $\overline{\mathrm{\upsilon}}_{\text {output}}$ defined for the exogenous activities.

$\upsilon_{\text {output}}$ = n_output/n_input: Success rate of effectively trained managers.

The outputs’ achievement is verified if:

$\bar{\upsilon}_{\text {output }}<\upsilon_{\text {output }}$

It is assumed that 18 managers participated in the course, then:

$\upsilon_{\text {output}}$ = 18/20 = 90% > 80%

The output is verified.

Endogenous activities from outputs to outcomes.

Capacity is a necessary but not sufficient condition for achieving the expected outcomes of the activities. It is therefore essential to assess the current level of knowledge and skills acquired by participants after training and/or exercise activities. To this end, the participants must pass a test to measure their level of capability.

Capability reflects the actual level of learning of the concepts covered during the activity.

A multiple-choice questionnaire with closed-ended questions on the topics illustrated can be used [39, 61].

In the introduced example, the endogenous activities from outputs to outcomes are represented by a test consisting of a questionnaire composed of 20 multiple-choice questions to verify the acquisition of the capabilities by participants.

Participants must respond to a multiple-choice questionnaire to demonstrate their capability acquisition. The answers provided by each participant demonstrate their level of understanding of the course content.

Outcomes

Endogenous activities produce outcome indicators, which correspond to a predefined level of capability and a minimum number of correct answers to the questions.

Outcome verification

Outcome verification is performed by introducing a threshold value of correct answers to determine whether the outcome has been achieved:

Having the answers to the multiple-choice questionnaire, it is possible to calculate.

n_outcome: Number of managers that acquired the capability, in relation to the threshold $\overline{\mathrm{k}}_{\text {outcome }}$.

It is possible to calculate an indicator to verify the level of achievement of the outcomes, which must be greater than or equal to $\bar{\upsilon}_{\text {outcome }}$.

$\upsilon_{\text {outcome}}=\mathrm{n}_{\text {outcome}} / \mathrm{n}_{\text {input}}$: Success rate of managers who have acquired the capability.

The outcomes’ achievement is verified if:

$\bar{\upsilon}_{\text {outcome}}<\upsilon_{\text {outcome}}$

It is assumed that 15 participants in the course passed the test, then:

n_outcome = 15

$\upsilon_{\text {outcome}}=\mathrm{n}_{\text {outcome}} / \mathrm{n}_{\text {input}}$ = 15/20 = 75% > 60%

The outcome is verified.

Endogenous activities from outcomes to goals

The final level of LFA implies the knowledge of TSM and TRA for managing evacuation procedures. A test is useful to verify the knowledge level of each manager about TSM and TRA available in the literature and taught during the course.

Each manager must apply the TRA-TSM model to a test road network and must calculate the values of the performance indicators.

In the example, the 15 managers who have acquired the capability are asked to simulate the evacuation planning procedures for a city.

Figure 8 represents the road network used to conduct the test, with quantitative characteristics for each link. The objective of the test is to estimate the DRR by calculating evacuation times, or in other words, the total time taken by users to reach the safe areas (‘assembly points’).

The travel time over each road link is equal to the ratio of the distance to the speed.

Each origin zone is assigned to an 'assembly point', which is the destination of the evacuation trip.

Total evacuation times must be calculated for each pair of origins and destinations (o-d). The demand flow to evacuate for each o-d couple is known and constant. There are 2000 users, one for each car, in A and 1000 in B.

For each o-d pair, managers must select the evacuation route to be indicated to users to minimise travel times.

Two different routes are available for each pair.

The proposed approach is that of prescriptive assignment, in which managers must assess the user flows that will be present in the network as a result of their choices.

For each o-d pair, managers must choose the road route compatible with the available information regarding the network arcs. The available information is shown in the table in Figure 8.

image017.png

Figure 8. Road network for test with quantitative characteristics

Goals

The exposure indicator measures the evacuation time produced by the managers’ decisions, with or without the knowledge of TSM – TRA methodologies.

For each o-d pair, it is assumed that managers select a single evacuation route for users to follow.

A lack of knowledge about TSM may lead to an underestimation of the capacity variable. Managers who lack this knowledge may underestimate capacity constraints, allocating the entire demand to the routes with the shortest travel times. Managers with this knowledge must assess the feasibility of the route to be selected, i.e., whether the demand for each route complies with the capacity constraint.

For the pair A–S1, if managers adopt the criterion of minimum travel time without taking the capacity constraint into account (unlimited capacity), they will select route I. However, when the capacity constraint on the 1–2 section is considered, the demand is not entirely compatible.

Therefore, up to 900 users can be assigned to Route I, whilst the remainder must be assigned to Route II.

For the origin-destination pair B–S1, the same capacity constraint issue arises on Route III (Table 8).

Table 8. Evacuation times in the three hypotheses

* Unfeasible path for the link capacity constraint

An understanding of TSM-TRA methods should guide managers in selecting evacuation routes on the road network.

Through the course content, managers should be able to apply the correct selection criteria.

This knowledge enables managers to undertake the final assessment, demonstrating that they have acquired the necessary skills and know how to apply them. The course therefore delivers a demonstrable benefit in the form of the ability to correctly select viable routes and the associated travel times.

Goal verification

The assessment of the objectives is carried out by considering one or more reference variables for TRA and TSM.

In this case, the variable is:

t_goals: Maximum time to evacuate a portion of a city.

Once the managers have completed their tasks, it is possible to calculate.

$\upsilon_{goal}$: Number of managers who calculated the correct time t_goals.

The goal’s achievement is verified if:

$\bar{\upsilon}_{\text {goal}}>\upsilon_{\text {goal}}$

In the example under consideration, it is assumed that 13 managers completed the exercise correctly. It is possible to calculate:

n_goal = 13

$\upsilon_{\text {goal}}=\mathrm{n}_{\text {goal}} / \mathrm{n}_{\text {outcome}}$ = 13/20 = 65% > 50%

The goal is verified.

A summary of the LFA components is shown in Figure 9.

image019.png

Figure 9. Logical Framework Approach (LFA) for evaluating exercise results

This work highlights the shortcomings of current guidelines, with various overlaps in definition and the lack of a general framework. The prescriptive approach is generally adopted. The work, following a performance-based approach, proposes a structure derived from LFA to classify the indicators available in the literature.

These crucial elements highlight the lack of studies that allow training and exercise activities to be analytically linked to the pursuit of the objectives identified in evacuation plans.

This gap is the element on which scientific research and technical structures can work.

In relation to the limitations mentioned above, this research may develop in different directions in the future. One possible direction is to analyse the experiences of national and local administrations in implementing training and exercise actions in a real context, verifying the results obtained against the plan targets. Further advances concern the in-depth study of advanced TSM methods and possible interactions with the potential offered by e-ICT technologies.