Implementation and Effectiveness of Crew Resource Management in the Medical Sector

Implementation and Effectiveness of Crew Resource Management in the Medical Sector

Lea C. KraftJana Benning Verena Schürmann Nicki Marquardt 

Department of Industrial and Organizational Psychology, Rhine-Waal University of Applied Sciences, Kamp-Lintfort 47475, Germany

Corresponding Author Email:
10 March 2020
28 May 2020
30 June 2020
| Citation



Crew Resource Management (CRM) is a simulation-based team training that strives to reduce human errors in emergencies and to increase patient safety by improving non-technical skills. This qualitative study examines the current implementation and effectiveness of medical CRM training in German-speaking countries (Germany and Switzerland). Data was collected through interviews with 20 experts who conduct CRM training in various disciplines and application contexts. The material was analyzed first, using qualitative content analysis, and second, a frequency analysis was conducted. In order to ensure inter-rater reliability, Cohen's kappa was calculated. The results are consistent with research and showed that CRM in German-speaking countries is mainly based on the same principles, and training is conducted similarly. However, CRM is not widespread yet and requires consistent standards. Improvement in behavior in everyday professional life after training sessions have been observed, but no clear evidence of effectiveness on the outcome of the training has been provided to this point. Utilizing this study, German-speaking CRM applicants can compare their training implementation with that of the presented sample. This study is the first to assess the current implementation and effectiveness of CRM in German-speaking countries from the perspective of different disciplines and professions in the medical sector.


crew resource management, human factors, non-technical skills, team training, training effectiveness

1. Introduction

Since the publication of the report To err is human: Building a safer health system [1] medical errors and human factors have gained importance. Such remarkable statistics were noticed in Germany as well and show that 1.499 medical errors and risk education deficiencies are responsible for the health damage suffered by patients in 2018 [2]. Moreover, according to a member survey of the German Society for Orthopaedics and Trauma Surgery, 65% of the participants admitted that they perceive their own mistakes at least once a week, which are mainly due to time pressure, inadequate communication, lack of personnel and stress [3]. Between 70% and 80% of all errors are due to human factors, and 80% of these are potentially preventable [4]. Human factors influence performance and safety, especially in complex situations [5]. Further, the more people work together, and the more work steps required, the higher the probability of medical errors [6]. The exploration of ways to reduce human errors and improve safety culture is therefore essential and can decide over life and death for every patient.

1.1 Execution of Crew Resource Management

Crew Resource Management is a team training that is applied in high-risk sectors and primarily deals with an emphasis on human error reduction [7] by improving coordination and performance in the team [8]. CRM is focusing on the so-called non-technical skills since most errors do not result from a lack of technical skills, but rather from the difficulty of transferring the learned technical skills under the intense pressure of everyday practice [7]. These non-technical skills are conceptualized as task management, team working, situational awareness, and decision making [9]. According to Fletcher and colleagues [9], communication is not explicitly represented in this system since it is assumed that communication is essential for each category.

During the training, CRM participants simulate a critical situation or an incident with, e.g., the help of a simulation puppet, thereby improving their non-technical skills [10]. CRM training is divided into three sections: briefing, simulation, and debriefing. The briefing includes an introduction to CRM, familiarizing CRM participants with the simulator and the simulation environment as well as defining the objectives [11]. Directly after the simulation, a debriefing takes place where the simulated situation is reflected with a videotaped analysis [12]. Debriefing is considered as the most critical part of the training because it determines the learning effects [13]. Besides, there is also empirical evidence revealing that debriefings can increase team performance by 25% [14]. In terms of training duration, large differences from ninety minutes until one and a half days can be detected, which indicates no distinct regulations [15, 16]. According to a recommendation, the training should be repeated at least once a year with the entire team [17]. In order to make it easier to implement non-technical skills, some tools, such as the 15 guidelines developed by Rall and Gaba [18], address all non-technical skills, including communication. One guideline contains the application of the 10-seconds-for-10-minutes principle (so-called 10-for-10 principle), which means that every team member should take a short break, collect the thoughts, and reconsider the decision. A further guideline addresses the use of checklists to ensure that all essential steps are followed in the right order. Another tool for improving communication is closed-loop communication, which supports effective communication between the sender and receiver through ensuring that the message has been correctly understood and interpreted [19].

So far, only a few clinics in German-speaking countries have conducted medical CRM training. There are hardly any mandatory CRM courses in medicine [4, 17], and legal regulations for emergency and simulator team training in everyday work [20]. The lack of obligation for hospitals to implement CRM is also reflected in a lack of financial support [17]. Therefore, the data situation is insufficient regarding the current application of CRM training in German-speaking countries. Since medical journals and studies usually view the training from the perspective of a particular discipline, no cross-disciplinary studies are available so far. From this, the first and third research questions are derived, which will be addressed below. In the following, based on the previously described state of research, studies that evaluate the effectiveness of CRM training are reported.

1.2 Effectiveness of Crew Resource Management

In order to structure research results regarding the effectiveness of CRM, the Kirkpatrick and Kirkpatrick Model [21] for the Evaluation of Training Programs is used. This model evaluates a training program on four levels of reactions, learning, behavior, and results.

On the level of reactions, two studies state that CRM participants responded positively to the training [15, 22]. Regarding the level learning, the study of Chan and colleagues [22] revealed that a single CRM training improved attitudes towards team behavior and patient safety. In particular, CRM participants increasingly recognized the importance of addressing errors associated with the threat to patient well-being. In the meta-analysis of O'Dea and colleagues [15], CRM training has little effect on the attitudes of CRM participants, which suggests that the possible change in the behavior of CRM participants may not be based on personal beliefs of those participants. Whether the safety climate can be changed sustainably through CRM is, therefore, still questionable. Evidence for transfer of what has learned into habitual behavior is provided by Fransen et al. [23] with their multicenter, randomized, and controlled study from the field of obstetrics. Six months after a CRM training, a positive effect on team performance was measured. Another finding for the successful transfer of learned skills into everyday life found Morgan and colleagues [24]. According to the authors, increased use of non-technical skills was observed in the operation theatre of CRM participants. Another possibility to prove the effectiveness of the training on the behavioral level is to consider the application of CRM supporting tools in everyday working life. The probability increases that safety-relevant tools, such as checklists, will be used in the operating theatre after a training session [25]. In terms of treatment outcomes, one study demonstrated a reduction in complication and mortality rates three years after the implementation of CRM training in an intensive care unit [26]. However, other results were found [24], which even reported an increase in the complication rate in the operating theatre of CRM participants. In the same study [24], an improvement in non-technical skills was observed, which could indicate that an improvement in non-technical skills does not necessarily lead to an improvement in treatment outcomes. Further studies also show ambivalent treatment outcomes [15, 27]. In summary, the current data situation in terms of outcomes of CRM is characterized by its inconsistency, which suggests a difficulty in the measurability of the construct.

Due to a lack of studies in this field, one focus of the present work is to examine the effectiveness of medical CRM training in German-speaking countries regarding the outcome level in practice. Besides, the behavioral level will also be investigated. These two levels were chosen because, on the one hand, they reflect the long-term learning success in everyday professional life and, on the other hand, they represent the final benefit of the training program. That results in the second research question, which will be presented in the next paragraph.

Based on the statements mentioned above, the following three research questions were conducted:

  1. How is CRM currently implemented in the medical sector in German-speaking countries?
  2. How effective is medical CRM training currently in German-speaking countries?
  3. How will CRM develop in the future in German-speaking countries?
2. Methodology

For this study, semi-structured subject matter expert interviews were conducted. To ensure that all relevant topics are addressed during the interview, an interview guideline was developed. Subject matter experts (in the following "experts") are defined here as persons who have conducted a medical CRM training course themselves at least once. First, potential experts were contacted via e-mail and asked for their participation. In a second step, after renewing some questions based on a pretest, telephone interviews were held in October and November 2018. Telephone interviews were conducted as it was not possible to visit each interviewee personally due to the physical distance. In order to evaluate and secure all information, every interview was recorded. In total 20 medical CRM training experts (men = 19), were included into the analysis which were working in different fields of medicine: anesthesia (n = 13), neonatology (n = 1), cardiology (n = 1), internal medicine (n = 1), ambulance service (n = 2), midwifery (n = 1) and one did not give any information. The sample was composed interprofessional, for example, by interviewing a doctor and a nurse from the field of anesthesia. Most of the experts were working in Germany, and three experts were working in Switzerland. On average, the experts had 15 years of experience with CRM training.

Two different researchers conducted the interviews, which took 46 minutes on average. After all interviews were conducted, the gained material from 20 interviews was analyzed with the help of qualitative content analysis [28]. The goal of this qualitative analysis is to inductively build up a category system that represents the collected data as accurate and at the same time as short as possible. In order to ensure better reliability, the reduction of the material was conducted based on previously determined rules. The whole procedure is defined by a step by step reduction of the data into categories, whereas at the same time, those defined categories were rechecked on the raw interview material. If necessary, categories were changed for a better fit. This circular method was performed until the data was reduced into a reasonable amount of representing categories. At the end of the analysis, the material was represented by 28 categories. In order to prove the inter-rater reliability for the two researchers who coded the interviews, Cohen's kappa [29] was calculated. The value was κ = 0.964, which is seen as almost perfect according to the standards of Landis and Koch [30]. A frequency analysis was also carried out, which allowed statements about frequencies of the superordinate categories.

3. Results

The results of the interviews on the first research question show that the training is structured in the triad of briefing, simulation, and debriefing in different professions and disciplines and is conducted both interdisciplinary and interprofessional. However, so far, there is no obligation to carry out CRM in the German-speaking countries, and resources are lacking to implement and disseminate the training satisfactorily. Concerning the second research question, it turns out that positives behavioral changes after CRM training can be observed in everyday working life, as well as an increased use of tools to improve non-technical skills. Communication has improved most through the training. Regarding the treatment results of patients, some experts perceived an improvement in the treatment outcomes, but could not name any concrete treatment results. Successful procedures could be observed through previous CRM training as well as a reduction of errors and adverse events. In the interviews, it became clear that it is difficult to measure the effects of the training on the treatment outcomes. The construct treatment outcome is too multifactorial to allow clear conclusions to be drawn. The third research question relates to the future of CRM. It turns out that a mandatory establishment of CRM training and a cost absorption by health insurance companies, hospitals, or Diagnosis Related Groups (DRG) is demanded. The experts also expressed their wish for early integration of CRM in vocational training and medical studies to build up knowledge of CRM principles at an early stage. For the next five years, an increase in CRM offers is expected, but no significant changes in terms of the legal situation and associated mandatory implementation of CRM.

In Table 1, detailed results of the conducted qualitative content analysis, structured according to the three research questions, are presented. Due to the frequency analysis, it can be presented precisely how many experts have made this statement, which is shown by the percentages in brackets.

Table 1. Result overview



Current Implementation of CRM

  • Training includes a briefing (90%), simulation (100%) and debriefing (100%)
  • The number of CRM participants differ from two to twenty
  • No standardized guidelines for the frequency and duration of the training
  • The trainer has a central role in CRM training (95%) through setting focal points and moderating the debriefing
  • Existing internal obligations for CRM training (20%)
  • Group composition mainly interdisciplinary and interprofessional (50%)
  • Medical CRM training is not carried out enough in German-speaking countries (65%)
  • Insufficient safety culture in German-speaking countries (20%)
  • Insufficient dissemination of CRM training (65%) due to resource bottlenecks (40%), e.g., financial, personnel, spatial and time bottlenecks
  • General observation of an increased implementation over time (50%)
  • Patient safety modules increasingly integrated into study programs (40%), but not enough are carried out in the entire German-speaking area
  • Steeper hierarchies in Germany than in Switzerland (10%) which make CRM implementation more difficult (15%)

Effectiveness of CRM

  • Positive behavioral changes in everyday working life (35%) through the increased use of non-technical skills and related tools, e.g., 10-for-10 principle (65%), closed-loop communication (25%)
  • Improvement of non-technical skills in everyday working life: communication (85%), team working and decision making (60%), situational awareness (55%) and task management (50%)
  • Perceived improvement of treatment outcomes (30%) in the execution of procedures (5%), reduction of errors (10%) and undesired events (5%)
  • Little perceived efficacy effects on treatment outcomes (5%) and reinforcement of the awareness of team culture (10%) through the integration of CRM in medical study programs
  • No observed improvement in duration of the procedure (5%) and mortality rate (5%)
  • No measurable improvements in treatment outcomes through CRM, because none of the experts conducted a study on treatment outcomes on the patient
  • Reason for the difficulty of the measurement: very high study effort (20%), no financial support (5%)

Future of CRM

Requirements of the experts

  • Absorption of costs (95%) through, e.g., health insurance companies, hospitals or by Diagnosis Related Groups (DRG)
  • Patient safety modules in study programs (95%)
  • Mandatory and permanent establishment of CRM in the entire medical sector (85%)
  • Provision of necessary resources (40%)
  • Standardization (20%) and improved quality (25%) of the training

Experts' expectations for the next five years

  • Increase of CRM offer (55%)
  • No significant changes in the next five years (25%)
4. Discussion

4.1 Discussion of key results

The present study is the first to evaluate the current implementation and perceived effectiveness of medical CRM training in German-speaking countries. These insights were assessed by interviewing CRM experts from a variety of disciplines and professions.

To answer the first research question, which deals with the current implementation of CRM training, it can be stated that training, as described in the literature [11, 12] almost always begins with a briefing, followed by simulations and afterward debriefings are carried out. Debriefing, which is regarded as particularly important in the literature [13], is also an essential component of the experts' point of view, as it discusses core issues of CRM and reviews experiences from the simulation. No differences could be identified between the disciplines and professions regarding the fundamental training structure. Data on the strongly varying training duration from one hour to two days correspond with the results of the meta-analysis of O'Dea and colleagues [15]. To date, there are no regulations regarding an optimal training period.

However, it turns out that implementing CRM in the entire German-speaking area is associated with many problems. Due to financial, personnel, space, and time bottlenecks, it is not possible to implement CRM training in many hospitals or departments, which can be responsible for the low medical safety culture. Therefore, it is not yet possible to speak of a widespread training concept. An expert stated, "Mh (...) I would say it is still not sufficiently widespread. The problem is the financing, and if it is not the financing, then it is the exemption of employees for courses". The small dissemination of the training is also reflected in different internal guidelines for the training. A positive finding is that patient safety modules are increasingly being integrated into study programs. Therefore, it can be assumed that younger generations will be more familiar with human factors and non-technical skills. Accordingly, increased implementation of CRM training can be expected in the future, if this generation is committed to it. It can be stated that the training is equally structured in different professions and disciplines, but the implementation and dissemination are not satisfactorily established nowadays.

The second research question focuses on the effectiveness of medical CRM training in German-speaking countries. In order to examine whether the behavior changed after CRM training, experts refer to observations from participants in everyday working lives. Improvement of behavior can be shown in the sense of an increased application of non-technical skills as an expert indicated, "Well, communication will definitely be improved by CRM. I see that all the time if it is trained, it will be improved in the end". Furthermore, applying appropriate tools (see Table 1) among CRM participants in different disciplines and professions can be observed. The use of tools suggests that the concept of CRM is accessible to CRM participants and can successfully be implemented in their daily work. Improvement in communication is particularly noticeable, as evidenced by Savage and colleagues [31]. This improvement may be due to the use of closed-loop communication, which provides better performance of tasks within the team [32]. A closer look at other non-technical skills reveals that an improvement in task management and situation awareness could not be observed as frequently. Task management is often determined in advance by the education and status of persons [33]. Thus, it is likely that the roles of CRM participants are already established in most training groups. Under these conditions, the training cannot contribute to any substantial modification. While discussing situational awareness, difficulties in practicing this skill, and its dependence on a variety of factors were mentioned. Other authors also classify situational awareness as very difficult to observe and measure compared to other non-technical skills [34]. However, it must be mentioned that differences between non-technical skills are not very big in this study, and therefore, the above interpretation should not be overestimated.

In order to examine the effects on a patient's outcome, treatment results for patients were examined. In the present study, in line with the current research situation [15, 27], ambivalent treatment outcomes were perceived. Some of the experts emphasized that they noticed an improvement in the implementation of successful procedures, the reduction of errors, and adverse events. No improvement was observed in the duration of the procedure or the mortality rate. These different findings can mainly be attributed to the difficulty of measurability, which leads to a considerable lack of empirical data in German-speaking countries. Again, the lack of sufficient resources (e.g., financial resources) becomes crucial, which is neither provided for the training nor its evaluation. However, it is crucial to conduct high-quality studies in order to convince critics and decision-makers. If this persuasion is successful, decision-makers can promote nationwide implementation and make the necessary resources available. A limiting factor in this idea is that training effects take a certain amount of time to become effective [21, 26, 31]. Without this evidence, the necessary impulsion could be lacking.

The third research question focuses on how CRM training will develop within the next five years. This question led to much discussion; thus, we can conclude that there is a high demand for change. Especially the need for integration of CRM into the whole medical career path, starting at the time of education, until obligatory training sessions for practicing staff were claimed. Furthermore, there is a high demand for distinct regulations of training sessions and the need for financial support. The positive consequences of regulation for CRM would be that time, space, personnel, and financial resources have to be made available for CRM to enable an implementation for every hospital. However, there is the risk that people who are not convinced by CRM also have to complete the training. The lack of motivation of those CRM participants could lead to a lower training result due to poor communication, cooperation, and coordination in the team [19]. A further claim of the experts is an improved quality of CRM training "...there needs to be a criterion of quality. Not every simulator training is a good simulator training.". This improvement could be achieved by standardizing the whole training concept. Nevertheless, if the training will be standardized in the future, a mixture of fixed guidelines and individual freedom for training design is recommended.

Another aspect that was heavily criticized by the experts was the missing widespread of the training. Considering CRM to be a hallmark of a hospital and informing society about the benefits of training, it might put hospital management under pressure as it could be a factor in the competition between hospitals. Falling case numbers could be the result of not implementing CRM, as patients would be more likely to choose the hospital that trains staff for human error.

4.2 Strengths and limitations

By evaluating the present study, it can be stated as a strength that a generalization of the results to a broad application context of medical CRM training is possible since experts from different fields, workplaces, professions, and countries were interviewed. The sample size of 20 experts is appropriate in the context of a qualitative interview study. In particular, CRM training in the medical sector is not standard and, therefore, not widespread, which results in a small number of available experts. With the help of an interview guideline, which was pretested, the formulated research questions can be answered. The duration of the interviews can also be considered as a quality criterion. Due to the relatively long interview duration (on average, 46 minutes), a high and detailed exchange of information could occur. On the interviewer's side, it was possible to ask further questions about important topics or uncertainties. The almost perfect value of the inter-rater reliability (κ = 0,964) shows a high internal validity for the performance of the data analysis and strengthens the quality of the investigation. Moreover, the additional frequency analysis made the results quantifiable and, therefore, more accessible. Accordingly, a mix of a qualitative and quantitative method was created.

Nevertheless, the study also has limitations. Concerning the sample, it should be noted that these results are more representative for Germany than for Switzerland as most experts practice in Germany. Furthermore, the effectiveness of CRM was analyzed based on statements and observations of experts. None of the interviewed experts conducted a long-term study on the effectiveness of CRM with the inclusion of control variables. The reason for that is the lack of various resources, as mentioned before. Besides, the interviewed experts are generally convinced of the success and effectiveness of the CRM concept as otherwise, they would not conduct CRM training. Taken together, both facts, it might be possible that the results of CRM training were overestimated or biased.

4.3 Implications for Crew Resource Management

Based on the findings of the present study and technical literature, recommendations can be made for the future design of CRM training. It turns out that how CRM training is currently implemented in German-speaking countries achieves positive behavioral changes. For this reason, the implementation of training in the classical triad (briefing, simulation, and debriefing), as well as the interdisciplinary and interprofessional formation of the participant group, should be maintained. That includes the usage of tools such as the 10-for-10 principle and closed-loop communication. The present paper shows that the non-technical skills situational awareness and task management showed the least improvement. Therefore, increased focus on these two aspects of the training curriculum could contribute to an improvement in the effectiveness of the training. Since the availability of resources also matters, supportive teaching methods, such as e-learning courses, could be used to prepare for simulations. That can be carried out from any desired location and reduces time and costs. Also, an intensive implementation phase is recommended to integrate CRM into the whole company. In this respect, additional analysis of the work context can be carried out in terms of work processes [31]. That has the advantage that existing arrangements, in combination with CRM training, can be simultaneously improved. In this way, the entire system is aligned to improve patient safety. For example, as in the study by Haerkens et al. [26], a CRM group can be formed that will ensure an increasing spread of CRM throughout the entire company by using various measures. The anchoring of the training with other departments, such as Risk Management, is another way to achieve this goal.

Future research should also examine process-related effects. The advantage of this approach is that it evaluates effectiveness; furthermore, it can account for possible causes at events where no sufficient effects could be found. A challenge for the following research consists in closing the gap between the behavioral change and the actual benefit for patients. For this purpose, enhanced research in the field of treatment outcomes is fundamental and appropriate parameters must be identified in advance to assess a strong cause-effect relationship.

5. Conclusion

The present study aimed to present and examine the current state of medical CRM training in the German-speaking area in various disciplines and professions. Furthermore, the current implementation of CRM and the effectiveness of the training were investigated. To conclude, an outlook on the future development of the topic was given based on the assessment of medical CRM experts. In order to answer the three research questions, an empirical study was conducted with the help of expert interviews and was then evaluated and interpreted according to the qualitative content analysis by Mayring [28].

When examining the current state of the training, it was observed that the classical training structure consisting of briefing, simulation, and debriefing is implemented in every application context. The observation described in the literature that the group size and the training frequency are very uneven could be confirmed. Furthermore, according to the experts, the German-speaking countries' safety culture is not yet perceived as sufficiently developed. The present study made it possible to observe the effectiveness of CRM training in terms of a positive change in the participants' behavior in their everyday work. The behavioral patterns concerning non-technical skills are increasingly adapted, and CRM specific tools are applied. Of all non-technical skills, communication develops best after training, while task management and situation awareness develop worst. These distinct findings are contrasted by the inconsistent observations regarding the outcome of the intervention. This finding is in line with current research, which is also determined by a lack of empirical studies, problems of measurability, and definition of suitable parameters. Therefore, future research in this field is indispensable to convince critics to establish CRM throughout the German-speaking countries through methodologically high-quality studies. Besides, CRM training should be considered in context and interaction with the entire system of its application.

This qualitative study was able to show that implementing CRM in hospitals is worthwhile. It can contribute to more patient safety, as evidenced by the observed behavioral changes and the partially perceived improvements in treatment outcomes. Nevertheless, it became clear that there is still potential for improvement. The implementation of CRM must be supported by the hospital management to overcome personnel, financial, spatial, and time bottlenecks. Once this barrier is passed, the training will probably expand automatically. Besides, it is necessary to standardize the training to guarantee quality and increase patient safety, regardless of hospital or geographical location. Finally, it is necessary to prove a cause-effect relationship with suitable parameters through long-term studies to make the effect of CRM on treatment outcomes of patients measurable. Therefore, CRM as an intervention should be implemented for the benefit of all patients. Neither efforts nor costs should be saved at this point.

It needs to be highlighted that the present study is the first to assess the current implementation and perceived effectiveness of medical CRM in German-speaking countries with experts from a variety of disciplines and professions. Based on the provided overview, CRM users have the opportunity to compare their implementation of the training. Interested people are hopefully inspired to integrate CRM into their working environment. For those who are interested, this article might help and simplify the start of the implementation phase. An increased offer might improve patient safety in medicine.


The authors would like to thank all experts for their helpful, detailed, and honest insights about their personal experiences. Without them, this research would not have been possible.



Crew Resource Management


Diagnosis Related Group


[1] Kohn, L.T., Corrigan, J.M., Donaldson, M.S. (2000). To Err Is Human: Building a Safer Health System. Washington DC: The National Academies Press.

[2] Bundesärztekammer. (2018). Statistische Erhebung der Gutachterkommissionen und Schlichtungsstellen für das Statistikjahr 2018. Retrieved from

[3] Münzberg, M., Rüssler, M., Egerth, M., Doepfer, A.K., Mutschler, M., Stange, R., Bouillon, B., Kladny, B., Hoffmann, R. (2018). Sicherheitskultur in orthopädie und unfallchirurgie - wo stehen wir heute? Zeitschrift für Orthopädie und Unfallchirurgie, 156(5): 579-585.

[4] Koppenberg, J. (2016). Der Faktor Mensch - Human Factors. In Neumayr, A., Baubin, M., Schinnerl, A. (Eds.). Risikomanagement in der prähospitalen Notfallmedizin: Werkzeuge, Maßnahmen, Methoden, pp. 15-20. Berlin, Heidelberg: Springer.

[5] Reason, J. (1995). Understanding adverse events: Human factors. BMJ Quality & Safety, 4(2): 80-89.

[6] Thomeczek, C., Ollenschläger, G. (2006). Fehlermeldesysteme - aus jedem Fehler auch ein Nutzen? Bedeutung von Fehler- und "Incident-Reporting-Systems" in Industrie und Medizin. Rechtsmedizin, 16(6): 355-360.

[7] Rall, M., Dieckmann, P. (2005). Safety culture and crisis resource management in airway management: General principles to enhance patient safety in critical airway situations. Best Practice and Research: Clinical Anaesthesiology, 19(4): 539-557.

[8] Salas, E., Wilson, K.A., Burke, C.S., Wightman, D.C. (2006). Does crew resource management training work? An update, an extension, and some critical needs. Human Factors, 48(2): 392-412.

[9] Fletcher, G., Flin, R., McGeorge, M., Glavin, R., Maran, N., Patey, R. (2003). Anaesthetists' Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. British Journal of Anaesthesia, 90(5): 580-588. 

[10] Gaba, D.M., Howard, S.K., Fish, K.J., Smith, B.E., Sowb, Y.A. (2001). Simulation-based training in anesthesia crisis resource management (ACRM): A decade of experience. Simulation Gaming, 32(2): 175-193.

[11] Rall, M., Schaedle, B., Zieger, J., Naef, W., Weinlich, M. (2002). Neue Trainingsformen und Erhöhung der Patientensicherheit: Sicherheitskultur und integrierte Konzepte. Unfallchirurg, 105: 1033-1042.

[12] Issleib, M., Zöllner, C. (2015). Simulationsbasiertes Training - Für den Notfall vorbereitet sein in Anästhesie und Notfallmedizin: Auf dem Weg zu neuen Standards der Ausbildung in Deutschland. Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz, 58(1): 67-73.

[13] Rall, M., Manser, T., Howard, S.K. (2000). Key elements of debriefing for simulator training [Abstract]. European Journal of Anaesthesiology, 17(8): 516-517.

[14] Tannenbaum, S.I., Cerasoli, C.P. (2013). Do team and individual debriefs enhance performance? A Meta-Analysis. Human Factors, 55: 231-245.

[15] O'Dea, A., O'Connor, P., Keogh, I. (2014). A meta-analysis of the effectiveness of crew resource management training in acute care domains. Postgraduate Medical Journal, 90(1070): 1-10.

[16] Weaver, S.J., Lyons, R., DiazGranados, D., Rosen, M.A., Salas, E., Oglesby, J., Augenstein J.S., Birnbach, D.J., Robinson, D., King, H.B. (2010). The anatomy of health care team training and the state of practice: A critical review. Academic Medicine, 85(11): 1746-1760.

[17] Rall, M., Lackner, C.K. (2010). Crisis Resource Management (CRM): Der faktor mensch in der akutmedizin. Notfall Und Rettungsmedizin, 13(5): 349-356.

[18] Rall, M., Gaba, D.M. (2009). Human performance and patient safety. In Miller, R.D. (Eds.). Miller’s Anesthesia, pp. 93-105. Philadelphia: Elsevier Churchhill Livingstone. 

[19] Wilson, K.A., Salas, E., Priest, H.A., Andrews, D. (2007). Errors in the heat of battle: Taking a closer look at shared cognition breakdowns through teamwork. Human Factors, 49(2): 243-256.

[20] Rall, M., Oberfrank, S. (2013). Critical incident reporting systems: Erhöhung der patientensicherheit. Zeitschrift Für Herz-, Thorax- Und Gefäßchirurgie, 27(3): 206-212.

[21] Kirkpatrick, D.L., Kirkpatrick, J.D. (2006). Evaluating Training Programs. The Four Levels (3. Eds.). Berrett-Koehler Publishers.

[22] Chan, C.K.W., So, H., Ng, W., Chan, P., Ma, W., Chan, K., Leung, S., Ho, L. (2016). Does classroom-based crew resource management training have an effect on attitudes between doctors and nurses? International Journal of Medical Education, 7(109): 109-114.

[23] Fransen, A.F., Van De Ven, J., Merién, A.E.R., De Wit-Zuurendonk, L.D., Houterman, S., Mol, B.W., Oei, S.G. (2012). Effect of obstetric team training on team performance and medical technical skills: A randomised controlled trial. BJOG: An International Journal of Obstetrics and Gynaecology, 119(11): 1387-1393.

[24] Morgan, L., Hadi, M., Pickering, S., Robertson, E., Griffin, D., Collins, G., Rivero-Arias, O., Catchpole, K., McCulloch, P., New, S. (2015). The effect of teamwork training on team performance and clinical outcome in elective orthopaedic surgery: A controlled interrupted time series study. BMJ Open, 5(4): 1-8.

[25] Sax, H.C., Browne, P., Mayewski, R.J., Panzer, R.J., Hittner, K.C., Burke, R.L., Coletta, S. (2009). Can aviation-based team training elicit sustainable behavioral change? Archives of Surgery, 144(12): 1133-1337.

[26] Haerkens, M.H.T.M., Kox, M., Lemson, J., Houterman, S., van der Hoeven, J.G., Pickkers, P. (2015). Crew resource management in the intensive care unit: A prospective 3-year cohort study. Acta Anaesthesiologica Scandinavica, 59(10): 1319-1329.

[27] Fransen, A.F., van de Ven, J., Schuit, E., van Tetering, A.A.C., Mol, B.W., Oei, S.G. (2016). Simulation-based team training for multiprofessional obstetric care teams to improve patient outcome: A multicentre, cluster randomised controlled trial. BJOG: An International Journal of Obstetrics and Gynaecology, 124(4): 641-650.

[28] Mayring, P. (2015). Qualitative Inhaltsanalyse: Grundlagen und Techniken (12. Eds.). Weinheim: Beltz.

[29] Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1): 37-46.

[30] Landis, J.R., Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1): 159-174.

[31] Savage, C., Gaffney, F.A., Hussain-Alkhateeb, L., Ackheim, P.O., Henricson, G., Antoniadou, I., Hedsköld, M., Härenstam, K.P. (2017). Safer paediatric surgical teams: A 5-year evaluation of crew resource management implementation and outcomes. International Journal for Quality in Health Care, 29(6): 1-8.

[32] Siassakos, D., Bristowe, K., Draycott, T.J., Angouri, J., Hambly, H., Winter, C., Crofts, J.F., Hunt, L.P., Fox, R. (2011). Clinical efficiency in a simulated emergency and relationship to team behaviours: A multisite cross-sectional study. BJOG: An International Journal of Obstetrics & Gynaecology, 118(5): 596-607.

[33] St.Pierre, M., Hofinger, G., Simon, R., Buerschaper, C. (2011). Crisis Management in Acute Care Settings (2. Eds.). Berlin, Heidelberg: Springer.

[34] Verbeek-van Noord, I., de Bruijne, M. C., Twisk, J. W. R., van Dyck, C., Wagner, C. (2014). More explicit communication after classroom-based crew resource management training: results of a pragmatic trial. Journal of Evaluation in Clinical Practice, 21(1): 137-144.