Validity and reliability of simulation-based assessment tools for European Training Requirement-defined competencies in anaesthesiology: a two-centre observational pilot study

Abstract

Despite ongoing efforts to harmonise postgraduate anaesthesiology training across Europe, significant variability persists among national programmes, and a unified certification process is lacking. Simulation is widely used in postgraduate anaesthesiology training, and several validated tools are available to assess both technical and nontechnical skills. However, use of such tools for summative assessments in specialist certification has not yet been established. This prospective, observational pilot study aimed to evaluate the validity and inter-rater reliability of two scenario-specific checklists, a global rating scale (GRS), and the Anaesthetists’ Non-Technical Skills (ANTS) tool in assessing competencies defined by the European postgraduate anaesthesiology curriculum. The study was conducted between May and September 2022 at two university hospitals in Germany and Spain. A total of 43 (23 third-year and 20 fourth-year) anaesthesiology trainees completed a knowledge test and managed two simulated perioperative emergencies. Performances were video-recorded and independently assessed by two blinded raters. Fourth-year trainees scored significantly higher than third-year trainees across all tools, providing validity evidence. Inter-rater reliability was moderate for Scenario 1 (intraclass correlation coefficient [ICC] range: 0.50 to 0.67) and moderate-to-good for Scenario 2 (ICC range: 0.67 to 0.76). These findings suggest that simulation-based assessments using structured tools can validly and reliably evaluate technical and nontechnical competencies outlined in the European anaesthesiology training curriculum. The results support the integration of such assessments into standardised summative evaluation frameworks in European postgraduate education.