A distributed learning strategy improves performance and retention of skills in neonatal resuscitation: A simulation-based randomized controlled trial

Introduction

Ten percent of babies born in the high-income world require medical intervention at birth¹. The Neonatal Resuscitation Program® (NRP®) developed by the American Academy of Pediatrics (AAP) reflects international consensus on best practice². Similar courses for neonatal, child and adult resuscitation are run elsewhere. They teach knowledge and skills, improve performance in simulated settings^3-6, and contribute to improved patient outcomes^6-8. Unfortunately, attrition of these skills occurs over time^5,9-11. The consensus document from the International Liaison Committee on Resuscitation (ILCOR) advocates educational strategies which improve retention¹².

There is consensus that training followed by long intervals without formal training is undesirable^2,12. Recent studies favor approaches which employ e-learning, modularity, and simulation^13-15. However, heterogeneity is an obstacle to evaluating and standardizing practice, and to achieving value for money. One fundamental problem is the lack of quantitative theoretical frameworks on which to base practical training structures. Cognitive psychology studies have established that distributed learning, where a fixed amount of study time is broken into intervals (e.g. one hour of training for five days) rather than all at once (e.g. five hours of training in a single day), benefits long-term retention of knowledge^16-18. Most of these studies investigated learning corresponding to declarative memory – knowledge that can be consciously recalled. In medicine, procedural memory is also important, but the evidence for distributed learning is mixed – for example, microvascular anastomosis was shown to benefit from a distributed learning strategy¹⁹, whereas no difference could be shown with bronchoscopy²⁰.

Learning during the NRP involves both declarative and procedural memory, executed under time-critical situations when rapid memory recall is imperative to patient survival. In an emergency context, the retention of procedural memory is poorer than declarative memory^21,22. The evidence in favor of distributed learning in resuscitation skills is not convincing. Distributed learning can be organised in the form of a refresher course after initial training. Established resuscitation training programs offer renewal courses after intervals of a year or more, but they do not provide a refresher session soon after initial certification. The effect of a resuscitation refresher has been studied^9,23, but most investigators have been unable to show benefit. These studies provided a refresher several months after initial teaching and tested for retention relatively soon after. Evidence from the field of cognitive psychology suggests that the interval between original training session and refresher (the inter-study interval, ISI), and the interval between original session and retention test (the retention interval, RI), are important determinants of success (Fig. 1)^24,25. In a review of literature, Cepeda et al. note ‘the optimal ISI increases as the duration over which information needs to be retained increases’²⁴; i.e. if one wants to retain information for a longer period, one must increase the ISI. In a subsequent review, Rohrer and Pashler (2007)²⁵ conclude that ‘the optimal ISI [lies] at a value of 10 to 30% of the RI’. This may explain why previous researchers looking at distributed resuscitation learning failed to find an effect – their ISI:RI ratios were greater than 50%^9,23.

• Download figure
• Open in new tab

Fig. 1. A distributed learning program, with initial training, refresher and retention test. Definitions of retention interval and inter-study interval are given.

Methods

Ethical approval was secured from the Ottawa Health Science Network Research Ethics Board (Protocol #20120849-01H). We obtained written informed consent from each participant.

Participants were University of Ottawa residents in the departments of anesthesiology, family medicine and emergency medicine. They were selected after a call for interested volunteers approved by the programme leads. A questionnaire was administered to collect demographic data, including specialty, postgraduate year, and experience in neonatal resuscitation management. Residents were excluded if they were unable to commit to all phases of the study, or if they were NRP instructors.

This was a prospective, single-blinded randomized controlled trial (Fig. 2). Prior pilot data was collected using volunteers, allowing us to ‘dry run’ scenarios. Videos collected from pilot sessions were used for rater training and calibration. Participants underwent a two-hour training session in neonatal resuscitation as per NRP 6^th edition guidelines (G.M. and D.A., NRP instructors). Before the session began on day 1, they completed a multiple-choice knowledge test (‘written pre-test’, with questions taken from the NRP bank) and performed a simulated standardized NRP scenario (‘simulation pre-test’). Following training, they took the same written and simulation tests (‘immediate post-tests’). Structured content debriefing using the NRP checklist was given after the simulation post-test to maximise educational benefit. Fifteen minutes were allocated for the scenario and fifteen minutes for debrief. All participants were then randomized to an early refresher (intervention) or late refresher group (control). Block randomization was done using sealed envelopes and was stratified by specialty and years of postgraduate experience. The intervention group received an early refresher session at three weeks post-training. The refresher consisted of the same simulated resuscitation scenario, but this time in situ at the participant’s usual place of work, followed by structured content debriefing. Again, fifteen minutes each were allocated to scenario and debriefing. This structure made use of two evidence-based strategies that have been shown to aid retention²⁶. Firstly, the ‘testing effect’ (the fact that actively recalling information from memory strengthens later retention); secondly, timely feedback. The control group received a late refresher session at two months post-training. From the work by Kaczorowski and colleagues, a refresher at this time appears to be ineffective, and therefore an appropriate control⁹. All participants underwent retention post-tests at 4 months. This consisted of the same simulation scenario and written test. A questionnaire was administered after the retention tests to collect information on clinical neonatal resuscitation experience gained since initial training.

• Download figure
• Open in new tab

Fig. 2. Trial structure. R = randomization.

Participants were asked to perform the resuscitation scenario in a simulated environment with a Newborn HAL S3010 simulator (Gaumard Scientific, Miami, FL.). In this scenario, a term neonate is delivered by caesarean section. The subject is told the child is ‘flat’, and asked to assess and initiate appropriate management. Simulator settings were: (i) apnoea; (ii) HR 50; (iii) SpO₂ 85% at delivery, deteriorating to 65% over 1min; (iv) cyanosis; (v) airway obstruction until appropriate manoeuvres performed. The performance was recorded for later analysis. All appropriate equipment was available as described in the AAP and Canadian Paediatric Society (CPS)’s NRP 6^th edition²⁷. Two confederates were available: one to assist with NRP tasks, the other to find equipment and follow instructions.

The standardized, previously validated NRP Basic Megacode Assessment checklist (Canadian adaptation)²⁸ (Appendix 1) was used to assess simulation performance, with a modification to reflect the fact that participants were not given the opportunity to demonstrate post-resuscitation care measures. We assessed non-technical skills using the Ottawa Global Rating Scale (GRS)²⁹, a validated 7-point Likert scale which assesses performance in leadership, problem solving, situational awareness, resource utilisation and communication.

Two expert blinded raters (G.M & A.J) independently assessed videos. An inter-rater correlation coefficient was calculated on the experimental data-set. Knowledge was assessed with thirty questions from the NRP multiple-choice question bank, administered immediately after each scenario (Appendix 2). These were scored according to standard NRP criteria, with assessors blinded to study group.

The primary outcome measure was NRP Basic Megacode checklist score. Secondary outcome measures were: (i) Knowledge Scores on written tests, and (ii) Ottawa GRS scores.

Data was analyzed using SPSS 21.0 (SPSS Inc., Chicago, IL). The inter-rater reliability was measured using single-measures intraclass correlation coefficients (for total scores). NRP Basic Megacode checklist performance at pre-test and immediate post-test was assessed using analysis of variance (ANOVA). The performance score was treated as the dependent variable. The independent variables were the type of refresher (early vs. late) as the between-subjects variable, and the test phase (pre-test vs. immediate post-test) as the within-subjects variable. At retention post-test, NRP Basic Megacode checklist performance was assessed using one-way analysis of co-variance (ANCOVA). The retention performance score was treated as the dependent variable with the type of refresher (early vs. late) as fixed factor and immediate-post test score as the covariate. Written test and GRS scores were similarly analyzed using ANOVA and ANCOVA.

Sample size was calculated as follows. Previously, Bould and colleagues used the neonatal resuscitation performance checklist developed by Lockyear and colleagues^30,31, and found that the mean score was 18/30 with a standard deviation of 4.16. Assuming an F test, an alpha error of 0.05, and a beta error of 0.20, we will have 80% power for detecting an increase to 22/30 on this checklist (corresponding to 2 additional items done correctly that were previously not done) with 19 subjects in each group at retention test³².

Results

Eighteen participants were randomized. Seventeen completed all stages. One participant was unable to return for the refresher. Fig. 3 shows participant flow through the study. Table 1 summarizes the characteristics of participants who completed the program. Of note, we were unable to reach the target sample size due to difficulties with recruitment and drop outs after a prolonged recruitment phase. For these logistical reasons, we took the decision to terminate recruitment, and analyze findings. There was no significant difference in out-of-study exposure to neonatal resuscitation after randomization (Fisher exact test value = 1).

• Download figure
• Open in new tab

Fig. 3. Patient flow chart

Inter-rater reliability as measured using intra-class correlation coefficients (ICC). ICC for the NRP checklist was 0.74; for the Ottawa GRS it was 0.45. Following guidelines suggested by Landis and Koch, these indicate substantial and moderate agreement respectively³³. Reduced inter-rater reliability for non-technical assessment has been seen in other studies, and our ICCs are in line with figures reported elsewhere for similar measures^13,34.

Table 2 shows NRP checklist, Ottawa GRS, and knowledge test scores for early and late refresher groups at pre-test, immediate post-test, and retention post-test stages. For pre-test and immediate post-test, ANOVA shows no difference between groups for any outcome measure, suggesting that the groups were equally distributed prior to randomization. Both groups improved their scores from pre-test to immediate post-test, but ANCOVA shows no significant difference between the groups in the degree of this improvement (NRP checklist F(1,14) = 2.93, p = 0.11; Ottawa GRS F(1,14) = 3.07, p = 0.10; Knowledge test F(1,14) = 3.19, p = 0.10).

Fig. 4 presents boxplots of outcome scores at each experimental stage. Both early and late refresher groups improved at retention post-test compared to pre-test scores. However, performance in the early refresher group was significantly better than the late refresher group for the primary outcome measure, NRP checklist scores. The early refresher group also demonstrated superior Ottawa GRS scores. No significant difference in retention was seen with written knowledge scores.

• Download figure
• Open in new tab

Fig. 4. Boxplots of NRP checklist, Ottawa GRS and knowledge test scores at each experimental stage.

Discussion

In this study, we found that an early refresher after NRP training was more effective at promoting long-term retention of technical and non-technical skills compared to a late refresher, but has no impact on knowledge retention.

We have shown that a quantitative theoretical framework underpinning retention in declarative memory can be translated to neonatal resuscitation – a high-stakes skill set that requires mixed declarative and procedural memory, executed under intense time and situational pressure. It is not enough that learning is distributed, as discovered by Kaczorowski et al⁹; instead, the 10-30% interval rule for ISI:RI ratios ²⁵ also applies to complex, mixed-memory skills such as NRP.

We have also shown that an early refresher improves retention of non-technical skills. It has been shown previously that technical and non-technical skills are correlated, and that residents who are proficient in one domain also demonstrate proficiency in the other³⁵. This group postulated that the reason may be due to the concept of ‘cognitive load’. Working memory is limited, whilst long-term memory has much greater capacity. Participants who are technically skilled have committed these aspects to long-term memory, and are therefore able to free cognitive resources to better confront non-technical challenges.

We found no evidence for improved knowledge retention in the early refresher group. This is perhaps surprising, given that multiple choice questions are the closest test of purely declarative memory. It is notable that the knowledge exam was both the highest scoring of the three outcome measures, and the one to suffer the least attrition from immediate post-test to retention. We speculate that the cognitive demands were too low to show an effect from the intervention, or that a ceiling effect applies whereby the benefit of the early refresher is only apparent when attrition of memory reaches a certain threshold.

Health systems invest a large amount of time and money so that professionals respond appropriately in resuscitation situations. One trial estimated that the cost of an adult cardiopulmonary resuscitation program per life saved was around $500,000 and that a significant proportion of this was due to training³⁶. For neonatal resuscitation, the evidence suggests that this investment in time and money does not lead to the retention of skills for a clinically desirable time period^9,37. At present, there is no consensus about the optimal way to structure training for high-stakes scenarios such as resuscitation and this is a recurrent focus of gaps-in-knowledge summaries from ILCOR^2,12. Most authorities currently take an interval-based approach to training and certification, but with varying time-frames. In North America, it is recommended that trainees complete the program every two years, whilst in the UK certification is valid for four. There is no nationally-mandated training to be completed in between these intervals. This heterogeneity in practice, and the paucity of the evidence base, is a major priority for future research efforts.

Our work suggests a way forward based on an evidence-based theoretical framework. Policy makers must first decide what time-period retention of performance is to be maintained over – this will depend on an analysis of service needs and cost-benefit ratio – and then structure a training program in such a way that refresher sessions are delivered at appropriately-distributed intervals. We used a retention interval of four months for the purposes of this study. In reality, healthcare systems are likely to desire longer recertification cycles. Future studies should aim to determine optimal refresher timings and retention intervals for real-world use, and to investigate alternative methods of delivering distributed learning (e.g. as online courses or as other forms of distance learning).

Although we studied neonatal resuscitation, these findings may also be generalizable to other resuscitation programs such as Adult Cardiac Life Support, and to other interventions where skill retention depends on mixed types of memory.

There are limitations to our study. We were unable to recruit as many participants as we had intended, and the study is therefore underpowered. However, as there was a significant difference in our primary outcome, the issue of type 2 error is minimal. We plan to address this with a larger study in future, and are considering strategies that will mitigate these problems – for instance, a multi-centre trial that will both increase both generalizability and the target recruitment pool. We studied performance in a simulated situation, which is common for resuscitation studies because of the difficulties in controlling for variables in real resuscitation scenarios. There is good evidence to suggest that performance in a simulated setting can be transferable to clinical practice³⁸.

Ultimately, the aim of such research should not just be about improving education, but about improving patient outcome. There is evidence that neonatal resuscitation training improves APGAR scores in the US⁸, decreases the incidence of meconium aspiration in France³⁹, shortens the duration of hospitalization in Turkey⁴⁰, and reduces neonatal mortality in China⁶. There is scope for future studies to build on our work, and look at changes in patient outcome following introduction of a teaching program guided by our conclusions.