Debiasing and Educational Interventions in Medical Diagnosis: A Systematic Review

Literature Search

Searches were conducted on April 15, 2022, through four electronic databases (MEDLINE, Embase, PsycINFO, and Healthstar) using the Firefox browser. Search results were exported in Research Information Systems (RIS) format and imported to Rayyan for Systematic Reviews (Ouzzani et al., 2016). See supplementary materials for the full search strategy.

Inclusion and Exclusion Criteria

Studies were included if they 1) reported the implementation of at least one intervention or educational initiative related to diagnosis, 2) focused on reducing the prevalence or effect of cognitive biases in medical diagnosis, 3) examined diagnostic decision-making in a clinical setting or hypothetical clinical setting (i.e., patient vignettes), 4) participants in the intervention included medical students, resident physicians, attending physicians, physician assistants (PA), nurse practitioners, (NP), PA students, or NP students, and 5) were classified as an original study or conference abstract. Studies were excluded if they 1) did not explicitly ascribe at least one cognitive bias to diagnostic error, 2) were classified as a letter, review, commentary, editorial, book, or protocol, or 3) only suggested potential debiasing strategies without implementing at least one debiasing intervention.

Two screeners independently reviewed all studies using the Rayyan software program through one round of title and abstract screening and one round of full-text review. Screeners were blinded to each other’s decisions until the end of each screening round, after which a discussion was held to resolve any conflicts.

Data Extraction

Two reviewers independently extracted the included studies and imported them into a preformatted Excel worksheet. Information extracted included: author names and publication year, medical specialty, number of participants and their medical experience, intervention type and nature, cognitive bias targeted, and debiasing intervention efficacy.

Quality Assessment

Internal validity and quality of included studies were assessed with JBI’s “Checklist for Randomized Control Trials” or “Checklist for Quasi-Experimental Studies (Non-Randomized Experimental Studies)” tool. Additionally, each study was given a subjective rating of poor, fair, or good to describe the raters’ overall appraisal.

Search Results

Database searches yielded 2232 articles, 1166 of which were identified as unique by the Rayyan software program. 1142 articles were excluded during the title and abstract screening. Inter-rater reliability (IRR, average agreement) during the abstract screening was 98.88%, with a Cohen’s kappa coefficient (κ) of 0.73, indicating substantial agreement. The remaining 24 articles were retrieved and underwent full-text screening, during which 7 additional articles were excluded. IRR in this round was 86.96%, with a κ of 0.59, indicating moderate agreement. 17 articles were included in the final analysis.

See figure 1 for PRISMA article selection information.

• Download figure
• Open in new tab

Figure 1.

PRISMA flowchart of selected studies

Study Characteristics

The final analysis included a total of 1530 participants across 17 studies: 849 medical students (849/1530, 55.49%), 485 residents (465/1530, 30.39%), 30 attending physicians (30/1530, 1.96%), and 186 other healthcare professionals (186/1530, 12.16%). 11 of the studies were randomized controlled trials (RCT) (11/17, 64.71%), and 6 were quasi-experimental studies (QES) (6/17, 35.29%). Among the 17 studies, 13 examined diagnostic accuracy in a hypothetical clinical setting via case vignettes (13/17, 76.47%), 2 examined knowledge of reasoning/diagnostic skills via assessments involving patient vignettes (2/17, 11.76%), 1 examined bias recognition ability in pre-diagnosed patient vignettes (1/17, 5.88%), and 1 examined attitudes towards a conference (1/17, 5.88%). Only one of the included studies involved an actual clinical aspect (1/17, 5.88%), but did not evaluate diagnostic decision-making in an actual clinical environment.

Data Quality

IRR in quality assessment was 76.5% (13/17). 4 studies were rated as poor (4/17, 23.53%), 6 studies were rated as fair (6/17, 35.29%), and 7 studies were rated as good (7/17, 41.18%). The majority of studies rated good were successful interventions; out of 7, 5 were successful (5/7, 71.43%). The inverse was observed in studies rated poor, in which the majority were unsuccessful; out of 4, 1 was successful. (1/4, 25%).

See supplementary materials for quality assessment details (McGuinness & Higgins, 2021).

Objective 1: Debiasing Strategies Implemented

Three broad categories of debiasing intervention types were identified during the analysis of the included studies. The distribution of them across the included studies was as follows: 8 of the studies aimed to increase awareness/education of a bias (8/17, 47.06%), 7 aimed to provide education of debiasing strategies (7/17, 41.18%), and 6 used a tool such as a debiasing checklist (6/17, 35.29%). 4 of the studies’ debiasing intervention utilized more than one of the three intervention types (4/17, 23.53%), but none of the studies used all three (0/17, 0%). Among the three intervention types, several subcategories were found, including classes, workshops, checklists, metacognitive strategies, and long-term educational curricula.

Refer to Table 1 for full details of the nature of implemented interventions and their effectiveness.

Objective 2: Cognitive Biases Targeted

8 different biases were targeted across the included studies. The three most reported biases, excluding general/non-specific cognitive bias, were as follows: availability bias, targeted in 5 studies (5/17, 29.41%); search satisfying, targeted in 4 studies (4/17, 23.53%); and confirmation bias, targeted in 3 studies (3/17, 17.65%). 9 studies (9/17, 52.94%) did not target specific biases but biases in general. Metacognition, whether by education of debiasing strategy or through the implementation of checklists designed to induce reflective thinking, was found to more commonly counteract availability bias, confirmation bias, and search satisfying. No trends were found regarding any other specific bias being commonly treated with any specific debiasing intervention.

See Table 2 for identified cognitive biases and other cognitive biases mentioned in this review (Croskerry, 2003).

Objective 3: Effectiveness of Interventions

Of the 17 studies, 8 reported an increase in diagnostic accuracy attributed to at least one debiasing intervention (8/17, 47.06%), and 5 of the studies reported no improvement in diagnostic accuracy post-intervention (5/17, 29.41%). 4 studies reported improvements in areas other than diagnostic accuracy, such as knowledge of cognitive biases and potential debiasing strategies (4/17, 23.53%). Note that studies could implement more than one type of intervention. The success rates of each of the three intervention types were as follows: among interventions aiming to increase awareness/education of bias, 6 of the 8 studies reported successful improvement of diagnostic accuracy (6/8, 75%) and 2 reported ineffectiveness (2/8, 25%); among interventions aimed at providing education of debiasing strategies, 4 of the 7 reported successful improvement of diagnostic accuracy (4/7, 57.14%), 2 reported ineffectiveness (2/7, 28.57%), and 1 study did not report effectiveness (1/7, 14.29%); and among interventions involving tool use, 4 of the 6 studies reported successful improvement of diagnostic accuracy (4/6, 66.67%) including 1 study which reported one effective checklist and one ineffective checklist, and 2 reported ineffectiveness (2/6, 33.33%). Of the 4 studies that did not directly aim to improve diagnostic accuracy (4/17, 23.53%), 2 of the studies focused on measuring participants’ knowledge of cognitive/diagnostic error (2/4, 50%), 1 study examined the information-gathering ability of participants (1/4, 25%), and one study did not have an outcome measure (1/4, 25%).

Characterizing identified debiasing interventions

Cognitive biases targeted by interventions

The most targeted cognitive biases were availability bias, search satisfying, and confirmation bias. All three biases are known to cause fixation on one leading diagnosis, which may lead to premature closure (Croskerry, 2003). Among the included articles, metacognitive principles delivered in the form of an educational intervention and/or checklist were shown to have been used to target all three biases in almost all instances. This is unsurprising, as metacognition encourages clinicians to reconsider the working hypothesis by analyzing the reasoning steps taken to arrive at that conclusion, which would help alleviate a fixation on one diagnosis. This finding, however, could also be the result of reporting bias.

Most interventions analyzed did not target specific cognitive biases or targeted a wide variety of biases. This might be a result of researchers attempting to develop solutions which can be applied to a wide array of situations, thus encouraging non-specific targeting of biases.

Ineffectiveness of some described interventions

5 studies reported at least one ineffective intervention, which collectively includes interventions from all three types of interventions discussed thus far. The authors’ suggested reasons for the negative results vary greatly between studies and are largely speculative, but nevertheless may provide valuable insight into what future studies must avoid in designing their interventions.

Shimizu and colleagues reported one ineffective checklist intervention where their medical student participants unsuccessfully utilized a general debiasing checklist developed by Ely and colleagues (Ely et al., 2011; Shimizu et al., 2013). The authors suggested that due to the limited clinical experience that the participants had, heuristic thinking among the students may have been sparse. Therefore, a debiasing checklist would not have affected performance as cognitive bias occurs due to a fault in heuristics usage (Committee on Diagnostic Error in Health Care, 2015). Thus, if a debiasing intervention were to be tested on participants with relatively limited experience, such as medical students or junior residents, researchers should first ensure that participants demonstrate the use of heuristics or evidence of cognitive bias in the clinical situations they intend on examining.

The same debiasing checklist was also reported to be ineffective by Sibbald and colleagues (Sibbald et al., 2019). However, due to the use of simpler case vignettes, the baseline performance of both novices and experts was higher than in previous studies, which may explain the small effect size (Sibbald et al., 2013). Furthermore, a previously validated content-specific checklist also failed to demonstrate significant results in the same study (Sibbald et al., 2013, 2014), which could suggest a flaw in the study methodology which caused both checklists to be ineffective. Alternatively, since the studies examined diagnosis specifically in ECG, these results could show that debiasing checklists designed for general use are ineffective in specialized environments. High pre-intervention performance was also a suggested reason for the low effectiveness of another intervention implemented by Smith and Slack. The educational workshop demonstrated a low effect size, which was speculated to be due to the inconsistent assessment pre-workshop by attending physicians who were unfamiliar with the study evaluations.

Other instances of low intervention effectiveness include a debiasing checklist intervention reported by DiNardo and colleagues and a metacognitive education intervention by Payne. DiNardo noted highly variable use of the implemented tool and that it was used mostly by non-participant clinicians (i.e., attending physicians rather than resident participants). These factors may have contributed to the not significant findings (DiNardo et al., 2016). Payne reported that participants were distracted by graphical representations of mental models included in the educational intervention and did not spend a great deal of time on the metacognitive feedback (Payne, 2012).

Concerns with current interventions and future research

While all three broad debiasing intervention types discussed demonstrated some promise in improving diagnostic accuracy, there are several problems which paint an unclear picture of their effectiveness in practice.

Most significantly, no intervention was evaluated in real clinical contexts. Thus, the useability and actual effectiveness of each intervention is unclear. Furthermore, all three debiasing interventions reported some instances of ineffectiveness; the exact reasonings for ineffectiveness are heterogeneous. Studies also examined the effectiveness of interventions immediately or soon after delivery of interventions. This experimental choice is logical for the purposes of research but puts the long-term effectiveness of interventions into question. Lastly, while it is immediately obvious why nearly all participants in debiasing interventions and initiatives are medical students and resident physicians, it is known that attending physicians are also vulnerable to cognitive bias (Croskerry, 2003). Therefore, the reported effectiveness of each intervention may not be accurate in attending physician populations. Moreover, a lack of attending physician participants in the included studies suggests a troubling lack of focus on the effects of cognitive biases in continued medical education.

Due to these flaws in the interventions examined in this review, we recommend that future studies should further examine the effectiveness of each intervention in 1) real clinical practice, 2) with participant populations which include attending physicians and other practicing clinicians responsible for diagnosis (i.e., NP and PA), 3) report the long term retention of educational materials or intervention usage, and 4) investigate reasoning behind ineffectiveness and low intervention usage. Future studies should also examine whether effectiveness and intervention usage in case vignettes translate directly to clinical practice.

Limitations

There are limitations to this review which should be acknowledged. First, articles were only included if the intervention specifically mentions that it is a debiasing intervention or reduces the negative impacts of cognitive biases in diagnostic decision-making. This criterion meant that some articles that may also reduce the negative impacts of cognitive biases were not included as they did not explicitly identify themselves as such. Second, due to the heterogeneity of the extracted data, we were unable to provide a formal meta-analysis. While we were able to categorize interventions into three different types, the details of each intervention and outcome measures between studies varied greatly. Third, while the inclusion criteria specified the inclusion of NP, PA, and attending physician participants, no NP and PA participants were included in the final analysis and a limited number of attending physician participants were included. Fourth, the scope of this review was limited by the number of databases chosen. A more comprehensive search could have led to more reliable and robust findings. Lastly, all studies analyzed decision-making in a vignette context which may not accurately reflect real-world practice.