Assessment of Deep Learning-Based Triage Application for Acute Ischemic Stroke on Brain MRI in the Emergency Room

Study Sample

We consecutively enrolled 1224 patients visiting our institution’s emergency department who underwent a brain MRI examination under suspicion of AIS from April to October 2021. Every individual underwent one MRI examination. These examinations were reviewed by three board-certified neuroradiologists (J. K., S.-W.O., and J.-Y.K. with 8, 16, and 21 years of experience in brain MRI interpretation, respectively) to ascertain their eligibility for the final study population. 391 examinations were excluded due to the absence of the mandatory DWI sequence required for the triage research application analyses, and 2 other examinations were also excluded due to poor image quality caused by metal artifacts and acute ischemia in the cervical cord. Finally, 831 examinations were included for the triage research application image analyses. The data of these brain MRI examinations were uploaded to a server other than the application server and retrospectively analyzed. The same three neuroradiologists (J. K., S.-W.O., and J.-Y.K.) established a gold standard for acute AIS. The reviewers were blinded to each patient’s clinical information, previous readings, or each other’s reports at the initial reading. If the reviewers disagreed, the final decision about a brain MRI was then made in consensus. The sample size of this study was determined statistically feasible based on data from a previous study with a statistical significance of 0.05 and a statistical power of 0.80 ⁷. Further details on the estimation of sample size are provided in the Data Supplement.

Diagnosis of AIS and lacunar AIS on MRI

We defined an AIS as a cerebral region with restricted diffusion due to cytotoxic edema in the brain parenchyma that is indicated by hyperintense signals on DWI with a corresponding low value on the ADC map ⁸. We also defined lacunar AIS noncortical infarcts less than 15 mm in diameter in the distal distribution of deep penetrating vessels, such as the lenticulostriate and thalamoperforating artery territories ^9,10. Figure S1 in the Data Supplement presents examples of AIS and potential artifacts mimicking AIS.

Network architecture and development of the triage research application

This application was developed utilizing a deep learning 3D convolutional neural network architecture for AIS, acute hemorrhagic stroke, and substantial mass effect. Using brain MRI inputs, the model calculates the percent probabilities for each incident’s critical findings. The application consists of three components; localization feature extraction modules, which are deep image-to-image networks with convolution and pooling layers; orientation-specific feature connection layers, which make the network robust to missing contrasts; and a global classifier that generates a probability score based on the presence of critical findings. To detect and determine the AIS network architecture, an axial DWI with the ADC map is mandatory, and axial T1- and T2-weighted images are optional. The network architecture is briefly illustrated in Figure S2 in the Data Supplement.

During its development, this triage research application utilized a total of 12,143 brain MRI studies for training (74%), tuning (9%), internal validation (8%), and external validation (8%). The flowchart of the development and validation for the triage research application is presented in Figure S3 in the Data Supplement. Additional heat maps were generated for each contrast to indicate the significance of the input pixel, and the localization feature extraction module generated these maps with pixel-wise significance in the input space ⁷.

According to Nael et al., a critical finding can be interpreted as positive if the triage research application calculates its risk probability as greater than 50% ⁷. Previous results for detecting AIS with this application are summarized in Table S1 in the Data Supplement.

MRI protocol

All patients were scanned on a 3T MRI scanner (MAGNETOM Vida, Siemens Healthineers, Erlangen, Germany) with a 64-channel head & neck coil. In every examination, a standard infarct MRI protocol that included the axial DWI was carried out. The DWI was acquired with the following parameters using readout-segmented echo-planar diffusion: repetition time, 3720 ms; echo time, 61 ms; generalized auto-calibrating partial parallel acquisition factor, 2; number of excitation, 1; matrix size, 180 ⨯ 180; flip angle, 180 °; field of view, 220 ⨯ 220 mm; slice thickness, 3.0 mm; gap, 1.0 mm; b-value, 1000 s/mm². Protocols of optional axial T1-weighted image and axial T2-weighted image sequences are summarized in Table S2 in the Data Supplement.

Statistical analyses

To assess the performance of the triage research application, sensitivity, specificity, and the area under the receiver operating characteristic curve (AUROC) were calculated. To overcome potential bias caused by an imbalanced dataset, additional statistical values including the area under precision (e.g. sensitivity)-recall (e.g. positive predictive value) curve (AUPRC), and maximum F1 score were analyzed ¹¹. For the inter-reviewer agreements of the three radiologists, the Fleiss Kappa was defined as follows; poor agreement, < 0.2; fair agreement, 0.2 ≤ and < 0.4; moderate agreement, 0.4 ≤ and < 0.6; good agreement, 0.6 ≤ and < 0.8; and excellent agreement,≥ 0.8. Using the Student’s t-test, the baseline characteristics were compared between positive and negative AIS groups according to mandatory sequence alone and with the addition of optional sequences. All statistical analyses were performed using MedCalc (version 20.215, MedCalc Software Ltd, USA). p < 0.05 was considered to indicate a statistically significant difference.

Study Sample characteristics

A total of 831 individuals (mean age, 64 years ± 16; 399 men, 432 women) were enrolled in the study. A flow diagram of the inclusion and exclusion process is illustrated in Figure 1 and the characteristics of the final study population are summarized in Table 1. In total, 201 (mean age, 69 ± 14[SD]; 123 men, 78 women) were positive for AIS, while 630 (mean age, 62 ± 16[SD]; 276 men, 354 women) were negative. The group with AIS was significantly older than the control group and AIS also occurred significantly more frequently in men than women (p < 0.001). However, there was no predilection of age or sex between the mandatory sequence alone group and optional sequences groups (p > 0.05).

• Download figure
• Open in new tab

Figure 1:

Flowchart of the study sample enrollment. MRI, magnetic resonance imaging; AIS, acute ischemic infarct; DWI, diffusion-weighted image.

Inter-reviewer agreement

The inter-reviewer agreements among the neuroradiologists for detecting AIS were calculated. The Kappa value for AIS was 0.94 (95% CI, 0.90-0.98, p < 0.001), indicating excellent agreement. The Kappa value for the mandatory sequence alone and optional sequences group was 0.85 (95% CI, 0.69-1.00, p < 0.001) and 0.94 (95% CI, 0.90-0.98, p < 0.001), respectively, indicating excellent agreement. The Kappa value from the sub-analysis for lacunar AIS and non-lacunar AIS was 0.89 (95% CI, 0.85-0.94, p < 0.001) and 0.94 (95% CI, 0.90-0.98, p < 0.001), respectively, indicating excellent agreement.

Performance for detecting AIS

The statistical values of the triage research application were as follows; sensitivity, 90%; specificity, 89%; AUROC, 0.95 (95% CI, 0.93-0.96); AUPRC, 0.91 (95% CI, 0.86-0.94); and maximum F1 score, 0.87. The average reporting time from uploading images to presenting results was 61 ± 31 seconds. In terms of the contribution of optional sequences, all statistical values were higher with the mandatory sequence alone than with the addition of optional sequences. Table 2 summarizes the results of this application, and Figure 2 shows the AUROC and AUPRC. Also, illustrations of cases in which the application identified AIS as true positives are presented in Figure 3.

• Download figure
• Open in new tab

Figure 2:

A. Receiver operating characteristic curve of the triage research application. B. Precision-recall curve of the triage research application.

• Download figure
• Open in new tab

Figure 3:

Examples of AIS detected by the triage research application. From left to right, axial DWI, and heat map images are displayed. A. Lacunar AIS in the left basal ganglia. B. Non-lacunar AIS in the left middle cerebral artery. C. Hypoxic ischemic encephalopathy with diffusely elevated DWI signal along the bilateral cerebral cortex and deep gray matter with cerebral parenchymal edema. On the heat map, the application partially identified these features. D. Embolic AIS in both hemispheres. On the heat map, the application partially identified these features. The white arrowheads in DWI indicate missing lesions.

Sub-analysis of performance for detecting lacunar and non-lacunar AIS

The proportion of lacunar AIS and non-lacunar AIS was 31% (63 of 201) and 69% (138 of 201), respectively. This sub-analysis demonstrated a lower sensitivity of 83% for detecting lacunar AIS compared to a sensitivity of 93% for non-lacunar AIS. The sub-analysis showed higher performance for non-lacunar AIS than lacunar AIS. More detailed results are summarized in the Data Supplement with Table S3.