Cross-Center Validation of Deep Learning Model for Musculoskeletal Fracture Detection in Radiographic Imaging: A Feasibility Study

1 Introduction

Bone fractures, defined as disruptions in bone continuity, arise from diverse etiologies including trauma, stress injuries, and pathological conditions such as osteoporosis [1]. These fractures, irrespective of the patient’s age, manifest a spectrum of clinical presentations ranging from mild symptoms like pain and swelling to severe complications including deformity and functional impairment of the impacted region [2]. Clinically, fracture diagnosis incorporates an injury history, physical examination, and symptom evaluation, notably pain, edema, morphological alterations, abnormal mobility, and occasionally crepitus – a palpable or audible friction in the fractured bone. However, symptomatology in certain fracture types, such as closed or stress fractures, might be subtle or non-specific [3].

Diagnostic approaches for fracture detection and confirmation predominantly utilize X-ray imaging as the initial modality, offering detailed bone structure visualization and fracture identification [4]. Supplementary imaging techniques, including computed tomography (CT), magnetic resonance imaging (MRI), or ultrasound, are employed for complex cases or specific fracture types where musculoskeletal (MSK) X-rays are inadequate. Fracture management strategies depend on fracture characteristics like type, location, and severity, ranging from non-invasive treatments like splint immobilization to surgical interventions using internal osteosynthesis with screws, pins, wires, or external fixation devices [5]. Rehabilitation, encompassing physical therapy and exercises, is crucial for restoring function and strength to the affected region.

In the realm of clinical decision-making, the integration of deep learning-based decision support software for radiograph interpretation marks a significant advancement. Despite promising developments, existing algorithms for fracture detection face practical limitations, including the inability to analyze all body parts simultaneously or detect multiple fractures in a single X-ray, which are common in clinical practice [6]. Leveraging sophisticated deep learning models, such software enhances fracture detection accuracy on radiographs, alleviates the diagnostic workload of radiologists, and augments patient outcomes through expedited and precise diagnostic processes [7].

2 Materials and Methods

2.1 Training Data

For this feasibility study, the publicly available MURA dataset, further described in the study “MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs” [8] (Figure 1) was selected to validate the design and development of the proposed deep learning algorithm. As this dataset contains MSK X-rays annotated only at the label-level, the images required to be annotated at the bounding box-level.

• Download figure
• Open in new tab

Figure 1:

Examples of MSK X-rays of the upper extremity from the MURA dataset, with each study manually labeled by radiologists as normal (absence of fracture) or abnormal (presence of fracture). These examples show a normal study of the elbow, an abnormal study of the finger with degenerative changes, an abnormal study of the forearm demonstrating surgical fixation with plate and screws for radius and ulna fractures, and an abnormal study of the humerus with a fracture.

A total of 720 images with positive fracture findings were randomly extracted from the primary dataset of 40,561 skeletal radiographs (14 863 studies in total). Subsequently, the selected dataset was annotated by an in-house radiologist at the bounding-box labeling level. For training our deep learning model, we utilized 30 images of the shoulder, 249 images of the upper arm, 123 images of the elbow, 117 images of the forearm, 68 images of the wrist, and 133 images of the hand and fingers, containing a total of 832 bounding boxes indicating the presence of a fracture and localization (Table 1).

View this table:

• View inline
• View popup
• Download powerpoint

Table 1: Number of training images and bounding boxes for the indication of fracture.

2.2 Model Architecture

YOLO (You Only Look Once) [9] (Figure 2, Figure 3) is a single-stage regression model for object detection. This model uniquely combines object localization and classification in a single pass through a convolutional neural network, enabling fast, real-time detection. During analysis, the entire radiographic image is passed through the network, which is crucial for higher detection accuracy in scenarios where contextual information about bone structure is vital.

• Download figure
• Open in new tab

Figure 2:

Train/loss and val/loss metrics for the indication of fracture.

• Download figure
• Open in new tab

Figure 3:

Precision-Recall Curve a F1-Confidence Curve for the indication of fracture.

Developed using the Darknet framework, YOLO is optimized for both CPU and GPU computation, making it suitable for diverse clinical settings. The model outputs bounding boxes, each representing a detected fracture site. The bounding box is defined by: where p_c is the probability of fracture presence, (b_x, b_y) are the coordinates of the bounding box center, (b_h, b_w) are the dimensions of the bounding box, and c represents the class of the object, in this case, a fracture. The deep learning model’s output is then layered on the original musculoskeletal X-ray image, highlighting the regions of interest with bounding boxes, thereby aiding radiologists in identification of fractures.

3 Results

The proposed deep learning algorithm correctly predicted 131 images (TP) containing one or more fractures out of a total of 144 positive skeletal X-ray images for Dataset 1, while it misclassified 308 negative images (FP) out of a total of 696 images. For Dataset 2, the algorithm correctly predicted 46 images (TP) out of a total of 74 positive examples, while incorrectly classifying 13 (FP) images without abnormality (Figure 4).

• Download figure
• Open in new tab

Figure 4:

Confusion matrix illustrating the performance of the proposed deep learning algorithm.

• Download figure
• Open in new tab

Figure 5:

Example predictions of the proposed deep learning algorithm trained to identify skeletal fractures in X-ray images. The examples show the true positive (TP) predictions of the algorithm on the input images from Dataset 1: (a) frontal image of the hand and wrist, (b) lateral image of the forearm, (c) frontal image of the hip and pelvis, (d) lateral image of the elbow.

• Download figure
• Open in new tab

Figure 6:

Example predictions of the proposed deep learning algorithm trained to identify skeletal fractures in X-ray images. The examples show the true positive (TP) predictions of the algorithm on the input images from Dataset 2: (a) frontal image of the hip, (b) frontal image of the knee, (c) frontal image of the wrist, (d) frontal image of the shoulder.

This corresponds to a sensitivity (Se) of 0.910 (95% CI: 0.852-0.946) and specificity (Sp) of 0.557 (95% CI: 0.520-0.594) on Dataset 1, and Se of 0.622 (95% CI: 0.508-0.724) and Sp of 0.740 (95% CI: 0.604-0.841) on Dataset 2 (Table 3).

The algorithm’s performance was further assessed using Positive Predictive Value (PPV) and Negative Predictive Value (NPV), which respectively indicate the probability that patients with positive and negative test results truly have and do not have the condition. For Dataset 1, the PPV was 0.298 (95% CI: 0.256-0.341) and NPV was 0.968 (95% CI: 0.950-0.985). For Dataset 2, the PPV was 0.780 (95% CI: 0.674-0.885) and NPV was 0.569 (95% CI: 0.449-0.690). Likelihood Ratios (LRs), which combine sensitivity and specificity, were also calculated. The Positive Likelihood Ratio (PLR) quantifies how much the odds of the disease increase when a test is positive, while the Negative Likelihood Ratio (NLR) indicates how much the odds of the disease decrease when a test is negative. For Dataset 1, the PLR was 2.056 (95% CI: 2.008-2.968) and NLR was 0.162 (95% CI: 0.140-0.187). For Dataset 2, the PLR was 2.391 (95% CI: 1.926-2.968) and NLR was 0.511 (95% CI: 0.447-0.585) (Table 4.

These values suggest that while the algorithm is highly effective in correctly ruling out fractures (high NPV), its ability to confirm the presence of fractures (PPV) varies significantly between datasets. The LRs indicate a moderate increase in the probability of a fracture being present with a positive test result, and a moderate to significant decrease in probability with a negative test result.

4 Discussion

The study’s findings highlight the deep learning algorithm’s proficiency in detecting musculoskeletal (MSK) fractures in X-ray images. The algorithm exhibited high sensitivity (Se) in Dataset 1, indicating its strong capability in correctly identifying fractures. However, the lower Se observed in Dataset 2 suggests potential limitations in diverse clinical scenarios or variations in fracture types and imaging qualities. Specificity (Sp) varied significantly between the two datasets, with a notably lower Sp in Dataset 1 compared to Dataset 2. This variation may reflect differences in the datasets’ characteristics, such as the distribution of fractures or differences in X-ray interpretation. These findings underscore the importance of considering dataset diversity when assessing algorithm performance in a clinical context.

5 Conclusion

In conclusion, this study validates the potential of the YOLO-based deep learning algorithm for fracture detection in MSK radiography. Its high sensitivity in some cases and consistently high negative predicitive value are promising, but the variability in specificity and positive predictive value indicates the need for further optimization and broad-scale validation. These findings advocate for the integration of such AI tools in radiographic analysis, complementing the expertise of radiologists to enhance diagnostic accuracy and improve patient care outcomes. Future work should focus on collecting extensive training dataset with including ground truth, refining the algorithm, and validating its performance in diverse, real-world clinical settings.