A Balanced Bagging Classifier machine learning model-based web application to predict risk of febrile neutropenia in cancer patients

INTRODUCTION

Febrile neutropenia (FN) remains a critical concern in oncology, characterized by a fever and a significant reduction in neutrophil count, typically induced by chemotherapy. It poses a life-threatening risk to patients, requiring immediate medical intervention (1). The accurate prediction of FN risk is crucial for optimizing prophylactic measures and patient management strategies. Traditional predictive models have leveraged clinical factors such as age, type of cancer, chemotherapy regimen, and baseline blood counts to estimate FN risk (2,3). In 2 previous publications from our group, we were able to delineate the predictors of febrile neutropenia in cancer (4,5). However, our models and also models from other groups lacked precision at times, mostly with low sensitivity potentially leading to mostly under-, or sometimes, overprediction of the risk of febrile neutropenia.

Machine learning (ML) approaches have increasingly been applied in healthcare to enhance predictive accuracy (6). Techniques such as random forests, support vector machines, and neural networks have demonstrated success in various medical predictions (6-8). A significant challenge in developing ML models for FN prediction lies in the imbalanced nature of clinical datasets, where the occurrence of FN is relatively rare compared to non-FN events. For example, the proportion of cases within the whole cohort was 1.5% and 4% in our previous model derivation cohorts (4,5). This imbalance can lead to models biased towards predicting the majority class, thereby underperforming in identifying FN cases.

The Balanced Bagging Classifier (BBC) is an ensemble learning technique specifically designed to address the issue of class imbalance (9). By resampling the training data, BBC aims to balance the class distribution, thereby improving the model’s sensitivity and specificity for the minority class. This study builds on our previous work to refine an existing predictive model for FN by incorporating BBC, which has been shown to enhance the model’s ability to accurately predict FN risk.

In addition to developing a robust predictive model, we have aimed to test the feasibility of a web application to facilitate its use in clinical settings. This application potentially provides healthcare providers with an accessible tool to estimate FN risk, aiding in the decision-making process for patient care.

MATERIALS AND METHODS

Model Development

The BBC was employed to address the class imbalance problem. BBC is an ensemble method that constructs multiple base classifiers trained on balanced subsets of the data (10,11). This method reduces the bias towards the majority class, improving the model’s ability to predict FN. Refer to Figure 1 for a demonstration of the working principles of BBC.

• Download figure
• Open in new tab

Figure 1.

Balanced Bagging Classifier (BBC)

Then, in order to identify the most influential features in the model, we used permutation importance analysis (12). The permutation importance algorithm involves computing importance i_j for feature f_j, computing the score s_kj for each feature j and for each repetition k, as follows:

The model’s parameters were optimized using cross-validation, which helps prevent overfitting and ensures the model generalizes well to unseen data. We evaluated the model’s performance using metrics such as the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity. The AUC provides a measure of the model’s discriminative power, while accuracy, sensitivity, and specificity offer insights into its overall performance and its ability to correctly identify FN and non-FN cases.

As a part of the analysis, we searched optimal value from a grid. To achieve this, we found the probabilities for the class label first, then looked for the optimal threshold to map the probabilities to its proper class label. The probability of prediction can be obtained from a classifier by using predict_proba() method from the Sklearn Python library (13). Using the optimal threshold, then we calculated the confusion matrix, and the accuracy, sensitivity and specificity figures from the sklearn. metrics module.

RESULTS

BBC model

The BBC model achieved an AUC of 0.97, indicating the discriminative ability. The model’s accuracy was 0.95, with a sensitivity of 0.93 and a specificity of 0.95. These results were related to the model’s effectiveness in terms of correctly identifying both FN and non-FN cases, to judge its potential utility in clinical practice. Refer to Figure 2 for the AUC plot.

• Download figure
• Open in new tab

Figure 2.

The AUC plot

Permutation importance analysis further highlighted the significance of specific clinical factors. Refer to Figure 3 for the results of the permutation importance analysis. Notably, patients with a history of FN were at a higher risk of experiencing FN during subsequent chemotherapy cycles. Cancer type also played a crucial role, with certain cancers associated with a higher risk of FN. Previous radiotherapy was another significant factor, that correlated with the risk of febrile neutropenia after chemotherapy.

• Download figure
• Open in new tab

Figure 3.

Permutation importance analysis

Three most influential factors from the BBC model can be viewed in Figure 4. These features are displayed in the plot with reference to the magnitude of their importance scores. Figure 4a focuses on the effect of previous febrile neutropenia, 4b on cancer type and 4c on radiotherapy history, in relation to febrile neutropenia risk.

• Download figure
• Open in new tab

Figure 4.

Most influential factors on the risk of febrile neutropenia

Web application

The web application developed as part of this study integrates the BBC model and provides a user-friendly interface for clinical use. It allows healthcare providers to input patient data and receive real-time predictions of FN risk after a specific cycle of chemotherapy. The application was evaluated for usability and found to be a practical tool in clinical settings. Web application user interfaces can be viewed in Figure 5. Figure 5a reflects the data entry part, and Figure 5b details the febrile neutropenia risk output part.

• Download figure
• Open in new tab

Figure 5.

Web application user interface

CONCLUSION

The implementation of the Balanced Bagging Classifier (BBC) in our study has yielded an accurate predictive model for febrile neutropenia (FN) in cancer patients undergoing chemotherapy. The model’s notable performance metrics, including an AUC of 0.97, accuracy of 0.95, sensitivity of 0.93, and specificity of 0.95, highlight its potential utility in clinical practice. These findings are consistent with existing literature that emphasizes the capability of ensemble learning methods in enhancing predictive accuracy, especially in the context of imbalanced clinical datasets. BBC, as an ensemble learning method, involves creating multiple versions of a predictor and using these to get an aggregated prediction. Specifically, BBC aims to address class imbalance by training each base classifier on balanced subsets of the data, which can improve the model’s sensitivity to the minority class. This method is particularly useful when the dataset contains a significantly lower number of instances of one class compared to another, as it helps prevent the model from becoming biased toward the majority class (16,17).

The high sensitivity and specificity observed in our model suggest that it is well-calibrated to distinguish between FN and non-FN cases. This accuracy is crucial in a clinical setting, where the consequences of both false positives and false negatives can be significant. For instance, a false positive could lead to unnecessary prophylactic interventions, which can be costly and potentially harmful due to side effects like myalgia, while a false negative could result in inadequate preparation for an FN episode, putting the patient at serious risk and may lead to complications (18,19).

The permutation importance analysis provided valuable insights into the most significant predictors of FN risk, with previous FN episodes, cancer type, and prior radiotherapy identified as the most influential factors. The strong association of previous FN episodes with subsequent risk is well-documented, suggesting that a history of FN is a critical marker for assessing patient vulnerability. Similarly, the type of cancer plays a pivotal role, with non-small cell lung cancer and small cell lung cancer, for example, being associated with higher FN risk due to the intensive nature of their treatment regimens. Prior radiotherapy’s contribution to FN risk may be linked to its cumulative effect on bone marrow suppression, which can be exacerbated by concurrent chemotherapy (20-23).

Our study also demonstrated the practical application of the BBC model through the development of a web application, which provides a user-friendly interface for healthcare providers. This tool allows for real-time FN risk assessment, facilitating timely and informed clinical decisions. The importance of integrating predictive models into clinical workflows has been highlighted in recent studies, which emphasize the need for accessible and interpretable tools to support medical professionals (24). The usability testing of our application suggests that it could be effectively implemented in routine clinical practice, potentially improving patient outcomes by enabling early intervention and tailored prophylaxis.

The success of our model aligns with the broader trend in healthcare towards the adoption of machine learning and artificial intelligence (AI) technologies (25). The ability of these technologies to process vast amounts of data and identify complex patterns offers unprecedented opportunities for personalized medicine. Recent advances in AI have shown promise in various domains, from diagnostic imaging to treatment planning, underscoring the transformative potential of these technologies (26). However, our study is not without limitations. The exclusion of missing data, while necessary to maintain model integrity, may introduce bias and limit the generalizability of our findings. Missing data is a common issue in clinical datasets, and strategies such as imputation or the use of advanced models that can handle missingness should be considered in future research (27). Additionally, while our model demonstrated high accuracy on the test set, its real-world applicability needs to be validated through prospective clinical trials. Such validation is crucial to ensure that the model’s performance translates effectively into practical clinical settings.

The promising results of this study suggest several avenues for future research. One potential direction is the incorporation of additional clinical and molecular markers, which could further enhance the model’s predictive power. For instance, genetic polymorphisms related to drug metabolism or immune response could provide additional layers of information, helping to identify patients at the highest risk of FN (28). Another area of interest is the exploration of model interpretability. As machine learning models become increasingly complex, ensuring that their predictions are interpretable and transparent is essential for clinical acceptance and trust.

In conclusion, our study demonstrates the efficacy of the Balanced Bagging Classifier in predicting febrile neutropenia risk in cancer patients undergoing chemotherapy. The model’s integration into a user-friendly web application represents a significant step towards practical clinical implementation, offering a valuable tool for febrile neutropenia risk assessment and management. From a broader point of view, as the field of AI in healthcare continues to evolve, it is imperative to explore and address the challenges and opportunities that these technologies present, including our model, as we highlighted in this study.

COMPETING INTEREST STATEMENT

No competing interest for any of the authors.

FUNDING STATEMENT

No funding support received.