Non-generalizability of biomarkers for mortality in SARS-CoV-2: a meta-analyses series ====================================================================================== * ME Rahman Shuvo * Max Schwiening * Felipe Soares * Oliver Feng * Susana Abreu * Niki Veale * Q Gao * William Thomas * AA Roger Thompson * Richard J Samworth * Nicholas W Morrell * Stefan J Marciniak * Elaine Soon ## ABSTRACT **Rationale** Sophisticated prognostic scores have been proposed for SARS-CoV-2 but do not always perform consistently. We conducted these meta-analyses to uncover why and to investigate the impact of vaccination and variants. **Methods** We searched the PubMed database for the keywords ‘SARS-CoV-2’ with ‘biomarker’ and ‘mortality’ for the baseline tranche (01/12/2020-30/06/2021) and either ‘SARS-CoV-2’ or ‘Covid19’ with ‘biomarker’ and either ‘vaccination’ or ‘variant’ from 01/12/2020 to 31/10/2023. To aggregate the data, the *meta* library in R was used, and a random effects model fitted to obtain pooled AUCs and 95% confidence intervals for the European/North American, Asian, and overall datasets. **Results** Biomarker effectiveness varies significantly in different continents. Admission CRP levels were a good prognostic marker for mortality due to wild-type virus in Asian countries, with a pooled area under curve (AUC) of 0.83 (95%CI 0.80-0.85), but only an average predictor of mortality in Europe/North America, with a pooled AUC of 0.67 (95%CI 0.63-0.71, *P<*0.0001). We observed the same pattern for D-dimer and IL-6. This variability explains why the proposed prognostic scores did not perform evenly. Notably, urea and troponin had pooled AUCs ≥0.78 regardless of location, implying that end-organ damage at presentation is a key prognostic factor. The inflammatory biomarkers (CRP, D-dimer and IL-6) have generally declined in effectiveness in the vaccinated and variant cohorts. We note a significant lag from the pandemic advent to data availability and this has no doubt impacted on patient care. **Conclusions** Biomarker efficacies vary considerably by region. It is imperative that the infrastructure for collecting clinical data should be put in place ahead of a future pandemic. ## To the Editor SARS-CoV-2 is a novel beta coronavirus of zoonotic origin first identified in Wuhan, China at the end of 2019, which led to a Public Health Emergency of International Concern between February 2020 and May 2023. SARS-CoV-2 differs from previous viral threats in showing marked transmissibility during the asymptomatic/very early symptomatic stage1 and person-to-person transmission by both airborne and fomite routes2. At the beginning of the pandemic, there was no previous immunity, no known effective antiviral treatment, and no vaccine, resulting in a global death toll of >6.9 million ([https://covid19.who.int/](https://covid19.who.int/)). Due to the overwhelming number of cases and the significant morbidity and mortality associated with SARS-CoV-2, reliable prognostic scores are critically important to maximize survivorship. Sophisticated scoring systems have been proposed but have not performed consistently3–6. For example, El-Solh3 tested 4 peer-reviewed prognostic models constructed to predict in-hospital mortality for SARS-CoV-2 patients; proposed by Chen7, Shang8, Yu9, and Wang10. All the models examined had validation area under curves (AUCs) which were significantly worse than the area under curves of their derivation cohorts. For example, the AUC of the validation cohort using the model proposed by Chen was at best 0.69 (95% confidence interval [CI] 0.66-0.72) compared to the derivation AUC, which was 0.91 (95% CI 0.85-0.97). A similar pattern was noted in the other three models. Gupta4 tested 20 candidate prognostic models using data derived from 411 consecutively admitted adults with a PCR-confirmed diagnosis of SARS-CoV-2 in a major London hospital. Five of these models were pre-existing point-based scores not specific for Covid19 (MEWS, REMS, qSOFA, CURB65 and NEWS2) and the remainder of which were a combination of point-based scores and logistic regression models specifically derived from SARS-CoV-2 patients. None of these methods overlapped with those previously tested by El-Solh. The most discriminating univariable predictor for in-hospital mortality was age (AUC 0.76 [95% CI 0.71-0.81]) and for in-hospital deterioration was oxygen saturation on room air (AUC 0.76 [95% CI 0.71-0.81]). More importantly, none of the models tested performed consistently better than these univariable predictors. We ran these meta-analyses to identify the source of these inconsistencies and to determine the effect of vaccination and virus variants on biomarker efficacy. We searched the PubMed database for the keywords ‘SARS-CoV-2’ in combination with ‘biomarker name’ and ‘mortality’. The period for the first data tranche was set from 01st December 2019 to 30th June 2021. We subsequently repeated the search from 01st December 2019 to 31st October 2023 using the following combinations of keywords: 1. SARS-CoV-2’ with ‘vaccination’ and ‘biomarker name’ 2. SARS-CoV-2’ with ‘variant’ and ‘biomarker name’ 3. Covid19’ with ‘vaccination’ and ‘biomarker name’ 4. Covid19’ with ‘variant’ and ‘biomarker name’ All papers reporting mortality data for hospitalized patients swab-positive for SARS-CoV-2 with a biomarker level within 48h of admission were examined. Ethical approval was obtained from the Integrated Research Application System (reference 281880) for analysis of the Cambridge (UK) data. To ensure biomarkers were applicable to acute adult general admissions, we excluded reports of patients already admitted to intensive care or restricted to specific groups (e.g. hemodialysis). Mortality (30-day or in-hospital) was used as the endpoint. This study is registered with PROSPERO (CRD42022366893). This process is summarized in Fig.1A. ![Figure 1:](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2023/12/12/2022.12.03.22282974/F1.medium.gif) [Figure 1:](http://medrxiv.org/content/early/2023/12/12/2022.12.03.22282974/F1) Figure 1: Panel (A) shows a modified PRISMA flow diagram for paper review, selection, and inclusion in these meta-analyses for both the original/wild-type virus (in purple) and for variant and vaccination-related data (in orange). Panel (B) shows a summary forest plot from the meta-analyses of the first tranche of data (original/wild-type virus) with comparison of pooled area under curves for the five biomarkers being meta-analysed (CRP, D-dimer, troponin, urea, and IL-6) and age. Panel (C) shows results from variant cohorts, with the relevant pooled AUC from (A) as a reference. *indicate that the AUCs shown were for severe/critical illness rather than mortality. Panel (D) shows results from vaccinated cohorts, with the relevant pooled AUC from (B) as a reference. Throughout the panels, for ease of comparison, pooled values for Asian countries are shown in red and values for European/North American countries are shown in blue. To aggregate the data on age and biomarkers from individual studies, the *meta* library in R was used to report overall mean values and 95% confidence intervals and the statistical significance of differences between mean values in the joint European and North American cohort and the Asian cohort. This analysis was based on estimates of standard errors for each study, obtained by assuming values for individual subjects were normally distributed in each study with a study-specific mean. In this way, measures of spread (IQR, SD and range) were converted into estimates of within-study standard deviations. Since the estimates of the study-specific means exhibited high levels of heterogeneity, a *random effects model* was fitted. We examined 1,930 articles that were published from the beginning of the SARS-CoV-2 pandemic on 01st December 2019 to 30th June 2021 for the baseline study and 2,758 papers from 01st December 2019 to 31st October 2023 to obtain vaccination and variant data. The first phase meta-analyses revealed different patterns in the effectiveness of biomarkers in different regions of the world (Fig.1B). For example, admission CRP levels were a good prognostic marker for mortality in Asian countries, with a pooled AUC (area under curve) of 0.83 (95% CI 0.80-0.85) from 34 studies, but only an average predictor of mortality in Europe and North America, with a pooled AUC of 0.67 (95% CI 0.63-0.71, *P<*0.0001). This also held true for admission D-dimer and IL-6 levels. This explains why the prognostic scores that are being proposed for SARS-CoV-2 do not perform evenly in different countries, as the ‘building blocks’ underpinning these prognostic scores have intrinsically different effectiveness in different populations. Interestingly, troponin and urea levels had universally ‘good’ pooled AUCs. This implies that end-organ damage at the time of presentation was a key prognostic indicator of severity for wild-type SARS-CoV-2 infection. We expected that multiple rounds of vaccinations and ongoing mutations into different strains would significantly impact biomarker efficacy. The ‘inflammatory’ biomarkers (CRP, D-dimer and IL-6) have generally declined in effectiveness in the vaccinated and variant cohorts (Fig.1C-D). This is particularly illustrated by the Jalali11 cohort (Delta variant), who are completely unvaccinated, with AUCs of 0.576 (0.434-0.713) for CRP and 0.620 (0.481-0.760) for D-dimer. This is consistent with reports that the pathogenicity of SARS-CoV-2 has decreased in the order wild-type>Delta>Omicron and that the Omicron variant in particular has decreased IL-6 production1213. It is also likely that the targeting of inflammation in general (e.g. dexamethasone) and IL-6 pathways in particular (e.g. tocilizumab) has had an effect. Sadly, urea and troponin now appear to perform unevenly as well14,15, although we are limited by the very small number of studies. We hypothesize that biomarker efficacy in any population at this given time now hinges on many factors including: * Population factors – the numbers vaccinated and/or previously infected, the age and co-morbidities present, * Variant factors such as pathogenicity and transmissibility, * Socio-economic factors such as access to healthcare and nutritional status, and that each country likely has its personalised ‘ingredient list’ which is why overarching patterns are now lost. Therefore we think that prognostic biomarkers/scores should be individualized to particular populations. A way to achieve this would be to set up information frameworks ahead of time, ideally online. For example, we mapped the root studies of the first phase on the following ([https://covid19.cimr.cam.ac.uk/](https://covid19.cimr.cam.ac.uk/)), with a free-to-use software programme. Given that staff are likely to be incredibly busy if another pandemic strikes, it would make sense to automate the process as much as possible*, e.g.* by using programmes to automatically harvest data from medical software systems. It could be agreed beforehand what data is needed. For example, basic requirements could be age, sex, biomarker levels, and outcome, while secondary outcomes (such as imaging data and complications) could be ‘bolt-on’ options. Furthermore, centres could be designated to collect data from specific groups, e.g. pregnant women, and immunocompromised people. In this way, data collection can be standardized rather than growing organically as was the case in SARS-CoV-2. The availability of such data would not just aid in biomarker identification but also allow the rapid organization of clinical trials by pinpointing ‘at-risk’ groups early who could be targeted for vaccination or intervention programmes. There are also significant geographical ‘black boxes’ where no published data is available. We would urge large organisations such as the World Health Organisation to pinpoint this as an area requiring further funding and development, ideally before another pandemic. ## Contributorship MERS, FS, WT, and NV collected data, reviewed source manuscripts, and helped analyze data. MS, FS, QG and SA contributed to the creation of online resources. OF and RJS performed statistical analyses. RT, NWM and SJM contributed to the planning, organization, and funding of the study. ES conceptualized and organized the study, performed data analyses, funded the study, and takes responsibility for the integrity of the work as a whole. All authors contributed to the writing and critical revision of the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. ## Supporting information Supplement [[supplements/282974_file03.pdf]](pending:yes) ## Data Availability All data produced in the present study are available upon reasonable request to the authors. ## Copyright for publication The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all forms, formats and media (whether known now or created in the future), to (i) publish, reproduce, distribute, display and store the Contribution, (ii) translate the Contribution into other languages, create adaptations, reprints, include within collections and create summaries, extracts and/or, abstracts of the Contribution, (iii) create any other derivative work(s) based on the Contribution, (iv) to exploit all subsidiary rights in the Contribution, (v) the inclusion of electronic links from the Contribution to third party material where-ever it may be located; and, (vi) licence any third party to do any or all of the above. ## Disclosures All authors have completed the ICMJE uniform disclosure form at [http://www.icmje.org/disclosure-of-interest](http://www.icmje.org/disclosure-of-interest) / and declare: no support from any commercial organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. ## Ethics approval Ethical approval was obtained from the National Health System (UK) Health Research Authority via the Integrated Research Application System (reference 281880) for analysis of the Cambridge (UK) data. All the other data has been published and is in the public domain. ## Funding ES and MS are supported by the UK Medical Research Council (MR/R008051/1); the British Medical Association (the Josephine Lansdell Award); and the Association of Physicians of Great Britain and Ireland (Young Investigator Award to ES); the Wellcome Trust ISSF and the Cambridge BHF Centre of Research Excellence (RE/18/1/34212). MES and WT are full-time NHS physicians who have volunteered their time for this work. FS received in-kind funding by the AWS Diagnostic Development Initiative and Google TPU Research Cloud. NV is supported by a BLF-Papworth Fellowship from the British Lung Foundation and the Victor Dahdaleh Foundation (VPDCF17-18). AART is supported by a British Heart Foundation Intermediate Clinical Fellowship (FS/18/13/33281). OF is funded by the StatScale programme (EP/N031938/1). RJS is supported by Engineering and Physical Sciences Research Council grants EP/P031447/1 and EP/N031938/1, as well as ERC Advanced Grant 101019498. SA and SJM are funded by the British Lung Foundation (VPDCF17-18), the Medical Research Council, UK (MR/V028669/1), the NIHR Cambridge Biomedical Campus (BRC-1215-20014) and the Royal Papworth NHS Trust. NWM is supported by the British Heart Foundation (SP/12/12/29836), the Cambridge BHF Centre of Research Excellence (RE/18/1/34212), the UK Medical Research Council (MR/K020919/1), the Dinosaur Trust, BHF Programme grants to NWM (RG/13/4/30107), and the NIHR Cambridge Biomedical Research Centre. The funders did not have a role in the study design, data collection or analyses or the decision to submit for publication. ## Data sharing The relevant anonymized data will be shared on reasonable request. ## Collaborators ## Acknowledgements We thank Ms Natalie Doughty, Mr Chris Davies, Dr Benjamin Dunmore, and Mr Nikita Zubkov who provided general public and patient input into accessibility of data, website, and software. Many thanks also to all root study authors, patients and essential workers who have selflessly shared their time and data. ## Footnotes * Additional data on vaccinated and variant cohorts has been added to update the manuscript. * Received December 3, 2022. * Revision received December 11, 2023. * Accepted December 12, 2023. * © 2023, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## References 1. 1.Yanes-Lane M, Winters N, Fregonese F, et al. Proportion of asymptomatic infection among COVID-19 positive persons and their transmission potential: A systematic review and meta-analysis. PLoS One 2020;15(11):e0241536. doi: 10.1371/journal.pone.0241536 [published Online First: 2020/11/04] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0241536&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33141862&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F12%2F12%2F2022.12.03.22282974.atom) 2. 2.Bak A, Mugglestone MA, Ratnaraja NV, et al. SARS-CoV-2 routes of transmission and recommendations for preventing acquisition: joint British Infection Association (BIA), Healthcare Infection Society (HIS), Infection Prevention Society (IPS) and Royal College of Pathologists (RCPath) guidance. J Hosp Infect 2021;114:79–103. doi: 10.1016/j.jhin.2021.04.027 [published Online First: 2021/05/04] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jhin.2021.04.027&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33940093&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F12%2F12%2F2022.12.03.22282974.atom) 3. 3.El-Solh AA, Lawson Y, Carter M, et al. Comparison of in-hospital mortality risk prediction models from COVID-19. PLoS One 2020;15(12):e0244629. doi: 10.1371/journal.pone.0244629 [published Online First: 2020/12/29] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1371/journal.pone.0244629&link_type=DOI) 4. 4.Gupta RK, Marks M, Samuels THA, et al. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study. Eur Respir J 2020;56(6) doi: 10.1183/13993003.03498-2020 [published Online First: 2020/09/27] [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiZXJqIjtzOjU6InJlc2lkIjtzOjEyOiI1Ni82LzIwMDM0OTgiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8xMi8xMi8yMDIyLjEyLjAzLjIyMjgyOTc0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 5. 5.Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. Bmj 2020;370:m3339. doi: 10.1136/bmj.m3339 [published Online First: 2020/09/11] [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzNzAvc2VwMDlfNy9tMzMzOSI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIzLzEyLzEyLzIwMjIuMTIuMDMuMjIyODI5NzQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 6. 6.Bradley P, Frost F, Tharmaratnam K, et al. Utility of established prognostic scores in COVID-19 hospital admissions: multicentre prospective evaluation of CURB-65, NEWS2 and qSOFA. BMJ Open Respir Res 2020;7(1) doi: 10.1136/bmjresp-2020-000729 [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYm1qcmVzcCI7czo1OiJyZXNpZCI7czoxMToiNy8xL2UwMDA3MjkiO3M6NDoiYXRvbSI7czo1MDoiL21lZHJ4aXYvZWFybHkvMjAyMy8xMi8xMi8yMDIyLjEyLjAzLjIyMjgyOTc0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 7. 7.Chen R, Liang W, Jiang M, et al. Risk Factors of Fatal Outcome in Hospitalized Subjects With Coronavirus Disease 2019 From a Nationwide Analysis in China. Chest 2020;158(1):97–105. doi: 10.1016/j.chest.2020.04.010 [published Online First: 2020/04/19] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.chest.2020.04.010&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F12%2F12%2F2022.12.03.22282974.atom) 8. 8.Shang Y, Liu T, Wei Y, et al. Scoring systems for predicting mortality for severe patients with COVID-19. EClinicalMedicine 2020;24:100426. doi: 10.1016/j.eclinm.2020.100426 [published Online First: 2020/08/09] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.eclinm.2020.100426&link_type=DOI) 9. 9.Yu C, Lei Q, Li W, et al. Clinical Characteristics, Associated Factors, and Predicting COVID-19 Mortality Risk: A Retrospective Study in Wuhan, China. Am J Prev Med 2020;59(2):168–75. doi: 10.1016/j.amepre.2020.05.002 [published Online First: 2020/06/23] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.amepre.2020.05.002&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F12%2F12%2F2022.12.03.22282974.atom) 10. 10.Wang K, Zuo P, Liu Y, et al. Clinical and Laboratory Predictors of In-hospital Mortality in Patients With Coronavirus Disease-2019: A Cohort Study in Wuhan, China. Clin Infect Dis 2020;71(16):2079–88. doi: 10.1093/cid/ciaa538 [published Online First: 2020/05/04] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciaa538&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32361723&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2023%2F12%2F12%2F2022.12.03.22282974.atom) 11. 11.Azarfar F, Abbasi B, Jalali A, et al. Investigation of the relationship between monocyte chemoattractant protein 1 rs1024611 variant and severity of COVID-19. Cytokine 2023;171:156367. doi: 10.1016/j.cyto.2023.156367 [published Online First: 2023/09/16] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.cyto.2023.156367&link_type=DOI) 12. 12.Barh D, Tiwari S, Rodrigues Gomes LG, et al. SARS-CoV-2 Variants Show a Gradual Declining Pathogenicity and Pro-Inflammatory Cytokine Stimulation, an Increasing Antigenic and Anti-Inflammatory Cytokine Induction, and Rising Structural Protein Instability: A Minimal Number Genome-Based Approach. Inflammation 2023;46(1):297–312. doi: 10.1007/s10753-022-01734-w [published Online First: 2022/10/11] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s10753-022-01734-w&link_type=DOI) 13. 13.Korobova ZR, Arsentieva NA, Liubimova NE, et al. Cytokine Profiling in Different SARS-CoV-2 Genetic Variants. Int J Mol Sci 2022;23(22) doi: 10.3390/ijms232214146 [published Online First: 2022/11/27] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/ijms232214146&link_type=DOI) 14. 14.Meletis G, Tychala A, Ntritsos G, et al. Variant-Related Differences in Laboratory Biomarkers among Patients Affected with Alpha, Delta and Omicron: A Retrospective Whole Viral Genome Sequencing and Hospital-Setting Cohort Study. Biomedicines 2023;11(4) doi: 10.3390/biomedicines11041143 [published Online First: 2023/05/16] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3390/biomedicines11041143&link_type=DOI) 15. 15.Homen-Fernandez JR, Valls A, García A, et al. Differences in the inflammatory response among hospitalized patients with distinct variants of SARS-CoV-2. Front Immunol 2023;14:1267991. doi: 10.3389/fimmu.2023.1267991 [published Online First: 2023/11/01] [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.3389/fimmu.2023.1267991&link_type=DOI)