The CoLab-score rapidly and efficiently excludes COVID-19 at the emergency department without need for SARS-CoV-2 testing: a multicenter case-control study

Study design

Routine laboratory results combined with age and gender from all patient presenting at the emergency department (ED) of the Catharina Hospital Eindhoven from July 2019 to July 2020 were combined with SARS-CoV-2 PCR test results in a development dataset. A model that could predict the presence of a COVID-19 infection was fit to this dataset. Performance of the model was assessed by 1) internal validation using bootstrapping, 2) temporal validation by using data from all patients presenting at the ED of the Catharina Hospital from July 2020 to November 2020, and 3) external validation by using data from the ED of three other hospitals in the Netherlands. The study was reviewed by the Medical research Ethics Committees United (MEC-U) under study number W20.071, which confirmed that the Medical Research Involving Human Subjects Act (In Dutch: WMO) does not apply to this study. The study was thereafter reviewed and approved by the internal hospital review board.

Development dataset

All ED patient presentations of the Catharina Hospital Eindhoven from July 2019 to July 2020 were included in the development dataset, provided that routine laboratory testing had been requested by the attending physician. The routine laboratory panel at the ED consists of 28 laboratory tests. However, in some cases not all tests in the routine panel were requested or one or more quantitative results were not available due to analytical interference (hemolysis, lipemia or icterus). Presentations with missing values in the routine panel were excluded. Presentations with one or more extreme lab results (> 10 times standard deviation) were also excluded. After the first case of COVID-19 in the Netherlands (28 February 2020) all patients with symptoms of COVID-19 (either fever and/or respiratory symptoms) were subjected to nasopharyngeal PCR testing for SARS-CoV-2 RNA. PCR testing was performed by commercial tests that were approved by the Dutch national institute of public health (RIVM). If a patient had a positive PCR result in the past, subsequent presentations were excluded as re-presentations might be clinically different from de novo presentations.

The ED routine lab panel results were merged with SARS-CoV-2 PCR results if the underlying nose or throat swap had been taken ≤ 1 day prior, or ≤ 1 week after initial blood withdrawal at the ED. If multiple PCR tests were performed in this window and at least one PCR test was positive, the presentation was labelled “PCR-positive”. If all PCR test results in the time window were negative, the presentation was labelled as “PCR-negative”. If no PCR tests were performed in the time window but the presentation occurred after the first case of COVID-19 in the Netherlands, the presentation was labelled as “Asymptomatic”. All presentations before the first case in our center were labelled as “Pre-COVID-19”.

Laboratory tests

The routine laboratory panel consisted of hemocytometric and chemical analyses. In the Catharina hospital, the hemocytometric tests were performed on Sysmex XN-10 instruments (Sysmex Corp., Kobe, Japan) and consisted of hemoglobin, hematocrit, erythrocytes, mean corpuscular volume (MCV), mean cellular hemoglobin (MCH), mean cellular hemoglobin concentration (MCHC), thrombocytes, leukocytes, neutrophils, eosinophils, basophils, lymphocytes and monocytes. The chemical analyses were performed on a Cobas 8000 Pro (Roche Dx, Basel, Switzerland) instrument and consisted of glucose, total bilirubin, aspartate aminotransferase (ASAT), alanine aminotransferase (ALAT), lactate dehydrogenase (LD), creatine kinase (CK), alkaline phosphatase (ALP), gamma-glutamyltransferase (gGT), blood urea nitrogen (BUN), creatinine, CKD-epi estimated glomerular filtration rate (eGFR), potassium, sodium, chloride, albumin (bromocresol green) and C-reactive protein (CRP). These results were combined with age and gender.

Modelling

All data were processed and analyzed in R version 4.0.3 (R Core Team, 2020). Laboratory results, combined with age and gender were used as covariates in the model. Cases were ED presentations labelled as “PCR-positive”, controls as “PCR-negative”, “Asymptomatic” or “Pre-COVID-19”. To minimize additional missing data, all non-numeric results at the extremes of the measuring range, were converted to numeric results by removing the “<” and “>” signs. For eGFR (CKD-epi) and CRP this caused a discontinuous distribution. Therefore, instead of random sampling for eGFR (CKD-epi) and CRP the raw precursor value was used instead of >90 ml/min/m² and <6 mg/L, respectively. Considering that laboratory results of bilirubin, ASAT, ALAT, LD, CK, AF and gGT can have heavy (right) tailed distributions, which in turn impacts model predictions, these variables were transformed logarithmically.

To achieve predictive accuracy, limit overfitting and emphasize sparsity, adaptive lasso regression was chosen (Tibshirani, 1996; Zou, 2006). Prior to model fitting, covariates were scaled to zero mean and unit variance, after model fitting coefficients were unscaled to obtain regression coefficients on the original scale. In adaptive lasso, weights are applied to each of the covariates present in the lasso constraint, the weight vector has to be calculated before the adaptive lasso regression is performed. Due to multicollinearity between laboratory tests in the routine lab panel, weights in the adaptive lasso were based on ridge regression estimates () as recommended by Zou (Zou, 2006). To obtain the optimal penalty (λ) for the ridge regression was chosen using 10 fold cross-validation (CV) with area under the ROC curve (AUC) as the loss function. The λ corresponding to the maximum AUC was selected to obtain . The weight vector () was calculated by . This weight vector was then used to fit an adaptive lasso regression where λ was chosen by the criterion ±1 SE of the maximum AUC. All models were fitted using the glmnet-package (Friedman et al., 2010).

Since this is a retrospective case-control study, the sample prevalence may not reflect the true/current prevalence, as is the case with COVID-19. Therefore the intercept term in the model must be adjusted according to the actual prevalence (if known, or estimated) to obtain well-calibrated probabilities (King and Zeng, 2001).

CoLab-score

Since manually updating the intercept term is not straightforward to implement in clinical practice, we developed the CoLab-score. The CoLab-score is a categorization of the linear predictor into a CoLab-score ranging from 0 to 5, where score 0 reflects low risk for COVID-19 and score 5 reflects high risk. The CoLab-score is based on a fixed number needed to test of 15, this implies that one is willing to test 15 patients to find one positive. A number needed to test of 15 corresponds to a threshold probability of 1/15 = 0.067, on the linear predictor (or log-odds) scale, this corresponds to a threshold of logit(0.067) = −2.639. This threshold is correct when the sample prevalence matches the actual prevalence. In reality the prevalence of COVID-19 in the ED varies, therefore thresholds are calculated for prevalence cut-points of 1%, 2%, 5%, 10% and 40% using the intercept adjustment formula by King (King and Zeng, 2001). The intervals obtained through these breaks correspond to CoLab-scores 5 to 0, respectively. The CoLab-score can thus be used as a “prevalence-dependent binary classification” (either positive or negative). If COVID-19 prevalence at the ED in the previous week is <1%, only CoLab-score 5 should be classified as positive and CoLab-score 0 till 4 as negative. If prevalence is 1% – 2%, CoLab-score 4 and 5 should be classified as positive and 1 – 3 negative. Similarly, with a prevalence of 2 – 5% the split is between CoLab-score 2 and 3 and with prevalence of 5 – 10% between CoLab-score 1 – 2. If the prevalence is higher than 10% only CoLab-score 0 is classified as negative. Using the CoLab-score in this fashion, aims to preserve a number need to test of 15.

Internal validation

To assess model performance while taking overfitting into account, bootstrapping was performed. 1000 bootstrap samples were generated from the original data. On each bootstrap sample, the full model fitting procedure and CoLab-score conversion were performed. Optimism adjusted performance measures of the CoLab-score were obtained by applying the 0.632 bootstrap rule to the in-sample and out-of-bag-sample performance (Efron, 1983). Performance measures included, AUC, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of each CoLab-score. The pROC-package was used to calculate performance measures (Robin et al., 2011). The performance of the prevalence dependent binary CoLab-classification was evaluated in terms of overall sensitivity and specificity, and potential reduction in the number of PCR-tests. Although the full inclusion period from July 2019 to July 2020 was used for model fitting, the performance was evaluated on the period starting from the first COVID-19 infection (24^th of February 2020) to July 2020. This was done to obtain performance measures that would reflect real world performance during COVID-19 waves.

Temporal validation

For temporal validation, results from our center were prospectively analyzed from July 2020 to November 2020. During this period, a second wave of COVID-19 infections developed. In this period there was also more widespread external PCR testing by municipal health services (In Dutch: GGD). Since the results of external conducted PCR tests were not available to our study, the outcome in the temporal validation cohort was chosen as a composite of the hospital registration of a confirmed COVID-19 infection or at least one positive PCR test results in the hospital (with the same time window as described above). The temporal validation consisted of assessing the AUC, sensitivity, specificity, PPV and NPV of each CoLab-score threshold. Calibration was assessed graphically by converting the CoLab-linear predictor to the predicted probability (through the inverse logit function) and plotting the proportion of observed outcomes versus expected probabilities. The rms-package was used to generate calibration plots (Harrell Jr, 2021).

External validation

For the external validation, several centers in the Netherlands were approached and asked if they collected the required panel of laboratory tests and had access to SARS-CoV-2 PCR test results. Seven centers responded and the three centers with the least inclusion bias (i.e. measuring routinely all laboratory tests underlying the CoLab-score for large unselected patient groups) were Gelre Hospitals (center 1), Atalmedial Diagnostic Centers, location Alrijne Hospital Leiderdorp (center 2) and Zuyderland Medical Center (center 3). The hematological parameters were measured with Sysmex XN10/XN20 (center 1), CELL-DYN-Sapphire (Abbott Laboratories) (center 2) and Sysmex XN10 instruments (center 3). The clinical chemistry parameters were measured with Architect c14100/c160000 (Abbott Laboratories) (center 1), Architect ci4100 (Abbott Laboratories) (center 2) and Cobas 8000 instruments (Roche Dx) (center 3). The external validation was similar to the temporal validation and consisted of assessing the AUC sensitivity, specificity, PPV and NPV of each CoLab-score threshold. Calibration was assessed graphically analogous to the temporal validation dataset. In addition, the distributions of the variables in the CoLab-scores was plotted for all control (i.e. negative) patients to determine if there were any systematic differences that could cause bias in the CoLab-score predictions.

Development dataset

For model building, 12.879 emergency department (ED) presentations of 10.327 patients from July 2019 to July 2020 were included. After excluding cases with an incomplete lab panel, patient presentations that occurred after a positive PCR test in the past (re-presentations) and presentations with extreme values (>10 times standard deviation) in any of the lab results, 10.417 presentations of 8610 patients remained (Figure 1A).

• Download figure
• Open in new tab

Figure 1: Inclusion flow of patients in the development (A) and temporal validation (B) dataset.

All patient admissions with routine venous blood sampling at the emergency department (ED) were included. For the development dataset, completeness of the lab panel was assessed for all the 29 laboratory tests (see Table 1), for the temporal validation dataset this was only necessary for 10 laboratory tests (see Table 2). The major causes of missingness are described in the text. In the development dataset, presentations with extreme values (>10 SD) were excluded. The same limits were applied to the temporal validation dataset (see Table 2 for limits).

Descriptive statistics of ED presentations are shown in Table 1. Presentations were stratified into four categories: “Pre-COVID-19” consists of all ED presentations before the first case of COVID-19 in our center (N = 5890). This period also includes a winter flu-season. The category “Asymptomatic” consists of ED presentations (N = 3303) after the first case of COVID-19, but who were not subjected to PCR testing. Finally, presentations tested for SARS-CoV-2 were categorized depending on the result as “PCR negative” (N= 945) or “PCR positive” (N= 279). In the latter category, 91% (95% CI: 88 to 94%) of the cases were tested positive in their first PCR. The remaining 24 patients were positive in their second (N = 18), third (N = 5) or fourth (N = 1) PCR.

Table 1 shows the mean/median concentrations of the ED laboratory tests for each category. Concentrations that clinically differed from the Pre-COVID-19 category are marked in grey. As expected, patients tested positive for SARS-CoV-2 showed multiple clinically relevant differences. Interestingly, some differences (like increased CRP and gGT) were – although to a lesser extent – also present in the COVID-19 negative-tested patient group.

CoLab-score

The model obtained through adaptive lasso regression contained eleven variables, which are depicted with their regression coefficients (weights) in Table 2. A larger β-coefficient does not imply that a variable is more important in predicting the odds of testing positive for SARS-CoV-2, since the coefficients are unscaled after model fitting and variables are on different scales. Therefore, the relative importance is calculated based on scaled coefficients. The absolute basophil count has the highest relative weight, followed by eosinophil count.

As shown in Figure 2, the linear predictor clearly discriminates between COVID-19 and non-COVID-19. Note that in this figure, the area under both curves is equal since these are probability density functions, in reality the weekly COVID-19 prevalence at the ED varies from <1% to 36% and hence the number of presentations in the PCR positive group. Therefore, the linear predictor was converted to CoLab-scores 0 – 5 (Figure 2). Subsequently, the actual prevalence of the previous week was used to obtain a threshold above which patients should be tested, this is hereafter referred to as the “prevalence-dependent binary classification”. CoLab-scores above the prevalence-dependent cut-off value, are classified as “positive” and warrant PCR testing, while CoLab-scores equal to or below the cut-off value (= ”negative”) discourage PCR testing. If the prevalence is not known, the CoLab-score can be used as-is, where patients with a CoLab-score of 0 can be considered non-COVID-19 and patients with a score 5 as highly suspicious for COVID-19.

• Download figure
• Open in new tab

Figure 2: Probability density plot of the CoLab-linear predictor.

The probability density plots for COVID (dark grey) and non-COVID patients (light grey) are plotted against the linear predictor (see table 2). The CoLab-score cut-offs (–5.83, –4.02, –3.29, –2.34 and –1.64) are depicted with vertical dashed lines. The white-boxed numbers (between the cut-offs) represent the corresponding CoLab-score. Note that while the area under both curves is identical (since these are probability density functions), in absolute numbers the “negative or untested”-group is about 36 times larger than the PCR positive group.

Internal validation

The model was validated on the period starting from the first COVID-19 infection to July 2020, in this period the mean prevalence was 7.2%. The AUC of the CoLab-score is 0.930 (95% CI: 0.909 to 0.945). A CoLab-score of 0 has a negative predictive value (NPV) of 0.997 (95% CI: 0.994 to 0.999, Table 3), one third (38.4%, 95% CI: 26.4 to 48.4%) of all ED presentations were assigned this score and can therefore be safely excluded. Conversely, 6.2% (95% CI: 6.3 to 7.2%) of the ED patients have a CoLab-score 5. Given the high positive predictive value (PPV) of this category (0.682, 95% CI: 0.622 to 0.740), subsequent PCR testing is advised.

For CoLab-scores 1 - 4, the threshold to guide PCR testing is dependent on COVID-19 prevalence. Therefore, the CoLab-score was dichotomized as positive or negative, based on the prevalence of the previous week. As shown in Table 3, the sensitivity of this binary CoLab-classification (0.923, 95% CI: 0.879 to 0.944) is similar to singular PCR testing (0.91, 95% CI: 0.88 to 0.94), and has a NPV of 0.992 (95% CI: 0.987 to 0.994). Therefore, the CoLab-score and the prevalence dependent binary classification could be used to guide PCR testing when rapid PCR testing is not available or test capacity is limited. Using the CoLab-score retrospectively on the development dataset to guide PCR testing, 92.3% (95% CI: 87.9 to 94.4%) of the COVID-19 patients could have been identified with 30.5% (95% CI: 14.5% to 37.6%) fewer PCR test. The largest reductions are seen in periods with low or rapidly decreasing prevalence, when more than half of the PCR tests may be superfluous (Figure 3).

• Download figure
• Open in new tab

Figure 3: CoLab-guided PCR testing versus routine practice.

The black bars represent the number of SARS-CoV-2-PCR tests performed on a weekly basis in the Catharina Hospital. The grey bars represent the number of positives by using the prevalence dependent binary CoLab-classification (in this setting the COVID-19 ED prevalence of the previous week determines the threshold above which a CoLab-score is considered positive in the week thereafter). If PCR testing was performed for all CoLab-positives, the sensitivity would not by lower than a single PCR-test (see Table 3) and this would result a reduction in the number of PCR tests.

Temporal validation

As the CoLab-score was developed in our center after the first COVID-19-wave in the Netherlands, the performance was evaluated in our center during the subsequent second wave. In the Netherlands, the second wave started in August 2020. During this period lab results from 5100 ED presentations were collected. After applying the inclusion flow as shown in Figure 1B, 4.333 presentations remained, of which 165 were associated with a COVID-19 infection.

The mean prevalence in this period was 3.8%, which is approximately half of the prevalence during the first period. The AUC of the CoLab-score in the temporal validation set is 0.895 (95% CI: 0.866 to 0.932). The performance in terms of specificity and NPV is similar to the development cohort, whereas the sensitivity and PPV are slightly lower, although 95% CIs overlap (see Tables 3 and 4). In terms of calibration, the observed fraction of COVID-19 positives appears slightly higher than expected (Figure 6A). When using the prevalence-dependent binary classification, the sensitivity is 0.868 and the NPV is 0.993 (Table 4), which is somewhat lower than the development dataset.

Based on these findings, we investigated the relation between the CoLab-linear predictor and duration of COVID-19-related symptoms at ED presentation in the ”PCR positive” presentations of both the internal and temporal validation datasets. As shown in Figure 4, the CoLab-linear predictor increases with the duration of symptoms and gradually decreases after day 7. As a consequence, some COVID-19 patients with early or late presentation after onset of symptoms are missed by the CoLab-score. Optimal performance of the CoLab-score is achieved when the onset of symptoms is > 1 and < 10 days prior to ED presentation.

• Download figure
• Open in new tab

Figure 4: Association between the CoLab-linear predictor and the duration of COVID-19-related symptoms.

For all PCR-positive ED presentations in the development and temporal validation dataset, the CoLab-linear predict is plotted against the duration of COVID-related symptoms as registered in the electronic patient records. Patients with unknown duration are not plotted. Patients without symptoms were plotted at 0 days. The solid horizontal lines represent the CoLab-score thresholds, the dashed line is a LOESS regression curve with 95% CI. As the duration of symptoms is an integer, some random jitter was added to the days, for visualization purposes. Note that only the first 14 days are shown in this graph.

External validation

For external validation, data obtained from three other centers were used. The data were processed analogous to the development dataset. The inclusion flow is summarized in Figure 5.

• Download figure
• Open in new tab

Figure 5: Inclusion flow of ED patients in three external centers.

All emergency department (ED) presentations with routine venous blood sampling were included. Missingness of lab panels was assessed for the 11 variables in the CoLab-score (see Table 2). Re-presentations after a positive PCR result or clinical COVID-19 registration were excluded as “previous COVID-19+”. Presentations with any laboratory result above the limits of the CoLab-score (see Table 2) were excluded.

The large number of patients that are excluded in the first step (incomplete laboratory panel) in center 1 and center 2 is due to the fact that only ED presentations for internal medicine were tested for all 10 laboratory tests of the CoLab-score. The ED lab panel of other disciplines (e.g. urology, surgery or pediatrics) differed and did not contain the required tests. Nevertheless, the majority of COVID-19 ED presentations were examined by clinicians of internal medicine, which is reflected by the low number of PCR-positive patients excluded. Other causes of missing lab results were mainly due to assay interference. COVID-19 prevalence differed between the three centers (4.0%, 3.4% and 9.5% respectively) and was lower in centers 1 and 2, and higher in center 3 than the internal validation prevalence. The AUC’s of the CoLab-score are 0.904 (95% CI: 0.866 to 0.942), 0.886 (95% CI: 0.851 - 0.922) and 0.891 (95% CI: 0.872 - 0.909), for centers 1, 2, and 3 respectively.

Performance in terms of sensitivity and specificity is shown in Table 4. Also the PPVs and NPVs are shown, although these are largely dependent on prevalence. Although the sensitivity of the prevalence dependent binary CoLab-classification in some centers is somewhat lower than in the development and temporal dataset, the sensitivity of CoLab-score 0 in all centers is ≥ 0.96. Therefore, the NPV of CoLab-score 0 was more than 99%. However, when only presentations were taken into account with duration of symptoms >1 and < 10 days prior to presentation, the NPV for all patients with CoLab-score 0 was >99.8% (Table 4).

Calibration plots for external centers are shown in Figure 6B-D, the observed fraction of COVID-19 positives is slightly lower than expected in centers 1 and 2. For center 3, low probabilities appear slightly underestimated and high probabilities slightly overestimated.

• Download figure
• Open in new tab

Figure 6: CoLab-score calibration plots of the temporal validation (A), external validation center 1 (B), external validation center 2 (C) and external validation center 3 (D).

In the calibration plots, the proportion of observed COVID-19 positives versus expected probabilities are plotted. Observations are grouped with an average of 150 observations per group. The expected probabilities follow from applying the inverse logit function to the CoLab-linear predictor calculated from Table 2. If the observed proportion in an external dataset is lower than the expected proportion, this means risks are over-estimated, if the observed fraction is higher, risks are under-estimated. Ideally, observed proportions are equal to expected proportions, this ideal-calibration-line is shown as a straight line through the origin with a slope of 1. The logistic calibration line is a logistic regression fit of the predicted probabilities. [Intercept, slope] for plots A-D: A [0.22, 0.92], B [-0.39, 0.92], C [-0.76, 0.77], D [0.08, 0.79]. Although no validation datasets show perfect calibration, this is the result of differences in COVID-19 prevalence in each center and differences in calibration of laboratory equipment (see Supplemental Figure 6).

Supplemental Figure 6: Probability density plots of laboratory parameters.

Probability density plots are shown for all control patients of the development dataset and the three external centers. Ideally all distributions should overlap since this implies that control patient populations are most likely similar in the development dataset to the external datasets.