Are commercial antibody assays substantially underestimating SARS-CoV-2 ever infection? An analysis on a population-based sample in a high exposure setting

Introduction

Coronavirus disease 2019 (COVID-19), due to the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), continues to be a global health challenge. As of November 22, 2020, the COVID-19 burden included 57.6 million confirmed cases and 1.3 million deaths worldwide [1]. Meanwhile, the true extent of exposure to the SARS-CoV-2 infection and how far different national populations are from herd immunity remain poorly understood. Commercial serological assays are increasingly being used to address this gap in evidence. The extent to which such assays can capture ever infection in a population remains to be elucidated.

Understanding who has been exposed and potentially acquired immunity against this virus may help healthcare providers and public health stakeholders in establishing and implementing more efficient and effective strategies and policies for managing the disease and economic burden associated with the COVID-19 pandemic.

Qatar experienced a large SARS-CoV-2 epidemic with a high rate of laboratory-confirmed infections at >60,000 infections per million population [2-4]. As part of the national response, the public health authorities expanded serological testing for SARS-CoV-2 antibodies for both healthcare and research purposes. Three automated main serological testing platforms are being used. The first is the Roche Elecsys® Anti SARS CoV 2 (Roche, Switzerland) [5] platform at Hamad Medical Corporation (HMC), the main public healthcare provider and the nationally-designated provider for all COVID-19 healthcare needs. The second is the Mindray CL-900i anti-SARS-CoV-2 IgG (Shenzhen Mindray Bio-Medical Electronics Co., China) [6] platform at Qatar University (QU), which is used for research purposes. The third is the BioMérieux VidasIII (BioMérieux, Marcy-l’Etoile, France) [7] platform at QU, which is also being used for research purposes.

To interpret the emerging results of serological testing and to inform the national response, this study was conducted to compare the performance of these three assays and to assess the implications for measuring SARS-CoV-2 ever infection. The novelty and strength of this study is that it is conducted based on a population-based sample [8] in a setting at a high exposure to this infection [2, 3, 9, 10].

Methods

Blood specimens were collected from 394 volunteering individuals between July 26 and September 9, 2020, as a sub-study of a nationwide survey [8] assessing SARS-CoV-2 seroprevalence (IgG antibodies) and current-infection prevalence (using polymerase chain reaction [PCR] testing) in the wider population of craft and manual workers who constitute 60% of the population of Qatar [11]. Informed by prior work [12, 13], a sample size of 400 was estimated to be sufficient to ensure narrow confidence intervals for the Cohen’s kappa statistic, but we were able to include and test only 394 specimens. The research work was approved by the ethics review boards at HMC, QU, and Weill Cornell Medicine-Qatar.

The automated serological testing was performed using the above indicated three commercial assays. The Roche Elecsys® Anti SARS-CoV-2 (“Elecsys” in short form) assay, our reference assasy, uses a recombinant protein representing the nucleocapsid (N) antigen for the determination of IgG antibodies against SARS-CoV-2 [5]. Anti-SARS-CoV-2 results were generated following the manufacturer’s instructions (reactive: optical-density cutoff index ≥1.0 vs. non-reactive: cutoff index <1.0) [5, 14].

The Mindray CL-900i® anti-SARS-CoV-2 IgG (“CL-900i” in short form) assay uses paramagnetic microplates coated with recombinant nucleocapsid (N) and spike (S) antigens for the determination of anti-SARS-CoV-2 IgG antibodies [6]. The analyzer automatically calculates the analyte concentration of each serum specimen according to a master calibration curve, and the results are shown in the units of U/mL. Anti-SARS-CoV-2 results were generated following the manufacturer’s instructions (reactive: optical-density cutoff index ≥10.0 vs. non-reactive: cutoff index <10.0) [6, 15].

The BioMérieux VidasIII assay (“VidasIII” in short form) uses a VIDASIII® analyzer for anti-SARS-CoV-2 IgG detection through a two-step sandwich ELFA assay [7]. The IgG in the serum specimen binds to a recombinant spike S1 sub-domain (containing the receptor-binding domain [S1-RBD]) of the SARS-CoV-2 virus coated on a solid phase. Alkaline phosphatase-conjugated anti-human IgG are then added. The fluorescence intensity generated by the substrate is then measured at a wavelength of 450 nm. The intensity of the signal is proportional to the level of IgG. The optical-density cutoff index was calculated according to the manufacturer’s instructions [7, 14]. The ratio between the relative fluorescence value (RFV) measured in the specimen and the RFV from the calibrator was interpreted as positive if the index value was ≥1.0 [7, 14].

All PCR testing was conducted at HMC Central Laboratory or at Sidra Medicine Laboratory, following standardized protocols. Nasopharyngeal and oropharyngeal swabs (Huachenyang Technology, China) were collected and placed in Universal Transport Medium (UTM). Aliquots of UTM were: extracted on the QIAsymphony platform (QIAGEN, USA) and tested with real-time reverse-transcription PCR (RT-qPCR) using the TaqPath™ COVID-19 Combo Kit (Thermo Fisher Scientific, USA) on an ABI 7500 FAST (ThermoFisher, USA); extracted using a custom protocol [16] on a Hamilton Microlab STAR (Hamilton, USA) and tested using the AccuPower SARS-CoV-2 Real-Time RT-PCR Kit (Bioneer, Korea) on an ABI 7500 FAST; or loaded directly to a Roche cobas® 6800 system and assayed with the cobas® SARS-CoV-2 Test (Roche, Switzerland). The first assay targets the S, N, and ORF1ab regions of the virus; the second targets the virus’ RdRp and E-gene regions; and the third targets the ORF1ab and E-gene regions.

Results of the serological and PCR testing were subsequently linked to the HMC centralized and standardized database comprising all SARS-CoV-2 PCR testing conducted in Qatar since the start of the epidemic [2, 17]. The database also includes data on hospitalization and on the World Health Organization (WHO) severity classification [18] for the hospitalized PCR-confirmed infections.

Results from the three types of serological testing were cross-tabulated. Four concordance metrics were estimated: overall, positive, and negative percent agreement, as well as Cohen’s kappa statistic. The latter is a robust metric that measures the level of agreement, beyond chance, between two diagnostic testing methods [19]. The kappa statistic ranges between 0 and 1; a value ≤0.40 indicates poor agreement, a value between 0.40 and 0.75 indicates fair/good agreement, and a value ≥0.75 indicates excellent agreement [19]. Level of significance was established at 5%, and a 95% confidence interval (CI) was reported for each metric. A nonparametric statistical method, Spearman correlation, was used to assess the correlation between the optical densities of each pair of antibody assays. Calculations were conducted using Microsoft Excel.

Results

SARS-CoV-2 seroprevalence was estimated at 42.9% (169/394; 95% CI 38.1%-47.8%) using the CL-900i assay, 40.6% (160/394; 95% CI 35.9%-45.5%) using the VidasIII assay, and 42.4% (167/394; 95% CI 37.6%-47.3%) using the Elecsys assay. A total of 183 specimens were seropositive in at least one of the assays for a total sample seroprevalence of 46.5% (183/394; 95% CI 41.6%-51.4%).

Table S1 shows the results of the serological and PCR testing for each of the 394 participants. A total of 35 specimens showed discordant results between the three antibody assays (Table 1). Of the 35 individuals with discordant antibody results, 9 were PCR-positive at the time of specimen collection. Eleven specimens were seropositive using the CL-900i assay but seronegative using the VidasIII and the Elecsys assays; among these, two were PCR-positive with cycle threshold (Ct) values of 23.9 and 27.0. Five specimens were seropositive using the Elecsys assay but seronegative using the VidasIII and the CL-900i assays; among these, one person was PCR-positive with a Ct value of 21.6. Two specimens were seropositive using the VidasIII assay but seronegative using the CL-900i and the Elecsys assays; among these, one was PCR-positive with a Ct value of 29.2.

The overall, positive, and negative percent agreements between the CL-900i and the Elecsys assays were estimated at 93.4% (95% CI 90.5%-95.5%), 92.8% (95% CI 87.9%-95.8%), and 93.8% (95% CI 89.9%-96.3%), respectively (Table 2A). Cohen’s kappa statistic was estimated at 0.87 (95% CI 0.83-0.90), indicating excellent agreement between the two assays (Table 2A). The Spearman correlation between the optical densities was at 0.751 (p-value<0.001) indicating strong correlation.

The overall, positive, and negative percent agreements between the VidasIII and the Elecsys assays were estimated at 95.7% (95% CI 93.2%-97.3%), 92.8% (95% CI 87.9%-95.8%), and 97.8% (95% CI 95.0%-99.1%), respectively (Table 2B). Cohen’s kappa statistic was estimated at 0.91 (95% CI 0.88-0.94), indicating excellent agreement between the two assays (Table 2B). The Spearman correlation between the optical densities was at 0.824 (p-value<0.001) indicating strong correlation.

The overall, positive, and negative percent agreements between the VidasIII and the CL-900i assays were estimated at 93.2% (95% CI 90.2%-95.3%), 89.3% (95% CI 83.8%-93.2%), and 96.0% (95% CI 92.6%-97.9%), respectively (Table 2C). Cohen’s kappa statistic was estimated at 0.86 (95% CI 0.82-0.90), indicating excellent agreement between the two assays (Table 2C). The Spearman correlation between the optical densities was at 0.804 (p-value<0.001) indicating strong correlation.

A total of 49 swabs were PCR-positive at the time of specimen collection during this study for a current-infection prevalence of 12.5% (49/392; 95% CI 9.6%-16.1%)—two individuals declined PCR testing (but not antibody testing) during this study. Figure 1A shows the distribution of the PCR Ct values among those PCR-positive, indicating broad distribution suggestive of these persons being diagnosed at the various stages of infection. The mean PCR Ct value was 25.8 with a standard deviation of 6.3.

• Download figure
• Open in new tab

Figure 1.

Distribution of polymerase chain reaction (PCR) cycle threshold (Ct) values of A) the 49 persons identified as SARS-CoV-2 PCR-positive at the time of specimen collection during the study, B) the 17 persons identified as SARS-CoV-2 PCR-positive, but were antibody-positive in at least one of the assays, and C) the 32 persons identified as SARS-CoV-2 PCR-positive, but were antibody-negative in all three assays.

Among all those PCR-positive at the time of specimen collection, 30.6% (15/49; 95% CI 19.5%-44.5%) were seropositive in the CL-900i assay, 22.5% (11/49; 95% CI 13.0%-35.9%) were seropositive in the VidasIII assay, 24.5% (12/49; 95% CI 14.6%-38.1%) were seropositive in the Elecsys assay, and 34.7% (17/49; 95% CI 22.9%-48.7%) were seropositive in at least one of the assays. Therefore, 32 individuals had a PCR-positive diagnosis at the time of specimen collection, but were antibody-negative in all three assays.

Among all those seropositive in at least one of the assays, 9.3% were PCR-positive (17/183; 95% CI 5.9%-14.4%) at the time of specimen collection. The mean PCR Ct value was 30.8 with a standard deviation of 5.2, indicative of mostly non-recent infections (Figure 1B). Among all those seronegative in all three assays, 15.2% were PCR-positive (32/211; 95% CI 11.0%-20.6%) at the time of specimen collection. The mean PCR Ct value was 23.0 with a standard deviation of 5.1, indicative of mostly recent infections (Figure 1C).

Through linking with the national SARS-CoV-2 PCR testing database [17], and of the 394 participants, 4.3% (17/394; 95% CI 2.7%-6.8%) had a record of SARS-CoV-2 PCR-confirmed diagnosis prior to this study. All but one of these were antibody-positive by at least one of the assays. The individual testing antibody-negative but had a prior PCR-confirmed diagnosis was diagnosed on July 23, 2020, that is four days prior to the antibody serological test date. This individual declined PCR testing during this study and at the time of the serological test.

Of the 183 persons with an antibody-positive status in at least one assay, 16 persons had a SARS-CoV-2 PCR-confirmed diagnosis prior to this study. Accordingly, the detection rate (the percentage of those antibody-positive who had a prior PCR-confirmed diagnosis) was 8.7% (16/183; 95% CI 5.5%-13.7%).

Based on the above, a total of 216 persons had a laboratory-confirmed infection at or prior to this study; that is an antibody-positive result in at least one assay (183 cases), a PCR-positive diagnosis prior to this study but with an antibody-negative status in all three assays (1 case), or a PCR-positive diagnosis at the time of specimen collection during this study but with an antibody-negative status in all three assays (32 cases). Accordingly, the percentage of persons with evidence of ever infection through either PCR or antibody testing was 54.8% (216/394; 95% CI 49.9%-59.7%). Moreover, only 78.2% of ever infections were antibody-positive by the CL-900i assay (169/216; 95% CI 72.3%-83.2%), 74.1% by the VidasIII assay (160/216; 95% CI 67.8%-79.5%), and 77.3% by the Elecsys assay (167/216; 95% CI 71.3%-82.4%).

Linking with the national COVID-19 hospitalization database [17] identified only one laboratory-confirmed infection through this study to have progressed to severe disease per WHO severity classification [18]. The person had also a diagnosis of diabetes, hypertension, and coronary artery disease. This person was diagnosed PCR-positive at time of specimen collection, was seronegative in the CL-900i and the VidasIII assays, but was seropositive in the Elecsys assay. No infection was critical per WHO severity classification [18] and no COVID-19 death was reported for any of the study participants.

Discussion

A primary finding of this study is that all three antibody assays had comparable performance and excellent agreement. This positive finding, however, conceals important shortcomings about the use and performance of commercial antibody assays in assessing ever infection with SARS-CoV-2 in population-based surveys, especially at times of high SARS-CoV-2 incidence, as is the case at present globally.

The first shortcoming is that each of these three assays missed ≥69% of those who were PCR-positive at the time of specimen collection. This finding is explained in large part by the 1-4 weeks delay in development of detectable antibodies after acquiring the infection [20, 21]. This explanation is supported by the low PCR Ct value among those PCR-positive but antibody-negative (Figure 1C), which indicates recency of infection. At the time of the study and in the population being studied, the outbreak was advancing, so there was a significant proportion of new infections, making the serology assay less useful for estimating population prevalence of ever infection. It is unknown whether the lower sensitivity could have been due in part also to commercial assay development preferentially opting to maximize the specificity of the assay, to avoid a false positive diagnosis with its clinical implications, but at the expense of the sensitivity of the assay.

The second shortcoming is that each of these three assays also missed other individuals with evidence of ever infection. Despite excellent agreement overall, nearly 10% of the total sample still showed discordant results between the three antibody assays. Differences in the sensitivity of the assays to diagnose recent infection explains only partially these discordant results. Indeed, most (74.3%) of these persons with discordant results were PCR-negative at the time of specimen collection (Table 1), and thus less likely to have had a recent infection. The extent to which false positivity may explain some of these discordant results is unknown, but the three manufacturers reported essentially perfect specificity for each of these assays [5, 7, 22, 23].

As a consequence of these findings, the use of any one of these antibody assays to assess ever infection in a population-based sample, especially at the times of high SARS-CoV-2 incidence, will substantially underestimate the ever infection prevalence in the sample. In the sample in the present study, at least 20% of ever infections were missed. A solution to this challenge is to combine PCR data and serology data together, or that the serology data cannot be adequately interpreted without knowledge of the PCR positivity data, or that serology is less useful when the epidemiology is rapidly changing. With the global pandemic continuing at high SARS-CoV-2 incidence, this finding suggests that ever infection in populations is possibly substantially higher than is currently believed.

This study has some limitations. Two out of the 394 participants included in the study declined PCR testing (but not serological testing) at time of specimen collection. The performance of these antibody assays was compared to each other (and to PCR testing), but not to a gold standard test of seropositivity, as such a test was not available to study investigators. Therefore, we were unable to measure ever infection prevalence to a gold standard, and use this to compare the performance of each assay to the gold standard, nor to assess the sensitivity and specificity of each assay in the study sample. The specificity of the Elecsys assay has previously been reported to be 99.98% and the sensitivity to be 98.80% on day 14 after PCR diagnosis [5]. A validation study by Public Health England reported a specificity of 100% and a sensitivity of 83.9% for the same assay [22]. As for the remaining assays, specificity and sensitivity were reported at 94.9% and 82.2%, respectively, for the CL-900i assay [23], and at 99.9% and 88.6%, respectively, for the VidasIII assay [7].

In conclusion, all three assays had comparable performance and excellent agreement when used in a high SARS-CoV-2 exposure setting, but still missed at least 20% of cases with laboratory-confirmed evidence of ever infection. This suggests that current growing use of commercial antibody assays to assess ever infection in population-based surveys, especially at times of high SARS-CoV-2 incidence when many infections are recent, is likely to substantially underestimate actual infection exposure. The findings demonstrate further the need to interpret the serology testing together with PCR testing.

Funding

The authors are grateful for support provided by Qatar University, Ministry of Public Health, Hamad Medical Corporation, and the Biomedical Research Program and the Biostatistics, Epidemiology, and Biomathematics Research Core, both at Weill Cornell Medicine-Qatar. Part of this work was made possible by grant No. RRC-2-032 from the Qatar National Research Fund (a member of Qatar Foundation) given to GKN. Finally, we would like to thank the BioMérieux Middle East Regional Office and their distributor in Qatar for providing us with the VidasIII reagent as in-kind support for this project. The statements made herein are solely the responsibility of the authors.

Author contributions

AAT conceived the study. LJA, GKN, and SRD co-designed the study and led the statistical analyses. GKN and PC led the laboratory testing. FS, DWA, and HA performed the CL-900i and the VidasIII testing. SRD, HC, and LJA performed the data analyses and wrote the first draft of the article. All authors contributed to data collection and acquisition, database development, discussion and interpretation of the results, and to the writing of the manuscript. All authors have read and approved the final manuscript.

Conflicts of interests

None.

Data and materials availability

All relevant data are available within the manuscript.