Evaluation of three rapid lateral flow antigen detection tests for the diagnosis of SARS-CoV-2 infection ======================================================================================================== * AE Jääskeläinen * MJ Ahava * P Jokela * L Szirovicza * S Pohjala * O Vapalahti * M Lappalainen * J Hepojoki * S Kurkela ## Abstract **Introduction** The COVID-19 pandemic has led to high demand of diagnostic tools. Rapid antigen detection tests have been developed and many have received regulatory acceptance such as CE IVD or FDA markings. Their performance needs to be carefully assessed. **Materials and Methods** 158 positive and 40 negative retrospective samples collected in saline and analyzed by a laboratory-developed RT-PCR test were used to evaluate Sofia (Quidel), Standard Q (SD Biosensor), and Panbio™ (Abbott) rapid antigen detection tests (RADTs). A subset of the specimens was subjected to virus culture. **Results** The specificity of all RADTs was 100% and the sensitivity and percent agreement was 80% and 85% for Sofia, 81% and 85% for Standard Q, and 83% and 86% for Panbio™, respectively. All three RADTs evaluated in this study reached a more than 90% sensitivity for samples with a high viral load as estimated from the low Ct values in the reference RT-PCR. Virus culture was successful in 80% of specimens with a Ct value <25. **Conclusions** As expected, the RADTs were less sensitive than RT-PCR. However, they benefit from the speed and ease of testing, and lower price as compared to RT-PCR. Repeated testing in appropriate settings may improve the overall performance. ## Introduction The superior analytical sensitivity and specificity makes RT-PCR the primary diagnostic tool for the detection of SARS-CoV-2 in respiratory samples [1-2]. Rapid antigen detection tests (RADTs) can be cheap and easy to use compared to RT-PCR. Contrary to PCR, antigen tests do not amplify the detected target, rendering them generally less sensitive [3]. The SARS-CoV-2 RADTs are relatively new in the market, without extensive evaluation. They appear to show very variable performance: according to a systematic review published in August 2020, RADT sensitivity values varied from 0% to 94% [4]. We evaluated three CE IVD marked SARS-CoV-2 RADTs: Sofia (Quidel), Standard Q COVID-19 Ag (SD Biosensor), and Panbio™ (Abbott). In each of these tests, the sample is suspended or diluted in a sample inactivation medium and transferred to a test device. If SARS-CoV-2 antigen is present above a threshold concentration, a line will appear in the test device within 15 minutes. For Panbio™ and Standard Q, the line is visible for a naked eye, and for Sofia, detection of the fluorescent signal is automated. ## Materials and methods The evaluation was performed at the HUS Diagnostic Center, the diagnostic laboratory of Helsinki University Hospital and Helsinki and Uusimaa Hospital District, Helsinki, Finland. Research permit HUS/157/2020 (Helsinki University Hospital, Finland) was obtained from the local review board. ### Evaluated tests We evaluated the performance of Quidel Sofia SARS FIA (Quidel, San Diego, CA), Standard Q COVID-19 Ag test (SD Biosensor, Republic of Korea), and Panbio™ (Abbott Diagnostic GmbH, Jena, Germany) according to each manufacturer’s guidelines for samples in virus transport medium. All evaluated tests are intended for fresh swab samples, so this is off-label use and leads to dilution of samples, see Table 1. View this table: [Table 1.](http://medrxiv.org/content/early/2021/01/04/2020.12.30.20249057/T1) Table 1. Main properties of the three RADTs evaluated in this study. ### Reference test The samples were originally analyzed with a laboratory-developed RT-PCR test (LDT) based on the method by Corman and others [5] and modified by us [6] to detect the N gene target of SARS-CoV-2. ### Patient samples A total of 198 nasopharyngeal swabs in 0.9% saline taken between April and November 2020 and stored in −20°C were available for the study. The analytical performance evaluation was conducted with 102 samples: 40 LDT negative and 62 positive samples, selected to cover a wide Ct value range. The Ct value of the positive control varied between 26.35-32.66. To investigate the test performance in an outpatient setting, another set of LDT positive samples from adults from outpatient clinics and drive-through testing sites was included by selecting samples systematically backwards from 18 November until 96 specimens were reached, i.e. November 1-18 2020. Samples from hospitals and occupational health services were excluded, as well as samples that had less than the required 750 µl for three RADTs left. The positive control Ct values varied from 25.56 to 27.96. Eighty-seven samples were analyzed by Sofia, 90 by Panbio™ and 96 by Standard Q. ### Virus culture The PCR positive subset of samples used for analytical performance evaluation was subjected to virus isolation experiments in Vero E6 TMRPSS2 cells as described in [7]. ### Statistics Concordance of the results was examined in McNemars’s test. Level of statistical significance was set at P<0.05. Cohen’s kappa coefficient was computed to assess the agreement between the methods by chance and Mann-Whitney U test was used to compare the Ct value medians. Statistical analysis was performed using SPSS/PASW statistical program package, version 25 (IBM SPSS Inc., Chicago, IL, USA). ## Results The performance of the Sofia, Standard Q and Panbio™ RADTs in comparison with the LDT RT-PCR was assessed by analysing 188, 198 and 190 specimens, respectively. The three RADTs provided valid results for all specimens. The specificity for all three tests was 100%: all RT-PCR negative samples tested negative. In all, the results of Sofia were 80.4% (119/148) consistent with the reference RT-PCR with a kappa value of 0.636 (P<0.001). The Standard Q and Panbio™ tests achieved overall agreements of 84.85% and 86.32% as compared to the RT-PCR with kappa values of 0.633 (P<0.001) and 0.660 (P<0.001), respectively. For all of the RADTs, the observed difference to the reference PCR was statistically significant (P<0.001). The median Ct values of the specimens positive with RADTs were significantly lower than those of the specimens with false negative results (P<0.001 for all RADTs) (Table 2 and Figure 1). View this table: [Table 2.](http://medrxiv.org/content/early/2021/01/04/2020.12.30.20249057/T2) Table 2. The performance of RADTs, all samples, all RT-PCR positive samples, and samples grouped according to Ct values of the N gene target in the LDT RT-PCR. Tested, n, number of tested samples. Same as ref: number of samples tested the same as reference LDT RT-PCR test, positive or negative. Tested pos, n, number of samples that were positive in RADT. Ag-/Ag+: Ct median of samples that ![Figure 1.](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2021/01/04/2020.12.30.20249057/F1.medium.gif) [Figure 1.](http://medrxiv.org/content/early/2021/01/04/2020.12.30.20249057/F1) Figure 1. Sensitivities of Sofia, Standard Q and Panbio rapid antigen detection tests for SARS-CoV-2. Total number of RT-PCR positive samples tested for Sofia: 148, Standard Q: 158, Panbio: 152; number of RT-PCR positive samples, Ct<25, tested for Sofia: 89, Standard Q: 97, Panbio: 92. Number of RT-PCR positive samples, Ct 25-29.99, tested for Sofia: 34, Standard Q: 35, Panbio: 34. Number of R-PCR positive samples, Ct ≥30, tested for Sofia: 25, Standard Q: 26, Panbio: 26. Table 3 shows the results for the analytical evaluation of the tests, 62 positive and 40 negative samples in LDT RT-PCR with results of virus culture experiments. Of 18 culture positive specimens, 17 were positive in RADTs. View this table: [Table 3.](http://medrxiv.org/content/early/2021/01/04/2020.12.30.20249057/T3) Table 3. The performance of the antigen tests and virus culture in a panel of selected 62 positive and 40 negative samples according to Ct values of the N gene target in the LDT RT-PCR. Tested, n, number of tested samples. Tested pos, n, number of samples that were positive in antigen test or virus culture. ND, not done. The Ct values in the 96 positive samples of the adult outpatients were 10.74-32.49. In all, 75 samples (78%) had a Ct value <25, sixteen (17%) had Ct values of 25-29.99 and five (5%) ≥30. The sensitivity values of RADTs were 87% (Panbio™), 89% (Standard Q) and 92% (Sofia) but reached 92-96% for samples with Ct <30. (Table 4 and Figure 2). View this table: [Table 4.](http://medrxiv.org/content/early/2021/01/04/2020.12.30.20249057/T4) Table 4. The performance of antigen tests in a panel of positive adult outpatient samples in November 1-18, 2020, grouped according to Ct values of the N gene target in the LDT RT-PCR. Tested, n, number of tested samples. Tested pos, n, number of samples that were positive in antigen test. ![Figure 2.](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2021/01/04/2020.12.30.20249057/F2.medium.gif) [Figure 2.](http://medrxiv.org/content/early/2021/01/04/2020.12.30.20249057/F2) Figure 2. Sensitivity of the RADTs in comparison to Ct values in RT-PCR. Samples include all the LDT RT-PCR positive nasopharyngeal samples from adult outpatients at HUSLAB in 1.-18. November 2020 that had enough sample volume left for three antigen tests. N= 96, however, in some cases, not all three RADTs were performed to all samples. Each dot indicates one samples. Ct <25: 75 samples; Ct 25-29.99: 16 samples; Ct ≥30: 5 samples. ## Discussion We evaluated three lateral-flow RADTs for the detection of SARS-CoV-2 infection in nasopharyngeal swab samples collected in 0.9% saline. Our results agree with previous observations that RADTs perform well in detecting culture positive samples and samples with high viral loads [8-11]. Compared to RT-PCR, the sensitivity of RADTs is low: in the present study the sensitivity was only 12-38% in samples with Ct≥30 (Table 2). However, the distribution of Ct values of RT-PCR positive samples in settings where antigen detection tests could primarily be used is unknown. We wanted to approach the real-life setting by testing all positive outpatient adult samples tested by LDT in our laboratory during 2.5 weeks in November 2020. The testing strategy in Finland in November 2020 assumed patients to have at least mild symptoms of SARS-CoV-2 infection. Thus, this panel is unlikely to include asymptomatic or presymptomatic patients. During the time of evaluation, the sampling was available in average within 12 hours of seeking for testing. Although medical records of outpatients were not available, we assume the majority of this panel to consist of samples taken within a couple of days from symptom onset. 78% of samples had a Ct value <25, and 95% <30, of the N gene in our LDT. This may justify using antigen tests as a diagnostic tool even when their sensitivity compared to nucleic acid testing is poor for samples with high Ct values. Careful consideration between the advantages (ease, speed, costs) and disadvantages (lower sensitivity) needs to be conducted in the selection of approriate testing strategies [11-12]. If RADTs are considered, laboratories need to perform independent clinical validations, as manufacturers’ declaration of performance may not include satisfactory selection of clinical samples. WHO has stated SARS-CoV-2 a biohazard risk for laboratory personnel [13], which needs to be accounted for in RADT testing. In addition, RATDs are not easily scalable for mass testing. The present study is limited by its retrospective nature. Ideally, samples for RADT evaluation should be collected according to manufacturer’s instructions. Instead, we used frozen samples in saline. This may lead to underestimation of the sensitivity of the antigen detection tests. All three RADTs, Sofia, Standard Q and Panbio™, were specific but less sensitive than RT-PCR. In appropriate settings, this disadvantage may be compensated by the ability to perform the test outside a central laboratory, the shorter turnaround time, and lower price. The sensitivity of the testing regimen as a whole can be increased by repeat testing [14]. Simple and cheap RADTs with a satisfactory performance, such as the three tests evaluated in this study, make repeated testing a realistic diagnostic approach in settings in which RT-PCR is not readily or quickly available. ## Data Availability All detailed data is available from the authors upon request. ## Acknowledgements We thank Marianne Ahola, Jenna Roivas and Laila Shakari for excellent technical assistance. * Received December 30, 2020. * Revision received December 30, 2020. * Accepted January 4, 2021. * © 2021, Posted by Cold Spring Harbor Laboratory This pre-print is available under a Creative Commons License (Attribution 4.0 International), CC BY 4.0, as described at [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) ## References 1. 1.Matheeussen V, Corman VM, Donoso Mantke O, McCulloch E, Lammens C, Goossens H, Niemeyer D, Wallace PS, Klapper P, Niesters HGM, Drosten C, Ieven M, on behalf of the RECOVER project and collaborating networks. International external quality assessment for SARS-CoV-2 molecular detection and survey on clinical laboratory preparedness during the COVID-19 pandemic, April/May2020. Euro Surveill. 2020;25(27):pii=2001223. [https://doi.org/10.2807/1560-7917.ES.2020.25.27.2001223](https://doi.org/10.2807/1560-7917.ES.2020.25.27.2001223) 2. 2.van Kasteren PB, van der Veer B, den Brink S, Wijsman L, Jonge J, van den Brandt A, Molenkamp R, Reusken Cbem, Meijer A. Comparison of seven commercial RT-PCR diagnostic kits for COVID-19. J Clin Virol 2020:128 [https://doi.org/10.1016/j.jcv.2020.104412](https://doi.org/10.1016/j.jcv.2020.104412) 3. 3.Merckx J, Wali R, Schiller I, Caya C, Gore GC, Chartrand C, Dendukuri N, Papenburg J. Diagnostic accuracy of novel and traditional rapid tests for influenza infection compared with reverse transcriptase polymerase chain reaction. A systematic review and meta-analysis. Annals of Internal Medicine 2017:6. Doi: 10.7326/M17-0848 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7326/M17-0848&link_type=DOI) 4. 4.Dinnes J, Deeks JJ, Adriano A, Berhane S, Davenport C, Dittrich S, Emperador D, Takwoingi Y, Cunningham J, Beese S, Dretzke J, Ferrante di Ruffano L, Harris IM; Price MJ, Taylor-Phillips S, Hooft L, Leeflang MM, Spijker R, Van den Bruel A, Cochrane COVID-19 Diagnostic Test Accuracy Group: Rapid, point-of-care antigen and molecular-based tests for diagnosis of SARS-CoV-2 infection. Cochrane Database of Systematic Reviews 2020:8. DOI: 10.1002/14651858.CD013705. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/14651858.CD013705&link_type=DOI) 5. 5.Corman VM, Landt O, Kaiser M, Molenkamp R, Meijer A, Chu DK, Bleicker T, Brünink S, Schneider J, Schmidt ML, Mulders DG, Haagmans BL, van der Veer B, van den Brink S, Wijsman L, Goderski G, Romette JL, Ellis J, Zambon M, Peiris M, Goossens H, Reusken C, Koopmans MP, Drosten C. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Euro Surveill. 2020 Jan;25(3):2000045. doi:10.2807/1560-7917.ES.2020.25.3.2000045. Erratum in: Euro Surveill. 2020 Apr;25(14): Erratum in: Euro Surveill. 2020 Jul;25(30): PMID: 31992387; PMCID: PMC6988269. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2807/1560-107917.ES.2020.25.3.2000045&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=31992387&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F01%2F04%2F2020.12.30.20249057.atom) 6. 6.Mannonen L, Kallio-Kokko H, Loginov R, Jääskeläinen AE, Jokela P, Antikainen J, Väre P, Kekäläine n E, Kurkela S, Jarva H, Lappalainen M. Comparison of two commercial platforms and a laboratory developed test for detection of SARS-CoV-2 RNA. 2020 (Preprint.) doi:[https://doi.org/10.1101/2020.07.03.20144758](https://doi.org/10.1101/2020.07.03.20144758) 7. 7.Rusanen J, Kareinen L, Szirovicza L, Uğurlu H, Levanov L, Jääskeläinen AE, Ahava M, Kurkela S, Saksela K, Hedman K, Vapalahti O, Hepojoki J. A generic, scalable, and rapid TR-FRET – based assay for SARS-CoV-2 antigen detection. 2020 (Preprint.) doi:[https://doi.org/10.1101/2020.12.07.20245167](https://doi.org/10.1101/2020.12.07.20245167) 8. 8.Corman VM, Haage VC, Bleicker T, Schmidt ML, Mühlemann B, Zuchowski M, Jó Lei WK, Tscheak P, Möncke-Buchner E, Müller MA, Krumbholz A, Drexler JF, Drosten C: Comparison of seven commercial SARS-CoV.2 rapid Pont-of-Care Antigen tests. 2020. (Preprint.) doi:[https://doi.org/10.1101/2020.11.12.20230292](https://doi.org/10.1101/2020.11.12.20230292) 9. 9.Merino-Amador P, Guinea J, Muñoz-Gallego I, González-Donapetry P, Galán J-C, Antona N, Cilla G, Hernáez-Crespo S, Díaz-de Tuesta J-L, Gual-de Torrella A, González-Romo F Escribano P, Sánchez-Castellano MÁ, Sota-Busselo M, Delgado-Iribarren A, García J, Cantón R, Muñoz P, Folgueira MD, Cuenca-Estrella M, Oteo-Iglesias J, Spanish Panbio™ COVID-19 validation group: Multicenter evaluation of the Panbio™ COVID-19 Rapid Antigen-Detection Test for the diagnosis of SARS-CoV-2 infection. 2020 (Preprint.) doi:[https://doi.org/10.1101/2020.11.18.20230375](https://doi.org/10.1101/2020.11.18.20230375) 10. 10.Toptan T, Eckermann L, Pfeiffer AE, Hoehl S, Ciesek S, Drosten C, Corman VM. Evaluation of a SARS-CoV-2 rapid antigen test: potential to help reduce community spread? 2020 (Preprint.) doi:[https://doi.org/10.1101/2020.12.04.20240283](https://doi.org/10.1101/2020.12.04.20240283) 11. 11.WHO: Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays. Interim guidance. Available at [https://www.who.int/publications/i/item/antigen-detection-in-the-diagnosis-of-sars-cov-2infection-using-rapid-immunoassays](https://www.who.int/publications/i/item/antigen-detection-in-the-diagnosis-of-sars-cov-2infection-using-rapid-immunoassays). Updated September 11, 2020. Accessed December 29, 2020. 12. 12.ECDC: Options for the use of rapid antigen tests for COVID-19 in the EU/EEA and the UK. Technical report. Available at [https://www.ecdc.europa.eu/sites/default/files/documents/Options-use-of-rapid-antigen-tests-for-COVID-19.pdf](https://www.ecdc.europa.eu/sites/default/files/documents/Options-use-of-rapid-antigen-tests-for-COVID-19.pdf). Updated November 19, 2020. Accessed December 29, 2020. 13. 13.WHO: Laboratory biosafety guidance related to coronavirus disease (COVID-19): interim guidance. Available at [https://apps.who.int/iris/handle/10665/332076](https://apps.who.int/iris/handle/10665/332076). Updated May 13, 2020. Accessed December 29, 2020. 14. 14.Mina MJ, Parker R, Larremore DB. Rethinking Covid-19 test sensitivity – a strategy for containment. N Engl J Med 2020; 383:e120 DOI: 10.1056/NEJMp2025631 [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMp2025631&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=32997903&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2021%2F01%2F04%2F2020.12.30.20249057.atom)