App-based symptom tracking to optimize SARS-CoV-2 testing strategy using machine learning

Introduction

The current COVID-19 pandemic caused by the SARS-CoV-2 requires extensive testing programs to understand the transmission, diagnose, and isolate the positive cases. Given the high mortality and absence of a specific consensual treatment or a reliable vaccine, large testing programs are an essential part of epidemic control. The frequency of testing, however, is very heterogeneous among countries. Brazil currently has the second-highest number of COVID-19 cases, even with lower test rates (59,252 tests per one million inhabitants, as of July 27, 2020)[1].

In a recent paper, Menni and colleagues[2] used information from an app-based symptom tracker from UK and USA and concluded that the combination of symptoms could be used as a screening tool to identify people with a possible positive result for COVID-19. However, little is known about this association and their potential usage as a screening tool in low- and medium-income countries (LMIC) such as Brazil. Thus, our study aims to use the combination of symptoms and machine learning techniques to develop a predictive model that identifies people and areas with a higher risk of SARS-CoV-2 infection. We used data from an app-based symptom tracker known as “Dados do Bem”[3], which is an initiative that started available for the state of Rio de Janeiro, one of the centers of the outbreak in the country. Applying our model, we can estimate the proportion of infected participants and then categorize risk levels of infection within the geographical area of the state of Rio de Janeiro, aiming to prioritize tests and optimize the testing program.

Materials and Methods

Statistical analysis

We described the characteristics and symptoms of positive and negative tested participants, displaying the mean and standard deviation for continuous variables and the frequency for categorical variables. We then analyzed the individual association between symptoms and the test result using a logistic regression model adjusted to age and gender. We provided the corresponding Odds Ratio (OR) with a 95% confidence interval.

We aim to identify a combination of symptoms to build a prediction model for determining a participant with SARS-CoV-2 infection. For that, we compared five different machine learning techniques: Logistic Regression (stepwise), Naïve Bayes, Random Forest, Decision Tree (C 5.0), and eXtreme Gradient Boosting. To address the imbalanced response variable (only 11.8% are positive tests) during model training, we also evaluated four different data balancing techniques (downsampling, upsampling, SMOTE, and ROSE).

The dataset was randomly divided into training and test sets (ratio: 80:20). During model training, for each combination of machine learning and data balancing techniques, we applied grid-search hyperparameter optimization with 5-fold cross-validation, using the Area Under the ROC Curve (AUC) as the target metric. We comparatively evaluated the models’ performance in the test set and selected the one with the highest Matthews Correlation Coefficient (MCC)[5] value. The cut-off point for predicted values was 50% (i.e., participants with a probability higher than 50% were classified as “positive”, and “negative” otherwise). After our final model was incorporated by the “Dados do Bem” app, we performed an external validation using data from users in the state of Goias.

Finally, we evaluated the distribution of SARS-CoV-2 infection risks over the geographic area of the state of Rio de Janeiro modeled as a grid map (each grid is a 400m x 400m square area). Along with the participants with confirmed test results, we applied the chosen model to the sample of participants that had not been tested to obtain their estimated test result. We then calculated the proportion of estimated SARS-CoV-2 infections for each grid according to Equation 1.

To avoid misinterpreting proportions in grids with scarce data, we considered grids with at least 10 participants (~94% of all observations). Then, we evaluated the distribution of the grid risks among all grids and classified them into five risk groups using the mean ± 0.5 and 1.5 standard deviations (SD) as thresholds: “very low” (< mean-1.5*SD), “low” (from mean-1.5*SD to mean-0.5*SD), “medium” (from mean-0.5*SD to mean+0.5*SD), “high” (from mean+0.5*SD to mean+1.5*SD), and “very high risk” (>mean+1.5*SD). Using this classification, we built a risk map for the state of Rio de Janeiro.

All analyses were performed in R 3.6.3, using ‘tidyverse’ package for data wrangling and plots; and ‘caret’ for the prediction models, with ‘ranger’ for Random Forest, ‘C50’ for Decision Trees, ‘xgbTree’ for the eXtreme Gradient Boosting, and ‘naivebayes’ for the Naïve Bayes model.

Results

Characteristics and self-reported symptoms associated with SARS-CoV-2 infection

From April 28, 2020, to July 16, 2020, 337,435 individuals registered their symptoms through the smartphone app. Of these, 49,721 users were already tested, from which 5,888 (11.8%) received a positive result for SARS-CoV-2 infection.

According to the self-reported information (Table 1), most participants were women (61.9%), health professionals (55.8%), with a median age of 41 years old (IQR: 33-51). Among those who tested positive for SARS-CoV-2 infection, cough was the most frequent symptom (59.6%), followed by myalgia (57.4%), coryza (56.3%), loss of smell/anosmia (52.9%), and fever (44.8%). When evaluating the association between each symptom and the test result, adjusted for age and gender (Fig 1), we found a similar result: loss of smell (odds ratio [OR]: 4.6; 95% CI: 4.4 - 4.9), fever (OR: 2.6; 95% CI: 2.5 - 2.8), and shortness of breath (OR: 2.1; 95% CI: 1.6 - 2.7) were associated with a positive result for SARS-CoV-2 infection.

View this table:

• View inline
• View popup
• Download powerpoint

Table 1: Characteristics and symptoms of the study population tested for SARS-CoV-2 infection

Results are displayed in median (interquartile range, IQR) for continuous variables and as percentage values for categorical variables.

• Download figure
• Open in new tab

Fig 1. Association between symptoms and the SARS-CoV-2 infection

The Odds Ratio (OR) with 95% confidence intervals for each symptom using a logistic regression model adjusted to age and gender.

Combination of symptoms and predictive modeling

Aiming to develop a model to predict positive participants based on the available dataset, we ran 25 different scenarios, combining machine learning techniques and sampling strategies. The logistic regression, gradient boosting and random forest techniques presented the best median MCCs (0.25), followed by the decision tree, and naïve Bayes (see the Supporting Information S2). Since the logistic regression model is more interpretable and parsimonious, we chose it as the final model (Equation 2).

Regarding the classification metrics, our model performed as follows: sensitivity of 0.60; specificity of 0.75; PPV of 0.25; and NPV of 0.93. Thus, evaluating the predicted probability values, we observed that our model correctly predicts almost all negative tests with only 7% of false-negative users among the predicted as negatives (NPV = 0.93). We focus on avoiding false negatives (thus maximizing NPV) since we aim to prioritize tests in people with a potential positive result for COVID-19.

The characteristics of the false-negative and false-positive cases predicted by our model can be seen in the Supporting Information S3. We observed that most of the false-positive cases present the top-four predictors with the highest positive coefficients. At the same time, only four of the false negatives reported the loss of smell - the strongest predictor of a positive test.

SARS-CoV-2 risk areas in Rio de Janeiro

Aiming to optimize the testing strategy, we applied the predictive model (Equation 1) to the 287,714 individuals registered in the app and still untested for SARS-CoV-2. According to our model, 99,431 (34,5%) of these participants were classified as a potential positive infection. We also visualized the geographical distribution of SARS-CoV-2 infection, calculating by Equation 2 the proportion of positive test results for each specific geographical area (named a grid) in the state of Rio de Janeiro. We calculated the predicted infection risk for each grid (Fig 2) and their risk groups. As of July 16, 2020, we observed that the southern areas in the city of Rio de Janeiro presented lower proportions of potential positive participants compared to the northern areas.

• Download figure
• Open in new tab

Fig 2. SARS-CoV-2 infection risk map (grid) of Rio de Janeiro state, displaying the city of Rio de Janeiro (capital, left) and the city of Niteroi (right)

The grid risk considered the proportion of potential positive infection (observed test results + estimated from the prediction model) for each grid (400mx400m area). The risk groups were obtained as very low (<17%), low (≥17% and <33%), medium (≥33% and <48%), high (≥48 and <63%), and very high (≥63%). The map shows the distribution of risks as of June 10, 2020.

Discussion

Extensive testing programs for SARS-CoV-2 are, in general, not available in low- and middle-income countries, conferring the under-reporting of confirmed cases into a problem. A previous study estimated that only 9,2% of the Brazilian cases are being notified [6]. Restricting tests hinders the monitoring of the epidemic progression, resource planning, and evaluation of the effectiveness of the control measures. Besides, it leads to false conclusions that the disease is under control.

Since it is not possible to test all individuals, some studies suggest that the combination of symptoms could be used as a screening tool to identify people with potential SARS-CoV-2 infection who could be selected for testing [2,7]. This can be useful for planning public policies and for preventing the spread of the pandemic. That said, our study used data on individual symptoms and demographics obtained from an app-based system, known as “Dados do Bem”, to develop a model that predicts individuals with a higher probability of testing positive for SARS-CoV-2 infection.

Some works criticize the use of symptom-based screening strategies to quantify an individual likelihood of having COVID-19, due to the non-specific nature of some symptoms and the existence of co-infections with other respiratory viruses [8]. However, our results evidenced that such a strategy contributes to optimizing the overall testing strategy. Out of the 287,714 new users still not tested, our model estimated that the virus could infect 99,431 who, therefore, should be prioritized for testing. It reduced the need for extensive testing to only 34.5% of the registered untested users. This is undoubtedly beneficial as a public policy, especially in Brazil, a country with the second-highest number of COVID-19 cases and one of the lowest test rates¹.

Our model was incorporated into the app and used to select patients for testing in the state of Goias. Of the first 465 users selected, 52% were tested positive. This positivity rate is extremely high compared to the observed positivity rate without a model (11.8%). It indicates that our model presented an excellent performance, improving the test strategy and selecting the users most likely to be positive in the actual scenario.

In addition to forecasting the likelihood of each user to acquire the virus, our model also assesses the geographical distribution of these participants, being a source of information to build a risk map for the state of Rio de Janeiro, as shown in Fig 2. The “Dados do Bem” app currently uses this map for categorizing risk areas, thus supporting decision-makers to identify areas with a higher risk of infection and accordingly refine testing and disease control strategies.

The risk map analysis developed in this work is exemplified in Fig 3, which presents the risk map of the south zone of the city of Rio de Janeiro. The chosen area includes both high-income neighborhoods (such as “Ipanema,” “Leblon,” and “Gávea”) and slums (such as “Rocinha”). The selected grid in Ipanema is classified as “low risk,” which means that the proportion of positive tests in this grid was between 17% and 33%. The other selected grid is in Rocinha, which, although located less than three miles from Ipanema, is classified as a “very high risk” grid, meaning that the proportion of positive tests living in this grid was higher than 63%. The existence of higher-risk areas in poor communities was also noted in other regions of the state (Fig 2). Many higher-risk grids were located in the north zone of the city, where the most deprived communities are located (“Complexo do Alemão” and “Complexo do São Carlos,” for example). The presence of social inequalities in Brazil has been pointed out by previous studies [9–11], which noted that it could be associated with the spreading of the disease.

• Download figure
• Open in new tab

Fig 3. Risk map of two neighborhoods of Rio de Janeiro (Rocinha – very high risk and Ipanema – low risk)

Regarding our results of the reported symptoms, loss of smell (anosmia) was the strongest indicator of SARS-CoV-2, followed by fever, shortness of breath, myalgia, cough, nausea, diarrhea, and coryza. The significant influence of loss of smell and cough is in line with previous studies carried out in high-income countries such as the US and UK [2,12–14], and the influence of fever, myalgia, and nausea was pointed as significant in some studies [13,14], but others noted as not associated [2,12]. Regarding sore throat, diarrhea, and shortness of breath, previous studies observed that they are not significant predictors for the SARS-CoV-2 infection [2,12–14].

Menni and colleagues[2,7] used real-time tracking of self-reported symptoms similar to ours to predict potential SARS-CoV-2 infection in a cohort of individuals from the US and UK. This model was applied to compare the incidence in the UK regions. The authors noted that, in southern Wales, users reported symptoms that predicted, 5 to 7 days in advance, two spikes in the number of confirmed positive SARS-CoV-2 infection reported by public health authorities. The prediction models presented NPV of 0.75 and 0.87 in the UK (15,638 participants) and in the US (2,763 participants), respectively. Compared to them, our best model obtained a competitive performance (NPV of 0.93).

Sebo and colleagues [14] studied a sample of 1,543 primary care patients tested in two laboratories in the Lyon area (France) and found that the two symptoms most strongly associated with a positive test were loss of taste (ageusia) and loss of smell and that the combination of these symptoms resulted in an even stronger association (i.e., the odds of having a positive test were six times greater than the odds of having a negative test). A recent literature review of studies analyzing the presence of loss of taste and smell in SARS-CoV-2 infected patients concluded that, from a total of 10,818 patients, 8,823 presented ageusia (81.6%) and 8,088 presented anosmia (74.8%) [15]. Our results reinforce the literature conclusions about the strong influence of loss of smell.

This study presents some limitations. First, the symptoms are self-reported. Hence, the participant may report apparent manifestations of the disease, which may not be precise as physiological evaluation by a physician. Second, we were unable to know when a symptom appeared to indicate the stage of the disease at the testing moment. Third, a non-negligible number of false negatives may be present, considering the sensitivity of the serological test. However, identification of potential clusters and optimization of testing resources using a combination of self-reported symptoms is a viable strategy for many countries. A similar combination of symptoms can explain the SARS-CoV-2 infections in developed countries, such as the United Kingdom and the United States, as well as in LMIC such as Brazil.

Conclusions

Our work used data regarding individual symptoms and demographics obtained from an app-based system to predict individuals with a higher probability of being infected by SARS-CoV-2. The model was applied in practice to select users from the state of Goias for testing, resulting in a positivity rate that was 4.4 more accurate than the positivity rate observed without using the model. In addition, the developed model was applied to new users of Rio de Janeiro to determine the risk of being infected in different geographical areas of this state. Our model estimated that 35.6% are potentially infected, reducing the need for extensive testing and optimizing the overall testing strategy. Based on the risk map derived from the model, the decision-makers may locate regions with a higher risk of positive tests, therefore allowing better testing and disease control policies.

Supporting information

S1 Fig. Pages of the COVID Symptom Tracker app.

S2 Table. Performance of the ML models computed from the independent test set for each combination.

S2 Fig. Box-plots are representing the MCCs of each strategy (points) for all methods.

S3 Table. Characteristics and symptoms of false-negative and false-positive users predicted from our model.

S3 Fig. Probability density function and frequency of the predicted values by the model using a testing set, compared to the real (observed) values. The black vertical line corresponds to the cut-off of 0.5, and the colored dashed vertical lines correspond to the expected average probability for the group of negative (red) and positive (blue) groups.