On Machine Learning-Based Short-Term Adjustment of Epidemiological Projections of COVID-19 in US

INTRODUCTION

The novel Coronavirus, SARS-CoV-2, was first detected in Wuhan, China on December 31, 2019 and by early January 2020, it spread to 21 countries. The first case in the United States (US) was reported on January 21, 2020 in Snohomish County in the state of Washington [1]. By the middle of March 2020, the number of COVID-19 infected cases started to peak up in the US [2], inducing panic about potential shortages of hospital capacities and supplies.

As more cases were detected in the US, many epidemiological models, which had been mainly developed for other pandemics such as influenza and season-long forecasting, started to adjust to and project COVID-19 growth patterns. Due to the critical role of forecasting COVID-19 in helping policymaker for future planning, the Center of Disease Control and Prevention (CDC) has initiated an effort for forecasting in form of challenge with the participation of several of the epidemiological models [3].

Although overall epidemiological models have been useful for understanding the future outlook of COVID-19 pandemic, they often have been criticized for overestimating the projections and inducing uncertainties [4]. The limitations in predicting future cases of COVID-19 stems from a combination of limited understanding and a lack of data about its infectious spread pattern due to its novel nature. Beside data limitations, region-specific factors were often not accounted-for in many published epidemiological models of the COVID-19 pandemic. Factors such as age distribution, comorbidities, and pre-excising conditions, as well as socioeconomic factors, are expected to play role in the severity and duration of this disease.

Given the importance of COVID-19 forecasting and considering the majority of epidemiological models are best suited for long-term projections, in this work rather than developing a new forecasting model, we aim at improving the overestimated projected numbers from epidemiological models on a short-term basis based on machine learning. In particular, we conducted a systematic analysis to study the importance of various data elements for our short-term prediction. Our predictive modeling included a carefully selected 40 geo-specific features for the US counties.

A previous study by Liu et al. on machine learning-based predictions of COVID-19 outbreak in China, showed that modeling for the regions clustered together based on geo-specific similarities resulted in improved predictive performance for majority of China provinces [5]. In this work, we studied four different clustering strategies, primarily based on demographic similarities, COVID activity trends, state boundaries, as well as a national cluster, to examine the effects of these boundary conditions in predictive power of our model.

Our main contribution is investigating the role of machine learning to adjust the short-term epidemiological projections with the ultimate aim of helping counties and hospitals better plan their resources. We are also investigating which additional region-specific features play a role in short-term adjustments of the COVID-19 projections.

1 RESULTS

For historical epidemiological projection, we used the CovidActNow model as one of the early epidemiological models that became opensource [6]. For the data augmentation purpose, we considered four clustering strategies: one national cluster including all the counties in our training set, second clustering based on counties state boundary, and another two clustering strategies were based on counties similarities in COVID-19 spread characteristics and demographic information as described in the Methods section.

The epidemiological historical projections were one-day ahead on any given day during March 23^rd-April 20^th, 2020 period. We trained an individual date-dependent models for each of the clusters within each clustering strategy. The out-of-sample test sets were evaluated from April 6^th to April 19^th, 2020 individually each with the date-appropriate trained model (the total of 14 date-dependent models for each cluster). The comparison and the impact of the four different clustering boundary conditions is shown in Table 1.

Regardless of the clustering strategy, our one-day-ahead prediction showed ~97% uplift compared to the epidemic projection with the same one-day-ahead horizon. Overall, the performance of the models based on each clustering strategy was similar.

Also consistent across all four different clustering strategies, the best model performance was achieved for the cluster in which New York County was included. Figure 1 shows the comparison of our model output to the CovidActNow projected “all infected” cases both compared with the ground truth (i.e., estimated all infected) for three example counties which clustered together with New York County for two different clustering strategies. Both COVID-based and demographic-based clustering achieved 97% uplift compared to the epidemiological projections for the cluster model that included New York County.

• Download figure
• Open in new tab

Figure 1: One-day-horizon predictions from our machine learning model and from the CovidActNow epidemiological model both compared to the ground truth for two different clustering strategies: a) one of the 12 clusters of the COVID-based clustering with 49 counties in the cluster, and b) one of the 6 clusters of the demographic-based clustering with 259 counties in the cluster.

Closer assessment of two major counties of New York and Los Angeles further revealed the similarities of four different clustering strategies, particularly for New York County as shown in Figure 2. The best performing cluster model for New York County was found to be state-based clustering with R²= 0.90. Los Angeles County was obtained from the COVID-based clustering method, which follows similar trajectory as the national clustering with comparable RMSE (676.34 and 695.61, respectively). Stateclustered model showed highest deviation with RMSE = 1249.78, followed by second highest RMSE = 723.75 from the demographicbased clustering model. We found that the level of COVID-19 inactivity or low activity in the cluster impacted the performance for the corresponding model. For example, comparing two of the clusters containing Los Angeles County from two different clustering strategies, the state-based cluster contained 36% zero cases as opposed to only 2% zero cases in the COVID-based cluster.

• Download figure
• Open in new tab

Figure 2: One-day-horizon predictions of our models trained individually according to the clustering strategy for two of the major counties.

As LASSO gives spare weights by driving small weights to zero, we can detect the most predictive features of our model. As walkforward split expands by every date and more training data gets included in the split, fewer features become prominent. For the national model, as shown in Figure 3, the most impactful features for the last trained model were population density, and the autoregressive features of new COVID cases from three previous date of the prediction.

• Download figure
• Open in new tab

Figure 3: LASSO selected features for the first (top panel) and last (bottom panel) trained models (out of 14 date-dependents models) trained for all of the counties included in our model (total of 1578 counties). Although majority are small values, none of the features in the first trained model have zero coefficients. The last model has 36 features with coefficients value of zero displayed for better comparison.

In the first trained model (with the least training data), in addition to population density and autoregressive features, workers in commuting flow and cumulative number of cases had positive predictive effects on predicting the next day’s infected cases. In contrast, population and next day projected cases from the CovidActNow output had negative predictive power. We speculate the negative predictive power of CovidActNow is due to its increasing trend as opposed to ground truth that has downward trend generally. It is also important to note for the first trained model (first expanding window iteration) with limited training data, LASSO attempts to distribute the weights to more features. As the training data expands in the last trained window, the effect of CovidActNow projections is eliminated. Other features had small weights (either negative or positive) and eventually weighted to zero for the last trained model. Similarly, we found that population density and three autoregressive features were the most important (non-zero and positive) features for the majority including best performing clusters, for other clustering strategies. For the demographic-based clustering due to homogeneous distribution of the, regardless of the date of trained model, none of the demographic features carried any weight with the exception of the population density and workers in commuting flow. Overall, no particular pattern was detected for the role of race and sex with few exceptions. For instance, the proportion of non-Hispanic black population showed positive predictive impact for the State of Wisconsin with the state-based clustering approach. For feature importance map of other three clustering strategy please refer to APPENDIX II.

2 Methods

For the purpose of data augmentation, we considered grouping counties four different ways to ensure the that the scarcity of available training data during early period of COVID-19 pandemic (window considered March 23^rd-April 20^th, 2020) did not impact the model performance. One grouping included counties of each of the US states, yielding 51 individual trained models. Another national approach involved grouping all the counties together and training one model for all. The two other groupings were conducted by clustering the counties based on their similarities in COVID-19 spread characteristics and demographic information as described below.

2.3 Predictive Model

For the predictive modeling, we fitted our data to a LASSO model, a multivariate linear model with L1-norm regularization (penalizing the absolute sum of the model coefficients). The feature vector included both time-dependent and static features for each county. The time-dependent were COVID-19 dependent features both from the epidemiological model output and from JHU reported parameters. The time-independent features were demographic and characteristics of each county. In total, static and time-varying, 40 features were included in our model. For the list of the features and their sources please refer to APPENDIX III.

The number of new confirmed COVID-19 cases for the next day was then predicted based on:

Where y_t+1, y_t, y_t−1, and y_t-2 are the new confirmed cases for the next day, same date, the day before, and two days prior of date t. d_t refers to the new death number at date t; i_t+1 is the new “all infected” cases projected for next day at date t from the epidemiological model, and C_t is the cumulative number of COVID-19 cases on date t. S_j refers to collection of 34 static features including, population density, county commute flow, age, gender, race, socioeconomic features such as unemployment and Area Deprivation Index, as well as disease prevalence and comorbidities.

Due to time-series nature and daily update of the data, a walk-forward with expanding window validation fashion was considered resulting in 15 splits with one day horizon and initial training window of 10 days. Since the split was performed according to the date of data, each cluster yielded 15 sets consistently; however, depending on number of counties in each cluster, different train-test splits were expected for each cluster. The total of 1578 counties were included in this study each with feature vector of size 40. To compare the predicted outcome of our model with the epidemiological projected cases, the predicted new confirmed cases were then summed with the JHU-reported “cumulative number of cases” from the previous date, and then converted to “estimated all infected” according to the conversion used in the CovidActNow model.

The best model parameters, including LASSO alpha, were selected in a Grid search manner. To simulate the real-life scenario in which the last trained model needs to be updated daily as the new data comes in, the model trained on the first n-1 splits (n=2, 3, 4, … 15), was evaluated on the test set of nth split (i.e. out-of-sample test set).

3 CONCLUSIONS

Our machine learning results showed significant improvement over the epidemiological projection with a one-day prediction horizon. Although small, the changes in model performance was detectable based on different clustering strategies. Assessing New York County, for example, we found state-based clustering achieved the highest performance for the state of New York (R² = 0.90). However, state-based clustering resulted in lowest performance for overall counties compared to other strategies (Table 1). This finding suggests an advantage to considering training individual models based on the geographic region.

The results shown for our model are from March-April 2020 period of time, a time of dramatic increases in COVID-19 spread in major cities with subsequent implementation of mitigation strategies, such as stay-at-home policies with California first to enforce such a policy on March 19^th, 2020. Our findings of different clustering strategies and feature importance were likely due to the highly dynamic nature of infectious spread and local policies at this time. Moreover, as more data has become available over time, epidemiological models are also improving and more scenarios are considered in the modeling methods, such as the impact of asymptomatic cases in the recent version of the ActNow model.

It is also important to note that for this work, our prediction is short and limited to one-day horizon. Although, it is not expected that all of the 40 features that we have included have predictive power, we suspect with longer prediction horizon, demographic and geo-specific features will have important roles.

With initially flattening and now new rises in the pandemic curve, we plan to expand our modeling prediction horizon to as long as two weeks ahead. Some of the challenges that we foresee with longer prediction horizon is the policy changes as the states would go through different phases of reopening. We suspect for longer prediction horizon, more COVID-specific trends for each state as well as information about the upcoming policies can improve the predictive powers. Also, we intend to incorporate several epidemiological models’ outputs to study their difference and impact in our modeling.