Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.

View ORCID ProfileEvan L Ray, Nutcha Wattanachit, View ORCID ProfileJarad Niemi, Abdul Hannan Kanji, Katie House, View ORCID ProfileEstee Y Cramer, View ORCID ProfileJohannes Bracher, Andrew Zheng, View ORCID ProfileTeresa K Yamana, Xinyue Xiong, Spencer Woody, Yuanjia Wang, View ORCID ProfileLily Wang, Robert L Walraven, Vishal Tomar, Katharine Sherratt, Daniel Sheldon, Robert C Reiner Jr, B. Aditya Prakash, View ORCID ProfileDave Osthus, Michael Lingzhi Li, Elizabeth C Lee, Ugur Koyluoglu, Pinar Keskinocak, Youyang Gu, Quanquan Gu, Glover E. George, Guido España, Sabrina Corsetti, View ORCID ProfileJagpreet Chhatwal, Sean Cavany, Hannah Biegel, Michal Ben-Nun, Jo Walker, Rachel Slayton, Velma Lopez, Matthew Biggerstaff, View ORCID ProfileMichael A Johansson, View ORCID ProfileNicholas G Reich on behalf of the COVID-19 Forecast Hub Consortium
doi: https://doi.org/10.1101/2020.08.19.20177493
Evan L Ray
1University of Massachusetts Amherst, School of Public Health and Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Evan L Ray
Nutcha Wattanachit
1University of Massachusetts Amherst, School of Public Health and Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jarad Niemi
2Iowa State University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jarad Niemi
Abdul Hannan Kanji
1University of Massachusetts Amherst, School of Public Health and Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Katie House
1University of Massachusetts Amherst, School of Public Health and Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Estee Y Cramer
1University of Massachusetts Amherst, School of Public Health and Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Estee Y Cramer
Johannes Bracher
3Heidelberg Institute for Theoretical Studies and Karlsruhe Institute of Technology
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Johannes Bracher
Andrew Zheng
4Operations Research Center, Massachusetts Institute of Technology
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Teresa K Yamana
5Columbia University, Mailman School of Public Health, Department of Environmental Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Teresa K Yamana
Xinyue Xiong
6Northeastern University, Network Science Institute
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Spencer Woody
7University of Texas at Austin, Department of Statistics and Data Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Yuanjia Wang
8Columbia University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Lily Wang
2Iowa State University
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Lily Wang
Robert L Walraven
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Vishal Tomar
9Auquan Ltd
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Katharine Sherratt
10London School of Hygiene and Tropical Medicine
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel Sheldon
11University of Massachusetts, College of Information and Computer Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Robert C Reiner Jr
12University of Washington, School of Medicine, Department of Health Metrics Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
B. Aditya Prakash
13Georgia Institute of Technology, College of Computing
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Dave Osthus
14Los Alamos National Laboratory, Statistical Sciences Group
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Dave Osthus
Michael Lingzhi Li
4Operations Research Center, Massachusetts Institute of Technology
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Elizabeth C Lee
15Johns Hopkins Bloomberg School of Public Health, Department of Epidemiology
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Ugur Koyluoglu
16Oliver Wyman
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Pinar Keskinocak
17Georgia Institute of Technology, H. Milton Stewart School of Industrial and Systems Engineering
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Youyang Gu
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Quanquan Gu
18University of California, Los Angeles, Department of Computer Science
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Glover E. George
19US Army Engineer Research and Development Center
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Guido España
20University of Notre Dame
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sabrina Corsetti
21University of Michigan, Physics Department
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jagpreet Chhatwal
22Massachusetts General Hospital, Harvard Medical School
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Jagpreet Chhatwal
Sean Cavany
20University of Notre Dame
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Hannah Biegel
23University of Arizona, Department of Mathematics
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michal Ben-Nun
24Predictive Science Inc.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jo Walker
25COVID-19 Modeling Team, U.S. Centers for Disease Control and Prevention
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Rachel Slayton
25COVID-19 Modeling Team, U.S. Centers for Disease Control and Prevention
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Velma Lopez
25COVID-19 Modeling Team, U.S. Centers for Disease Control and Prevention
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Matthew Biggerstaff
25COVID-19 Modeling Team, U.S. Centers for Disease Control and Prevention
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael A Johansson
25COVID-19 Modeling Team, U.S. Centers for Disease Control and Prevention
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Michael A Johansson
  • For correspondence: eyq9{at}cdc.gov nick{at}umass.edu
Nicholas G Reich
1University of Massachusetts Amherst, School of Public Health and Health Sciences
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Nicholas G Reich
  • For correspondence: eyq9{at}cdc.gov nick{at}umass.edu
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

Abstract

Background The COVID-19 pandemic has driven demand for forecasts to guide policy and planning. Previous research has suggested that combining forecasts from multiple models into a single “ensemble” forecast can increase the robustness of forecasts. Here we evaluate the real-time application of an open, collaborative ensemble to forecast deaths attributable to COVID-19 in the U.S.

Methods Beginning on April 13, 2020, we collected and combined one- to four-week ahead forecasts of cumulative deaths for U.S. jurisdictions in standardized, probabilistic formats to generate real-time, publicly available ensemble forecasts. We evaluated the point prediction accuracy and calibration of these forecasts compared to reported deaths.

Results Analysis of 2,512 ensemble forecasts made April 27 to July 20 with outcomes observed in the weeks ending May 23 through July 25, 2020 revealed precise short-term forecasts, with accuracy deteriorating at longer prediction horizons of up to four weeks. At all prediction horizons, the prediction intervals were well calibrated with 92-96% of observations falling within the rounded 95% prediction intervals.

Conclusions This analysis demonstrates that real-time, publicly available ensemble forecasts issued in April-July 2020 provided robust short-term predictions of reported COVID-19 deaths in the United States. With the ongoing need for forecasts of impacts and resource needs for the COVID-19 response, the results underscore the importance of combining multiple probabilistic models and assessing forecast skill at different prediction horizons. Careful development, assessment, and communication of ensemble forecasts can provide reliable insight to public health decision makers.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

ELR, NGR, EC and others were supported by the US Centers for Disease Control and Prevention (U01IP001122). NGR, NW, and others were supported by the National Institutes for General Medical Sciences (R35GM119582). JB was supported by the Helmholtz Foundation via the SIMCARD Information & Data Science Pilot Project. The content is solely the responsibility of the authors and does not necessarily represent the official views of CDC or NIGMS. BAP and others were partially supported by the National Science Foundation (Expeditions CCF-1918770, CAREER IIS-1750407, RAPID IIS-2027862, Medium IIS-1955883, NRT DGE-1545362), funds from Georgia Tech Research Institute (GTRI) and funds/computing resources from Georgia Tech. The content is solely the responsibility of the authors and does not necessarily represent the views of the funding agencies, Georgia Tech, GTRI, University of Iowa, UIUC or IQVIA. SC and others, as well as computing resources, were funded by the University of Michigan. The content is solely the responsibility of the authors and does not necessarily represent the views of the University of Michigan. PK and others were supported in part by the William W. George and the Virginia C. and Joseph C. Mello endowments at Georgia Tech, the NSF grant MRI 1828187, and research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at Georgia Tech. ECL and others were supported by the State of California, the U.S. Department of Health and Human Services and the U.S. Department of Homeland Security. This work was additionally supported by the Office of the Dean at the Johns Hopkins Bloomberg School of Public Health, the Johns Hopkins Health System, and with computing service credits from Amazon Web Services. These thoughts and opinions are our own and do not represent the views of the U.S. federal government. YW and others were supported by NIH grant GM124104. MBN and others were supported by the Defense Threat Reduction Agency (Award Number: HDTRA1-19-D-0007) and by the National Science Foundation (Award Number: 2031536). KS and others were supported by the Wellcome Trust [210758]. JC and others were supported by National Science Foundation awards 2035360 and 2035361. GE and others were supported by NSF RAPID award 2027718. RCR and others were supported by the Bill & Melinda Gates Foundation and National Science Foundation award 2031096. GEG and others were supported by funding provided by the US Army Corps of Engineers Geospatial Task Force.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

UMass-Amherst IRB.

All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

All data and code referred to in the manuscript are publicly available.

https://github.com/reichlab/covid19-forecast-hub/

https://github.com/reichlab/covidEnsembles

https://zoltardata.com/project/44

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available for use under a CC0 license.
Back to top
PreviousNext
Posted August 22, 2020.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.
Evan L Ray, Nutcha Wattanachit, Jarad Niemi, Abdul Hannan Kanji, Katie House, Estee Y Cramer, Johannes Bracher, Andrew Zheng, Teresa K Yamana, Xinyue Xiong, Spencer Woody, Yuanjia Wang, Lily Wang, Robert L Walraven, Vishal Tomar, Katharine Sherratt, Daniel Sheldon, Robert C Reiner Jr, B. Aditya Prakash, Dave Osthus, Michael Lingzhi Li, Elizabeth C Lee, Ugur Koyluoglu, Pinar Keskinocak, Youyang Gu, Quanquan Gu, Glover E. George, Guido España, Sabrina Corsetti, Jagpreet Chhatwal, Sean Cavany, Hannah Biegel, Michal Ben-Nun, Jo Walker, Rachel Slayton, Velma Lopez, Matthew Biggerstaff, Michael A Johansson, Nicholas G Reich
medRxiv 2020.08.19.20177493; doi: https://doi.org/10.1101/2020.08.19.20177493
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Ensemble Forecasts of Coronavirus Disease 2019 (COVID-19) in the U.S.
Evan L Ray, Nutcha Wattanachit, Jarad Niemi, Abdul Hannan Kanji, Katie House, Estee Y Cramer, Johannes Bracher, Andrew Zheng, Teresa K Yamana, Xinyue Xiong, Spencer Woody, Yuanjia Wang, Lily Wang, Robert L Walraven, Vishal Tomar, Katharine Sherratt, Daniel Sheldon, Robert C Reiner Jr, B. Aditya Prakash, Dave Osthus, Michael Lingzhi Li, Elizabeth C Lee, Ugur Koyluoglu, Pinar Keskinocak, Youyang Gu, Quanquan Gu, Glover E. George, Guido España, Sabrina Corsetti, Jagpreet Chhatwal, Sean Cavany, Hannah Biegel, Michal Ben-Nun, Jo Walker, Rachel Slayton, Velma Lopez, Matthew Biggerstaff, Michael A Johansson, Nicholas G Reich
medRxiv 2020.08.19.20177493; doi: https://doi.org/10.1101/2020.08.19.20177493

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Epidemiology
Subject Areas
All Articles
  • Addiction Medicine (349)
  • Allergy and Immunology (668)
  • Allergy and Immunology (668)
  • Anesthesia (181)
  • Cardiovascular Medicine (2648)
  • Dentistry and Oral Medicine (316)
  • Dermatology (223)
  • Emergency Medicine (399)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (942)
  • Epidemiology (12228)
  • Forensic Medicine (10)
  • Gastroenterology (759)
  • Genetic and Genomic Medicine (4103)
  • Geriatric Medicine (387)
  • Health Economics (680)
  • Health Informatics (2657)
  • Health Policy (1005)
  • Health Systems and Quality Improvement (985)
  • Hematology (363)
  • HIV/AIDS (851)
  • Infectious Diseases (except HIV/AIDS) (13695)
  • Intensive Care and Critical Care Medicine (797)
  • Medical Education (399)
  • Medical Ethics (109)
  • Nephrology (436)
  • Neurology (3882)
  • Nursing (209)
  • Nutrition (577)
  • Obstetrics and Gynecology (739)
  • Occupational and Environmental Health (695)
  • Oncology (2030)
  • Ophthalmology (585)
  • Orthopedics (240)
  • Otolaryngology (306)
  • Pain Medicine (250)
  • Palliative Medicine (75)
  • Pathology (473)
  • Pediatrics (1115)
  • Pharmacology and Therapeutics (466)
  • Primary Care Research (452)
  • Psychiatry and Clinical Psychology (3432)
  • Public and Global Health (6527)
  • Radiology and Imaging (1403)
  • Rehabilitation Medicine and Physical Therapy (814)
  • Respiratory Medicine (871)
  • Rheumatology (409)
  • Sexual and Reproductive Health (410)
  • Sports Medicine (342)
  • Surgery (448)
  • Toxicology (53)
  • Transplantation (185)
  • Urology (165)