Infoveillance study on the dynamic associations between CDC social media contents and epidemic measures during COVID-19

Data Acquisition and Preprocessing

Using the Twitter academic API and search query (see search query in Appendix), we retrieved a total of 17,524 English tweets posted by seven official CDC-affiliated Twitter accounts to January 15, 2022. These accounts are: @CDCgov: CDC’s official Twitter source for daily credible health and safety updates from Centers for Disease Control & Prevention; @CDCEmergency: CDC Emergency, which tweets ways to for public health preparedness during emergency responses; @CDCDirector: account of the CDC director of the time; @CDCGlobal: CDC Global health, which tweets about how CDC strives to contribute to saving lives, reducing disease, and improving global health around the world; @CDCtravel: CDC Travel health, which voices to help travelers and their clinicians prevent illness and injury during international travel; @DrKhabbazCDC: past Twitter account from former director of CDC emerging infections (NCEZID), Dr. Rima Khabbaz. NCEZID works to protect people from emerging and zoonotic infectious diseases, from anthrax to Zika; @CDCMMWR: MMWR (Morbidity and Mortality Weekly Reports), which is CDC’s primary vehicle for scientific publication of timely, authoritative, and useful public health information and recommendations. We consolidated these tweets into a data frame of daily tweet counts of each of the seven CDC- affiliated accounts during the time period. The associated metadata include tweet posting dates, textual data of the tweets, tweet account ID, account types (organization or individual), public engagement metrics (e.g., number of retweets, replies, likes, and quotes), referenced tweet type (retweeted, replied to, and quoted), and referenced tweet IDs.

We also acquired the COVID-19 epidemic metric data in the U.S. from Johns Hopkins University – Center for Systems Science and Engineering (CSSE) public GitHub repository ([13]–[15]). Four sets of important COVID-19 time series data were retrieved, including daily cumulative confirmed cases, deaths, testing, and vaccination. The data were all at U.S. nation- level. The original four sets of COVID-19 time series data consist of dates and their cumulative targeted measurements. Case series includes daily cumulative number of confirmed COVID-19 reported cases, and it has 751 records, ranging from January 22, 2020 to February 10, 2022.

Death series reports daily cumulative number of confirmed COVID-19 death cases, and it has 908 records, ranging from January 22, 2020 to July 17, 2022. Testing data reports daily cumulative number of completed PCR tests or other approved nucleic acid amplification test (NAAT), and it has 760 records, ranging from January 13, 2020 to February 10, 2022. Vaccination data includes daily cumulative number of people who received a complete primary series of vaccine doses, from the CDC Vaccina Tracker; vaccination series has 428 records, ranging from December 10, 2020 to February 10, 2022.

For consistency in analysis, all CDC tweets time series and U.S. COVID-19 variable time series were standardized to the same time span in this study, ranging from the start-date of reported cases data, January 22, 2020 (the start-date of reported cases data), to the end-date of CDC tweet collection on January 15, 2022, with total of 725 records for each data. Since vaccination data were not available until late 2020, missing values in the vaccination series were filled with 0s. In summary, we have four time series from four different COVID-19 U.S. epidemic metrics, and another social media time series of tweets from all seven CDC-associated Twitter accounts.

Natural Language Processing (NLP)

In order to identify major topics in CDC’s COVID-19 tweets, we performed various natural language preprocessing (NLP) analyses. NLP, especially topic modeling, provides granular characterization of textual inputs such as CDC’s COVID-19 communications.

Regular expressions were first applied to process tweet texts by removing @mentions, hashtags, special characters, emails, punctuations, URLs, and hyperlinks. Tokenization was performed to break down sentences into individual tokens, which can be individual words or punctuations.

For example, the sentence “As COVID19 continues to spread, we must remain vigilant.” becomes tokens of “As”, “COVID19”, “continues”, “to”, “spread”, “,”, “we”, “must”, “remain”, “vigilant” after tokenization. Next, lemmatization, a structural transformation where each word or token was turned to its base or dictionary form of their morphological information, was performed. For example, for words “studies” and “studying”, their base form, or lemma, was the same “study”. In addition to stop words removal via Python NLTK library, we also created our own list of stop words and removed them from the texts (see stopwords list in Appendix). With help from domain experts, we excluded top words that did not contribute to topic mapping.

N-grams, phrases with n words, were developed with a threshold value of 1 to form phrases from tweets. Phrase-level n-grams were applied here because phrases offer more semantic information than individual words [16]. Higher threshold value resulted in fewer phrases to be formed. The texts were mapped into a dictionary of word representations, which was a list of unique words, and was then used to create bag-of-words presentations of the texts. A TF-IDF (term frequency- inverse document frequency) model was implemented to evaluate the importance and relevancy of the words to a document. It was calculated by multiplying term frequency, which was the relative frequency a word within a document, with the inverse document frequency, which measures how common or rare a word is across a corpus. Higher TF-IDF values indicates that the word is more relevant to the document it is in ([17], [18]). Words that were missing and lower than the threshold value 0.005 from the TF-IDF model were excluded. Figure 1 shows the process of data collection and pre-processing, and Figure 2 shows the steps of subsequent NLP and statistical analyses.

• Download figure
• Open in new tab

Figure 1.

Interactive mapping of topic 4 generated by LDA

• Download figure
• Open in new tab

• Download figure
• Open in new tab

Figure 2.

Time series of topic 2 against two COVID-19 metrics: a) confirm case counts, b) death counts

Topic Modeling with Latent Dirichlet Allocation (LDA)

To identify more specific topics from all the COVID-19 tweets posted by CDC, we performed topic modeling via latent Dirichlet allocation (LDA). LDA automatically generates non- overlapping clusters of distributions of words that represent different topics based on probabilistic distributions. LDA was developed to find latent, or hidden topics from a collection of unstructured documents, or corpus with text data. Topic models are probabilistic models that performs on three levels of documents: a word, a document, and a corpus. Details of LDA and topic models are provided in supplementary information (SI). We investigated and compared across 3 to 8 potential topics, and determined the optimal number of topics based on both topic model evaluation and domain expert interpretations of the identified topic clusters.

Model perplexity and topic coherence scores were calculated as performance metrics of LDA. Perplexity is a decreasing “held-out log-likelihood” function that assesses LDA performance using a set of training documents. The trained LDA model is then used to test documents (held- out set). The perplexity of a probability model q on how well it predicts a set of samples x₁, x₂, …, x_N drawn from an unknown probability distribution p, is defined as:

An ideal q should have high probabilities q(x_i) for the new data. Perplexity decreases as the likelihood of the words in new data increases. Therefore, lower perplexity indicates better predictability of an LDA model.

Topic coherence assesses the quality of the topics, measured as the understandability and semantic similarities between high scoring words (i.e., the words that have high probability of occurring within a particular topic) in topics generated by LDA [20]. We used UMass coherence score [21], which accounts for the order of a word appears among the top words in a topic. It is defined as:

where N = the number of top words of a topic of a sliding window, P(w_i) is the probability of the i-th word w appearing in the sliding window that moves over a corpus to form documents, P(w_i, w_j) is the probability of words w_i and w_j appear together in the sliding window. According to the study from UMass, coherence decreases initially and becomes stationary as the number of topics increases [20].

Human domain experts are involved to further interpret the generated topics. Representations of all topics were presented in word-probability pairs for the most relevant words generated by the topics. Interactive visualizations were produced using pyLDAvis package in Python 3.7 to examine the topics generated by LDA and their respective associated keywords. A data frame of all dominant, key topics was created. The original unprocessed full texts of the CDC tweets, IDs, and posting date were combined into a data frame along with their corresponding key topic number label and topic keywords. In addition, daily percentage of each topic from LDA were also calculated for further time series analysis. For instance, vaccine/vaccination is an identified key topic in CDC’s COVID-19 tweets, so the percentage of vaccine-related tweets in each day was calculated for the entire study period to construct the vaccine/vaccination specific topic time series. Since LDA is technically an unsupervised clustering method, we further labeled and interpreted the topics using domain knowledge. Four sets, each containing 20 tweet samples from each identified topic were randomly selected. Domain experts then examined and discussed the sampled tweets from each topic and labeled the topics (e.g., vaccine/vaccination). The final agreement on the interpretation of LDA-generated topics were reached after multiple iterations and discussions of the above process.

Multivariate Time Series Analyses between Identified CDC Tweet Topics and COVID-19 Epidemic Metrics

Topic Modeling and Content Results

A total of 17,524 English tweets posted by CDC were retrieved and analyzed. LDA topic model generated 4 key topics based on the reference evaluation scores (i.e., perplexity and coherence score). These topics were then examined and categorized to themes by domain experts. The themes of the topics and their top 10 unique associated keywords are presented in the table below:

Visualization of the topics were displayed through an 2D plane, or an interactive chart. Based on LDA visualization results, these four identified key topics had the largest distances and minimal dimensional overlapping in the reduced two-dimensional plane. Nevertheless, from a public health perspective, CDC’s online health communication of COVID-19, the largest health emergency in the 21^st century, is relatively cohesive and comprehensive. CDC disseminated up- to-date, evidence-based, and accurate health information regarding COVID-19 to the public and to curb the pandemic. Therefore, the four key topics identified via LDA were not completely mutually exclusive. In addition, the 4-topic model had the balance of separation of topics from a computational perspective, and clear interpretability from a health perspective. Increasing the number of topics yields a substantial amount of topic overlapping, which was also challenging to provide explicit interpretations.

Visualization for key topics generated by LDA were shown in an interactive chart using pyLDAvis. In the interactive chart, topics were plotted as circles and displayed on the left panel; the most relevant terms, or associated keywords with their corresponding topics were displayed in frequency bars on the right panel, which showed each term’s frequency from each topic across the corpus (i.e., all CDC COVID-19 tweets sampled) [32] (see supplementary materials for more detailed information about visualizations in pyLDAvis). Size of the circle indicated prevalence of that topic in the corpus. Visualizations for all topics, displayed in circles on the left panel, and their top 15 corresponding relevant terms or associated keywords, displayed in frequency bars on the right panel, were provided in Appendix (Figure A1-A5 in Appendix).

Figure 1 below illustrated an example of topic 4. A list of associated terms of topic 4 and their overall frequency of the term in the corpus were displayed on the right panel. “booster”, “school”, “increase”, “parent”, and “country” were the five key terms from topic 4 based on overall frequency across all tweets.

Multivariate Time Series Analysis Results

Cross-correlation Functions (CCF)

Time series of CDC topics and COVID-19 metrics were plotted to visually examine patterns and potential associations. A total of 16 time series plots (four topics x four COVID-19 epidemic metrics) were generated (Figure A14-A29 in Appendix). Cross correlation functions (CCF) were computed to quantify the dynamic association between each CDC key topic series and each of the four COVID-19 epidemic metrics. Quantitative results were presented (Table A3-A6 in Appendix). Visualizations (Figure A30-A44 in Appendix) illustrated CCFs between both type of time series. CCF values and plots showed that CDC’s key COVID-19 tweet topic series were not substantially correlated with confirmed COVID-19 case counts series. As an example, there were no specific patterns between topic 2 and daily confirmed COVID-19 cases (Figure 2a).

COVID-19 confirmed cases and death time series had very similar dynamic patterns in the U.S. across time span (Figure 2b). Consequently, they also showed similar CCF with CDC key topic series (Figure A45 in Appendix). COVID-19 deaths had no substantial correlations with any of the four CDC key topics (Figure A18-A21 in Appendix) based on CCFs. There were no substantial correlations between any of the four key topics and COVID-19 testing series, as well as fully vaccinated rate series. Examples showed the CCFs between those and topic 2 (Figure 3-4). These results indicated potential discrepancy between CDC’s health communication focus and actual COVID-19 epidemic dynamics in the U.S. during the pandemic.

• Download figure
• Open in new tab

Figure 3.

CCF between completed COVID-19 tests series and topic 2 tweets

• Download figure
• Open in new tab

Figure 4.

CCF between completed COVID-19 vaccinations series and topic 2 tweets

Principal Findings

Our study took a unique approach of infoveillance by identifying potential associations between COVID-19 epidemic outcome metrics in the U.S. and CDC’s key topic dynamics during different stages of the pandemic. This innovative framework significantly expanded the original infoveillance approach that generally relied on number of posts (i.e., posting dynamics) without further extracting more detailed and meaningful content topics and sentiments from the textual data. Our study was able to further provide practical and useful health communication strategies for public health agencies to effectively communicate to the public with timely and accurate information. We systematically investigated and comprehensively identified CDC’s key topics, COVID-19 epidemic outcome metrics, and dynamic associations between the two types of data series based on more than 1.7 million COVID-related English tweets from CDC since January 2022. LDA topic model was built to characterize and identify the dynamic shifts of topics in CDC’s COVID-19 communication over more than 2-year period. For the first time, we were able to identify the four key topics: 1) general vaccination information and education; 2) pediatric intervention that also involved family and school safety; 3) updates on the COVID-19 epidemic situation such as numbers of cases, deaths, etc.; 4) research studies that was able to curb the pandemic.

Our study is the first of the kind to comprehensively evaluate the impact of online communications, especially on Twitter, one of the major social media platforms, during different phases of a large health emergency. We used NLP, especially topic modeling to explicitly explore CDC’s COVID-19 communication on social media. This approach and findings significantly reduced the gap of previous studies that overlooked the dynamic association between detailed topics discussed by public health agencies on social media and the real-world epidemic outcome metrics. We examined the dynamic associations between the four identified key topics, and four COVID-19 epidemic outcome metrics. Among the four major topics, topic 1, which covered information on vaccination and adverse health outcomes of COVID-19, had substantially strong associations with death counts and testing records during the Alpha phase (December 29, 2020 – June 14, 2021). Topic 3, which provided updates on three of the COVID- 19 measures (testing, cases, and deaths) and their relevant information, aligned better with case series during the Delta phase (June 15, 2021 – November 29, 2021). It also matched with death series during the declared pandemic phase (original variant, March 11, 2020 – December 28, 2020) and Delta phase, classified by WHO on May 11, 2021. Furthermore, even though topic 3 did not demonstrate visible temporal association with the testing series, timely communication from CDC was actually strongly associated with the trend over the entire study period based on the multivariate time series analysis.

According to these findings of this study, we suggest that aligning the content topics of health communication from public health agencies with the temporal dynamics of COVID-19, or other public health emergencies (e.g., major epidemic outcome metrics) can help health agencies provide more timely and relevant information to the public. Therefore, we recommend that CDC and other public health agencies to monitor the epidemic outcome metrics in real-time. Health agencies can post timely updates about the emergency, most recent findings, and interventions on social media according to the dynamic changes of these outcome metrics. Public health agencies can regain trust from the public by not only helping the public better understand the complex dynamics of the health emergency, but also informing the public with evidence-based guidance and recommendations more effectively.

Conclusions

This study investigated the dynamic associations between CDC’s detailed COVID-19 communication topics on Twitter and epidemic metrics for almost two years during the pandemic. Using LDA topic modeling, we were the first to comprehensively identify and explore various COVID-related topics tweeted by the federal public health agency during the pandemic. We also collected daily COVID-19 epidemic metrics (confirmed case counts, death counts, completed tests records, and fully vaccinated records) and performed various multivariate time series analysis to unravel the temporal patterns and associations with CDC’s COVID-19 communication patterns, i.e., investigating the dynamic associations between the time series of each topic generated by LDA model and the time series of each epidemic metrics. The results suggested that some topics were strongly associated with certain COVID-19 epidemic metrics, indicating that advanced social media analytics (e.g., natural language processing, NLP) could be a valuable tool for effective infoveillance. Based on our findings, we recommend that CDC, along with other public health agencies, could further optimize their health communications on social media platforms by posting contents and topics that aligns with the temporal dynamics of key epidemic metrics. Effectively disseminating timely and accurate information about an ongoing health emergency to the general public is crucial for health agencies to inform and educate the public, and eventually curb the ongoing health emergency. For example, we suggest increasing online health communications on health practices and interventions during high-level epidemic periods with large numbers of cases and deaths. Our findings also highlighted the importance of health communications on social media platforms to better respond to and tackle future health emergencies and issues.