Exploring the Application of Target Trial Emulation in Vaccine Evaluation: Scoping Review
=========================================================================================

* Toshiaki Komura
* Miwa Watanabe
* Kayoko Shioda

## ABSTRACT

**Background** Target trial emulation (TTE) has gained popularity in evaluating treatments and health interventions. Its application to infectious disease outcomes requires careful consideration, as infectious disease transmission violates the assumption of no interference. We conducted a scoping review to understand how TTE approaches have been applied to vaccine evaluation.

**Methods** We conducted a systematic search of literature published in PubMed, Embase, and Web of Science until May 2024, using keywords related to TTE, infectious diseases, and vaccines. Three independent reviewers screened titles and abstracts for relevance. Full-text articles meeting inclusion criteria were further assessed for eligibility.

**Results** Our keyword-based search and citation search identified a total of 240 studies. Of these, 34 original research studies used TTE approaches to evaluate vaccines, predominantly published from 2022 to 2024. Most studies (n=32, 94%) were conducted in high-income countries. The majority (n=31, 91%) evaluated the effect of COVID-19 vaccines, with one study each evaluating influenza, mpox, and rotavirus vaccines. Nationwide healthcare databases were used in 17 studies (50%). Twenty-one studies (62%) conducted analysis among adults aged ≥18 years, while six studies (18%) focused on children <18 years. Most studies did not define which of the four effects of vaccination they evaluated (direct, indirect, total, or overall effect), and none incorporated interference in vaccine evaluation.

**Conclusions** Our review highlights the increasing popularity of TTE in vaccine evaluation following the COVID-19 pandemic. Further discussions are needed to establish TTE approaches to estimating four effects of vaccination, using large, routinely collected data.

Keywords
*   vaccine evaluation
*   target trial emulation
*   scoping review
*   interference

## Key Messages

*   - This study performed a scoping review on how target trial emulation (TTE) approaches have been applied to vaccine evaluation, which requires careful consideration of interference.

*   - Among 34 studies in our scoping review, the majority evaluated the effect of COVID-19 vaccines, with one study each evaluating influenza, mpox, and rotavirus vaccines. Most studies were conducted in high-income countries, predominantly published from 2022 to 2024.

*   - Most studies did not define which of the four effects of vaccination they evaluated (direct, indirect, total, or overall effect), and none incorporated interference in vaccine evaluation.

*   - Our scoping review indicates a need for careful discussions to estimate the four effects of vaccination, using large, routinely collected data in the TTE framework.

## INTRODUCTION

Target trial emulation (TTE) is a causal inference framework that allows us to emulate a hypothetical target trial using observational data. TTE has become a popular research method to evaluate treatment and health interventions due to its practical and methodological advantages, serving as an important alternative to randomized controlled trials (RCTs) (1). While head-to-head RCTs are widely regarded as the gold standard for intervention evaluation, they are time-consuming, and operational and ethical challenges often hinder the randomization of participants in RCTs. Even when RCTs are feasible, they are generally restricted to small sample sizes and short follow-up periods, making it difficult to detect rare or long-term outcomes (2). RCTs also face constraints in comparing a large number of different protocols with various doses and treatment schedules due to resource limitations. Additionally, RCTs are primarily conducted in urban settings in high-income countries with selected population groups (3,4), leaving underrepresented groups reliant on findings from contexts that may not align with their own (5,6).

Observational studies could become useful to address these challenges posed by RCTs, especially when leveraging existing surveillance systems or large-scale health data (7). Existing surveillance systems can provide affordable and readily available data for intervention evaluation, helping us overcome operational and ethical challenges with RCTs. If surveillance systems cover the whole nation for a long time period, they could provide important information on short- and long-term outcomes of various diseases across demographic characteristics and geographic locations. However, as widely discussed, observational studies come with their own set of analytic challenges, including selection bias, confounding, and immortal time bias (which occurs when a time during which the outcome could not have happened is incorrectly handled in the analysis, often due to the misalignment of eligibility criteria and treatment assignment) (7–12).

TTE is a promising tool to harness the benefits of observational data while mitigating their analytic challenges (1). TTE facilitates causal inference using observational data, by mimicking a hypothetical target trial. By clearly specifying "time-zero" for treatment and control groups, TTE aims to effectively remove selection bias and immortal time bias. With the blurred lines between methodological advances in RCTs and observational studies, TTE has garnered attention as a convenient design for evaluating interventions (13).

While TTE has promising applications, it requires careful attention when used in the context of infectious diseases, where outcomes often do not satisfy the fundamental assumption of causal inference—no interference (14,15). Infectious diseases, which spread from infected individuals to susceptible individuals, challenge the assumption that the treatment of one person does not affect another person (16,17). This concern becomes particularly important when evaluating the effect of vaccines, because vaccination reduces population-level transmission, subsequently lowering the risk of infection among both vaccinated and unvaccinated individuals. In fact, vaccine evaluation has well-established guidelines categorizing vaccine effects into the following four types (18,19): 

1.  *direct effect*, which is the difference between the outcome in the vaccinated individual and what the outcome would have been without vaccination in the same person;

2.  *indirect effect*, which is the difference in the outcome in the unvaccinated individual in a population with a vaccination program and what the outcome would have been in the unvaccinated individual in a population without a vaccination program;

3.  *total effect,* which is the difference between the outcome in the vaccinated individual in a population with a vaccination program and what the outcome would have been in the unvaccinated individual in a comparable population without a vaccination program; and

4.  *overall effect,* which is the difference in the outcome in an average vaccinated individual in a population with a vaccination program compared with the outcome of an unvaccinated individual in a comparable population without a vaccination program.

In light of these considerations, we explored how the TTE framework has been applied to vaccine evaluation. This scoping review investigated the recent trends in the use of TTE for assessing vaccine effectiveness and safety. We summarized the target populations, data sources, vaccine products, comparators, and clinical outcomes. Additionally, we examined specific TTE and statistical methods used in each study, and investigated how studies defined specific types of vaccine effects within the TTE framework. Finally, we evaluated whether and how studies addressed the dependent nature of infectious diseases. Based on findings from the scoping review, we highlighted methodological gaps and provided recommendations for future studies using TTE in the field of vaccine evaluation.

## METHODS

### Search strategy

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for a scoping review checklist to design this scoping review (20). The protocol of this scoping review was registered with the Open Science Framework Registries on January 29, 2024 (21). Studies were extracted from three databases: PubMed, Embase, and Web of Science. The databases were searched from January 1, 2012, following the protocol of a previous systematic review (22), to May 21, 2024. Studies published in English were included. The search term included keywords related to vaccine evaluation and target trial emulation and was determined through group consensus (**Supplementary Table 1**). As TTE approaches are relatively new, especially in the field of vaccine evaluation, studies may have not explicitly stated that TTE approaches were used. The authors aimed to identify these studies by going through the literature and citations of identified studies.

### Eligibility Criteria

Three independent reviewers (TK, MW, and KS) screened titles and abstracts based on the following inclusion criteria: 1) original research articles, 2) primary objectives that include the evaluation of vaccines, 3) study designs that adhere to a target trial emulation framework, and 4) peer-reviewed articles published in scientific journals.

### Data Extraction

To examine study characteristics, we extracted the following information: authors, year of publication, countries of origin, target populations, data sources, treatments (i.e., evaluated vaccine regimen), comparators, outcomes, types of vaccine effects evaluated, specific TTE methods, applied statistical methods, and whether the interference was adjusted, incorporated, or discussed. Three reviewers (TK, MW, and KS) independently conducted the full-text screening.

## RESULTS

### Characteristics, target populations, and data sources of included studies

A total of 234 studies were identified from the keyword-based literature search, and 28 of them met the inclusion criteria after the screening (**Figure 1**) (23–50). Additionally, we identified six studies through citation search that did not explicitly state that the TTE approach was used but applied it for vaccine evaluation (51–56). Therefore, a total of 34 studies were included in the analysis. Most studies were published from 2022 to 2024 (2021: n=5, 15%; 2022: n= 10, 29%; 2023: n= 12, 35%; and 2024: n=7, 21%) (**Table 1**). The United States (n= 14, 41%)(24,27–29,32,38–40,44,45,47,48,50,51) and the United Kingdom (n= 5, 15%)(26,30,36,37,43) were the most common countries represented in the studies. According to the World Bank classification (57), almost all studies (n=32, 94%) were conducted in high-income countries (23–33,35–48,50–56), except for two (6%) in upper-middle-income countries (34,49). Regarding the target population, nine studies (26%) conducted analysis among veterans (24,27–29,32,38,39,45,47), and one study (3%) was conducted among nursing home residents (40). Twenty-one studies (62%) conducted analysis among adults aged ≥18 years (24,26–30,32–34,36–39,41,43–49), three of which (9%) specifically focused on older adults aged ≥60 years (26,34,43). Six studies (18%) focused on children <18 years (23,31,35,51,55,56). Among studies conducted in the U.S., the most common data source was the national healthcare databases of the Department of Veterans Affairs (n= 9, 64%) (24,27–29,32,38,39,45,47), whereas, in other countries, national healthcare service data, primary care data, cohort data, surveillance data, vaccine registry data, and insurance service data were used. Nationwide healthcare databases were used in 17 studies (50%) (23–25,27–29,32,35,36,38,39,41,42,45–47,49).

![Figure 1.](http://medrxiv.org/http://medrxiv.stage.highwire.org/content/medrxiv/early/2024/08/28/2024.07.26.24311066/F1.medium.gif)

[Figure 1.](http://medrxiv.org/content/early/2024/08/28/2024.07.26.24311066/F1)

Figure 1. 
Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) 2020 flow diagram for the systematic review.

View this table:
[Table 1.](http://medrxiv.org/content/early/2024/08/28/2024.07.26.24311066/T1)

Table 1. Characteristics of included studies.

### Evaluated vaccines, outcomes, and comparators

Almost all studies (n= 31, 91%) evaluated COVID-19 vaccines (23–34,36–47,49,50,52–56), with one study each evaluating influenza, mpox, and rotavirus vaccines (35,48,51). Among the COVID-19 vaccine studies, 18 (53%) examined the effectiveness against hospitalization and death (24,25,28–30,32,34,36,38–44,49,52,54), eight (24%) evaluated the effectiveness against infection only (23,26,31,37,45,50,55,56), one (3%) evaluated the effectiveness against death only (47), and one (3%) evaluated the effectiveness against infection and death but not hospitalization (27). One study (3%) focused on long COVID symptoms (46), and two studies (6%) evaluated the safety of COVID-19 vaccines against adverse events (46,53). Fourteen studies (41%) compared outcomes among vaccinated and unvaccinated individuals (23,25–27,31,33,34,43,45,47,52,53,55,56), while seven studies (21%) compared outcomes among those who received a booster dose and those who did not (28,38,40,41,44,49,54). Ten studies (29%) compared outcomes among those who received two different types of COVID-19 vaccines, such as BNT162b2 and mRNA-1273 (24,29,30,32,36,37,39,41,46,49). Two studies (6%) used individuals who had previous infections as a comparator (42,53), and one study (3%) evaluated the comparative effectiveness of different dosing intervals (50).

### TTE approaches and statistical methods

Twenty one studies (62%) used a matched cohort (23,24,26–29,32–36,38,41–43,45,46,49,52–54), while three studies (9%) used the clone-censor weight analysis to emulate target trials (47,50,51). One study (3%) prospectively collected observational data to conduct TTE analysis (37). Various statistical methods were used to quantify and compare vaccine effectiveness and safety across comparators. The majority (n= 25, 74%) used the Kaplan-Meier estimator, Cox proportional hazards model, or both (23–32,34,37,38,41–45,49,50,52–56). Seven studies (21%) applied the cumulative incidence function, the Fine-Gray subdistribution hazards model, or both to explicitly account for competing risks (30,38–40,44,47,51). Four studies (12%) used Poisson regression (35,36,46,54). Five studies (15%) applied a causal survival analysis framework to evaluate vaccine effectiveness (33,40,47,48,51).

### Evaluated types of vaccine effects and interference

Among the included studies, three studies (9%) defined the type of vaccine effect that they evaluated, all of which were direct effects (23,50,51). We did not find any study that explicitly incorporated indirect effects in vaccine evaluation. Only three studies (9%) mentioned the lack of adjustment for interference as a limitation (40,50,51).

## DISCUSSION

Our review showed that TTE approaches have increasingly become a popular method for vaccine evaluation in recent years, particularly following the introduction of COVID-19 vaccines. The majority of the included studies focused on the evaluation of COVID-19 vaccines, but TTE started to be applied to other vaccines, such as rotavirus vaccines, influenza vaccines, and mpox vaccines. Many studies did not specify which type of vaccine effects – direct, indirect, total, or overall effect – they evaluated. Interference of infectious diseases was not incorporated or addressed in vaccine evaluation in the included studies.

TTE approaches prove useful to evaluate the comparative effectiveness and safety of different vaccines (e.g., BNT162b2 and mRNA-1273 for COVID-19) or various dosing schedules (e.g., single dose vs. two doses of rotavirus vaccines; with vs. without a booster dose of COVID-19 vaccines), using observational data. RCTs cannot practically evaluate the full spectrum of dosing schedules (combination of a total number of doses and timing of each dose) in fine scale due to resource constraints. Real-world observational data, in contrast, have great variations in actual dosing schedules followed by vaccine recipients, which could be used in TTE approaches for evaluation. Selection bias and immortal time bias are major concerns when using observational data for intervention evaluation, as observational studies inherently assume that individuals successfully completed their assigned protocols, with no dropouts or outcomes during the course of intervention. This assumption is frequently violated when evaluating vaccines, especially for those requiring multiple doses, because vaccine schedules themselves affect the completion rate of the schedules. For example, the completion rate would be lower for schedules with a larger number of doses (e.g., with vs. without booster dose). Clone-censor weight analysis was specifically designed to address these issues by aligning the beginning of the follow-up time period with the date when individuals became eligible for and assigned to each protocol, rather than the date when they completed each protocol (58). These modern techniques are a promising tool to evaluate the impact of vaccines, especially when RCTs are not feasible or suitable.

Many of the included studies evaluated vaccines among the general population. This is a significant advantage of TTE approaches using large, routinely collectedd data, compared to RCTs that generally focus on smaller populations in selected settings. However, global disparity remained in the TTE studies. Most studies in our scoping review were from the U.S. and Europe. Since the impact of vaccines can vary across different settings due to varying population characteristics, disease burden, and transmission levels, data from diverse populations and regions are critical to guide global and local vaccine policy. TTE could be an affordable tool to achieve this if countries have surveillance databases and vaccine registry databases (3). To facilitate the use of TTE, it is important to teach these methods in workshops or include them in public health training curricula worldwide.

Despite various advantages of TTE, challenges persist in the application of TTE to vaccine evaluation, particularly concerning interference. The assumption of no interference is violated in studies of vaccines (59,60), because vaccines reduce population-level transmission and subsequently lower the risk of infection among both vaccinated and unvaccinated individuals.

As different vaccine products and their dosing schedules may produce varying levels of indirect effects, it is crucial to incorporate these effects into vaccine evaluation to identify optimal vaccine strategies. None of the studies in our scoping review addressed or incorporated interference in their vaccine evaluation, although a few mentioned it as a limitation. A few studies specified the types of vaccine effects evaluated, all of which were the direct effect.

Further discussions are needed to establish formal TTE methods for estimating the four effects of vaccination in the presence of interference, using large, routinely collected data. Halloran and Hudgens have discussed ways to estimate these four effects of vaccination with big data, addressing specific data requirements (61). Perez-Heydrich, *et al*. have estimated the four effects of cholera vaccination by applying inverse-probability weighted estimators to an individually-randomized controlled trial (62,63). This approach could potentially be extended to the TTE methodology. Additionally, agent-based modeling or other types of simulation modeling could be alternative approaches to addressing interference within the TTE framework. For example, agent-based modeling has been used to investigate the causal effects of preexposure prophylaxis among men who have sex with men in the presence of interference or spillover (64,65). The issue of interference is not restricted to infectious disease research; it is also relevant in fields such as health promotion programs, social epidemiology, mental health, and drug use disorder (66–70). Further work is needed to explore methods that address interference in various contexts of the target trial emulation framework.

The reliance on conventional survival analysis methods like Kaplan-Meier curves and Cox proportional hazards models in TTE studies raises concerns regarding potential biases, especially concerning competing risks (71). This is of particular concern when using surveillance data in which studies generally assume that if individuals do not show up in surveillance databases they did not have an outcome. To enhance the robustness of vaccine evaluation, we advocate for the adoption of causal survival analysis frameworks, such as pooled logistic regression with inverse probability weighting, which mitigate bias in long-term follow-up settings (72). Novel measures like restricted mean survival time (RMST) offer alternatives to evaluate vaccine effectiveness on an absolute scale, complementing traditional hazard ratio estimations (73).

A limitation of our study is the potential omission of studies that applied TTE approaches but did not explicitly label them as such. As TTE methodologies are relatively new, with standardized protocols recently being proposed and implemented (22), our keyword-based search might not have captured all relevant studies. To mitigate this issue, we conducted a review of the literature and examined the citations of identified studies to include any that were initially missed in our keyword-based search. Nonetheless, this approach may not have been exhaustive.

In summary, TTE has various practical and methodological advantages and is a useful approach for evaluating vaccine programs, especially when RCTs are not feasible. Clone-censor weight analysis is particularly effective for evaluating multi-dose vaccines to avoid selection bias and immortal time bias. Challenges related to interference need to be addressed to incorporate indirect effects or spillover effects in the evaluation of interventions, including vaccines. Moving forward, further collaboration between causal inference researchers and infectious disease experts will be beneficial for developing new approaches and guidelines on the application of TTE for vaccine evaluation in the presence of interference.

## Data Availability

All data produced in the present study are available upon reasonable request to the authors

## CONFLICT OF INTEREST

The authors have declared no competing interest.

View this table:
[Supplementary Table 1.](http://medrxiv.org/content/early/2024/08/28/2024.07.26.24311066/T2)

Supplementary Table 1. Search terms and results of the literature review.

## ACKNOWLEDGEMENT

We thank Kristen Sheridan for her support with the literature review. We also thank Dr. M. Elizabeth Halloran, Dr. Marc Lipsitch, and Katherine Jia for their invaluable feedback. This study was also made possible by cooperative agreement CDC-RFA-FT-23-0069 from the CDC’s Center for Forecasting and Outbreak Analytics. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the Centers for Disease Control and Prevention.

## Footnotes

*   Papers identified by citation search have been added.

*   Received July 26, 2024.
*   Revision received August 27, 2024.
*   Accepted August 28, 2024.


*   © 2024, Posted by Cold Spring Harbor Laboratory

This pre-print is available under a Creative Commons License (Attribution-NonCommercial-NoDerivs 4.0 International), CC BY-NC-ND 4.0, as described at [http://creativecommons.org/licenses/by-nc-nd/4.0/](http://creativecommons.org/licenses/by-nc-nd/4.0/)

## REFERENCES

1.  1.Hernán MA, Wang W, Leaf DE. Target Trial Emulation: A Framework for Causal Inference From Observational Data. JAMA. 2022 Dec 27;328(24):2446.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/jama.2022.21383&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=36508210&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

2.  2.Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Soc Sci Med. 2018 Aug;210:2–21.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

3.  3.Nsanzimana S, Gupta A, Uwizihiwe JP, Haggstrom J, Dron L, Arora P, et al. The Need for a Practical Approach to Evaluate the Effectiveness of COVID-19 Vaccines for Low- and Middle-Income Countries. Am J Trop Med Hyg. 2021 Jul 16;105(3):561–3.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.4269/ajtmh.21-0482&link_type=DOI) 

4.  4.Madhi SA, Baillie V, Cutland CL, Voysey M, Koen AL, Fairlie L, et al. Efficacy of the ChAdOx1 nCoV-19 Covid-19 Vaccine against the B.1.351 Variant. N Engl J Med. 2021 May 20;384(20):1885–98.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2102214&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=33725432&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

5.  5.Equity must be baked into randomized controlled trials. Nature. 2022 Jun 23;606(7915):624–624.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/d41586-022-01675-2&link_type=DOI) 

6.  6.Habibzadeh F. Disparity in the selection of patients in clinical trials. The Lancet. 2022 Mar;399(10329):1048.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-6736(22)00176-3&link_type=DOI) 

7.  7.Hernán MA, Robins JM. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available: Table 1. Am J Epidemiol. 2016 Apr 15;183(8):758–64.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwv254&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=26994063&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

8.  8.García-Albéniz X, Hsu J, Hernán MA. The value of explicitly emulating a target trial when using real world evidence: an application to colorectal cancer screening. Eur J Epidemiol. 2017 Jun;32(6):495–500.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=doi:10.1007/s10654-017-0287-2&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=28748498&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

9.  9.Caniglia EC, Zash R, Jacobson DL, Diseko M, Mayondi G, Lockman S, et al. Emulating a target trial of antiretroviral therapy regimens started before conception and risk of adverse birth outcomes. AIDS. 2018 Jan 2;32(1):113–20.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/QAD.0000000000001673&link_type=DOI) 

10. 10.Hernán MA, Sauer BC, Hernández-Díaz S, Platt R, Shrier I. Specifying a target trial prevents immortal time bias and other self-inflicted injuries in observational analyses. J Clin Epidemiol. 2016 Nov;79:70–5.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=doi:10.1016/j.jclinepi.2016.04.014&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=27237061&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

11. 11.Danaei G, García Rodríguez LA, Cantero OF, Logan RW, Hernán MA. Electronic medical records can be used to emulate target trials of sustained treatment strategies. J Clin Epidemiol. 2018 Apr;96:12–22.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jclinepi.2017.11.021&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

12. 12.Antihyperglycemic Therapy and Cardiovascular Risk: Design and Emulation of a Target Trial Using Healthcare Databases. In 2019. Available from: [https://api.semanticscholar.org/CorpusID:210926774](https://api.semanticscholar.org/CorpusID:210926774)
    
    

13. 13.Fernainy P, Cohen AA, Murray E, Losina E, Lamontagne F, Sourial N. Rethinking the pros and cons of randomized controlled trials and observational studies in the era of big data and advanced methods: a panel discussion. BMC Proc. 2024 Jan 18;18(S2):1.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12919-023-00285-8&link_type=DOI) 

14. 14.Cox DR. Planning of experiments. Oxford, England: Wiley; 1958. (Planning of experiments.).
    
    

15. 15.Rubin DB. Discussion of “Randomization Analysis of Experimental Data in the Fisher Randomization Test. J Am Stat Assoc. 1980;75:591–593. by D. Basu.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/2287653&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1980KL55500018&link_type=ISI) 

16. 16.Hudgens MG, Halloran ME. Toward Causal Inference With Interference. J Am Stat Assoc. 2008 Jun 1;103(482):832–42.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1198/016214508000000292&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=19081744&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

17. 17.Halloran ME, Struchiner CJ. Causal Inference in Infectious Diseases: Epidemiology. 1995 Mar;6(2):142–51.
    
    

18. 18.Halloran ME, Struchiner CJ. Study Designs for Dependent Happenings: Epidemiology. 1991 Sep;2(5):331–8.
    
    

19. 19.Halloran ME, Longini IM, Struchiner CJ. Design and analysis of vaccine studies. New York: Springer; 2010. 387 p. (Statistics for biology and health).
    
    

20. 20.Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018 Oct 2;169(7):467–73.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7326/M18-0850&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=30178033&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

21. 21.OSF [Internet]. [cited 2024 Jul 3]. Available from: [https://osf.io/m6b9a](https://osf.io/m6b9a)
    
    

22. 22.Hansford HJ, Cashin AG, Jones MD, Swanson SA, Islam N, Douglas SRG, et al. Reporting of Observational Studies Explicitly Aiming to Emulate Randomized Trials: A Systematic Review. JAMA Netw Open. 2023 Sep 27;6(9):e2336023.
    
    

23. 23.Chemaitelly H, AlMukdad S, Ayoub HH, Altarawneh HN, Coyle P, Tang P, et al. Covid-19 Vaccine Protection among Children and Adolescents in Qatar. N Engl J Med. 2022 Nov 17;387(20):1865–76.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2210058&link_type=DOI) 

24. 24.Dickerman BA, Gerlovin H, Madenci AL, Kurgansky KE, Ferolito BR, Figueroa Muñiz MJ, et al. Comparative Effectiveness of BNT162b2 and mRNA-1273 Vaccines in U.S. Veterans. N Engl J Med. 2022 Jan 13;386(2):105–15.
    
    

25. 25.Gazit S, Shlezinger R, Perez G, Lotan R, Peretz A, Ben-Tov A, et al. The Incidence of SARS-CoV-2 Reinfection in Persons With Naturally Acquired Immunity With and Without Subsequent Receipt of a Single Dose of BNT162b2 Vaccine : A Retrospective Cohort Study. Ann Intern Med. 2022 May;175(5):674–81.
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

26. 26.Català M, Burn E, Rathod-Mistry T, Xie J, Delmestri A, Prieto-Alhambra D, et al. Observational methods for COVID-19 vaccine effectiveness research: an empirical evaluation and target trial emulation. Int J Epidemiol. 2024 Feb 1;53(1).
    
    

27. 27.Ioannou GN, Locke ER, O’Hare AM, Bohnert ASB, Boyko EJ, Hynes DM, et al. COVID-19 Vaccination Effectiveness Against Infection or Death in a National U.S. Health Care System : A Target Trial Emulation Study. Ann Intern Med. 2022 Mar;175(3):352–61.
    
    

28. 28.Ioannou GN, Bohnert ASB, O’Hare AM, Boyko EJ, Maciejewski ML, Smith VA, et al. Effectiveness of mRNA COVID-19 Vaccine Boosters Against Infection, Hospitalization, and Death: A Target Trial Emulation in the Omicron (B.1.1.529) Variant Era. Ann Intern Med. 2022 Dec;175(12):1693–706.
    
    

29. 29.Ioannou GN, Locke ER, Green PK, Berry K. Comparison of Moderna versus Pfizer-BioNTech COVID-19 vaccine outcomes: A target trial emulation study in the U.S. Veterans Affairs healthcare system. EClinicalMedicine. 2022 Mar;45:101326.
    
    

30. 30.Wei J, Zhang W, Doherty M, Wallace ZS, Sparks JA, Lu N, et al. Comparative effectiveness of BNT162b2 and ChAdOx1 nCoV-19 vaccines against COVID-19. BMC Med. 2023 Feb 28;21(1):78.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/s12916-023-02795-w&link_type=DOI) 

31. 31.Gazit S, Saciuk Y, Perez G, Peretz A, Ben-Tov A, Stuart EA, et al. Hybrid immunity against reinfection with SARS-CoV-2 following a previous SARS-CoV-2 infection and single dose of the BNT162b2 vaccine in children and adolescents: a target trial emulation. Lancet Microbe. 2023 Jul;4(7):e495–505.
    
    

32. 32.Dickerman BA, Gerlovin H, Madenci AL, Figueroa Muñiz MJ, Wise JK, Adhikari N, et al. Comparative effectiveness of third doses of mRNA-based COVID-19 vaccines in US veterans. Nat Microbiol. 2023 Jan;8(1):55–63.
    
    

33. 33.Tran VT, Perrodeau E, Saldanha J, Pane I, Ravaud P. Efficacy of first dose of covid-19 vaccine versus no vaccination on symptoms of patients with long covid: target trial emulation based on ComPaRe e-cohort. BMJ Med. 2023;2(1):e000229.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1136/bmjmed-2022-000229&link_type=DOI) 

34. 34.Monteiro HS, Lima Neto AS, Kahn R, Sousa GS, Carmona HA, Andrade JSJ, et al. Impact of CoronaVac on Covid-19 outcomes of elderly adults in a large and socially unequal Brazilian city: A target trial emulation study. Vaccine. 2023 Sep 7;41(39):5742–51.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2023.07.065&link_type=DOI) 

35. 35.Kildegaard H, Lund LC, Pottegård A, Stensballe LG. Effectiveness of the quadrivalent live attenuated influenza vaccine against influenza-related hospitalisations and morbidity among children aged 2 to 6 years in Denmark: a nationwide cohort study emulating a target trial. Lancet Child Adolesc Health. 2023 Dec;7(12):852–62.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S2352-4642(23)00225-0&link_type=DOI) 

36. 36.Kerr S, Bedston S, Bradley DT, Joy M, Lowthian E, Mulholland RM, et al. Waning of first- and second-dose ChAdOx1 and BNT162b2 COVID-19 vaccinations: a pooled target trial study of 12.9 million individuals in England, Northern Ireland, Scotland and Wales. Int J Epidemiol. 2023 Feb 8;52(1):22–31.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/ije/dyac199&link_type=DOI) 

37. 37.Nguyen VG, Yavlinsky A, Beale S, Hoskins S, Byrne TE, Lampos V, et al. Comparative effectiveness of different primary vaccination courses on mRNA-based booster vaccines against SARs-COV-2 infections: a time-varying cohort analysis using trial emulation in the Virus Watch community cohort. Int J Epidemiol. 2023 Apr 19;52(2):342–54.
    
    

38. 38.Butt AA, Talisa VB, Shaikh OS, Omer SB, Mayr FB. Relative Vaccine Effectiveness of a Severe Acute Respiratory Syndrome Coronavirus 2 Messenger RNA Vaccine Booster Dose Against the Omicron Variant. Clin Infect Dis Off Publ Infect Dis Soc Am. 2022 Dec 19;75(12):2161–8.
    
    

39. 39.Kelly JD, Leonard S, Boscardin WJ, Hoggatt KJ, Lum EN, Austin CC, et al. Comparative mRNA booster effectiveness against death or hospitalization with COVID-19 pneumonia across at-risk US Veteran populations. Nat Commun. 2023 May 23;14(1):2976.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41467-023-38503-8&link_type=DOI) 

40. 40.McConeghy KW, Bardenheier B, Huang AW, White EM, Feifer RA, Blackman C, et al. Infections, Hospitalizations, and Deaths Among US Nursing Home Residents With vs Without a SARS-CoV-2 Vaccine Booster. JAMA Netw Open. 2022 Dec 1;5(12):e2245417.
    
    

41. 41.Andersson NW, Thiesson EM, Baum U, Pihlström N, Starrfelt J, Faksová K, et al. Comparative effectiveness of bivalent BA.4-5 and BA.1 mRNA booster vaccines among adults aged ≥50 years in Nordic countries: nationwide cohort study. BMJ. 2023 Jul 25;382:e075286.
    
    [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE5OiIzODIvanVsMjRfNy9lMDc1Mjg2IjtzOjQ6ImF0b20iO3M6NTA6Ii9tZWRyeGl2L2Vhcmx5LzIwMjQvMDgvMjgvMjAyNC4wNy4yNi4yNDMxMTA2Ni5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

42. 42.Chemaitelly H, Ayoub HH, AlMukdad S, Coyle P, Tang P, Yassine HM, et al. Protection from previous natural infection compared with mRNA vaccination against SARS-CoV-2 infection and severe COVID-19 in Qatar: a retrospective cohort study. Lancet Microbe. 2022 Dec;3(12):e944–55.
    
    

43. 43.Hulme WJ, Williamson E, Horne EMF, Green A, McDonald HI, Walker AJ, et al. Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data. Ann Intern Med. 2023 May;176(5):685–93.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.7326/M21-4269&link_type=DOI) 

44. 44.Hanberg JS, Fu X, Wang X, Patel NJ, Kawano Y, Schiff A, et al. Effectiveness of a fourth dose of COVID-19 mRNA vaccine in patients with systemic autoimmune rheumatic diseases using disease-modifying antirheumatic drugs: an emulated target trial. Lancet Rheumatol. 2024 Jan;6(1):e21–30.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S2665-9913(23)00272-2&link_type=DOI) 

45. 45.Li G, Gerlovin H, Figueroa Muñiz MJ, Wise JK, Madenci AL, Robins JM, et al. Comparison of the Test-negative Design and Cohort Design With Explicit Target Trial Emulation for Evaluating COVID-19 Vaccine Effectiveness. Epidemiol Camb Mass. 2024 Mar 1;35(2):137–49.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/EDE.0000000000001709&link_type=DOI) 

46. 46.Choi MJ, Na Y, Hyun HJ, Nham E, Yoon JG, Seong H, et al. Comparative safety analysis of mRNA and adenoviral vector COVID-19 vaccines: a nationwide cohort study using an emulated target trial approach. Clin Microbiol Infect Off Publ Eur Soc Clin Microbiol Infect Dis. 2024 May;30(5):646–52.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.cmi.2023.12.010&link_type=DOI) 

47. 47.McConeghy KW, Hur K, Dahabreh IJ, Jiang R, Pandey L, Gellad WF, et al. Early Mortality After the First Dose of COVID-19 Vaccination: A Target Trial Emulation. Clin Infect Dis Off Publ Infect Dis Soc Am. 2024 Mar 20;78(3):625–32.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/cid/ciad604&link_type=DOI) 

48. 48.Rosen JB, Arciuolo RJ, Pathela P, Boyer CB, Baumgartner J, Latash J, et al. JYNNEOSTM effectiveness as post-exposure prophylaxis against mpox: Challenges using real-world outbreak data. Vaccine. 2024 Jan 25;42(3):548–55.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.vaccine.2023.12.066&link_type=DOI) 

49. 49. Lazar Neto F, Hitchings MDT, Amin AB, de França GVA, Lind ML, Scaramuzzini Torres MS, et al. Effectiveness of the fourth dose of COVID-19 vaccines against severe COVID-19 among adults 40 years or older in Brazil: a population-based cohort study. Lancet Reg Health Am. 2024 Jun;34:100755.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.lana.2024.100755&link_type=DOI) 

50. 50.Shioda K, Breskin A, Harati P, Chamberlain AT, Komura T, Lopman BA, et al. Comparative effectiveness of alternative intervals between first and second doses of the mRNA COVID-19 vaccines. Nat Commun. 2024 Feb 9;15(1):1214.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1038/s41467-024-45334-8&link_type=DOI) 

51. 51.Butler AM, Breskin A, Sahrmann JM, Brookhart MA. Estimating the Effectiveness of Rotavirus Vaccine Schedules. Epidemiology. 2021 Jul;32(4):598–606.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1097/EDE.0000000000001363&link_type=DOI) 

52. 52.Dagan N, Barda N, Kepten E, Miron O, Perchik S, Katz MA, et al. BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. N Engl J Med. 2021 Apr 15;384(15):1412–23.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMOA2101765&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

53. 53.Barda N, Dagan N, Ben-Shlomo Y, Kepten E, Waxman J, Ohana R, et al. Safety of the BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Setting. N Engl J Med. 2021 Sep 16;385(12):1078–90.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMoa2110475&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

54. 54.Barda N, Dagan N, Cohen C, Hernán MA, Lipsitch M, Kohane IS, et al. Effectiveness of a third dose of the BNT162b2 mRNA COVID-19 vaccine for preventing severe outcomes in Israel: an observational study. The Lancet. 2021 Dec;398(10316):2093–100.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/S0140-4266736(21)02249-2&link_type=DOI) 

55. 55.Reis BY, Barda N, Leshchinsky M, Kepten E, Hernán MA, Lipsitch M, et al. Effectiveness of BNT162b2 Vaccine against Delta Variant in Adolescents. N Engl J Med. 2021 Nov 25;385(22):2101–3.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1056/NEJMc2114290&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=34670036&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

56. 56.Cohen-Stavi CJ, Magen O, Barda N, Yaron S, Peretz A, Netzer D, et al. BNT162b2 Vaccine Effectiveness against Omicron in Children 5 to 11 Years of Age. N Engl J Med. 2022 Jul 21;387(3):227–36.
    
    

57. 57.The World Bank [Internet]. [cited 2024 Aug 12]. World Bank Country and Lending Groups. Available from: [https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups](https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups)
    
    

58. 58.Gaber CE, Hanson KA, Kim S, Lund JL, Lee TA, Murray EJ. The Clone-Censor-Weight Method in Pharmacoepidemiologic Research: Foundations and Methodological Implementation. Curr Epidemiol Rep [Internet]. 2024 Feb 17 [cited 2024 Jul 14]; Available from: [https://link.springer.com/10.1007/s40471-024-00346-2](https://link.springer.com/10.1007/s40471-024-00346-2)
    
    

59. 59.Robins J. A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect. Math Model. 1986;7(9–12):1393–512.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/0898-1221(87)90238-0&link_type=DOI) 

60. 60.Rubin DB. Randomization Analysis of Experimental Data: The Fisher Randomization Test Comment. J Am Stat Assoc. 1980 Sep;75(371):591.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.2307/2287653&link_type=DOI) 
    
    [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=A1980KL55500018&link_type=ISI) 

61. 61.Halloran ME, Hudgens MG. Estimating population effects of vaccination using large, routinely collected data. Stat Med. 2018 Jan 30;37(2):294–301.
    
    

62. 62.Perez-Heydrich C, Hudgens MG, Halloran ME, Clemens JD, Ali M, Emch ME. Assessing effects of cholera vaccination in the presence of interference. Biometrics. 2014 Sep;70(3):731–41.
    
    

63. 63.Chakladar S, Rosin S, Hudgens MG, Halloran ME, Clemens JD, Ali M, et al. Inverse probability weighted estimators of vaccine effects accommodating partial interference and censoring. Biometrics. 2022 Jun;78(2):777–88.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/biom.13459&link_type=DOI) 

64. 64.Murray EJ, Marshall BDL, Buchanan AL. Emulating Target Trials to Improve Causal Inference From Agent-Based Models. Am J Epidemiol. 2021 Aug 1;190(8):1652–8.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwab040&link_type=DOI) 

65. 65.Buchanan AL, Bessey S, Goedel WC, King M, Murray EJ, Friedman SR, et al. Disseminated Effects in Agent-Based Models: A Potential Outcomes Framework and Application to Inform Preexposure Prophylaxis Coverage Levels for HIV Prevention. Am J Epidemiol. 2021 May 4;190(5):939–48.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwaa239&link_type=DOI) 

66. 66.Morozova O, Crawford FW, Cohen T, Paltiel AD, Altice FL. Cost-effectiveness of expanding the capacity of opioid agonist treatment in Ukraine: dynamic modeling analysis. Addiction. 2020 Mar;115(3):437–50.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1111/add.14797&link_type=DOI) 

67. 67.Komura T, Tsugawa Y, Kondo N, Inoue K. Depression Onset After a Spouse’s Cardiovascular Event. JAMA Netw Open. 2024 Apr 12;7(4):e244602.
    
    

68. 68.Tracy M, Cerdá M, Keyes KM. Agent-Based Modeling in Public Health: Current Applications and Future Directions. Annu Rev Public Health. 2018 Apr 1;39(1):77–94.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1146/annurev-publhealth-040617-014317&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=29328870&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom) 

69. 69.Cerdá M, Keyes KM. Systems Modeling to Advance the Promise of Data Science in Epidemiology. Am J Epidemiol. 2019 May 1;188(5):862–5.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1093/aje/kwy262&link_type=DOI) 

70. 70.Steeves S, Acciai F, Tasevska N, DeWeese RS, Yedidia MJ, Ohri-Vachaspati P. The Special Supplemental Nutrition Program for Women, Infants, and Children Spillover Effect: Do Siblings Reap the Benefits? J Acad Nutr Diet. 2020 Aug;120(8):1288–94.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1016/j.jand.2020.02.013&link_type=DOI) 

71. 71.Coemans M, Verbeke G, Döhler B, Süsal C, Naesens M. Bias by censoring for competing events in survival analysis. BMJ. 2022 Sep 13;e071349.
    
    

72. 72.Murray EJ, Caniglia EC, Petito LC. Causal survival analysis: A guide to estimating intention-to-treat and per-protocol effects from randomized clinical trials with non-adherence. Res Methods Med Health Sci. 2021 Jan;2(1):39–49.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1177/2632084320961043&link_type=DOI) 

73. 73.Royston P, Parmar MK. Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome. BMC Med Res Methodol. 2013 Dec;13(1):152.
    
    [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1186/1471-2288-13-152&link_type=DOI) 
    
    [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24314264&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2024%2F08%2F28%2F2024.07.26.24311066.atom)