Abstract
Transparency shortcomings can undermine confidence in the safety and efficacy of vaccines. This study assesses the publication status and disclosure gaps in a cohort of 71 clinical trials assessing the safety and efficacy of 3 COVID-19 vaccines developed by Chinese biopharmaceutical companies that received a World Health Organization Emergency Use Listing (EUL) and have been marketed globally. We searched trial registries and the scientific literature to assess the completion status of those 71 trials, and to determine whether the outcomes of completed trials have been made publicly available.
The 71 trials in our cohort were initiated by sponsors headquartered in 17 different countries and aimed to enrol a total of 313,750 participants from across 27 countries. Out of those 71 trials, 49 trials (69%) had verifiably been concluded. We were unable to determine the completion status of the remaining 22 trials (31%) with certainty. Depending on whether those 22 trials were assumed to have been concluded or not, we found that between 13 completed trials (27%) and 35 completed trials (49%) remained unreported. At least 9 trials (13% of the total) had not made their results public more than one year post completion. According to registry data, between 36,498 people (12% of participants across all trials) and 89,224 people (28%) had participated in trials that had been concluded and whose outcomes remained unreported. There were no tabular summary results available on trial registries for any of the trials.
Our findings suggest that there are significant gaps in clinical trial governance, especially in countries that have only recently emerged as significant players in biomedical research. Maintaining global progress in clinical trial reporting will require legislators and regulators to adopt and effectively enforce clinical trial reporting requirements that reflect WHO best practices. The global clinical trial registry infrastructure needs to be strengthened so that users can reliably determine whether studies have been withdrawn, are still ongoing, or have been concluded.
What is already known on this topic The results of many clinical trials are only made public partially, after long delays, or not at all. Such disclosure gaps can make it difficult to assess the benefits and harms of treatments, and can undermine public trust in health interventions, including vaccines. In North America and Europe, disclosure has significantly improved in recent years.
What this study adds Previous research in this field has overwhelmingly focused on clinical trials run by sponsors in North America and Europe that usually involved patients in these regions. Our study provides a global perspective on the problem, using a cohort of trials with high salience for global public health. More than 1.3 billion doses of 3 COVID-19 vaccines originally developed in China have been exported to dozens of countries worldwide. We assessed the publication status and disclosure gaps of 71 clinical trials of these vaccines. We found widespread research activity in newly emerging hubs of biomedical innovation. In total, sponsors from 17 different countries and participants from 27 countries were involved in relevant trials. None of the concluded trials had reported outcomes in line with global best practice standards set out by the World Health Organisation.
How this study might affect research, practice or policy Clinical trial activity is increasingly becoming global. Our research points to significant gaps in clinical trial governance in many emerging hubs of biomedical innovation. Legislators and regulators in these countries should ensure that all clinical trial results are rapidly made public in line with World Health Organisation best practices to prevent the growth of gaps in the global medical evidence base. In addition, there is an urgent need to strengthen the global clinical trial registry infrastructure.
1. Introduction
After the emergence of the SARS-CoV-2 virus in late 2019, the rapid development of vaccines became a global health priority, despite concerns that regulators might rush through approvals without due regard to vaccine safety and efficacy (1). Transparency in clinical trials of vaccines is essential to foster public trust and boost vaccine acceptance (2, 3).
However, the existing literature suggests that disclosure of the designs and outcomes of some COVID-19 vaccine trials has been suboptimal. A 2021 global report by Transparency International found that clinical trial protocols had been made public for only 12% of 86 clinical trials of 20 COVID-19 vaccines (4). Several Western pharmaceutical companies have been criticised for incomplete disclosure of trial designs (5). Vaccine approvals by the Indian drug regulator did not follow established pathways and decision-making processes were opaque (6). Chinese companies in particular have been criticised for only releasing partial trial results (7, 8).
As of August 2023, a total of 3 COVID-19 vaccines developed by 3 different Chinese companies had successfully completed the World Health Organisation’s (WHO’s) Emergency Use Listing (EUL) evaluation process, opening the door to international markets (9, 10). As of December 2022, more than 1.3 billion doses of these 3 vaccines had been exported to dozens of countries, significantly increasing the diversity and volume of global vaccine supply (11, 12).
This paper examines a key element of trial transparency by assessing the publication status of all clinical trials involving the 3 COVID-19 vaccines developed by Chinese companies that received an EUL from the WHO. While Chinese law requires vaccine trial results to be shared with the national medicines regulator, it does not require them to be made public (13).
However, irrespective of national laws, publication of trial results is an ethical obligation under the Declaration of Helsinki (14). According to global best practices set out in the WHO’s Joint statement on public disclosure of results from clinical trials, the results of all clinical trials should be published on a public trial registry within 12 months of trial completion, and in a peer-reviewed journal within 24 months of trial completion (15). Sharing vaccine trial results with the public can support a better understanding of vaccine effectiveness and potential adverse effects (16-18).
The objective of this study was to assess whether the outcomes of clinical trials involving the 3 COVID-19 vaccines had been made public.
2. Methodology
The study protocol was preregistered on OSF (https://osf.io/4f9k7). No UK Health Research Authority NHS REC ethics approval was required as the study exclusively used publicly available study-level data. Patients and the public were not involved in this research. Outcomes are reported in line with the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) guideline for cohort studies (19). The study protocol and dataset are publicly available on OSF (https://osf.io/r3vp9/). This study did not receive external funding. The authors have no conflicts of interest to declare.
2.1 Cohort selection
We extracted the registry numbers of 95 clinical trials involving the 3 COVID-19 vaccines from McGill University’s COVID-19 Vaccine Tracker website (https://covid19.trackvaccines.org/), which had last been updated on 02 December 2022 (11). We de-duplicated 20 entries for 16 trials, creating a study cohort of 75 unique trials.
Data collection and search strategy
First, we entered the trial registry ID numbers sourced from the COVID-19 Vaccine Tracker into the ICTRP search function (https://trialsearch.who.int/Default.aspx) to identify potential additional registry entries for each of the 75 trials, and captured those.
Second, we manually extracted the following data from registry entries for each trial: (lead) sponsor, sponsor country, trial locations, trial phase, current trial status, trial completion date (primary if available, else overall), and number of trial participants (actual if available, else planned enrolment). If a trial was registered on multiple registries, we used registry data from registries in the following order of priority: ClinicalTrials.gov, ChiCTR, other registries. Registry data were first extracted in August 2023.
Third, two researchers independently searched for the outcomes of each of the 75 trials using a detailed search strategy set out in the study protocol (https://osf.io/4f9k7). Briefly, we (a) scanned the “results” section of all relevant trial registries for tabular summary results or hyperlinked publications, (b) searched PubMed and Google Scholar for the clinical trial ID number(s), (c) searched PubMed and Google Scholar for the trial title and principal investigator. The three steps above were completed between 23 August 2023 and 05 September 2023.
We then reviewed all journal publications to verify that they described the outcome of the trial by matching the 4 PICO criteria (patient inclusion criteria, intervention, comparison and outcomes). As per protocol, we only counted tabular summary results on trial registries and full-length publications of final trial results in peer-reviewed journals as ‘reported’. We classified publications presenting only interim trial outcomes, preprints, conference abstracts, press releases or other grey literature as ‘unreported’. For each trial, 2 team members, independently and in duplicate, reviewed the search results. Any inconsistencies and ambiguities were resolved by consensus or by a third team member. We then extracted the date of each publication.
Between 20 March 2024 and 13 April 2024, to provide additional quality assurance, we updated the spreadsheet to capture recent changes in registry entries, repeated the PICO match for previously located publications, and fully replicated the search for outcomes of trials for which no outcome had previously been located. We removed 4 trials that according to registry data had been withdrawn and never recruited participants (NCT05156632, NCT05254236, NCT05308602, NCT05442684), leaving a study cohort of 71 trials.
2.2 Outcome measures
The prespecified primary outcome measure was the number and percent of concluded (i.e. completed or terminated) trials whose results remained unreported. Secondary outcome measures were the number and percentage of participants enrolled (planned or actual enrolment, depending on registry data availability for each trial) in concluded trials whose results remained unreported, and the number and percentage of trials that met both WHO best practices in clinical trial reporting, i.e. results posted onto a public registry within 12 months of trial conclusion and results published in a peer-reviewed journal within 24 months of trial conclusion.
For the purpose of calculating the metrics above, as per our protocol, we considered trials to have been concluded if their status was marked as ‘completed’ or ‘terminated’ on a trial registry, and/or if the final outcomes had been made public on a trial registry or in a peer-reviewed journal (even if the registry still listed the trial as ongoing).
3. Results
3.1 Study population
According to registry data, the 71 trials in our cohort aimed to enroll a total of 313,750 participants from across 27 countries (Argentina, Bahrain, Brazil, Chile, China, Colombia, Egypt, Ghana, Guinea, Indonesia, Iran, Jordan, Laos, Madagascar, Malaysia, Mexico, Mozambique, Pakistan, Panama, Peru, Philippines, Russia, South Africa, Thailand, Turkey, United Arab Emirates, United States). The trials were initiated by a diverse set of commercial and non-commercial sponsors headquartered in a total of 17 countries (Argentina, Brazil, China, Chile, France, Indonesia, Iran, Pakistan, Peru, Philippines, Russia, Singapore, South Korea, Thailand, Turkey, United Arab Emirates, United States). The earliest and latest ‘actual’ trial (primary) completion dates listed in registries were in July 2020 and January 2024, respectively.
3.2 Trial status
According to registry data, 31/71 trials (44%) had been completed or terminated as of 13 April 2024. A trial is only listed as ‘completed’ or ‘terminated’ on a registry if the record holder has actively performed an update after the initial registration, so those trials had verifiably been concluded. The remaining 40/71 trials (46%) were not listed as having been concluded, including 12 trials whose status was listed as ‘unknown’.
3.3 Publication status
We located publications in peer-reviewed journals containing the final outcomes for 36/71 trials (51%); only 18 of those were listed as ‘completed’ or ‘terminated’ in registries. As per protocol, we did not classify 3 trials as reported as available publications only reported interim outcomes (NCT05323461, NCT04992260) or were only available as a preprint (NCT05137418). We only located outcomes uploaded onto registries for two trials (IRCT20140818018842N24, IRCT20201214049709N3); the outcomes of those trials had previously already been reported in peer-reviewed journals.
The outcomes of a further 13/71 (20%) verifiably concluded trials (11 completed and 2 terminated) had not been made public, including 9 trials that according to their ‘actual primary completion date’ field in ClinicalTrials.gov had concluded more than a year before our assessment.
3.4 Combining trial status and publication status
By combining trial status and publication status, we determined that 49/71 trials (69%) had definitely been concluded. Of these 49 concluded trials, 36/49 (73%) had published outcomes while 13/49 (27%) had not. Of the 13 unreported trials, 4 had been concluded less than a year ago; the remaining 9 had failed to meet the WHO benchmark of making outcomes public within one year of concluding the trial.
The remaining 22/71 trials (31%) were not listed as concluded in trial registries and had not made their outcomes public. However, the true status of these 22 trials was impossible to verify because trial status information in registries was often incorrect. For example, 18/36 trials with published outcomes (50%) in our cohort remained listed as not concluded in registries because their sponsors had failed to update their status after trial end. Thus, we categorised the status of the remaining 22 trials as ‘uncertain’.
3.5 Primary outcome
The pre-specified primary outcome measure of this study was the number and percentage of concluded trials whose outcomes remained unreported. As we could not reliably determine whether 22 trials had concluded or not, we present our findings using two different baseline scenarios. The first scenario assumes that all 22 trials with uncertain status were still ongoing. The second scenario assumes that these 22 trials had all been concluded, i.e. completed or terminated. While we were unable to determine the precise number and percentage of concluded trials whose outcomes remained unreported, the actual figure lies within the range bracketed by these two scenarios.
Thus, out of 71 clinical trials of 3 COVID-19 vaccines developed by Chinese companies that received a World Health Organization Emergency Use Listing (EUL) and have been marketed globally, a minimum of 13 trials (27%) and a maximum of 35 trials (49%) remained unreported as of 14 April 2024. Note that these figures include 4 recently concluded trials that had not yet exceeded the one-year results disclosure timeframe set out by the WHO and may yet report their results in a timely manner.
3.6 Secondary outcomes
This study had two pre-specified secondary outcome measures. The first was the number and percentage of participants in concluded trials whose outcomes remained unreported. Again, we present minimum and maximum figures using the two baseline scenarios described above, using the actual (or where not available, planned) enrolment figures as stated in trial registries.
According to these calculations, out of the 313,750 people who participated in all 71 trials according to registry data, a minimum of 36,498 people (12%) and a maximum of 89,224 people (28%) had participated in trials that had been concluded and whose outcomes remained unreported as of 14 April 2024.
Our other secondary outcome measure was the number and percentage of trials that met both of two key WHO best practices in clinical trial reporting: posting outcomes onto an ICTRP-linked trial registry within 12 months of trial conclusion, and publishing outcomes in a peer-reviewed journal within 24 months of trial conclusion.
We expected to locate tabular summary results for some trials on the European EudraCT registry and/or on ClinicalTrials.gov. However, none of the trials in our cohort were registered on EudraCT, and none of the 62 trials registered on ClinicalTrials.gov had tabular summary results available there. We only located outcomes uploaded onto registries for 2 trials (IRCT20140818018842N24, IRCT20201214049709N3), both on the Iranian trial registry. In both cases, the responsible parties had copied and pasted a manuscript abstract into the results section of the registry following the manuscript’s publication in a peer-reviewed journal, rather than uploading tabular summary results. This approach improves the findability of trial outcomes. However, it fails to realise two other key advantages of registry reporting in tabular summary results format compared to journal publication: accelerated reporting timelines, and improved completeness and accuracy of outcome data (20-25). Thus, while between 2/49 (4%) of trials (Scenario 1) and 2/71 (3%) of trials (Scenario 2) technically met WHO best practices in clinical trial reporting, no trial fully delivered on the benefits of registry reporting.
4. Strengths and limitations
Our study has high relevance to public health and regulatory policy because it assesses the publication status and reporting gaps in clinical trials of 3 vaccines that have collectively been exported to dozens of countries worldwide and administered to hundreds of millions of people. However, it has several limitations. First, we relied on an external dataset generated by a different team of researchers to compile our initial study cohort of trials. Second, our follow-up period ranged from only 4 months to just under 4 years, making it likely that additional results will be published in future. Third, we may have missed relevant publications if these did not include the trial registry number and were in a language other than English. Finally, responsible parties had frequently failed to update trial registry entries as recommended by the WHO. This made it impossible to determine with certainty whether 22 of the trials in our cohort had been concluded. It also made it impossible to determine precisely how many people had participated in some concluded trials; we used planned enrolment figures where final enrolment numbers were not available.
5. Discussion
Our study shows that out of 71 clinical trials of 3 COVID-19 vaccines developed by Chinese companies that received a World Health Organization Emergency Use Listing (EUL) and have been marketed globally, a minimum of 13 trials (27%) and a maximum of 35 trials (49%) remained unreported post conclusion as of 14 April 2024, including at least 9 trials that had been concluded more than one year earlier. At first glance, the range of 27%-49% of trials remaining unreported appears to be broadly in line with non-reporting rates documented for large cohorts of clinical trials conducted in the United States and European Union (26-28). However, such direct comparisons would be misleading as those previous studies had longer follow-up periods and included a wide variety of trial types, including investigator-initiated studies of non-drug interventions that typically have below average publication rates.
Due to the lack of consistent updating of registry data, our finding that up to 89,224 people (28% of all trial participants) had been enrolled in trials that remained unreported should be treated with caution. The outcomes of only 2 trials (3% of all 71 trials) had been uploaded onto trial registries as recommended by the WHO.
Our findings suggest that there are significant gaps in clinical trial governance, especially in countries that have only recently emerged as significant players in biomedical research. Both commercial and non-commercial sponsors in the United States and many European countries have in recent years significantly improved their compliance with clinical trial disclosure requirements (29, 30). Legislation in both the United States and the European Union requires sponsors to disclose the outcomes of many interventional studies, including vaccine trials, in the form of tabular summary results uploaded onto a trial registry within a year of concluding a trial (31-33). The WHO also recommends this reporting format and timeframe.
In contrast, we found that the final outcomes for 9 COVID-19 vaccine trials remained unreported more than one year after the end of the trial. All of these trials had recruited participants exclusively in emerging hubs of biomedical innovation: Argentina, China, Chile, Peru, the Philippines, and the United Arab Emirates. Either these jurisdictions do not have adequate disclosure laws, or they do not effectively enforce legal compliance, or both. Similarly, none of the 36 reported trials in our study had reported outcomes on a registry using the tabulated summary results format recommended by the WHO; one of them (NCT05293665, sponsored by the American company Vaxxinity) appeared to be in violation of United States disclosure laws (34). Maintaining global progress in clinical trial reporting will require legislators and regulators worldwide, and in particular in emerging innovation hubs, to adopt and effectively enforce clinical trial reporting requirements that reflect WHO best practices.
Finally, our research highlights the urgent need to strengthen and adequately fund the global clinical trial registry infrastructure. The WHO-run International Clinical Trials Registry Platform (ICTRP) currently aggregates data from a network of 19 individual registries. Largely due to widespread data quality issues within individual registries, this infrastructure failed to provide a comprehensive, accurate and up-to-date overview of the global medical research landscape during the early months of the COVID-19 pandemic. This led to the proliferation of redundant studies and research waste on a large scale (35). Our study documents that these data quality issues persist. We were unable to reliably determine how many trials of 3 widely administered vaccines were still ongoing, how many had been concluded, how many people had participated in them, and what harms and efficacies had been detected. Notably, we were completely unable to access the Peruvian trial registry’s online platform during both assessment phases of this study. We were also unable to determine which countries two trials (NCT04984408, NCT05308576) were being conducted in. While there is no substitute for adequate funding, the Thai registry’s recent introduction of automated email reminders (36) and the ISRCTN registry’s forthcoming transparency tracker (37) illustrate that in some cases, significant improvements can be achieved at minimal cost.
Data Availability
Key data produced in the present work are contained in the manuscript (Table 7). All data produced are available online on OSF: https://osf.io/r3vp9/
Footnotes
↵† joint first authors
Clinical trials were de-duplicated, additional searches for publications were run, sponsor names and countries were added, and the entire dataset was updated. These updates led to changes in the conclusions section.
References
