Abstract
Objectives Prospectively registering study plans in a permanent time-stamped and publicly accessible document is becoming more common across disciplines and aims to reduce risk of bias and make risk of bias transparent. Selective reporting persists, however, when researchers deviate from their registered plans without disclosure. This systematic review aimed to estimate the prevalence of undisclosed discrepancies between prospectively registered study plans and their associated publication. We further aimed to identify the research disciplines where these discrepancies have been observed, whether interventions to reduce discrepancies have been conducted, and gaps in the literature.
Design Systematic review and meta-analyses.
Data sources Scopus and Web of Knowledge, published up to 15 December 2019.
Eligibility criteria Articles that included quantitative data about discrepancies between registrations or study protocols and their associated publications.
Data extraction and synthesis Each included article was independently coded by two reviewers using a coding form designed for this review (osf.io/728ys). We used random-effects meta-analyses to synthesize the results.
Results We reviewed k = 89 articles, which included k = 70 that reported on primary outcome discrepancies from n = 6314 studies and, k = 22 that reported on secondary outcome discrepancies from n = 1436 studies. Meta-analyses indicated that between 29% to 37% (95% confidence interval) of studies contained at least one primary outcome discrepancy and between 50% to 75% (95% confidence interval) contained at least one secondary outcome discrepancy. Almost all articles assessed clinical literature, and there was considerable heterogeneity. We identified only one article that attempted to correct discrepancies.
Conclusions Many articles did not include information on whether discrepancies were disclosed, which version of a registration they compared publications to, and whether the registration was prospective. Thus, our estimates represent discrepancies broadly, rather than our target of undisclosed discrepancies between prospectively registered study plans and their associated publications. Discrepancies are common and reduce the trustworthiness of medical research. Interventions to reduce discrepancies could prove valuable.
Registration osf.io/ktmdg. Protocol amendments are listed in Supplementary Material A.
Introduction
In 2000, ClinicalTrials.gov and the ISRCTN Registry were launched with several aims, including aiding participant recruitment, facilitating knowledge synthesis, and reducing duplication, publication bias and selective reporting (Deborah A. Zarin et al., 2017). In 2005, the International Committee of Medical Journal Editors (ICMJE) made prospective registration a condition of consideration for publication (De Angelis et al., 2004). Thousands of journals now claim to follow this policy (ICMJE, 2021). In parallel, the World Health Organization International Clinical Trials Registry Platform established a minimum set of required information for a trial to be considered fully registered, including experimental design elements such as the conditions being studied, intervention, key inclusion and exclusion criteria, sample size, primary outcomes, and key secondary outcomes (Sim et al., 2006). While the relatively widespread uptake of clinical trial registration has substantially improved transparency, many trials remain unregistered, are registered after enrollment of participants begins or analyses are complete (i.e., retrospective registration), are never published, or publish outcomes discrepant with those in the registration without disclosing the discrepancy (e.g., Chan et al., 2017; Scott et al., 2015). Nevertheless, the existence of registries allows researchers to identify and quantify these issues.
In research disciplines other than clinical trials, study registration is becoming more common, but remains far from standard practice (Hardwicke et al., 2021; Hardwicke, Wallach, et al., 2020; Scoggins & Robertson, 2023; Thibault et al., 2023). For example, starting around 2011 the field of psychology has increasingly taken the “replication crisis” seriously and many researchers and journals now use registration to reduce bias and make risk of bias transparent. Other disciplines have created dedicated registries, such as PROSPERO for systematic reviews and the American Economic Association RCT Trial Registry.
In this manuscript, we systematically reviewed articles that quantify the prevalence of discrepancies between registrations or study protocols and their associated publications (e.g., in primary outcome measures). Our analysis extended beyond the three systematic reviews already published on this topic in several ways (Dwan, Gamble, et al., 2013; C. W. Jones et al., 2015; G. Li et al., 2018). First, registration has expanded beyond clinical trials; we included all research disciplines and used key word searches for registries including the Open Science Framework, the American Economic Association RCT Trial Registry, and PROSPERO. Second, we extracted more fine-grained information about a wide range of discrepancies (e.g., outcomes, analysis, sample size), as well as which version of the registration was surveyed and whether discrepancies were disclosed (we believe disclosed discrepancies present little reason for concern). Third, our review included over twice as many studies as previous systematic reviews on this topic, provides meta-analytic estimates, and used meta-regression and additional analyses to attempt to identify predictors of discrepancies.
Methods
Terminology
We present a systematic review of k = 89 articles that assessed a wide range of outcome discrepancies and non-outcome discrepancies across over n = 7,000 studies. To avoid confusion, this report consistently uses the terms studies to refer to the over n = 7,000 individual studies that were assessed, and the term article to refer to the k = 89 articles that assessed these studies, and that we reviewed. We restrict our usage of the term publication to refer to the publications stemming from the studies (not to refer to the articles).
We use the term discrepancy to refer to any incongruity between the content of a publication and its associated registration (e.g., on clinicaltrials.gov) or study protocol (e.g., submitted to an ethics review board or funding agency)—see Box 1 for examples. We use the term prospective registration broadly to include terms used in different research disciplines, such as prospective trial registration, preregistration, and pre-analysis plans. All these terms indicate the registration of study details before commencing a study, or in some cases, before viewing the data or removing the blind. They are in contrast to retrospective registration, which occurs after participant enrollment begins or analyses are complete. We use the term outcome discrepancy to indicate a discrepancy in the outcome measure registered versus the outcome measure reported in a publication (not to indicate a discrepancy in the value of a reported outcome between these documents).
Box 1. Examples of discrepancies
We coded 10 types of outcome discrepancies (listed in Table 2) and 10 types of non-outcome discrepancies (listed in Table 4). The degree to which the information in a registration is discrepant with the information in a publication can range widelya. The associated concern about risk of bias can also range widely and often requires domain expertise to assess. We present two examples from Calméjane et al. (2018) in this box. Many more examples are available in the appendix of Calméjane et al. (2018) and at https://www.compare-trials.org/results.
Registered: Primary outcome: Visual Acuity [Time Frame: 6 months].
Published: Primary outcome: 3-year cumulative incidence rate of myopia. Myopia was defined as a spherical equivalent refractive error (sphere +½ cylinder) of at least -0.50 D
Coded as: Timing of outcome measurement changed.
Registered: Primary outcome: Severity of Device and Procedure related complications [Time Frame: At the time of ExAblate Transcranial thalamotomy procedure]. Secondary outcome: Effectiveness of the ExAblate Transcranial MRgFUS treatment determined using the Clinical Rating Scale for Tremor (CRST) [Time Frame: Participants will be followed from the date of treatment until study completion, approximately up to 12 months].
Published: Primary outcome: Change from baseline to 3 months in the tremor score for the hand derived from the CRST, Part A (three items: resting, postural, and action or intention components of hand tremor), and the CRST, Part B (five tasks involving handwriting, drawing, and pouring).
Coded as: Secondary outcome promoted to primary outcome; Timing of outcome measurement changed; Primary outcome omitted.
↵a Some researchers suggest that people checking for discrepancies should compare publications to clinical trial study protocols instead of registrations (Annals of Internal Medicine Editors, 2016). Study protocols are generally far more thorough and are more likely to be updated when trialists change their plans (which may occur before starting a study, but after registering). Study protocols contain timestamps, yet they often become public only at the time of publication.
Eligibility criteria
We included articles that reported quantitative data about discrepancies between registrations or study protocols and their associated publication. We excluded conference proceedings and articles written in a language other than English (for full inclusion and exclusion criteria, see our preregistered protocol).
Study selection
We searched Scopus and Web of Science on 15 December 2019 using the queries in Appendix A and Appendix B of our preregistered protocol (osf.io/ktmdg). Briefly, our queries included (1) variations of the terms preregistration, pre-analysis plans, and prospective registration in the title or keyword fields; (2) terms indicating discrepancies such as “outcome switching” in the title, keywords or abstract; (3) names of registration or protocol repositories such as “clinicaltrials.gov” in the title or keywords; and excluded overlapping but irrelevant terms (e.g., “nursing preregistration”). To limit the number of irrelevant articles, we did not search for variations of the term preregistration or for repository names in the abstract field.
Our search returned 4,283 articles after duplicates were removed (see Figure 1 for a PRISMA flowchart). Articles were screened independently by two reviewers in two stages. In Stage One, reviewers screened titles and, if necessary, briefly examined abstracts of articles to determine inclusion in the systematic review or in a scoping review on prospective registration1. If at least one of the reviewers deemed an article potentially relevant, it was included in Stage Two screening. In Stage Two, the reviewers independently examined the remaining 464 abstracts in greater detail for eligibility. Disagreements were resolved through discussion between the two reviewers and eventual consensus. Inter-rater agreemend for the 464 articles was Cohen’s k = 0.67 for inclusion in the systematic review (the list of articles and coding is available at osf.io/wa62f). Interrater agreement for all 4,283 articles was Cohen’s k = 0.72. We then used a snowball method and identified 33 additional articles that met our inclusion criteria, mostly through citations in G. Li et al. (2018) and C. W. Jones et al. (2015). These 33 additional articles are not included in the inter-rater agreement scores. After After full-text review, we included 89 articles in our systematic review.
PRISMA flowchart of article inclusion
Coding items
Each included article was independently coded by two of four reviewers (RTT, RC, OvdA, SW) using a coding form designed for this review. The form consisted of five sections that assessed (1) article characteristics, (2) study registration details, (3) 10 types of outcome discrepancies (listed in Table 2), (4) 10 types of non-outcome discrepancies (listed in Table 4), and (5) any additional descriptive or inferential statistics on discrepancies. The form details the operationalization of each variable we coded, and is available at osf.io/728ys. We chose items to code based on a pilot test of our protocol, as well as the categories used in a seminal paper (Chan, Hr’objartsson, et al., 2004) and a systematic review on discrepancies (G. Li et al., 2018).
The data we extracted was often presented as summary results in a table and sometimes as text in the results section. To be included in our meta-analyses this data had to include at least two of (1) the number of studies assessed (denominator), (2) the number of studies with a given discrepancy (numerator), and (3) the percentage of studies with a given discrepancy—from which we could calculate an unreported numerator or denominator. We did not access the raw data. If an article did not report data for a certain measure (e.g., secondary outcome discrepancies), then we did not include that article in the meta-analysis for that measure (this is why k varies among the meta-analyses we present—see Tables 2 and 4). For the meta-regressions we performed, we could not find data on (1) the version of the registry that publications were compared to for 35 articles, and (2) the number of studies that disclosed discrepancies for 57 articles. We coded these cases as “not reported” and included “not reported” as a factor in the meta-regressions. All other meta-regression data was complete. Further coding details are available in Supplementary Material B.
The complete dataset, including the coding of each reviewer and the resolved coding, is available at osf.io/ue2c6. A cleaned dataset with only the resolved coding is available at osf.io/6cn9m.
Statistical analyses
We performed two main random effects meta-analyses: one on the proportion of studies with at least one primary outcome discrepancy, and another on the proportion of studies with at least one secondary outcome discrepancy. We used random effects models because they allow for the true effect to vary across the populations the articles sampled from, and the articles we reviewed differ in their methodologies and the research disciplines that they assess. We used a random intercept logistic regression model with the Knapp-Hartung adjustment for the synthesis of proportions (J. Hartung & Knapp, 2003). We used the maximum-likelihood method for estimating the between-study heterogeneity (Tau). We also performed meta-regressions to test whether article characteristics are associated with the proportion of studies with at least one primary or secondary outcome discrepancy.
For pooled estimates, we report both confidence intervals and prediction intervals. Whereas researchers are likely more familiar with confidence intervals, interpreting confidence intervals can be unintuitive (Hoekstra et al., 2014), and their pooled-estimate does not incorporate uncertainty due to the between-article heterogeneity. If we assume that we could resample from our population, 95% of the resampled meta-analyses would yield a 95% confidence interval that contains the true value of the parameter being estimated (e.g., proportion of articles with at least one primary outcome discrepancy). Alternatively, if we are interested in the results that would come from another article assessing discrepancies, we would want a 95% prediction interval. In other words, of 100 articles drawn from the same population, we could expect the results from 95 of them—on average—to fall within the 95% prediction interval. While prediction intervals are not commonly reported, methodologists recommend reporting them for random effects meta-analysis, particularly when few articles are included or, as in our case, included articles are highly heterogeneous (Higgins et al., 2009; Riley et al., 2011).
Whereas we did not perform a formal risk of bias assessment—because our review differed substantially from the purpose these tools were built for—we shed light on a few potential sources of bias with additional analyses that consider the funding source, statistical significance, and the timing of registration of included studies. These additional analyses were not prospectively registered. We made a few amendments to our preregistered study protocol which are listed in Supplementary Material A.
Results
Articles characteristics
We identified and reviewed k = 89 articles that report at least one type of discrepancy. Articles that checked for outcome discrepancies assessed a median of 68 studies (IQR: 33 to 112). Article characteristics are outlined in Table 1. All articles except for two, one preprint in economics (Ofosu & Posner, 2019) and one preprint in psychology (Claesen et al., 2019), focused on clinical trials or systematic reviews. All but k = 10 articles were solely observational. Only one article attempted to correct published discrepancies (Goldacre et al., 2019). The authors assessed all trials published in five journals over a six-week period and sent a letter to the editor for each trial that published a discrepant outcome (for 58 letters in total). In the time since our literature search, at least two more interventional studies were published. One reports a trial that attempted to reduce discrepancies at medical journals by sending peer reviewers information about the study registration (C. W. Jones et al., 2022). They found null results. The other was a feasibility study that assigned a peer reviewer to specifically check for discrepancies in manuscripts submitted for publication (TARG Meta-Research Group & Collaborators, 2022). Further details about article characteristics are available in Supplementary Material C.
Article characteristics
Registration timing
Articles varied in the level of detail they provided about whether and when studies were registered. For example, whereas some articles presented their sample only after selecting for prospectively registered studies, other articles detailed their selection process including how many studies were registered and if so, when they were registered. Using the terminology in the articles we reviewed, articles identified studies that were registered retrospectively (k = 29), registered during participant enrollment (k = 17), registered after participant enrollment was complete (k = 14), and studies that were not registered (k = 36)2. Many articles were ambiguous regarding when some studies were registered (k = 47) and whether or not some studies were registered at all (k = 24). While these data do not provide fine-grained detail, they highlight two overarching issues: many studies are not registered, and many registered studies are registered retrospectively. These studies fail to meet the Declaration of Helsinki (World Medical Association, 2013) (item 35) requirement that “Every research study involving human subjects must be registered in a publicly accessible database before recruitment of the first subject” and the equivalent International Committee of Medical Journal Editors (ICMJE) policy (ICMJE, 2019), which thousands of journals claim to follow (ICMJE, 2021).
Eighty of the k = 89 articles we reviewed report at least one type of outcome discrepancy. Of these, 23 report only on studies that were unambiguously prospectively registered, 51 do not unambiguously distinguish between prospectively and retrospectively registered studies, and 6 report outcome discrepancies separately for each of prospectively and retrospectively registered studies. Separate meta-analyses for unambiguously prospectively registered studies and studies with unclear timing of registration are presented in Supplementary Material D.
Forty-six of the k = 89 articles report at least one non-outcome discrepancy (e.g., in sample size or analyses). Of these, 12 report only on studies that were unambiguously prospectively registered, 33 do not unambiguously distinguish between prospectively and retrospectively registered studies, and one reports non-outcome discrepancies separately for each of prospectively and retrospectively registered studies.
Primary outcome discrepancies
An estimated 29-37% (95% confidence interval) of the population of studies contained at least one primary outcome discrepancy (Figure 2). This equates to a 95% prediction interval of 10-68%.
Forest plot of articles reporting the proportion of assessed studies with at least one primary
This meta-analysis had high heterogeneity (I 2 = 86%), suggesting that the broad range of estimates across the articles stem largely from differences in the methodology of the articles or populations they sample from, rather than from chance. Heterogeneity could not be explained by meta-regression of any of the following article-level characteristics: discipline (p = 0.28)3, whether the publications were compared to registry entries versus other protocol formats (e.g., ethics applications) (p = 0.46), sources searched to identify studies (p = 0.65), version of the registry analyzed (p = 0.77), whether discrepancies were disclosed (p = 0.97), and year of article publication (p = 0.83). The meta-regression on discipline had low power because 63 articles assessed medical research and 7 assessed studies across dentistry, psychology, physical therapy, and economics. To increase statistical power, we reran this meta-regression after dichotomizing discipline and found that nonmedical disciplines may have a greater proportion of studies with at least one primary outcome discrepancy (p = 0.09; OR 95% CI: 0.91-3.19). We ran another meta-regression after dichotomizing the source which publications were compared to into registrations versus other protocols and did not find evidence to suggest this moderator played a role (p = 0.42). We also conducted a sensitivity analysis that included all six article-level characteristics in a single meta-regression. We found that publications compared to the most recent version of a registration may have a smaller proportion of studies with at least one primary outcome discrepancy, realtive to publications that were compared to the original version of a registration (p = 0.09; OR 95% CI: 0.32-1.07). All meta-regression model summaries are presented in Supplementary Material E.
The high heterogeneity in this meta-analysis may stem from genuine differences among the articles, including the sub-disciplines surveyed, specific sources searched, definition of a discrepancy (e.g., whereas some articles considered a change in the timing of an outcome as a discrepancy, others did not), and other article characteristics that may or may not have been reported. Our dataset contains more fine-grained information about the specific sub-discipline surveyed and specific sources searched. While we do not further explore these potential moderators in the present report, we note that, whereas some sub-disciplines and sources were highly specific (e.g., cystic fibrosis, lung cancer immunotherapy, Global Resource of Eczema Trials database), others were broad (e.g., medicine, clinicaltrials.gov, core clinical MEDLINE journals). We did not collect information on the exact definitions an article used to identify a primary outcome discrepancy. However, we did collect information on the proportion of articles with sub-categories of outcome discrepancies, which are more strictly defined and listed in Table 2 (e.g., promoting a secondary outcome to a primary outcome). We ran meta-analyses on these sub-categories of outcome discrepancies and found they also had high heterogeneity (Table 2). Thus, varying definitions are unlikely to be the main driver of the high heterogeneity in the present analysis on primary outcome discrepancies.
Meta-analytic estimates for the proportion of studies that contain various types of outcome discrepancies
Secondary outcome discrepancies
An estimated 50-75% (95% confidence interval) of the population of studies contained at least one secondary outcome discrepancy (Figure 3). This equates to a 95% prediction interval of 13-95%.
Forest plot of articles reporting the proportion of assessed studies with at least one secondary outcome discrepancy.
This meta-analysis also had high heterogeneity (I 2 = 90%) which could not be explained by meta-regression of the version of the registry analyzed (p = 0.8) or the year of article publication (p = 0.72). Metaregression of the sources searched to identify studies explained some heterogeneity, in that searches stemming from journals, compared to registries, had a greater proportion of publications with at least one secondary outcome discrepancy (p = 0.03; OR 95% CI: 1.89-13.45). Meta-regressions on discipline (p = 0.29), whether discrepancies were disclosed (p = 0.68), and whether the publications were compared to registry entries versus other protocol formats (p = 0.08) had very low statistical power because almost all articles had the same characteristic. All meta-regression model summaries are included in Supplementary Material E.
Descriptively, omitting secondary outcomes and adding secondary outcomes appears to occur more frequently than omitting primary outcomes, adding primary outcomes, or demoting primary outcomes, which in turn appear to occur more frequently than promoting a secondary outcome (see Table 2).
Parameters potentially related to outcome discrepancies
A subset of articles contained information on parameters potentially related to the proportion of outcome discrepancies. These include the disclosure of discrepancies, presence of a ‘statistically significant’ result, funding source, and timing of registration (Table 3).
Additional analyses regarding discrepancies
Non-outcome discrepancies
The meta-analyses for the non-outcome discrepancies had high heterogeneity, and wide confidence intervals and prediction intervals (Table 4). Articles varied in the criteria they used to identify non-outcome discrepancies and there were fewer articles than for outcome discrepancies. Prediction intervals can be particularly imprecise when few articles are included in a meta-analysis (IntHout et al., 2016). Whereas our coding procedure divided outcome discrepancies into 10 sub-categories, it did not employ the same level of granularity for non-outcome discrepancies. Supplementary Material H contains additional information on non-outcome discrepancies.
Meta-analytic estimates for the proportion of studies that contain various types of non-outcome discrepancies
Gaps in the literature
We identified several gaps in the literature on discrepancies. There exists little research on: (1) the prevalence of discrepancies in fields other than clinical research, (2) the prevalence of discrepancies in a representative sample across clinical disciplines, (3) the level of specificity in registrations, and (4) interventions to reduce undisclosed discrepancies (see Supplementary Material J for additional information about these gaps). We also identified several themes from surveying the conclusions of the articles we reviewed. These include the need for awareness surrounding discrepancies, the need for mandates, enforcement, and/or new initiatives to address discrepancies, and the benefit of registering additional information such as analysis plans (Supplementary Table J1 contains additional details).
Discussion
We found that outcome measures in registrations and study protocols often differ from published outcome measures, that these discrepancies are rarely disclosed, that the prevalence of discrepancies varies substantially across the articles we reviewed, and that this heterogeneity is not easily assigned to specific article characteristics.
Limitations
Given the wide range of discrepancy prevalence across individual articles, point estimates and confidence intervals may provide false precision when extrapolating our findings to the registered literature at large. Moreover, because heterogeneity could not be explained by meta-regression of article characteristics, more precise estimates cannot be derived for subsets of the literature. The prediction intervals can reasonably be used to extrapolate to another article in the registered medical literature at large, although the included studies do not necessarily form a representative sample.
Comparisons to previous research
Our main findings are in line with those from previous systematic reviews. These reviews included 27 articles each and found that 31% of studies had a primary outcome discrepancy in the median article they reviewed (C. W. Jones et al., 2015) and 54% of studies had any outcome discrepancy in the median article they reviewed (G. Li et al., 2018). The latter review did not distinguish between primary and secondary outcomes, and many articles they reviewed only assessed primary outcomes. Our review included all the articles contained in these systematic reviews, except for a few that did not meet our inclusion criteria (e.g., a PhD thesis, an abstract).
Implications
Our review raises broader issues regarding the efficiency of the research ecosystem and the trustworthiness of research outputs. We identified articles that documented discrepancies between publications and all of registrations, protocols, ethics applications, funding applications, and marketing approval applications. The existence of multiple documents outlining the same study raises the likelihood of discrepancies and, in the absence of a clearly demarcated ‘master’ document, leaves ambiguity regarding which document is ‘correct’. Rehashing the same study details for different audiences may also be an inefficient use of researchers’ time. Identifying a single publicly accessible document as the version of record (this could be the registration) and having all other documents point to this version of record for key information could reduce ambiguity and improve efficiency.
As for trustworthiness, registration has had a clearly positive influence on medical research (DeVito, 2022; Kaplan & Irvin, 2015). At the same time, some registration policies have poor adherence (e.g., many trials are registered retrospectively, and many trial results are never reported (DeVito et al., 2019)). The existence of research policies that are regularly overlooked, rarely monitored, and come with no consequence for non-compliance, can be damaging in at least two ways. They risk devaluing research policies altogether and they can reduce the trustworthiness of research outputs by creating a false impression that rigorous research practices were employed. Conceiving research as a complex ecosystem comprised of various agents with diverse incentives (e.g., funders, publishers, institutions, individual researchers) can help to comprehend why some policies have poor adherence and to develop and implement effective research infrastructure.
Conclusion
Registrations provide the evidence to detect selective reporting and outcome switching, which we found to be common. Nearly all articles we reviewed focused on documenting issues. Future efforts regarding discrepancies—and research improvement broadly—could prove more fruitful by shifting focus towards developing and testing solutions to these now well-documented issues.
Supplementary material
Funding
Robert Thibault was supported by a general support grant awarded to METRICS from Arnold Ventures and postdoctoral fellowships from the Canadian Institutes of Health Research (CIHR) and the Fonds de recherche du Québec - Santé (FRQS). Hugo Pedder was supported by funding from the National Institute of Health and Care Excellence’s Guidelines Technical Support Unit and Bristol’s National Institute for Health and Care Research Technology Assessment Group. Olmo van den Akker was supported by Consolidator Grant (IMPROVE) from the European Research Council (ERC; grant no. 726361). Marcus Munafò, Robert Thibault, Jacqueline Thompson, and Robbie Clark were part of the MRC Integrative Epidemiology Unit (MC_UU_00011/7). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Transparency statement
Robert Thibault, the manuscript’s guarantor, affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned have been explained.
Data, data dictionaries, analysis script and materials related to this study are publicly available on the Open Science Framework at https://osf.io/5gfty/. The study protocol and materials were registered on 17 January 2020 at https://osf.io/ktmdg. Discrepancies between this manuscript and the registered protocol are outlined in the Supplementary Material A. To facilitate reproducibility, this manuscript was written by interleaving regular prose and analysis code using R Markdown. The relevant files are available in a Code Ocean container (https://doi.org/10.24433/CO.4753181.v1) which recreates the software environment in which the original analyses were performed. This container allows this manuscript to be reproduced from the data and code with a single button press.
Competing interests
None declared
Contributors
Conceptualization: Robert T. Thibault(1,2,†), Jacqueline M. Thompson(2,9), and Marcus R. Munafò(2,3).
Data curation: Robert T. Thibault.
Formal analysis: Robert T. Thibault and Hugo Pedder(4,5).
Funding acquisition: Robert T. Thibault.
Investigation: Robert T. Thibault, Robbie Clark(2,3), Olmo van den Akker(6,7), and Samuel Westwood(8).
Methodology: Robert T. Thibault, Robbie Clark, and Marcus R. Munafò.
Project administration: Robert T. Thibault.
Software: Robert T. Thibault and Hugo Pedder.
Supervision: Robert T. Thibault and Marcus R. Munafò.
Validation: Robert T. Thibault.
Visualization: Robert T. Thibault.
Writing - original draft: Robert T. Thibault.
Writing - review & editing: Robert T. Thibault, Robbie Clark, Hugo Pedder, Olmo van den Akker, Samuel Westwood, Jacqueline M. Thompson, and Marcus R. Munafò.
(1) Meta-Research Innovation Center at Stanford (METRICS), Stanford University
(2) School of Psychological Science, University of Bristol
(3) MRC Integrative Epidemiology Unit at the University of Bristol
(4) Population Health Sciences, University of Bristol
(5) Bristol Medical School, University of Bristol
(6) Department of Methodology & Statistics, Tilburg University
(7) QUEST Center for Responsible Research, Berlin Institute of Health
(8) Department of Psychology, Institute of Psychiatry, Psychology, Neuroscience, King’s College London
(9) Bodleian Libraries, University of Oxford
†Correspondence should be addressed to Robert Thibault: robert.thibault{at}stanford.edu
Data Availability
Data, data dictionaries, analysis script and materials related to this study are publicly available on the Open Science Framework at https://osf.io/5gfty/. The study protocol and materials were registered on 17 January 2020 at https://osf.io/ktmdg. Discrepancies between this manuscript and the registered protocol are outlined in the Supplementary Material A. To facilitate reproducibility, this manuscript was written by interleaving regular prose and analysis code using R Markdown. The relevant files are available in a Code Ocean container (https://doi.org/10.24433/CO.4753181.v1) which recreates the software environment in which the original analyses were performed. This container allows this manuscript to be reproduced from the data and code with a single button press.
Footnotes
Content moved from the supplementary material to the main manuscript. Additional supplementary material provided. Text revised for clarity. The data and analyses remain the same as in version 1.
↵1 We had planned to report on this scoping review separately. However, the scoping review has been terminated (details provided at https://osf.io/ktmdg).
↵2 By definition, non-registered studies cannot be assessed for discrepancies between their registration and publication. Several articles identified non-registered studies in their sampling process, but did not include these studies in their final sample.
↵3 If readers want to find if a study on discrepancies exists within a particular sub-discipline, we invite them to explore the variable ‘disciplineDetail’ in our dataset.
References
