ABSTRACT
Research transparency is crucial for ensuring the relevance, integrity, and reliability of scientific findings. However, previous work indicates room for improvement across transparency practices. The primary objective of this study was to develop an extensible tool to provide individualized feedback and guidance for improved transparency across phases of a study. Our secondary objective was to assess the feasibility of implementing this tool to improve transparency in clinical trials.
We developed study-level “report cards” that combine tailored feedback and guidance to investigators across several transparency practices, including prospective registration, availability of summary results, and open access publication. The report cards were generated through an automated pipeline for scalability. We also developed an infosheet to summarize relevant laws, guidelines, and resources relating to transparency. To assess the feasibility of using these tools to improve transparency, we conducted a single-arm intervention study at Berlin’s university medical center, the Charité – Universitätsmedizin Berlin. Investigators (n = 92) of 155 clinical trials were sent individualized report cards and the infosheet, and surveyed to assess their perceived usefulness. We also evaluated included trials for improvements in transparency following the intervention.
Survey responses indicated general appreciation for the report cards and infosheet, with a majority of participants finding them helpful to build awareness of the transparency of their trial and transparency requirements. However, improvement on transparency practices was minimal and largely limited to linking publications in registries. Investigators also commented on various challenges associated with implementing transparency, including a lack of clarity around best practices and institutional hurdles.
This study demonstrates the potential of developing and using tools, such as report cards, to provide individualized feedback at scale to investigators on the transparency of their study. While these tools were positively received by investigators, the limited improvement in transparency practices suggests that awareness alone is likely not sufficient to drive improvement. Future research and implementation efforts may adapt the tools to further practices or research areas, and explore integrated approaches that combine the report cards with incentives and institutional support to effectively strengthen transparency in research.
Introduction
The inability to reproduce scientific findings (Baker, 2016; Begley and Ellis, 2012; National Academies of Sciences, Engineering, and Medicine, 2019; Open Science Collaboration, 2015) and the non-or incomplete reporting of research results (Bassler et al., 2016; Franco et al., 2014; Ioannidis et al., 2014; Song et al., 2010; Toews et al., 2017) have raised concerns about research waste, and highlighted the need to promote responsible research practices as a means to increase research value. Transparency is core to responsible research: transparency practices allow the quality of research to be appraised, mitigate biases, discourage poor methodological practices, and facilitate comprehensive evidence synthesis (Bradley et al., 2020; Glasziou et al., 2014; Munafò et al., 2017; Nosek et al., 2015). The relevance of these practices is emphasized in initiatives to reform research assessment (Hicks et al., 2015; “San Francisco Declaration on Research Assessment (DORA),” 2012), which have included calls to increase recognition of researchers who commit to robust and transparent practices (Moher et al., 2020). One of the principles underlying the 2022 Agreement on Reforming Research Assessment is to focus research assessment criteria on quality, where “research is carried out through transparent research processes and methodologies” (Coalition for Advancing Research Assessment, 2022).
Reforming how research is performed and assessed requires evaluation and communication about the current uptake of responsible research practices, including transparency. Such monitoring efforts largely focus on a single phase of the research process (e.g., results reporting) and communicate findings at a group level, such as institution or country. For example, the EU Trials Tracker shows compliance of clinical trials with results reporting requirements and presents these data as a dashboard at the level of institutional sponsors (Goldacre et al., 2018). In turn, the French Open Science Monitor shows the evolution at the national level of several open science practices, such as open access and mentions of data sharing in publications (Jeangirard, 2019). These initiatives and others (Barbers et al., 2022; Diprose et al., 2023; Franzen et al., 2023; QUEST Center for Responsible Research, Berlin Institute of Health at Charité Universitätsmedizin Berlin, n.d.) provide baseline information against which change can be measured and empower efforts to drive research transparency.
Policy and monitoring activities reflect top-down approaches to inform and trigger change in research transparency. Yet improvement ultimately requires researchers to take specific actions, such as report results and methods in a timely manner and in sufficient detail. Thus, effective research reform also requires bottom-up approaches to empower researchers through feedback and guidance on transparency at the level of individual studies. Efforts to increase the transparency of individual studies largely focus on publications. Journals have experimented with awarding authors open science badges to incentivize practices such as open data (Kidwell et al., 2016; Rowhani-Farid et al., 2020). The ScreenIT pipeline screens preprints in medRvix and bioRxiv for common problems in how research is reported and provides rapid feedback to help authors improve transparency and reproducibility (Schulz et al., 2022).
Building on these efforts, we aimed for a complementary approach to provide feedback and guidance to investigators on transparency practices associated with different phases of a study. While such a feedback tool would be beneficial across research areas and study types, our feasibility study focused on clinical trials. Clinical trials are well suited to evaluate the feasibility and usefulness of such a tool for both normative and technical reasons. The significance of clinical trials to patient health has inspired ethical guidelines and legal requirements for transparency practices across the research lifecycle, from prospective registration to timely results in registries and journal articles (International Committee of Medical Journal Editors, 2023; World Health Organization, 2017a; World Medical Association, 2013). In turn, clinical trial registries provide a ready list of studies and collect data relating to those trials, from study start to completion, which enables evaluation of these practices (Saberwal, 2021; World Health Organization, 2017b; Zarin et al., 2017). In previous work, we developed tools to monitor and communicate performance on these practices at the level of institutions, which could be adapted to the level of individual studies (Franzen et al., 2023).
This study’s primary objective was to develop a scalable tool to provide feedback and guidance on transparency across research phases of an individual study. We designed individualized “report cards” that summarize the performance of a given clinical trial across several transparency practices and provide tailored guidance for improvement. We also created an infosheet with an overview of recommended transparency practices and relevant guidelines and regulations.
The secondary objective was to investigate the feasibility of implementing these tools to improve trial transparency. We conducted an exploratory, single-arm intervention and disseminated individualized trial transparency report cards and the infosheet to investigators at the Charité – Universitätsmedizin Berlin, in collaboration with the institutional Clinical Study Center. We surveyed investigators on the self-reported usefulness of the materials and evaluated the included trials for improvements in transparency.
Results
Tools: report cards and infosheet
We developed study-level report cards that combine individualized feedback and guidance on the transparency of a given trial (Figure 1 (a)). The report cards provide feedback on the following transparency practices: a) prospective registration, b) availability and timeliness of summary results in the registry, c) availability and timeliness of a results publication (earliest if multiple), and if a publication exists, d) links between registration and the results publication, and e) open access status of the publication. In case of a validated cross-registration in the EU Clinical Trials Register (EUCTR), the report card additionally displays the following practices based on the EUCTR registration: a) prospective registration, and b) availability of summary results in the registry. For transparency practices that can still be improved after trial completion, the report card includes a call-to-action and a link to resources to implement the practice (e.g., instructions in the relevant registry).
We also developed an infosheet with an overview of guidelines and laws relating to clinical trial transparency, tailored to the national and institutional contexts (Figure 1 (b)). The infosheet summarizes transparency practices relating to trial registration and reporting, their normative basis, and available resources to implement each practice.
The final versions of the report card and infosheet incorporated feedback from expert reviews and cognitive interviews. Suggestions for improvement from participants largely related to the visual design, scope, and implementation of the tools. Themes are elaborated in Supplement 1 Table 1.
Feasibility study
Sample characteristics
To investigate the feasibility of using report cards and an infosheet to improve trial transparency, we started with a dataset of 2,909 trials generated in previous studies (known as “IntoValue”; see details in the Methods). After limiting this set to trials led by the Charité and completed between 2014 and 2017, we contacted 128 investigators responsible for 168 trials. After excluding unreachable investigators, our analysis sample included 92 investigators responsible for 155 trials. Figure 2 (a) details screening of both trials and investigators. Characteristics of the included trials and investigators are in Supplement 2 Table 1.
Survey
A total of 37 researchers (40% of the 92 contacted) completed the survey in the 12 weeks between survey launch on 25 May 2022 and study close on 17 August 2022. Figure 2 (b) shows a flowchart of investigators and exclusion from analysis at each stage. Supplement 3 Figure 1 shows the response timeline across the survey fielding period. The survey took respondents generally just under 4 minutes, with a median response time of 237 seconds (IQR: 124 - 402). The majority of respondents self-reported as study leads or study doctors (n = 32, 86%); see Supplement 3 Figure 2 for additional details on respondent The survey asked trial investigators to rate their agreement with each of eight statements about the report card and infosheet (for example, “The infosheet is clear.”). Respondents shared overall positive perceptions about the report card and infosheet, with at least 49% agreeing or strongly agreeing with all provided statements, and mean scores on all individual items ranging from 3.2 to 3.7, on a scale from 1 to 5. A peak of 73% of respondents found the infosheet with an overview of relevant laws, guidelines, and resources generally valuable. The statement with the lowest agreement (49%) and highest disagreement (38%) was whether the report cards were helpful to improve their own trial’s transparency. Figure 3 shows responses to each item; means with 95% confidence intervals are presented in Supplement 3 Figure 3.
A subgroup analysis of 3 participants who reported not reviewing the materials before starting the survey showed the most negative option (i.e., strongly disagree) across all statements on both the report card and infosheet. Another subgroup analysis showed no meaningful difference between study leads and doctors (n = 32) versus other respondents (n = 5). See Supplement 3 for further details on these subgroup analyses.
Respondents were invited to share comments on the materials. After removing non-substantive responses (e.g., “I have no comment”), 17 remained. Comments included positive remarks and suggestions for improvement of the materials, as well as broader challenges around the management and coordination of clinical trials. Supplement 3 Table 1 details the suggestions and challenges provided in comments.
Investigators also had the opportunity to provide corrections to their report card. In our manual review of the 18 provided corrections, we found that only 5 were valid, while the remaining were incorrect understandings of the practices or additional information on the trials beyond the scope of the report card. See Supplement 3 for details.
Improvement in transparency practices
Trial transparency improved minimally across the follow-up period for eligible trials (see Figure 2 (c) for a screening flowchart for eligible trials). The largest improvement was in linked publications, which increased 5.9%: 5 trials added links to their publications in the registry in the 3 months following our intervention, and 1 additional trial did so within 1 year of the intervention. One of these trials also posted summary results in the registry. While these changes were small, no changes were seen in the 3-months preceding the intervention (Figure 4). Additionally, one publication became openly accessible on the journal website and not a repository, suggesting this was not triggered by an investigator and was hence independent of our intervention.
Methods
Tools: report cards and infosheet
Design
We set out to develop report cards with individualized feedback on the transparency of a given trial. The report cards aimed to: a) provide feedback on registration and reporting best practices at the level of an individual trial, b) indicate visually (e.g., tick mark, cross) whether a practice was followed, c) include the trial details relevant to each practice, d) be succinct (max. 1 page), and e) be generated with open source code for scalability. We also designed an infosheet that provides an overview of guidelines and laws relating to clinical trial transparency. The infosheet aimed to: a) provide an overview of the recommended or required transparency practices and their normative basis (i.e., guidelines or laws), b) include links to resources for each practice, c) be tailored to the German and institutional context, and d) be succinct (max. 1 page). The design of the report cards and infosheet was shaped by feedback obtained in cognitive interviews and from experts in clinical trial transparency Supplement 1. The report cards and infosheet were presented in English.
Trial transparency dataset
The report cards require a structured trial transparency dataset, i.e., a dataset of clinical trials and their performance across several transparency practices. We previously developed methods to assess transparency practices for a cohort of trials using a combination of automated and manual approaches, which are described in detail elsewhere (Franzen et al., 2023; Franzen, 2023; Riedel et al., 2022; Salholz-Hillel et al., 2022). Supplement 4 summarizes the main steps to prepare a trial transparency dataset for use with the report cards.
Technical implementation: translating the dataset into study-level report cards
We developed a pipeline to automatically generate individualized report cards based on the trial transparency dataset (see Supplement 4 Figure 1). In the first step, we designed a template of the report card in Affinity Designer, a vector graphics editor. The report card template is a visual representation of a trial’s performance across transparency practices. Since trials perform differently across transparency practices, the report card template includes all possible outcomes of a trial’s performance represented as layers. The report card template was exported as Scalable Vector Graphics (SVG), which is an image format based on Extensible Markup Language (XML). This allows the report card template to be accessed and modified using code. In the second step, report cards for each trial were assembled automatically using a custom-made Python script that selects the correct layers to include for each trial based on the trial transparency dataset. In the third step, each report card was automatically exported as a PDF for dissemination to study participants. An example report card is displayed in Figure 1 (a). The infosheet (Figure 1 (b)) was generated in Affinity Designer and Affinity Publisher and adapted on an iterative basis.
Feasibility study
We conducted a feasibility study to pilot the use of the tools at the Charité – Universitätsmedizin Berlin, in collaboration with the institutional Clinical Study Center. The study used a cross-sectional, interventional design and included a descriptive analysis of two main outcomes: self-reported evaluation of the tools via a survey, and improvement in transparency practices following the intervention of the tools. All trials received the intervention (single-arm), as the number of trials eligible to be included in our analysis was too small for a control group and sufficient power to detect a minimum effect size of interest.
Ethical approval and study protocol
This study was approved by the ethics review board of the Charité – Universitätsmedizin Berlin (#EA1/337/21). The study protocol was preregistered and is openly available in the Open Science Framework (OSF) at: https://doi.org/10.17605/osf.io/stnp5. Deviations to the protocol are outlined in OSF: https://osf.io/wxayg.
Sample development
This study used a cohort of trials and associated investigators. Screening criteria were assessed for both trials and investigators, in part prior to study launch and in part prior to analysis. Criteria are outlined below and detailed in Supplement 2 Table 2.
Trials
This study draws from two cohorts of registered clinical trials and associated results, referred to as “IntoValue” (Riedel et al., 2021). The IntoValue dataset consists of clinical trials led by a German UMC (i.e., as either sponsor, responsible party, or host of the principal investigator), registered in ClinicalTrials.gov or the German Clinical Trials Register (DRKS), and considered as complete per the registry between 2009 and 2017. For this feasibility study, the IntoValue dataset was limited to trials led by the Charité with a completion date between 2014 and 2017 (i.e., the second IntoValue cohort); these more recently completed trials were selected, given the higher likelihood of reaching investigators and increased feasibility of implementing practices. To reflect the most up-to-date status of trials, we downloaded updated registry data for the trials in this cohort on 19 May 2022 and reapplied the original IntoValue exclusion criteria: study completion date before 2014 or after 2017, not considered as complete based on study status, and not interventional. Trials were excluded if no Charité investigators were found or reached.
Trial investigators
We developed a list of investigators for our sample of trials based on contact information retrieved from clinical trial registries. We used the following registry fields: from ClinicalTrials.gov, responsible party (if this included principal investigator), and overall study officials (principal investigator, study director, study chair); from DRKS, primary sponsor, and contacts for scientific and public queries. As contact information is often removed from registrations after trial completion (O’Neill et al., 2014; Viergever et al., 2014), we used the R package 
(Carlisle, 2022) to retrieve the current (as of 19 May 2022) as well as all historical versions of registrations. If no contact information was available in the registry, we searched Google and the Charité address book using the investigator’s name in the registry and the Charité email pattern, as well as trial details such as trial registration number and title keywords as needed. If multiple investigators or multiple emails for the same investigator were available, all were used; if the registration provided an email for the project or research group, this was included in addition to the investigator email(s). Investigators were excluded if no contact information was found, and trials were excluded if no contact information for any trial investigator was found. Per our preregistration, after the email-based intervention, investigators were excluded if all emails were undeliverable or if auto-replies indicated extended leave throughout our study timeframe; trials were then excluded if no trial investigator remained.
Materials
Transparency practices and report cards
In line with the requirements for the report cards (see “Trial transparency dataset”), the IntoValue trial cohort includes data on transparency practices. Data obtained through automated methods were updated at several time points during the study. Registry data for ClinicalTrials.gov was queried from the AACT (Clinical Trials Transformation Initiative, n.d.), and DRKS was webscraped prior to the start of the intervention and at the end of follow-up. At study end, we additionally used 
(Carlisle, 2022) to get historical versions of registrations in order to complete the analysis of each study time point. To determine the open access status, we queried the Unpaywall API (using UnpaywallR (Riedel and Franzen, 2022)) at each study time point; to determine publisher permissions for self-archiving, we queried the Shareyourpaper API (using a custom-made Python script (Franzen, 2023)) at study launch. Additional manual validations were done for the open access status of publications as well as for cross-registration in the EUCTR. Protocols for these validations were prospectively uploaded on OSF (https://osf.io/3upgy and https://osf.io/mjpzh), and methods are summarized in Supplement 5. The resulting trial transparency dataset was used to generate the report cards. After study launch, we found a bug in the code and subsequently made 6 amendments to the dataset and report cards, namely we added: 1 cross-registration in EUCTR, 4 publication links in DRKS, and 1 summary result in DRKS. More detailed information can be found in the project repository in GitHub: https://github.com/quest-bih/trackvalue.
Questionnaire development and pretesting
We conducted an anonymous survey to assess the usefulness of the report cards and infosheet. The survey was generated in LimeSurvey and hosted on a server at the QUEST Center for Responsible Research at the Berlin Institute of Health at Charité – Universitätsmedizin Berlin. After informed consent and prior to starting the survey, respondents were asked whether they had reviewed the materials, and if not, they were prompted to do so and return to the survey. Respondents were asked to evaluate the report card and infosheet using a 5-point Likert scale across four items for each tool. Respondents were also asked to indicate their role(s) in the trial, and were invited to provide corrections to the report card(s), if applicable, as well as general comments. The survey was presented in German. The design of the survey was shaped by feedback obtained in cognitive interviews (see Supplement 1). The survey is provided in OSF (https://osf.io/5z463).
Administration of the intervention and survey
Investigators were sent an invitation email including: a letter (co-signed by the Charité Clinical Study Center) listing all their trials, a report card for each trial they led, and an infosheet. The email and letter also included an invitation to participate in the anonymous survey as well as a notice that the transparency of their trials would be re-evaluated 3 months after the first email (see a sample invitation in OSF: https://osf.io/9qkdp). Emails were sent from a functional mailbox created for the study that was regularly monitored over the course of the study. Any auto-replies indicating undeliverable emails or extended leave were preserved. Follow-up emails were sent at weeks 1, 5, and 11. As the survey was anonymous, and we did not know who had already completed it, all reminders were sent to all investigators included in the study. We reviewed the survey responses on an ongoing basis to identify any valid corrections to the report cards provided by investigators. For valid corrections, we updated our dataset and generated and sent amended report cards to all investigators associated with the trial.
Analysis
Analysis of perceptions of the report card and infosheet
We describe investigators’ perception of the report card and infosheet per the survey. Our primary outcome is how much respondents agree with four positive statements about each tool. Statements (for example, “The infosheet is clear.”) were presented with a 5-point Likert-type response scale, with options ranging from 1 = strongly disagree, to 5 = strongly agree. We report the proportion and means with 95% confidence intervals of each Likert response. Survey responses were considered complete if all Likert items were completed, and as all Likert items were required, we did not evaluate missingness. We report the survey response rate, the response timeline across the survey fielding period, and exclusion from analysis at each stage (e.g., due to undeliverable emails). As the survey was anonymous, we could not evaluate non-response error or adjust for possible non-representativeness of the respondents; we could not determine whether unique respondents came from unique visitors. We evaluated response time as a quality check for active participation and to avoid counting bots. We report the number of respondents who reported not reviewing materials before starting the survey and perform a subgroup analysis of these respondents. We also perform a subgroup analysis of study leadership. We report the number of corrections, and evaluate and report the number which are valid; corrections were reflected in subsequent report card versions. We also report the number of comments, after removing non-responses and describe common themes identified in respondent feedback. Corrections and comments were reviewed and adjudicated by consensus by two researchers (MSH a
Analysis of improvement in transparency practices
We assessed trials for three transparency practices that could still be changed after trial completion: 1) reporting of summary results in the registry, 2) publication links in the registration, and 3) open access status of publication. The publication-based analyses were limited to trials with publications available at the time of intervention launch; the open access analysis was further limited to trial publications that could be self-archived in an institutional repository based on publisher policies (“green open access”) at the time of intervention launch. We report the overall proportion of trials performing each practice at the following time points relative to our intervention: 3 months prior to launch, at launch, 3 months after launch. As a protocol deviation, we additionally report rates a year after the intervention.
Software, code, and data
All the code generated in this study is openly available under an open license in GitHub: https://github.com/quest-bih/trackvalue. Data collection was conducted in Python [Version 3.9] and R [Version 4.3.2], and all analysis was conducted in R. This manuscript was written by interleaving regular prose and analysis code in Quarto [Version 1.4.549]. With the exception of investigator contact information and survey comments and corrections, all the data is openly available in the GitHub repository of the project and Zenodo: https://doi.org/10.5281/zenodo.10467054.
Reporting guidelines
This study uses both an intervention and a survey, and follows the CROSS Consensus-Based Checklist for Reporting of Survey Studies (Sharma et al., 2021) and CONSORT for Pilot and Feasibility Trials (Eldridge et al., 2016). Reporting checklists are available in Supplement 6.
Discussion
We developed scalable, open source tools to provide individualized feedback and guidance on research transparency across phases of a clinical trial. We investigated the feasibility of using these tools to improve clinical trial transparency in a sample of completed trials led by researchers at the Charité – Universitätsmedizin Berlin, one of the largest university hospitals in Europe (https://www.charite.de/en). Overall, researchers found the tools helpful to inform them of the transparency of their trial and to raise awareness of relevant guidelines, laws, and resources. However, improvement across transparency practices one year after dissemination of the tools was limited. This study demonstrates the feasibility of generating and communicating study-level feedback to researchers, and can inform future interventions to improve transparency and the adoption of responsible research practices more broadly.
Summary of findings
Perceptions of the report card and infosheet
Survey responses indicated that the report cards and infosheet were generally well received by study participants. The infosheet, in particular, stood out as being a valuable resource, suggesting that succinct visual summaries of what is expected of investigators (based on applicable laws, guidelines, and policies) may be a fruitful approach to build awareness of transparency requirements alongside other forms of training. Participants’ views on the report cards were largely positive. Lower ratings on the usefulness of the report cards to improve trial transparency may reflect the need for additional support or the removal of barriers to implement changes in practice. We attempted to bridge this gap by linking to detailed guidance on how to perform each practice.
Transparency improvements in practice
Trial transparency improved minimally over the follow-up period, with improvements largely limited to linked publication in the registry. The discrete improvement for this practice, which involves manually adding a link to a results publication in the registration, may reflect the lower time and effort needed compared to other practices, or perceived hurdles associated with summary results reporting (e.g., fear of jeopardizing journal publication (Bruckner, 2019)) and self-archiving (e.g., fear of copyright infringement (Swan and Brown, 2005)). Furthermore, our study called for action on trials reported as complete in the registry up to 8 years prior to this intervention, which may have been too long ago for these practices to be feasibly addressed. Prospective reminders while a study is ongoing or for recently completed studies, as already implemented in several registries (e.g., ISRCTN (Taylor, 2019)), may be more effective in driving change.
Perceived barriers to implementing transparency practices were noted in investigators’ survey responses. While the survey did not aim to solicit feedback beyond the tools, numerous respondents volunteered several challenges (see Supplement 3 Table 1), including lack of clarity around best practices (e.g., applicable guidelines and/or laws in case of multiple registrations of a single trial), diffusion of responsibility (e.g., limited control over transparency practices due to journal guidelines), and obstructive institutional processes and bureaucracy around the conduct of trials, which take time and resources away from other activities. Taken together, investigators’ positive perception of the materials yet limited improvement in transparency practices, as well as unprompted comments on challenges, suggests that implementing research transparency is a multifaceted issue and that interventions likely need to be embedded within systemic changes.
Research in context
Previous empirical research has explored the impact of interventions at the level of individual studies to increase clinical trial transparency. Maruani and colleagues conducted a randomized controlled trial in 2014 to evaluate the impact of sending email reminders to investigators of completed Phase IV trials about their requirements to post results (Maruani et al., 2014). At six months following their intervention, results had been posted for 24% of trials that had received the intervention versus 14% in the control group, suggesting that reminders may effectively prompt results reporting. A 2023 study at the German registry DRKS similarly found that email reminders were effective in prompting investigators to complete draft registrations: 5 weeks after the intervention, 11% of trials that had been sent an email had completed their registrations, compared with 4% of trials that received no communication (Bieselt, 2023).
Approaches such as report cards or reminders may have greater impact when embedded within a holistic behavior change approach that brings together capability, motivation, and opportunity (Michie et al., 2011), as suggested by previous work across research, policy, and practice. For example, the launch of the EU Trials Tracker as a public audit tool was complemented by an advocacy campaign, which eventually spurred policymakers and other stakeholders to demand change from institutions (Lamb, 2019). Research performing institutions subsequently rapidly improved their capacities to address these practices, and within a few years, universities policies had been strengthened (Keestra et al., 2022), and trial reporting increased from 50% in 2018 to 84% in September 2023 (Bennett Institute for Applied Data Science, n.d.). Additional external factors further reinforced opportunity: in February 2020, the Make It Public strategy was adopted in the United Kingdom, whereby clinical trials are automatically and centrally registered by the Health Research Authority following ethics approval, which saves researchers this extra step (Health Research Authority, 2020).
In the United States, efforts to understand existing support structures for trial registration and reporting (Mayo-Wilson et al., 2018) were accompanied by improvements in transparency at academic organizations. Institutional programs to support compliance with registration and reporting requirements launched a variety of activities, including audits to identify problem records and bring them into compliance, prospective reminders, policy development, enforcement, standard operating procedures, centralized resources, training, and consultation (Keyes et al., 2021; O’Reilly et al., 2015; Snider et al., 2020). Following the implementation of such a program at Johns Hopkins University School of Medicine, non-compliance for mandated registration and reporting dropped from 44% (339/774) in 2015 to 2% (32/1,304) in 2020 (Keyes et al., 2021). More broadly, the emergence of interventions implemented by key stakeholders at different levels offers a rich opportunity to investigate effective, multifaceted strategies for research improvement.
Strengths, limitations, and challenges
Our study has several strengths. The report cards offer a single resource that brings together individualized feedback and guidance on the implementation of specific practices. The approach to generate the report cards is openly available and can be scaled to other institutions and contexts that have, or are seeking to develop, an overview of their studies and associated practices. Importantly, this approach leverages automated approaches to drive transparency at scale, while not being tied to a specific stakeholder or infrastructure. Moreover, we collaborated with an institutional core facility: this approach may be a strength by increasing institutional buy-in and the likelihood of sustainable adoption of such a tool; however, it may also have shifted investigators’ attention away from the tools towards institutional aspects.
Our study also faces several limitations. The report cards rely on the information in the registry being accurate and up-to-date, which is not always the case (DeVito and Goldacre, 2022; Fleminger and Goldacre, 2018; Viergever et al., 2014). For EUCTR trials, prospective registration was assessed based on dates available in the registry; however, in principle all EUCTR trials should be prospectively registered, as registration is linked to regulatory approval by national competent authorities. The small sample size in this study precluded randomization and a control group with sufficient power to detect a minimum effect size of interest. The design therefore did not allow us to test whether our intervention drove the changes (i.e., causality), which may also have been prompted by researchers themselves or other co-occurring pressures in the German clinical trial landscape (Bruckner, 2022). However, as our aim was to test the feasibility of delivering such an intervention, the lessons learned may inform a larger randomized controlled trial. Another limitation of our study is that the invitation letter and survey were only provided in German, while the report card(s) and infosheet were provided in English. This may have limited the reach or understandability of the materials. Future studies may consider generating multilingual materials.
We identified several challenges related to material development and the intervention, which may impact the scalability of this type of intervention. Despite the use of automated approaches, additional manual checks were still required to validate the data. Another challenge was balancing comprehensiveness and clarity in the infosheet; this could be supported by engaging visual design experts and by using dynamic formats such as HTML instead of static PDFs. Additionally, step-by-step instructions on how to implement transparency practices were not available across all registries at the time of the intervention. Contact information of investigators was not always readily available in the registry, and in some cases required searching historical versions of a trial or internet searches.
Scaling beyond transparency in clinical trials
The report cards were designed for scalability and extensibility. They may be adapted to provide individualized feedback and guidance on further transparency or responsible research practices, such as data sharing, reporting of robustness measures (e.g., randomization), measures to improve patient-orientedness (e.g. patient engagement in study design), or reporting of ethical safeguards (e.g., conflict of interest disclosure). The report cards may also be adapted for use in other types of studies and disciplines. This requires a core set of practices to monitor, an overview of studies, and automated and/or manual approaches to extract, process, and validate the data to include in the report cards (Weissgerber et al., 2021). While many study types and disciplines do not yet have formalized best practices for responsible research comparable with those for clinical trials, a core set of practices to monitor within a discipline may be established through community consensus, as done in biomedicine (Cobey et al., 2023). We expect broader applicability of tools that provide study-level feedback, as registration becomes more common across certain fields (e.g., animal research (Heinl et al., 2022)) and serves as a source of study information complementary to publications.
Implications and future work
Future work is needed to further evaluate the use of report cards to address transparency, and iteratively transfer this potential solution to policy and practice. Our feasibility study may inform a larger randomized controlled trial at the level of multiple institutions, and institution-led behavior change interventions more broadly. These studies may consider following up with investigators who implement practices to understand whether the intervention indeed prompted the behavior. Future work may also consider evaluating prospective use of the tools: report cards that prompt investigators before a specific practice is due; an infosheet disseminated at study start (e.g., ethics approval) that sets out expectations for researchers before the work is conducted. The report cards may be particularly well suited to help funders monitor and support compliance with grant requirements, which typically span the entire project lifecycle.
Future studies should adopt a systemic approach and consider the role of capability, motivation, and opportunity when designing an intervention to improve transparency (Michie et al., 2011). For example, combining the report cards with initiatives that reinforce motivation, such as awards or broader recognition of responsible research practices in research assessment (Moher et al., 2020), may be an effective approach. Proposed solutions will also need to be tailored to the specific context and resources of a given institution. Engaging with researchers and other stakeholders, who have first-hand experience of challenges and needs, will be key towards identifying actionable areas for improvement to drive sustainable change.
Data Availability
All the code generated in this study is openly available under an open license in GitHub: https://github.com/quest-bih/trackvalue. Data collection was conducted in Python [Version 3.9] and R [Version 4.3.2], and all analysis was conducted in R. This manuscript was written by interleaving regular prose and analysis code in Quarto [Version 1.4.549]. With the exception of investigator contact information and survey comments and corrections, all the data is openly available in the GitHub repository of the project and Zenodo: https://doi.org/10.5281/zenodo.10467054.
Funding
This project was funded by a Wellcome Trust translational partnership with the Charité/BIH (218358/Z/19/Z). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Declarations
The authors declare no competing interests.
Author Contributions
Delwen L Franzen: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Software, Validation, Visualization, Writing - Original Draft, Writing - Review & Editing; Maia Salholz-Hillel: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Project Administration, Resources, Software, Validation, Visualization, Writing - Original Draft, Writing - Review & Editing; Stephanie Müller-Ohlraun: Conceptualization, Methodology, Writing - Review & Editing; Daniel Strech: Conceptualization, Funding Acquisition, Methodology, Supervision, Writing - Review & Editing.
1 Expert reviews and cognitive interviews on the materials and survey
Methods
We solicited expert reviews to get feedback on the content and design of the invitation letter, report cards, infosheet, and survey. We contacted content experts, experts in questionnaire development, and experts in statistical analysis (Dillman et al., 2014, Guideline 7.7). Reviews were unstructured and tailored to each person’s expertise. Content experts included researchers and administrators involved in the conduct and management of clinical trials. Colleagues with extensive experience conducting surveys reviewed and consulted on the questionnaire. A statistical consultant at the Charité reviewed the analysis plan prior to preregistration of the protocol.
We also performed think-aloud cognitive interviews to identify challenges with the visual design, wording, and navigation of the materials, and to evaluate whether the respondents interact with the materials and understand the questions as intended by the research team (Dillman et al., 2014, Guideline 7.8). Participants (n = 3) were identified within our networks and were selected based on previous or ongoing experience conducting or administering clinical trials at the Charité, in order to maximize similarity with the sample population. Participants received an invitation email to participate in a 1-hour interview conducted over Microsoft Teams. One team member led the interview (MSH) while another team member (DLF) observed and made notes of the interview. After informed consent, participants were asked to walk through the invitation letter, report card, infosheet, and survey and think aloud to capture their thought processes when engaging with the study materials. The cognitive interview guide is available in OSF (https://osf.io/xtpjc). Based on the feedback shared in each interview, we modified the materials prior to the following interview.
Results
Themes (Table 1) for the design of the report card included: provide actionable information by combining feedback and guidance, and by highlighting practices that can still be improved; contextualize feedback and guidance by providing study details and the normative basis for each practice; use clear visuals and accessible language to convey the message quickly. The main theme for the design of the infosheet was balancing comprehensiveness and clarity, by tailoring the information to the audience (e.g., acknowledge institutional guidelines and resources), and by carefully considering the structure and layout. With a few exceptions due to feasibility, the feedback was incorporated into the report cards and infosheet.
2 Screening criteria and characteristics of trials and investigators
3 Survey administration and detailed results
Response timeline across the survey fielding period
Survey respondents roles
The majority of respondents self-reported as study leads (n = 24, 65%), while an additional 6 (16%) self-reported as both study leads and doctors. The remaining respondents identified as either doctors, postdocs, or another position (per group, n = 2, 5.4%).
Themes of comments provided by respondents
Trial investigator corrections to the report card
Investigators had the opportunity to provide corrections to their report card. Respondents suggested 18 corrections on 11 trials across the survey and emails. In our manual review, we found that many were in fact not corrections but rather incorrect understandings of the practices or additional information on the trials beyond the scope of the report card. This, for example, included links to additional results publications, or information that a publication was open access when in fact it was only accessible via an institutional subscription. After limiting these to valid corrections, 5 corrections regarding 4 trials remained: 1 missed earlier results publication, 1 missed results publication, 1 missed cross-registration in EUCTR, and 2 missed publication links in DRKS.
Subgroup analyses of survey responses
We conducted two post-hoc subgroup analyses. Figure 3 shows means and 95% confidence intervals for agreement (i.e., Likert-type response item) with each of eight statements about the report card and materials, across all respondents as well as the for the subgroup analyses.
Study role
We explored perceptions of the report card and infosheet across different study roles. We did not observe a meaningful difference between study leads and doctors (n = 32) versus other respondents (n = 5), who were a smaller group with a wide spread of responses.
Review of report card and infosheet
A total of 3 respondents reported not reviewing the report card and infosheet prior to starting the survey and were asked to do so prior to continuing the survey. Per our prespecified inclusion criteria, we did not exclude these responses. However, response time did not significantly differ between those who were and were not asked to review the materials (
), which suggests that they did not pause the survey to review the materials. We therefore conducted a post-hoc subgroup analysis. Perceptions of the report card and infosheet were substantially lower among those who reported not reviewing the materials prior to starting the survey: all Likert responses across all 8 items for all 3 respondents were the most negative option, and their free-text responses also included negative comments.
4 Steps to generate individualized trial transparency report cards
Main steps to prepare a trial transparency dataset for use with the report cards
Additionally, the following code repositories provide more detailed information (e.g., codebook) on the automated steps used to generate components for the report cards:
Automated pipeline to generate individualized report cards
The pipeline is based on a report card template that includes all possible outcomes of a trial’s performance represented as layers. The report card template is exported as SVG, which is an image format based on XML. This means that the report card template can be accessed and modified using code. A custom-made Python script selects the correct layers to include for each trial based on a trial transparency dataset. Each report card is automatically exported as PDF.
5 Methods for manual validation of open access status of publications and cross-registration in the EUCTR
The report cards developed for this study included references to any known EUCTR cross-registration (additional registrations of a single trial in a different or same registry). To best ensure that our dataset accurately captured cross-registrations, we manually validated suspected cross-registrations based on the primary registration, and for trials with published results, in the full-text as well as PubMed metadata and abstract. A detailed protocol of these cross-registration checks is available at https://osf.io/mjpzh.
The report cards also called for action on open access (OA) and thus required an accurate OA status of publications. The OA status of publications was determined by querying the Unpaywall API using UnpaywallR. Publications were considered as OA if the peer-reviewed version was openly accessible (without fees or log-in) on either the journal website or a repository. Social networking platforms for researchers (e.g., ResearchGate) or personal websites were not considered as longterm openly accessible venues. Unpaywall has been reported to provide a conservative estimate of the actual percentage of OA in the literature (Piwowar et al., 2018), and changes in a publication’s OA status are reflected in Unpaywall with a delay. Therefore, we performed manual checks throughout the study of the OA status of publications that were marked as closed in Unpaywall. Manual checks of the OA status of publications were limited to the following study time points: launch, 3 months post-launch, and 12 months post-launch. Any corrections in OA status based on the manual check performed at launch were also applied to the pre-intervention study point for a given publication. Moreover, any new publications shared by investigators were manually checked before sending updated report cards; in these cases, this OA status was also applied to previous study time points. A detailed protocol of these OA checks is available at https://osf.io/3upgy.
6 Reporting checklists
CROSS Checklist for Reporting Of Survey Studies
Sharma, A., Minh Duc, N. T., Luu Lam Thang, T., Nam, N. H., Ng, S. J., Abbas, K. S., Huy, N. T., Marušić, A., Paul, C. L., Kwok, J., Karbwang, J., de Waure, C., Drummond, F. J., Kizawa, Y., Taal, E., Vermeulen, J., Lee, G. H. M., Gyedu, A., To, K. G., … Karamouzian, M. (2021). A Consensus-Based Checklist for Reporting of Survey Studies (CROSS). Journal of General Internal Medicine, 36(10), 3179–3187. https://doi.org/10.1007/s11606-021-06737-1
CONSORT Checklist for Pilot and Feasibility Trials
Eldridge, S. M., Chan, C. L., Campbell, M. J., Bond, C. M., Hopewell, S., Thabane, L., Lancaster, G. A., Altman, D., Bretz, F., Campbell, M., Cobo, E., Craig, P., Davidson, P., Groves, T., Gumedze, F., Hewison, J., Hirst, A., Hoddinott, P., Lamb, S. E., … on behalf of the PAFS consensus group. (2016). CONSORT 2010 statement: Extension to randomised pilot and feasibility trials. Pilot and Feasibility Studies, 2(1), 64. https://doi.org/10.1186/s40814-016-0105-8.
Table adapted from: https://figshare.com/articles/journal_contribution/CONSORT_2010_checklist_of_information_to_include_when_reporting_a_pilot_or_feasibility_trial_/21519771.
Acknowledgements
We would like to thank the Clinical Study Center of the Charité – Universitätsmedizin Berlin for supporting this intervention. We are grateful to the participants of our cognitive interviews as well as Sarah White and Tony Keyes from the US-based Clinical Trials Registration and Results Reporting Taskforce for their valuable feedback on the report cards and the infosheet. Many thanks to Benjamin Gregory Carlisle for assisting us with the automated email invitations and reminders, Tamarinde Haven and Christiane Wetzel for their input on the survey design, Susan Abunijela for support with performing manual checks of the open access status, Samruddhi Yerunkar for performing manual checks of cross-registrations and visualizing the survey responses, Nicole Hildebrand for manually searching for investigator contact information not available in the registries, as well as Martin Holst, Martin Haslberger, and Friederike Kohrs for their support with language translation. We also thank the Service Unit Biometrics of the Institute for Biometrics and Clinical Epidemiology (iBikE) of the Charité – Universitätsmedizin Berlin for consulting on the statistical analysis. Finally, we would like to thank those who gave their time to participate in our study.
Footnotes
Updated GitHub code repository link.
References
