Abstract
Background High-quality peer reviews are often thought to be essential to ensuring the integrity of the scientific publication process but measuring peer review quality is challenging. Although imperfect, review word count could serve as a simple, objective metric of review quality. We aimed to examine the prevalence of very short reviews and how often they inform editorial decisions on research articles in leading general medical journals.
Methods We compiled a data set of peer reviews from published full-length original research articles in The BMJ, BMC Medicine, and PLOS Medicine for the years 2003 to 2022. In our primary analyses, we used a threshold of <200 words to calculate the prevalence of very short reviews. In secondary analyses, we also used thresholds of <100 and <300 words. In addition to disaggregating results by journal and year, we plotted the proportion of articles for which the first editorial decision was made based on a set of peer reviews in which very short reviews constituted 100%, ≥50%, ≥33%, or ≥20% of the reviews.
Results In this sample of 11,466 reviews corresponding to 4,038 published articles, the median review word count was 425 (Interquartile Range=253–675), and the mean was 520 (Standard Deviation=401). The overall prevalence of very short (<200 words) reviews was 17.1% [95% CI: 16.4%–17.8%]. Across the three journals, 20.9% [95% CI: 19.6%–22.2%] of initial editorial decisions were based on review sets containing ≥50% very short reviews. The prevalence of very short reviews and share of editorial decisions based on review sets containing ≥50% very short reviews was highest for BMC Medicine at 26.8% [95% CI: 25.1%–28.5%] and lowest for The BMJ at 7.3% [95% CI: 5.7%–8.9%].
Conclusions A substantial proportion of initial editorial decisions for published articles in these three leading general medical journals was based on peer reviews of such short length that they were unlikely to be of high quality. Future research should determine whether monitoring peer review length is a useful approach to improving the quality of the peer review process and which interventions, such as incentives and norm-based interventions, are most effective in soliciting more detailed reviews.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
↵* Joint first authorship
Data Availability
All data produced in the present study are available upon reasonable request to the authors
