A Survey of Open Science Attitudes and Behaviors among US Pharmacy Faculty

2.1. Study Design and Sample

This cross-sectional study included a random sample of faculty from the AACP Roster of Pharmacy Faculty as of February 2022. This roster includes an estimated 6,500 faculty across pharmacy practice and the pharmaceutical sciences. Participants were eligible for inclusion if they had at least a 0.8 full-time equivalent faculty appointment in pharmacy practice or one of the pharmaceutical sciences at a school or college of pharmacy in the US. There were no other restrictions placed on type of faculty position (clinical, teaching, research, tenure-track, etc.). Chief executive officer (CEO) deans listed in the Directory of Professional Programs of Colleges and Schools of Pharmacy maintained by the Accreditation Council for Pharmacy Education in March 2022 were excluded from the roster prior to sampling faculty for questionnaire administration; however, respondents who identified as CEO deans were retained in the data. Based on a total population size of 6,500, a 95% confidence level, and a 3% margin of error, a total sample of 363 was needed for accurate and reliable results. Using a conservative estimated response rate of 15%, which is consistent with previous online surveys of pharmacy faculty,^22,23 a total sample of 3,200 was drawn randomly from the AACP faculty roster.

2.2. Study Questionnaire and Administration

The study questionnaire was based on the Open Scholarship Survey (OSS), a modular survey developed by the Center for Open Science.²⁴ The modules on data sharing, code sharing, materials sharing, preregistration, preprints, and open access publishing were used for this study. Questions within each module related to respondents’ attitudes, behaviors, and perceived disciplinary norms (i.e., the percentage of other faculty within their discipline who have engaged in various open science activities and the distribution of attitudes within their discipline towards various open science activities). As part of a larger study, the full study questionnaire included a total of 217 to 220 questions depending on the responses selected as skip logic used as part of the questionnaire flow. The questionnaire was broken into 4 sections: open science attitudes and behaviors, open science incentives and barriers, open science knowledge and future use, and demographics. Only the questionnaire sections on open science attitudes and behaviors and demographics are reported here.

Recruitment took place in 8 waves of 400 randomly selected faculty. Each wave lasted 2 weeks. As the current best, evidence-based practice for conducting survey research, Dillman’s tailored design method²⁵ was used as a framework for survey administration. The online questionnaire was open for 2 weeks within each recruitment wave. Each sampled faculty member received an introductory email that included a personalized link to the online questionnaire to prevent duplicate responses. Reminder emails were sent to non-respondents 3, 6, and 10 business days after the introductory email. Questionnaire administration occurred between April 26 and June 26, 2022, using the Qualtrics XM online survey platform (Qualtrics LLC; Provo, UT). The order of the OSS modules was randomized for each respondent to avoid any potential ordering effects. All other questionnaire sections were presented in the same order across respondents. Individuals who completed the survey were offered a $10 gift card to a selection of nationwide retailers. Identifiable information (names, email addresses, IP addresses, etc.) was not collected as part of questionnaire administration. Mailing information for gift cards was collected separately and not linked to any questionnaire responses.

To approximate attentiveness when completing the questionnaire, time spent on each questionnaire page was measured through embedded timer questions. Respondents spending >1 standard deviation below the mean response time across all respondents for the questionnaire page were excluded. If a respondent exceeded the threshold for ≥80% of survey pages, all their responses were excluded.

2.3. Study Variables

The OSS modules collected information on attitudes, behaviors, and perceived disciplinary norms related to open science attitudes and behaviors. Attitudes towards open science activities were obtained from the relevant OSS modules using the following response options: Very much against, Against, Neither in favor nor against, In favor, Very much in favor, and No opinion. For open science behaviors, respondents provided the percentage of their scholarly works that involved the 6 open science practices examined in this study. Disciplinary norms for attitudes were assessed by asking respondents to estimate the approximate percentage of faculty within their discipline who fell into each attitude category (Very much against, Against, etc.) for each open science practice. For behavioral norms, respondents estimated the percentage of faculty within their discipline who had engaged in each open science practice.

Individual characteristics (time since highest degree, gender, pharmacy discipline, administrative position, etc.) were obtained from each respondent. Institutional characteristics were obtained from publicly available resources. The standardized set of pharmacy disciplines maintained by AACP was used for this study. Research funding from the 2021-2022 fiscal year research was obtained from AACP Funded Research Grant Institutional Tables.²⁶

2.4. Statistical Analysis

Representativeness of the study sample was assessed by comparing respondent characteristics to the 2021-2022 academic year AACP Profile of Pharmacy Faculty.²⁷ The potential for non-response bias was assessed by comparing respondent characteristics and attitudes and behaviors for early and late responders within waves (individuals responding in the first 3 days of the wave vs. individuals responding on or after the last reminder within the wave) and across the data collection period (Waves 1 and 2 vs. Waves 7 and 8).

Current attitudes were described using means and standard deviations and percentage of responses in each response category. Behaviors were described as the percentage of faculty who had any scholarly works using the open science practice. For the comparison of attitudes and perceived attitudinal norms, the “Very much in favor” and “In favor” responses were combined to describe the total percentage of respondents with favorable attitudes towards each open science practice. Differences in attitudes towards open science practices across pharmacy disciplines were analyzed via ordinary least squares regression with response on the attitude item for each open science practice as the outcome variable and pharmacy discipline as the predictor variable.

For open science behaviors, the proportion of scholarly outputs using the open science practice of interest was the outcome variable for a generalized linear model (GLM) with a log link and binomial distribution. For open science behaviors, three models were constructed: Model 1 (pharmacy discipline only), Model 2 (pharmacy discipline and respondent characteristics), and Model 3 (pharmacy discipline, respondent characteristics, and institutional characteristics). Robust standard errors were estimated to account for clustering of respondents within institutions for all models.

Statistical analyses were conducted in R version 4.4.0 (R Foundation, Vienna, Austria) with p < 0.05 denoting statistical significance. As a pre-specified analysis, comparisons of attitudes across pharmacy discipline were adjusted for multiple comparisons using the Bonferroni method. No additional adjustments were made for the multiple testing across the analyses conducted for this study.²⁸ Disciplinary norms for attitudes and behaviors were compared to respondents’ attitudes and behaviors without formal statistical analysis. Respondents not providing their pharmacy discipline, indicating “No opinion” for attitudes towards an open science practice, or not providing any response for attitudes or behavior for an open science practice were excluded from statistical analyses; however, all eligible respondents were included when reporting respondent demographics.

2.5. Ethical Approval and Study Registration

This study was reviewed by the Midwestern University Illinois Campus Institutional Review Board and determined to qualify for an exemption (File Number 22011). The study protocol was registered at Open Science Framework Registries (https://osf.io/68rvy) prior to data collection.

3.1. Response Rate and Missing Data

Of the 3200 emails sent, 49 were unable to be delivered. Of those delivered, 116 individuals opted out of participation, and 59 did not meet the eligibility criteria upon completing the screening questions.

Complete responses were obtained from 502 faculty with an additional 161 providing partial responses yielding the following response rates using the American Association of Public Opinion Research²⁹ definitions: 16.0% (RR1, complete responses only) and 21.2% (RR2, complete and partial responses). Given the availability of additional financial resources after study initiation, recruitment was not concluded once the original sample size requirement was met.

No respondents were excluded because of insufficient attentiveness. Missing attitudes ranged from 103 for open access publishing to 219 for sharing code. Missing frequency of open science practices ranged from 134 for open access publishing to 177 for sharing code. By randomizing the order of the open science attitude and behavior sections, missingness was assumed to be at random for those variables.

Pharmacy discipline was not provided by 175 respondents. The effective sample sizes are provided in the appropriate tables for each analysis.

3.2. Sample Representativeness and Non-Response Bias

Most characteristics of the respondents were within a 5 percentage-point difference of the pharmacy faculty profile with a few exceptions. Faculty from the economic, social, and administrative sciences were more common in the sample (15.9%) than in the faculty profile (7.2%). Faculty at the rank of assistant professor were somewhat underrepresented in the sample (24.6%) compared to the faculty profile (31.7%). There were more tenured faculty responding (36.7%) and fewer tenure-eligible faculty (14.4%) compared to the faculty profile (14.3% and 32.3%, respectively). The full comparisons for representativeness can be found in Supplemental File 1.

Based on the comparisons of early and late responders, there was little concern for non-response bias. Respondent characteristics were similar across early and late respondents both within waves and across the data collection period (Waves 1/2 vs. Waves 7/8). No statistically significant differences were noted in the comparisons and the absolute differences were generally within 5 percentage points. When comparing open science attitudes and behaviors, there were some differences to note. For within-wave comparisons, the mean attitudes were not statistically significantly different; however, late respondents appeared to be more divided on attitudes toward data sharing (i.e., fewer “neither in favor nor against” responses) and were more positive with respect to material sharing. For comparisons across the data collection period, late wave respondents appeared to have slightly less positive attitudes surrounding code sharing than early wave respondents. Although late wave respondents had a statistically higher mean percentage of scholarly works published under open access pathways, there was no difference in the percentage of respondents who had any scholarly work published as open access. The full non-response assessment can be found in Supplemental File 2.

3.3. Respondent Characteristics

Respondents were primarily women (53.5%) with White racial identities (72.3%). Over half of the respondents were from the pharmacy practice discipline. Respondents were approximately equally distributed across academic rank with slightly higher representation of associate professors. Time in academia was approximately equally distributed with slightly fewer respondents in the 16- to 20-year category (14.5%). Respondents reported having published a mean (SD) of 27.3 (40.2) studies (median (IQR): 15 (4, 32)). The reported mean (SD) of total publications was 42.9 (58.9) with a median (IQR) of 25 (11, 50). Additional respondent characteristics are provided in Table 1. Based on the distribution of responses, pharmacy disciplines were collapsed into the following groups: pharmaceutical sciences (biological and biomedical sciences; medicinal or pharmaceutical chemistry and pharmacognosy; pharmaceutics; pharmacology and toxicology); economic, social, and administrative sciences (ESAS); pharmacy practice; and clinical sciences (pharmacokinetics, pharmacodynamics, genetics, pharmacotherapeutics, translational research). This was done to streamline analysis and reporting while maintaining groupings that were generally similar in nature with respect to research topics and approaches.

3.4. Attitudes Towards Open Science Practices

Attitudes were generally neutral (neither in favor nor against) to positive for all open science practices. The most positive attitudes were related to open access publishing (overall mean [SD]: 4.1 [0.9]) with the lowest attitudes related to study preregistration (3.2 [0.9]) and posting preprints (3.1 [1.1]). The mean (SD) attitudes across pharmacy disciplines are presented in Table 2. Statistically significant differences in attitudes across pharmacy disciplines were identified for data sharing, code sharing, and study preregistration. The largest differences across disciplines were noted for study preregistration where the mean (SD) attitudes for clinical sciences was 3.5 (1.5) compared to 2.9 (1.0) for pharmaceutical sciences. There were only 2 instances where mean responses were “against” an open science practice: study preregistration among pharmaceutical sciences faculty and posting preprints for clinical sciences faculty.

Respondents generally reported more positive attitudes for themselves compared to their estimates of the open science attitudes for other faculty within their discipline (Figure 1). Faculty estimated that other faculty within their discipline had the most positive attitudes towards open access publishing (ranging from 42% for ESAS faculty to 57% for clinical and translational sciences faculty). With the exception of code sharing for clinical and translational sciences faculty (41%), estimates of favorable attitudes were below 40% for all other open science practices across pharmacy disciplines. Faculty generally estimated that their colleagues held less favorable attitudes towards open science practices. The differences between their own attitudes versus estimated favorable attitudes ranged from -37.1 percentage points for sharing code in clinical and translational sciences faculty (i.e., respondents estimated that their own attitudes were more favorable than their colleagues) to +2.1 percentage points for posting preprints in clinical and translational sciences faculty (i.e., respondents estimated that their colleagues’ attitudes were more favorable than their own attitudes).

• Download figure
• Open in new tab

Figure 1:

Comparison of Respondents’ Favorable Attitudes and Estimated Favorable Attitudes by Pharmacy Discipline

3.5. Open Science Behaviors

Respondents reported less than half of their prior scholarly works had involved open science practices (Figure 2). The most common open science practice was open access publishing (overall mean [SD], 27.7% [29.1%]), which ranged from 23.2% (27.9%) of works for pharmacy practice faculty to 37.5% (30.1%) for clinical and translational sciences faculty. The least common open science practice was study preregistration (overall mean [SD], 1.7% [7.0%]). Aside from open access publishing, there were only 3 instances where the reported percentage of scholarly works was below 10%: data sharing among pharmaceutical sciences faculty, materials sharing among pharmaceutical sciences faculty, and data sharing among clinical and translational sciences faculty.

• Download figure
• Open in new tab

Figure 2:

Comparison of Respondents’ Open Science Practices and Estimated Open Sciences Practices by Pharmacy Discipline

The comparison of open science behaviors across pharmacy disciplines is presented in Table 3. For these generalized linear models, pharmacy practice faculty were chosen as the reference group since they were the most common group of respondents and because that group was assumed to be somewhat different from the other disciplines given their primary focus on practice rather than research. From Model 1 (pharmacy discipline as the only predictor), the odds of pharmaceutical sciences or clinical and translational sciences faculty engaging in data sharing, code sharing, or open access publishing were from 2 to almost 8 times higher than pharmacy practice faculty. ESAS faculty had an almost 70% lower odds of sharing data compared to pharmacy practice faculty. Additionally, pharmaceutical sciences faculty had over 3 times higher odds of sharing materials than pharmacy practice faculty and clinical and translational sciences faculty had an almost 6 times greater odds of preregistering studies compared to pharmacy practice faculty.

In Model 2, gender, identifying as a racial or ethnic minority, current enrollment in a training program, possessing a research-based doctoral degree, percent effort allocated to research, academic rank, and tenure status were included as confounders based on results from previous studies^18,30–34 and conceptual relevance. Other combinations of individual characteristics were considered but resulted in adverse effects on model performance. The findings from Model 2 showed similar patterns to Model 1.

Three-year total funding for the school or college of pharmacy and a collapsed Carnegie classification were included in Model 3. Each institution’s basic Carnegie classification³⁵ was collapsed in the following manner: baccalaureate institution (arts and science focus or diverse field), master’s institution (M1, M2, or M3), doctoral professional university, medical or health sciences university (Special Focus- Medical Schools and Centers, Special Focus-Other Health Professions Schools), and research-intensive universities (R1, R2, or Special Focus-Research Institution). Other combinations of institutional characteristics were considered but resulted in statistical challenges with model estimation. The results from Model 3 were somewhat different than for Models 1 or 2. The odds of data sharing among pharmaceutical sciences faculty were over 12 times higher than for pharmacy practice faculty (OR [95% CI]: 12.30 [3.04–49.72]). ESAS and clinical and translational sciences faculty had a higher odds of study preregistration compared to pharmacy practice faculty (OR [95% CI]: 5.2 [1.22–22.24] and 21.50 [2.10– 220.14], respectively). Clinical and translational sciences faculty had 70% lower odds of posting preprints (OR [95% CI]: 0.31 [0.10–0.97]) than pharmacy practice faculty. There were no statistically significant differences by pharmacy discipline for materials sharing, code sharing, or open access publishing in Model 3. It is important to note the wide confidence intervals for Model 3 suggesting imprecise estimates potentially associated with small sample sizes resulting from additional variables being added to the models.

Figure 2 provides a comparison of the percentage of respondents’ scholarly works using various open science practices with their estimates of the percentage of faculty within their discipline who have used open science practices. Compared to their own scholarly work, respondents estimated that more faculty within their discipline had engaged in open science practices. This ranged from a 1.3 percentage point difference for data sharing in clinical and translational sciences faculty to a 22.9 percentage point difference for open access publishing in ESAS faculty.