Changes in English medication safety indicators throughout the COVID-19 pandemic: a federated analysis of 57 million patients’ primary care records in situ using OpenSAFELY

Study design

We conducted a retrospective cohort study using general practice primary care electronic health record (EHR) data from all GP practices in England supplied by the EHR vendors TPP and EMIS.

Data source

Primary care records managed by the GP software providers TPP and EMIS are available in OpenSAFELY, a data analytics platform created by our team with the approval of NHS England to address urgent COVID-19 research questions (https://opensafely.org). OpenSAFELY provides a secure software interface allowing the analysis of pseudonymized primary care patient records from England in near real-time within the EHR vendor’s highly secure data centre, avoiding the need for large volumes of potentially disclosive pseudonymized patient data to be transferred off-site. This, in addition to other technical and organisational controls, minimises any risk of re-identification. Similarly pseudonymized datasets from other data providers are securely provided to the EHR vendor and linked to the primary care data. The TPP dataset analysed within OpenSAFELY (hereafter referred to as OpenSAFELY-TPP) is based on 24.2 million people currently registered with 2546 GP surgeries using TPP SystmOne software; the EMIS dataset analysed within OpenSAFELY (hereafter referred to as OpenSAFELY-EMIS) is based on 32.6 million people currently registered with 3821 GP surgeries using EMIS. It includes pseudonymized data such as coded diagnoses, medications and physiological parameters. No free text data are included. Further details on our information governance can be found under information governance and ethics.

Study population

We included all patients who were: alive; aged 18-120; registered with an OpenSAFELY-TPP or OpenSAFELY-EMIS practice; and recorded as at risk of at least one potentially hazardous prescribing indicator as defined by the SNOMED-CT and NHS dictionary of medicines and devices (dm+d) codes developed by PRIMIS for the PINCER programme (https://www.opencodelists.org/codelist/pincer/) between September 2019 and September 2021 to adequately cover both the period of onset and subsequent recovery of the COVID-19 pandemic.

Study measures

Definitions of the hazardous prescribing indicators are provided in Table 1. The percentage for each indicator is formed of a numerator and denominator pair, where the numerator is the set of patients deemed by the indicator to be at risk of a potentially hazardous prescribing event, and the denominator is the set of patients for which assessment of the indicator is clinically meaningful. Higher indicator percentages represent potentially poorer performance on medication safety. Indicators belong to three groups: (1) those associated with gastrointestinal bleeds; (2) those associated with cautioned medications; and (3) those associated with blood test monitoring; these groups are used to summarise results.

Each indicator was specified in analytic code using PRIMIS SNOMED-CT codelists using the OpenSAFELY framework. We generated the numerator and denominator for each indicator per month between September 2019 and September 2021, and then calculated monthly percentages for each practice. For indicators assessing numeric values, only unambiguous results were used in the calculation of indicator percentages (for example, an eGFR value of “>30” was considered ambiguous for an indicator requiring the identification of patients with an eGFR value less than 45); note that this functionality was not available in OpenSAFELY-EMIS at the time of the study, so results for the CRF & NSAID indicator are reported for OpenSAFELY-TPP practices only.

The monthly indicator percentages were summarised as deciles and presented as decile charts across all practices each month. We also calculated the mean rate across practices in Q1 2020 and 2021 for each indicator as well as total counts of the numerator and denominator for each indicator across the two year period; note that in this cumulative data, repeated events will be counted for each month the event occurs (e.g., if a heart failure patient is prescribed an oral NSAID in two separate months (HF & NSAID), this is represented as two separate events). Across this period we also calculated the ratio of hazardous prescribing events to unique patients experiencing those events (to give an indication of the extent of repeated hazardous prescribing) and the number and percentage of practices with at least one instance of potentially hazardous prescribing at any point across the period.

Each blood test monitoring indicator has an associated monitoring window (e.g. lithium concentrations are required to be checked within 3 months). We annotated each decile plot to indicate this monitoring window, as calculated from the onset of COVID-19 in March 2020, for each indicator. We can interpret these lines as the point at which all patients would be subject to delayed blood test monitoring (a ‘worst case scenario’), should no action have been taken to rectify any COVID-19 related delays.

Software and Reproducibility

Data management and analysis was performed using the OpenSAFELY software libraries and Python, both implemented using Python 3.8. A federated analysis involves carrying out patient-level analysis in multiple secure datasets, then later combining them: codelists and code for data management and data analysis were specified once using the OpenSAFELY tools; then transmitted securely to the OpenSAFELY-TPP platform within TPP’s secure environment, and separately to the OpenSAFELY-EMIS platform within EMIS’s secure environment, where they were each executed separately against local patient data; summary results were then reviewed for disclosiveness, released, and combined for the final outputs. All code for the OpenSAFELY platform for data management, analysis and secure code execution is shared for review and re-use under open licences at github.com/OpenSAFELY. Decile charts were drawn using Seaborn and matplotlib.

All analytical code used in this study is openly available for inspection and re-use at https://github.com/opensafely/pincer-measures. All codelists used are openly available at OpenSAFELY Codelists (https://www.opencodelists.org/codelist/pincer/). Results are available as an online report https://reports.opensafely.org/reports/changes-in-pincer-measures-throughout-the-covid-19-pandemic/.

Patient and Public Involvement

This analysis relies on the use of large volumes of patient data. Ensuring patient, professional and public trust is therefore of critical importance. Maintaining trust requires being transparent about the way OpenSAFELY works, and ensuring patient voices are represented in the design of research, analysis of the findings, and considering the implications. For transparency purposes we have developed a public website (https://opensafely.org/) which provides a detailed description of the platform in language suitable for a lay audience; we have participated in two citizen juries exploring public trust in OpenSAFELY (13); we are currently co-developing an explainer video; we have ‘expert by experience’ patient representation on our OpenSAFELY Oversight Board; we have partnered with Understanding Patient Data to produce lay explainers on the importance of large datasets for research; we have presented at a number of online public engagement events to key communities; and more. To ensure the patient voice is represented, we are working closely with appropriate medical research charities.

Indicators associated with gastrointestinal bleeding

The six indicators of potentially hazardous prescribing in relation to GI bleeds exhibit a decreasing trend across the study period with minimal impact of the pandemic (Figure 1a, OpenSAFELY-TPP only and OpenSAFELY-EMIS only decile charts are provided in Supplementary Figures 1a and 2a respectively). The mean percentage of Warfarin/DOAC & NSAID was 1.39% in Q1 2020; the equivalent rate in 2021 was 1.18%. Similarly, the percentage of PU & antiplatelet fell from 4.24% to 3.85% between Q1 2020 and Q1 2021 (Table 3). There were only marginal impacts on this improving trend during the peak waves of the COVID-19 pandemic.

• Download figure
• Open in new tab

Figure 1 Practice level decile plots for PINCER prescribing indicators, specifically in relation to (a) GI bleeding and (b) cautioned medications.

The percentage of patients identified as at risk of potentially hazardous prescribing as measured by each indicator is reported for the period September 2019 to September 2021 (inclusive). The median percentage is displayed as a thick blue line and deciles are indicated by dashed blue lines. The month of national lockdown in England as a response to the onset of COVID-19 (March 2020) is highlighted with an orange dashed vertical line. Deciles for CRF & NSAID are calculated across 2546 OpenSAFELY-TPP practices; all other deciles are calculated across 6367 practices (2546 OpenSAFELY-TPP + 3821 OpenSAFELY-EMIS practices).

Indicators associated with cautioned medications

Time trends and variation for all three “cautioned medication” indicators are presented in Figure 1b (OpenSAFELY-TPP only and OpenSAFELY-EMIS only decile charts are provided in Supplementary Figures 1b and 2b respectively): COVID-19 had minimal impact on compliance for all indicators.

Indicators associated with blood test monitoring

All blood test monitoring indicators exhibited an increase in delayed monitoring immediately following the onset of COVID-19 (May-July 2020; Figure 2, OpenSAFELY-TPP only and OpenSAFELY-EMIS only decile plots are provided in Supplementary Figures 3-4 respectively).These increased rates showed considerable recovery by August-September 2020, in the case of Lithium and no level recording (31.47% Q1 2020 vs 38.44% Q1 2021), Methotrexate and no FBC (18.64% Q1 2020 vs 22.73% Q1 2021) and Methotrexate and no LFT (19.62% Q1 2020 vs 23.27% Q1 2021). The Amiodarone and no TFT indicator behaved similarly (36.20% Q1 2020 vs 39.23% Q1 2021), though the initial post-COVID-onset recovery period extended into October-November 2020 (Table 3).

• Download figure
• Open in new tab

Figure 2 Practice level decile plots for PINCER blood test monitoring indicators.

The percentage of patients identified as at risk of potentially blood test monitoring as measured by each indicator is reported for the period September 2019 to September 2021 (inclusive). The median percentage is displayed as a thick blue line and deciles are indicated by dashed blue lines. The month of national lockdown in England as a response to the onset of COVID-19 (March 2020) is highlighted with an orange dashed vertical line. The monitoring window, as measured from the onset of COVID-19, for each indicator is shown by a green dashed vertical line. All deciles are calculated across 6367 practices (2546 OpenSAFELY-TPP + 3821 OpenSAFELY-EMIS practices).

The “ACEI or loop diuretic, no blood tests” indicator exhibited a noticeably different COVID-19 response pattern than the other blood test monitoring indicators. Here, the monitoring worsened steadily over a longer period of time, increasing from a mean of 5.16% to 12.14% between Q1 2020 and Q1 2021, beginning to recover in June 2021. The assessment window for this indicator is significantly wider than the windows for the other blood test monitoring indicators: within 15 months of prescription compared to three (lithium and methotrexate) or six (amiodarone) months.

Summary

Despite substantial barriers to the delivery of primary care during the COVID-19 pandemic, good performance was maintained across a diverse range of widely evaluated and nationally adopted measures of safe prescribing. There were evident delays in delivering some medication-related blood test monitoring within the time-window specified in the safety measure; especially for those blood tests where the time-window for compliance is itself already very long, and tests infrequent. However all indicators exhibited considerable recovery by the end of the study period. It was possible to describe PINCER indicators for almost all patients and GP practices in England with a single analysis in OpenSAFELY.

Strengths and limitations

This study has a range of strengths, some unprecedented. The scale and completeness of the data in the OpenSAFELY platform is greater than that through any other route for accessing GP data: other approaches either offer substantially fewer active current patients, in a data download onto researchers’ local machine (such as Clinical Practice Research Datalink); or only contain a small derived subset of information in the GP record (such as the GPES dataset extracted and disseminated by NHS Digital). Previous audits for compliance with PINCER or similar measures and indicators in primary care rely on manual audit within a practice, or analyses on data downloaded from a group of practices. By contrast OpenSAFELY executes analyses in a secure environment inside the EHR provider data centre, across the full set of all structured data in the GP record including all tests, prescriptions, diagnostic codes and referrals. In addition, although the underlying GP data is stored in two very different settings (TPP and EMIS) analyses written once in OpenSAFELY then execute in each setting identically, with the outputs aggregated afterwards, in a process known as “federated analytics”. Overall this represents a unique, national platform able to capture the patient journey for 57 million people in England whilst prioritising patient privacy.

A second strength is the transparency and reproducibility of the analysis: as with all OpenSAFELY analyses, the complete set of all code for the platform and for all data curation and analysis for every study from raw data to completed output is shared openly on GitHub in standard formats for scientific review and efficient re-use under open licences by all.

A third strength is the robustness of the measures for each safety behaviour: all eligible patients and targeted clinical safety behaviours were developed for the national PINCER medication safety programme which has been extensively peer reviewed and evaluated throughout the NHS over many years with strong support from clinicians and commissioners.

We also note some limitations. We note that our data will only include test results carried out in primary care, or those in secondary care that are returned to GPs as structured data: this may therefore not include test results communicated by letter or phone (such as tests requested while a person is in hospital, or other settings like psychiatric outpatients). However this is in line with the methodology already used in the national PINCER programme to evaluate compliance with the targeted safety behaviours using primary care data alone.

Comparison of existing literature

A recent systematic review of healthcare usage during the pandemic, encompassing 81 studies across 20 countries found that healthcare utilisation (including visits, admissions, diagnostics and therapeutics) reduced by 37% during the pandemic, highlighting a substantial reduction in April-May 2020 (14). The WHO also identified significant disruption to countries’ healthcare capacity for non-communicable diseases in a rapid assessment in May 2020 (15). A population cohort-based study conducted using the OpenSAFELY platform reported that clinical activity in relation to blood tests declined in the months following COVID-19 onset, but also reported recovery of these same tests by September 2020 (16). These findings are in line with our observations of the blood test monitoring indicators, where substantial delays were experienced in the same period of time. Interrupted service delivery leading to reduced NSAID prescriptions following acute presentations may also explain the temporary reduction of the first GI bleed indicator (prescription of an oral NSAID, without co-prescription of an ulcer-healing drug, to a patient aged ≥65 years) in April-July 2020; this is supported by data for this period in OpenPrescribing (17) and lower than predicted rates of prescribing for Naproxen and Ibuprofen in this period (18). Elsewhere we have found evidence of prioritisation of anticoagulant services, with blood tests to manage high-risk anticoagulants being prioritised during the initial stages of the COVID-19 pandemic (16); data from the current study also suggests that prescribing in relation to anticoagulants is a priority, with all GI bleed indicators being unaffected, and continuing to decline, following COVID-19 onset.

In the early stages of the pandemic, in recognition of the increased risk of medication related harm during the COVID-19 pandemic, NHS England and local Clinical Commissioning Groups (CCGs) revised guidance regarding blood test monitoring, extending the recommended monitoring window for some patient populations (e.g., in relation to lithium (19) and methotrexate (20)) or advising that blood monitoring for lower risk medications should be carried out “if possible” (e.g., ACE inhibitors (21)), if clinically safe to do so. There is some evidence in our data that practices did adopt this revised guidance, with post-recovery blood test monitoring often falling just short of pre-pandemic levels (particularly in the case of methotrexate and lithium).

Implications for policy and research

The variation in service recovery observed in the blood test monitoring indicators may in part be due to the assessment window for each indicator, and clinicians prioritising urgent work during the pandemic. For example, it is possible that the protracted recovery of the ACE inhibitor monitoring was due to primary care services proactively prioritising monitoring of higher risk prescriptions such as methotrexate so as to minimise the impact of service disruption on patient care. It is also likely that the systems put in place around the monitoring of high risk drugs (e.g., clinical system alerts) contributed towards expedited recovery of the other blood test monitoring indicators, particularly in the case of lithium and methotrexate. The decile chart for this indicator starts to plateau well before the ‘worst case scenario’ timepoint, suggesting that the majority of primary care providers successfully implemented recovery programmes in this clinical domain. Further areas for research include using innovative change detection methodologies (22) to ascertain practice-level features that influence recovery and resilience in the context of service disruption to inform WHO and NHS England’s recommendations to “build back better”.

The implications of this analysis for data usage in the NHS are very substantial. Historically, due to practical and privacy challenges around accessing GP data at scale, each practice participating in the PINCER programme has been required to manually execute the necessary computerised searches before individually uploading their results for central oversight; in some centres data for a group of practices and patients can be downloaded and analysed in larger volumes. This manual approach introduces delays and increases the resource cost of monitoring safety. Using the OpenSAFELY framework we were able to execute a single analysis for almost the entire population of England in near-real-time, while leaving data in situ. This approach is efficient: analyses can be easily updated, and expanded, because they are executed in a single framework from re-executable code. It also preserves patient trust: OpenSAFELY was the single most highly trusted COVID-19 data project in a rigorous Citizens Jury sponsored by the NHS and the National Data Guardian (13). Furthermore, the additional data also securely accessible through the OpenSAFELY tools can be used to describe PINCER measures in fine-grained demographic or clinical sub-populations. These tools can facilitate near real-time audit and feedback in the context of rapidly evolving pressures on the health service and are readily extendable to other clinical and challenges.

More broadly this analysis demonstrates the power of collaborative working with shared open source code in the NHS: it built upon the work of PRIMIS in establishing and then publicly releasing the full code for a set of rigorously tested medication safety indicators; and then implemented the open code from PINCER in the open source framework of OpenSAFELY, to assess a critical public health question on a national scale. Open working as demonstrated here is strongly supported by senior stakeholders in multiple sectors (23–25) and can bring substantial benefits: it facilitates efficient re-use of prior technical work; it ensures fidelity through the consistent implementation of data curation and analysis across all organisations; it supports complete reproducibility; it enables error-checking by all interested parties; and it facilitates public and professional trust.

Conclusion

NHS GP data can be analysed at national scale to generate insights on service delivery with strong public support. Potentially hazardous prescribing was largely unaffected by COVID-19 in a dataset of 57 million patients’ full primary care health records in England.