Regulating AI Adaptation: An Analysis of AI Medical Device Updates

1. Introduction

While the number of AI products developed for commercial applications is rapidly growing, the implementation of robust regulatory frameworks still lags behind (Larson et al., 2021; Wirtz et al., 2020; Wu et al., 2021a). Recently, high-profile accidents involving Boeing (?) and Tesla (Corfield et al., 2023) have been attributed to issues with software and AI updates in their systems. Applications of AI to consumer lending (Johnson et al., 2019) and hiring systems (Bogen and Rieke, 2018) has also led to calls for more flexible regulatory systems that can anticipate algorithmic changes and biases. Such cases high-light the inherent challenges regulators face due to the adaptive nature of software and especially AI products: while model adaptation and updates are a necessary step in maintaining or improving their performance, they can also introduce unknown safety risks (Babic et al., 2019; Gilbert et al., 2021).

In the US, the Food and Drug Administration (FDA) has been an early mover in AI regulation, with over 500 approved submissions for AI devices as of 2022 (Center for Devices and Radiological Health, 2022). The FDA faces unique challenges with regard to model updating, as adverse events can directly compromise patient well-being. As such, the FDA has traditionally not allowed any changes to a model once it has been approved (Gerke et al., 2020). At the same time, AI models are well-known to be prone to distribution shifts, whereby variations in factors such as medical practice, patient demographics, or disease prevalence can significantly affect a model’s performance (Raghu et al., 2019; Wiens et al., 2019; Wong et al., 2021). For example, researchers recently found that Epic’s widely used sepsis prediction model performed much worse than initially reported after being deployed in a new hospital setting (Wong et al., 2021). Such cases demonstrate that fixed AI models that never receive updates can likewise compromise patient safety. Recently, the FDA has taken action to address the limitations of a fixed-model approach by providing guidelines for a potential Predetermined Change Control Plan (PCCP) (Center for Devices and Radiological Health, 2023a), as well as a document describing best practices in machine learning published jointly by US, Canadian, and UK health authorities (Center for Devices and Radiological Health, 2023c). Under this provision, developers can make a limited set of changes to their models without a new submission as long as it is pre-specified in their initial approval. Such proposed measures by the FDA underscore the importance of the ongoing discussion around the appropriate levels of regulation regarding the adaptive nature of AI.

Despite the importance of model updating in AI medical devices, little is known about how often such devices are currently being updated. While AI developers may individually publish press releases about changes to their model, there does not exist a systematic analysis of updating across all AI medical devices. FDA approvals by the same developer often contain variants of company and product names, making it difficult to automatically link devices together. Furthermore, devices under the same name often vary widely according to their use cases and are actually different products. In this study, we aim to resolve these issues by organizing and grouping FDA-approved AI medical devices by their updates and performing an analysis of the frequency and nature of model updates. Our study explores the extent to which developers choose to update their devices given current regulatory, economic, and technological factors. Additionally, we perform an illustrative case study on AI models designed to predict pneumothorax, evaluating whether model updates consistently yield improved performance when re-trained on target populations.

2. Methods

3. Results

3.1. Device Update Frequency and Types

Among our dataset of 416 unique devices, we found that 101 devices report having been updated at least once (Figure 1). However, the vast majority of these updates expand the functionality or marketing claims of the device, essentially constituting a new device rather than a true model update. Of these 101 devices, only six of the updated devices report retraining in the model with new data. For each of the six devices, details on the types of data used in retraining are limited, with only three providing how much training data was used. For AI devices, retraining on new data is central to and distinctive of the technology, leading to our focus on the novel regulatory issues here. For example, Syngo.CT CaS-coring (K221219), which analyzes calcified coronary lesions, only references that “the algorithm was retrained on a larger database”. AI-Rad Companion (K213096), which analyzes lung CTs, references “additional training data was added”, while Briefcase (K230020), a rib fracture triage device, mentions that the updated device differs “due to training the subject device on a larger data set”. The remaining three devices reference the scale of the re-training dataset. For example, Quantib Prostate (K230772), which analyzes prostate MRIs, reports that the updated algorithm has been trained on “400 scans”, while Genius AI (K221449), a breast cancer detection device, reports a “two-fold” increase. Finally, Caption Ejection Fraction (K210747), a cardiac ultrasound AI device, reports an “additional 30% training data from three ultrasound devices and two clinical sites”. Details on these devices are also included in Figure 2. For the other 95 updated devices, we found several different update subtypes. The most common type of reported updating occurs when the manufacturer adds a new or additional prediction task to an existing model (55 total devices). For example, whereas FractureDetect’s original device only works on wrists, its update has expanded to ankles, elbows, and other body parts. Next, we found that 21 devices have received updates to their accepted input signal. For example, recent mammography products such as Mammoscreen have included the ability to process Digital Breast Tomosynthesis (DBT)/3D scans, whereas previous versions only accepted Full-Field Digital Mammography (FFDM)/2D scans. An additional 13 devices report changes to the model design or architecture, such as a change from a fully connected neural network to a convolutional neural network. Five devices report a change to the intended target population for the device. For example, EndoSleep expanded its population to pediatric patients, whereas the previous device only allowed for patients 18 or older. We found 22 devices that report changes to the model but do not specify the exact nature of the change. For example, approvals may report “additional algorithmic enhancements” or “improved quality of algorithms”, but not reference whether the improvements come from re-training or model design. Finally, 37 devices report updates unrelated to the model or its usage. Namely, these include software or hardware changes that pertain to its interoperability or output interface. Examples include the UI/UX of the device which is visible to physicians, or a hardware configuration that allows the device to be installed on new machines. We provide a list of examples of these update types in Table 1.

• Download figure
• Open in new tab

Figure 1:

(Top) Proportion of devices that report model re-training and other update types. The devices with any updates are a subset of the total AI medical devices, and the devices updated with model re-training are a subset of devices with any updates. The call-out provides details on the six devices that received model re-training, along with device name, device description, and details provided within the FDA approval regarding the type of model re-training applied. (Bottom) Graph of the number of times devices have been updated. The x-axis refers to the number of successive updates, and the y-axis refers to the count of devices in each group.

View this table:

• View inline
• View popup
• Download powerpoint

Table 1: Examples of updating types present in follow-up devices. The table provides the update type, along with an example of each subtype.

• Download figure
• Open in new tab

Figure 2:

(Top) Time to device update (in years), represented as a box plot where each vertical bar represents the 25th and 75th percentile, respectively. The red line represents the distribution of time to update for model re-training, while the blue line refers to time to update of any type. (Bottom) The estimated rate of updating as a function of device age, which is estimated through the Kaplan-Meier function in order to account for the right censorship in the dataset. For example, devices at two years old are updated across all types approximately 20% of the time, whereas the same-age devices are updated with model re-training around 1.4% of the time. Both plots have been cut-off at seven years as the longest device update observed is 5.9 years. Shading indicates the 95% confidence interval around the updating rate at any given time.

• Download figure
• Open in new tab

Figure 3:

Schematic of database curation steps, starting from the FDA’s official list of AI medical device approvals to the final set of 416 unique devices.

3.3. Case Study

We carried out a case study to illustrate and quantify the tradeoffs with AI adaptation through model retraining. We investigated the potential benefits and challenges of re-training on additional data from external sites in pneumothorax AI algorithms (Wu et al., 2021b,a). We found that external evaluation of models can result in an AUC decrease of up to 0.18, while re-training and evaluating on data from external sites improves model performance in all scenarios, with an average of 0.075 and a maximum of 0.23 AUC (Figure 4, Middle). However, after re-training on external sites, we also found that model performance degrades an average of 0.176 AUC (and up to 0.268 AUC) when re-evaluated on the original site (Figure 4, Bottom). This suggests that it can be challenging to have a single AI model that works well across heterogeneous settings.

• Download figure
• Open in new tab

Figure 4:

A case study on the effect of re-training pneumothorax AI models on other sites reveals that although re-training improves external site performance, performance consequentially degrades on the originally trained sites. Top Figure: An AI model is trained on pneumothorax cases from site A and then evaluated on held-out cases from site A and an external site (site B). Then, the model is updated by fine-tuning on 5K additional cases from site B and re-evaluated on sites A and B. This procedure is performed for three clinical sites (SHC, BID, NIH) across six total scenarios. The results of re-training and re-evaluation are shown in B and C. Middle Table: each cell shows the AUROC scores of the model evaluated on site B before and after re-training on site B. On average, models improved by 0.075 AUC after re-training. Bottom Table: each cell shows the AUROC scores of the model evaluated on site A before and after re-training on site B. Across both panels, we perform bootstrapped one-sided tests for each cell and indicate with asterisks (***) where p < 0.001.

4. Discussion

Currently, FDA-approved AI models are “locked” after approval, whereby making new changes requires undergoing a brand-new submission process, with most of the same regulatory burden (Gerke et al., 2020). Correspondingly, we observe in our analysis that only six out of 416 devices report actually received re-training updates, which is an essential approach for AI adaptation. On the other hand, nearly a quarter of devices receive updates in the form of additional marketing or functionality claims. Such disparity suggests a much stronger economic incentive for developers to increase the adoption of their devices through marketing new features rather than improving the original model through re-training.

One significant barrier to re-training is development costs, which may include acquiring new datasets (Chen et al., 2019; Wu et al., 2023a), computational resources (Wiens et al., 2019), data groundtruthing (Rahimi et al., 2021; Willemink et al., 2020), and regulatory hurdles (Kelly et al., 2019; Sertkaya et al., 2022). After models are updated, the manner in which they are deployed can also affect a device’s ultimate clinical impact. First, while previous-generation AI devices for mammography were clinically evaluated to improve detection rates, subsequent studies showed limited benefits to women due to changes in how clinicians interacted with the devices, as well as the transition from film to digital mammograms (Lehman et al., 2015; Fenton, 2015). Second, economic forces such as reimbursement rates can affect how the frequency and extent to which these devices are adopted (Parikh and Helmchen, 2022b; Abràmoff et al., 2022). AI adoption is still in a nascent stage, with very few widely adopted products and underdeveloped commercial payment pathways (Chen et al., 2021; Parikh and Helmchen, 2022a; Wu et al., 2023b). In such an environment, companies with few customers may not be able to dedicate resources toward regular model updating and maintenance. Currently, FDA cleared products exist in a similar band of risk profiles, with a previous study showing all devices currently categorized as risk class II (medium-risk) ((Zhu et al., 2022)). The lower-risk class I is largely exempt from the regulatory process, with the higher-risk class III reserved for devices that “sustain or support life, are implanted, or present potential unreasonable risk of illness or injury” ((for Devices and Health)). Whereas minimal-risk products like mobile health apps can introduce frequent updates without any regulatory hurdles, the medium-risk designation may encourage a trend towards more conservative updates that are more likely to be cleared rather than ambitious updates that may be rejected.

The FDA has recognized the high regulatory hurdles associated with model updating. In a recently proposed draft guidance from April 2023, model developers may be allowed to include a PCCP (Predetermined Change Control Plan) along with their device submission, which would allow them to simply document subsequent model updates rather than requiring a new submission every time (Center for Devices and Radiological Health, 2023b), potentially alleviating some of the regulatory burden and shortening update intervals. However, even under these proposed changes, developers are still required to complete rigorous evaluation and documentation of the algorithm changes, which incur much of the same prohibitive time and costs mentioned above (Allen, 2022; Evans, 2022). Furthermore, evaluating an updated model is an inherently difficult task due to various types of distribution shifts and heterogeneous data collection methods that are outside the control of developers (Schrouff et al., 2022; Chen et al., 2018). As such, future guidance documents should consider the challenges inherent in ensuring and evaluating fairness under fine-tuning and data shift. Our case study illustrates the tug-and-pull nature observed in AI models – when trained on data from a specific site, they can perform well, but this may trade-off with performance on other sites. Although our models are trained on only a few datasets and do not comprehensively represent the gamut of available training data sources and model architectures on the market, the results illustrate how one instantiation with commonly used data and architecture choices exhibits characteristic behaviors of performance shift. In the status quo, model developers are locked into one model, creating scenarios where they may have to optimize for one population at the expense of another. To compound this issue, the actual performance on new, unseen populations is not even reported since the FDA does not require postmarket surveillance for 510(k)-approved devices (Wu et al., 2021a). To alleviate this problem, future regulatory guidelines should move beyond a “one-model-fits-all” approach, and instead consider allowing site-specific re-training and deployment. By allowing developers to deploy and validate multiple models under a single device, they can optimize model performance for each intended population without incurring performance tradeoffs. This would ensure that developers verify that their models perform well on each deployed clinical site while allowing them to perform the necessary site-specific documentation and evaluation as they mature. There are various design decisions that can affect how an AI model is re-trained: factors like whether to freeze layers, mix new training data, hyperparameter tuning, and validation processes can all influence how much re-training improves model performance (Pham et al., 2021; Picard, 2021; Qian et al., 2021). Indeed, in our case study, even though the individual models used in our ensemble approach only varied by the random seed used during training, performance across models still differed by up to 0.056 AUC. In a study by Watson et al. (2022), chest X-ray deep learning models trained on the same BIDMC dataset across different random seeds and hyperparameters were found to disagree in their explanations up to two-thirds of the time. Such studies on specific datasets represent potential pitfalls of algorithms applied to a particular clinical domain, but do not necessarily mean they generalize to all other types of devices. Regulatory guidelines should include consideration of appropriate fine-tuning schemes used when evaluating models.

Furthermore, we find that among models that have been updated with re-training, details on the data used in training are very limited, with basic descriptions such as “additional training data”, or “larger database”. A limitation of our study lies in the limited details reported in FDA clearances. For example, while only 6 devices report retraining on new data, 5 devices report updates to their target population and 21 devices report unspecified improvements to their algorithm. As such, the true rate of retraining on new data may be higher than reported. In order for consumers and users to make informed decisions on the impacts of model updates, regulators should ensure that information about the data used for original model development and updates is transparent and accessible. Information such as patient demographics, hospital locations, disease subtypes, and healthcare settings are important covariates that can significantly influence model performance (Duffy et al., 2022; Wu et al., 2021b).

As the medical AI field matures, regulation should progress in lockstep with fully exploiting the technical benefits of adaptive learning systems while curbing risks to safety and efficacy. Looking beyond the US, regulatory bodies, such as the European Union, are similarly developing guidelines for regulating medical AI (Muehlematter et al., 2021). We believe that the trends and challenges in medical AI also extend to regulating other AI-transformed industries such as transportation and law, and offer important insights into how to appropriately foster AI innovation.