Research Collective’s Summary of FDA’s AI-Enabled Device Draft Guidance

Many professionals in the healthcare human factors field have a close relationship with regulatory documents. In the United States, this often means we invest significant time reading and interpreting the Food and Drug Administration’s (FDA) guidance documents. When a new guidance document is released, it can feel daunting, as we know we need to understand it in great detail. That’s why we took some time to review the FDA’s draft guidance Artificial-Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations and create a summary to help clarify its contents, making it easier to grasp.

Human Factors Implications

As a healthcare human factors consultancy, we tend to read FDA guidance documents through the lens of human factors engineering implications. For those interested, the following are what we believe to be the key human factors considerations:

• There are sources of use-related risk that are unique to medical devices that leverage AI, specifically “AI-enabled devices may introduce distinct information comprehension risks that might not be found with other types of devices.”
• The FDA emphasizes transparency and strong descriptions of how AI in the product works. Because of this, labeling is more important than ever. This includes:
  • What data contributes to the AI’s output
  • What the output means
  • How the output is intended to be used. This is especially important when it impacts user and/or patient safety. 
• All of this should be considered in the device’s Use-Related Risk Analysis (URRA).

If you’re looking for a more comprehensive summary, the following provides an overview of the FDA’s new draft guidance document in its entirety.

Summary

The FDA’s latest release, titled Artificial-Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations has 2 main objectives:

1. Recommendations on what to include in marketing submissions for AI-enabled devices
2. Recommendations for the design and development of AI-enabled devices 

The document is organized into 13 chapters and five appendices. The first four chapters offer an introductory overview. Chapters 5 through 13 focus on the recommended contents of AI-enabled device marketing submissions (objective 1) and feature detailed design and development recommendations (objective 2). The appendices offer additional details and are referenced throughout the document.

Introduction (Chapters 1-4)

The first part of the document outlines its objectives (see above) and how these objectives support the FDA’s total product life cycle (TPLC) approach to overseeing medical devices. This means that the FDA’s goal is to ensure the safety of medical devices throughout their entire life cycle, not just before they receive marketing clearance or approval.

Key to these first four chapters is the overview of terminology that may seem straightforward at first glance but carries specific definitions as established by the FDA, including:

• Function: The distinct purpose of a product, encompassing all or part of its intended use.
• Device software function: A software function that can be considered a device as defined by the FD&C Act [U.S. Congress. (1934) United States Code: Federal Food, Drug, and Cosmetic Act, 21 U.S.C. §§ 301-392 Suppl.]. This includes any instrument or article for diagnosis, treatment, or affecting bodily functions, not relying on chemical action or metabolism.
• Artificial Intelligence (AI) Model: A mathematical construct that generates predictions based on new input data.
• AI-enabled device software function (AI-DSF): A device software function implementing one or more AI models.
• AI-enabled devices: Devices incorporating one or more AI-DSF.

The introduction also explains how to use the document, providing an overview of the upcoming chapters.

Marketing Submission Contents (Chapters 5-13)

The remaining chapters of the document outline the information and documentation recommended for AI-enabled device marketing submissions. Notably, much of the robust recommended content is intended to supplement what is already expected to be included in non-AI device marketing submissions. 

To help make sense of these recommendations, each chapter includes the following subsections:

• Why the information should be included in a marketing submission
• What specifically should be included in that section of the marketing submission
• Where (i.e., which section) in the submission it should be included

This article provides some highlights and notable mentions from each marketing submission content chapter.

Device Description 

• The device description helps the FDA understand the general characteristics of the AI-enabled device by clarifying functionality, user interaction, and intended use. 
• Some of the information recommended to include in this section of the marketing submission includes a statement that AI is used in the device, how AI is implemented to achieve the device’s intended use, and a description of intended users (note that “users” include anyone who will interpret output data).

User Interface (UI) and Labeling

• A written description and graphical representation of the device’s UI helps the FDA understand what the device is intended to do and how users are intended to interact with it; a description of device labeling helps the FDA determine if all labeling requirements applicable for the type of marketing submission have been met.
• The FDA recommends including labeling that states AI is used in the device and detailed information about the AI model (e.g., inputs and outputs, architecture, development data, performance data, performance metrics, performance monitoring, and limitations).

Risk Assessment

• While all device marketing submissions should include a comprehensive risk assessment to help the FDA understand how the relevant risks have been identified and managed, AI-enabled devices may introduce distinct information comprehension risks that might not be found with other types of devices.
• During the risk assessment, the FDA recommends considering all risks throughout the device’s lifecycle (e.g., during installation, performance maintenance, and the interpretation of results) regarding both use and information comprehension.

Data Management

• For the FDA to understand the AI-enabled device they need a clear characterization of the data used for both developing and validating the AI model and an explanation of data management practices (e.g., how the data is collected, processed, and stored).
• Notable terminology in this section:
  • The data utilized for the development of the AI model is commonly referred to as development data or training data
  • Validation data, also known as performance validation data, test data, or clinical data, is utilized to assess the effectiveness of the AI model.

Model Description and Development

• AI Model and device design information, including biases and limitations, is crucial for the FDA’s evaluation of the safety and effectiveness of AI-enabled devices, ensuring compliance with performance testing specifications.
• FDA recommends including information describing each model and its development, and if applicable, how model outputs combine to create the device output, in the marking submission.

Validation

• Validation data offers the FDA insight into how the device may be used and how it will perform under real-world circumstances.
• Validation of an AI-enabled device involves ensuring that:
  1. The device performs its intended use safely and effectively for users. This is evaluated through either Human Factors Validation Testing or Usability Testing:
     • Human Factors Validation Testing includes evaluating critical tasks, which if performed incorrectly or not at all could cause serious harm [U.S. Food and Drug Administration, 2016].
     • Usability Testing is performed for devices without any critical tasks but may help support the control of risks.
  2. The device consistently meets relevant performance specifications. This is evaluated through Performance Validation, which uses testing and monitoring methods that assess how well the AI model performs under specific conditions. The resulting data is referred to as validation data, performance validation data, test data, or clinical data.
• The AI-enabled device marking submission should include study protocols and results from all validation testing conducted.
• It is important for the FDA to understand how the device performs overall in the intended use population as well as in subgroups of interest. They recommend stratifying and analyzing subgroups of interest when performing Performance Validation.

Device Performance Monitoring

• AI-enabled medical devices reliant on ongoing training data may experience performance changes over time, potentially affecting patient safety. In alignment with the FDA’s TPLC approach, it’s a good idea for manufacturers to implement a post-market performance monitoring plan to keep an eye on these changes even after their device has been cleared or approved. Though by no means required, by including this plan in marketing submissions, manufacturers can offer valuable insights that can assist the FDA in evaluating risk management strategies.

Cybersecurity

• Because AI-enabled medical devices can pose cybersecurity risks, the FDA offers helpful recommendations for designing and maintaining cybersecurity along with key information for marketing submissions in another guidance document titled “Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions [U.S. Food and Drug Administration, 2023].” This resource outlines important security objectives that can be applied to medical devices, including those that use AI technology.

Public Submission Summary

• Public submission summaries provide valuable information about the device and the supporting evidence used for regulatory decisions. To promote transparency regarding the FDA’s determination of substantial equivalence or safety and effectiveness, it’s important that details about the AI-enabled device are clearly included. This section outlines the information sponsors should consider including in the public submission summary and offers a suggested format (i.e., a “model card”).

Appendices

The guidance document includes six helpful appendices that are referenced throughout. These appendices offer a variety of valuable information, such as a table outlining the recommended sections of the marketing submission for the details covered in chapters 5-13, transparency design considerations, performance validation considerations, usability evaluation considerations, as well as an example model card and an example 510(k) public summary featuring a model card.

Conclusion

This newly released guidance offers a wealth of information on how to navigate regulatory concerns related to the development of AI-enabled medical devices and effectively submit a marketing application for clearance or approval. Although it is highly detailed and may initially appear complex, it serves as an invaluable resource for manufacturers of these advanced devices and will greatly assist those of us involved in the marketing submission process.

References:

U.S. Congress. (1934) United States Code: Federal Food, Drug, and Cosmetic Act, 21 U.S.C. §§ 301-392 Suppl.

U.S. Food and Drug Administration/Center for Devices and Radiological Health. (2016). Applying Human Factors and Usability Engineering to Medical Devices Guidance for Industry and Food and Drug Administration Staff. https://www.fda.gov/media/80481/download

U.S. Food and Drug Administration/Center for Devices and Radiological Health/Center for Biologics Evaluation and Research. (2023). Cybersecurity in Medical Devices: Quality System Considerations and Content of Premarket Submissions Guidance for Industry and Food and Drug Administration Staff. https://www.fda.gov/media/119933/download

For more resources on Medical Device Human Factors please check out our blog and YouTube channel.