AI/ML Model Risk Management

Problem and Need for the Study

Despite a rapid increase in applications of artificial intelligence (AI) and machine learning (ML) in society, its implementation in medicine lags behind. A key barrier is the lack of trust in AI/ML models, which depends on their reliability and robust risk management processes.

ISO 14971, Application of Risk Management to Medical Devices, is the FDA recognized consensus standard, which was originally developed for medical devices and is being used for AI/ML models. Unlike medical devices that serve  homogeneous populations, AI/ML models can be applied across broad and varied patient populations. As a result, risk management for AI/ML must target individual patients to account for unique patient-level differences within diverse populations.

Innovation and Impact

This project focuses on developing an individualized risk management platform and a process that implements the ISO 14971 standard for two clinical use cases at M Health Fairview and Mayo Clinic: clinical deterioration and postoperative complications.

Our first aim is to develop computational approaches through the ENTRUST AI Platform. This platform includes a suite of models designed to assess the reliability of the clinical AI predictions and provide individualized insights into patient-specific harms and benefits from interventions.

Our second aim extends current risk management best practices across the entire clinical AI lifecycle, building upon the ISO 14971 standard for new software devices and implementing individualized risk management processes (ENTRUST AI Process) at M Health Fairview and Mayo. This approach ensures that benefits and harms are carefully balanced for each patient using  information from the risk management platform.

These early steps aim to integrate AI into clinical workflows to achieve safe, trustworthy, and fair AI systems that support reliable AI in healthcare.