Research Brief: Researchers determine that AI-based tools have not yet reached full diagnostic potential in COVID-19

Study findings underperform radiologist prediction

MINNEAPOLIS/ST. PAUL (07/28/2022) — Published by the Journal of Radiology: Artificial Intelligence, a prospective observational study across 12 hospital systems from the University of Minnesota Medical School evaluated the real-time performance of an interpretable artificial intelligence (AI) model to detect COVID-19 from chest X-rays.

Participants with COVID-19 had a significantly higher COVID-19 diagnostic score than participants who did not have COVID-19. However, researchers found the real-time model performance was unchanged over the 19 weeks of implementation. The model sensitivity was significantly higher in men, while model specificity was significantly higher in women. Sensitivity was significantly higher for Asian and Black participants compared to white participants. The COVID-19 AI diagnostic system had significantly worse accuracy as compared to radiologist predictions.

“This study, which represents the first live investigation of an AI COVID-19 diagnostic model, highlights the potential benefits but also the limitations of AI,” said Christopher Tignanelli, MD, MS, FACS, FAMIA, an associate professor of surgery at the University of Minnesota Medical School and general surgeon at M Health Fairview. “While promising, AI-based tools have not yet reached full diagnostic potential.”

The research findings were informed by an AI algorithm developed by Ju Sun, an assistant professor at the U of M College of Science and Engineering, and his team in collaboration with M Health Fairview and Epic.

• COVID-19 diagnostic models perform well for participants with severe COVID-19 effects; however, they fail to differentiate participants with mild COVID-19 effects. 

• Many of the early AI models in the pandemic that were published boasted overly optimistic performance metrics using publicly available datasets. 

• The AI model’s diagnostic accuracy was inferior to the predictions made by board-certified radiologists.

“We saw the same overly optimistic performance in this study when we validated against two publicly available datasets; however, as we showed in our manuscript, this does not translate to the real world,” Dr. Tignanelli said. “It is imperative moving forward that researchers and journals alike develop standards requiring external or real-time prospective validation for peer-reviewed AI manuscripts.”

Researchers hope to develop a simpler diagnostic AI model by integrating data from more than 40 U.S. and European sites and multi-modal models that leverage structured data and clinical notes along with images.

This study was funded by grants from the National Institutes of Health (NHLBI T32HL07741), as well as the Agency for Healthcare Research and Quality (AHRQ) and Patient-Centered Outcomes Research Institute (PCORI), grant K12HS026379 (CJT) and the National Institutes of Health’s National Center for Advancing Translational Sciences, grants KL2TR002492 (CJT) and UL1TR002494 (EK).

###

Contact: Kat Dodge, Medical School, kdodge@umn.edu.

About the University of Minnesota Medical School
The University of Minnesota Medical School is at the forefront of learning and discovery, transforming medical care and educating the next generation of physicians. Our graduates and faculty produce high-impact biomedical research and advance the practice of medicine. We acknowledge that the U of M Medical School, both the Twin Cities campus and Duluth campus, is located on traditional, ancestral and contemporary lands of the Dakota and the Ojibwe, and scores of other Indigenous people, and we affirm our commitment to tribal communities and their sovereignty as we seek to improve and strengthen our relations with tribal nations. For more information about the U of M Medical School, please visit med.umn.edu.