Skip to main content
Move the World.
ai models for diagnosing covid-19

Lead image © Chan2545 / Adobe Stock

There is a growing trend in medicine toward using artificial intelligence to assess the risk of a disease, like diabetes or cancer, and researchers are now training algorithms to help predict the severity of COVID-19 by looking for patterns in x-rays of lungs or CT scans.

But is AI ready for COVID-19? A global team of 40 (and growing) researchers think not.

In a living review published in the BMJ, an international team of statisticians is evaluating COVID-19 prediction models using a set of standardized criteria — essentially a litmus test for how good the machine learning algorithms are. As new models appear, the researchers add their assessments to the review.

The effort is co-led by Maarten van Semden, a statistician at the University Medical Center Utrecht. On June 25, he tweeted, "If you are building your first clinical prediction model for COVID, Stop it. Seriously, stop it. /end rant."

Clearly, van Semeden has a few things to say about AI for COVID-19 prediction. And he should, having been involved in reviewing the 4,909 publications and 51 studies describing 66 prediction models. He can tell if the various AIs are smart enough to predict the severity of the disease reliably.

It turns out, they're not.

An AI is only as smart as the data it is built on — something the team says is seriously lacking for COVID-19. For an algorithm to be effective, it needs a ton of data. The review reports that many of the models used small sample sizes, which are simply not big enough to draw reliable conclusions.

This problem is likely exacerbated because research groups aren't sharing their underlying data. Instead, many of the AI models for diagnosing COVID-19 evaluated in the review rely on the limited data they have available to them, in their local area, a school, or a hospital. Some of the studies also had unclear reporting, meaning they weren't up-front on the data or method they used to calibrate their algorithm.

"They're so poorly reported that I do not fully understand what these models have as input, let alone what they give as an output," van Smeden told Discover Magazine. "It's horrible."

In a 2019 pre-pandemic opinion piece, van Smeden and his team refer to poor calibration as the Achilles' heel of risk prediction models, like those built on machine learning. "Poorly calibrated predictive algorithms can be misleading, which may result in incorrect and potentially harmful clinical decisions."

The team warns that even claims of AI models to predict COVID-19 that have "good to excellent discriminative performance" are likely biased. The bias can come from study patients who may not adequately represent the general population or from model "overfitting" — meaning the algorithm exaggerates the importance of every random idiosyncrasy in the data. Using AI to make medical decisions before it is vetted is dangerous, and it could lead to poor medical decisions, overspending, and — even worse — more patient suffering.

An AI's promise to offer a little assistance is likely appealing to the overtaxed hospitals around the world. But the team cautions, "We do not recommend any of the current prediction models to be used in practice," and stresses the urgency for more data sharing.

Efforts to amass better COVID-19 data from across the country and world are underway. Efforts include observational data listed by the global Cochrane COVID-19 Study Register or the AWS Data Lake for COVID-19, while the NIH is compiling a register of COVID-19 open-access data and computational resources.

But public health data sharing is often challenging. Common barriers include inconsistent data collection methods, problems with consent agreements (where patients agree to participate in one study, but not necessarily more), or lack of "metadata" (the context the data was collected under). To create reliable AI models of the disease, not only will researchers need to share their data, but they will also need to resolve these other issues regarding data collection and sharing techniques.

Up Next

Public Health
Doctors Use AI to Test New Coronavirus Treatments on Patients
Coronavirus Treatments
Public Health
Doctors Use AI to Test New Coronavirus Treatments on Patients
The University of Pittsburgh Medical Center launched a new trial that uses artificial intelligence to test promising coronavirus treatments as quickly as possible.

The University of Pittsburgh Medical Center launched a new trial that uses artificial intelligence to test promising coronavirus treatments as quickly as possible.

Computing
World’s Fastest Supercomputer Helps Battle COVID-19
supercomputer
Computing
World’s Fastest Supercomputer Helps Battle COVID-19
The Summit supercomputer made quick work of complicated simulations to identify 77 compounds that could be promising COVID-19 treatments.

The Summit supercomputer made quick work of complicated simulations to identify 77 compounds that could be promising COVID-19 treatments.

Health
Data Scientists Are Making It Easier to Track COVID-19
Track COVID-19
Health
Data Scientists Are Making It Easier to Track COVID-19
Teams of computer scientists across the globe are working tirelessly to help track COVID-19 through the use of computer modeling and data dissemination.

Teams of computer scientists across the globe are working tirelessly to help track COVID-19 through the use of computer modeling and data dissemination.

Computer Science
Crowdsourcing the Seed for Coronavirus Antiviral Medications
antiviral medications
Computer Science
Crowdsourcing the Seed for Coronavirus Antiviral Medications
Foldit players are solving a protein structure puzzle that could help kickstart coronavirus antiviral medications.

Foldit players are solving a protein structure puzzle that could help kickstart coronavirus antiviral medications.

Sonification
Why Scientists Are Turning the Coronavirus’ Structure Into Music
Coronavirus Structure
Sonification
Why Scientists Are Turning the Coronavirus’ Structure Into Music
MIT scientists have translated a key part of the coronavirus’ structure into music — and the song could help researchers find a way to stop the virus.

MIT scientists have translated a key part of the coronavirus’ structure into music — and the song could help researchers find a way to stop the virus.

Public Health
Wearables Can Detect Coronavirus Symptoms Early
Wearables that Detect Coronavirus Symptoms
Public Health
Wearables Can Detect Coronavirus Symptoms Early
A study involving more than 600 people found that the Oura Ring wearables can detect coronavirus symptoms 72 hours prior to onset with 90% accuracy.

A study involving more than 600 people found that the Oura Ring wearables can detect coronavirus symptoms 72 hours prior to onset with 90% accuracy.

Public Health
A Smart Thermometer Is Helping Fight the Coronavirus
fight the coronavirus
Public Health
A Smart Thermometer Is Helping Fight the Coronavirus
Kinsa Health is helping fight the coronavirus by sharing data collected by its smart thermometers as quickly as possible.

Kinsa Health is helping fight the coronavirus by sharing data collected by its smart thermometers as quickly as possible.

Science
What to Expect In a Post-Meat Future
What to Expect In a Post-Meat Future
Science
What to Expect In a Post-Meat Future
From advanced plant-based meat alternatives to real meat grown in a lab, the days of eating meat from once-living...
By Mike Riggs

From advanced plant-based meat alternatives to real meat grown in a lab, the days of eating meat from once-living animals could be numbered.