Skip to main content
Move the World.
Phone Cameras Measure Pulse Breathing

Lead Image © National Cancer Institute

Telemedicine has taken off since the pandemic, since healthcare through a camera carries no COVID-19 risk. 

There are limits to what doctors and nurses can learn remotely, however — diagnostic tests and medical measurements that may require physical contact, or close examination of nuances that can be lost over the internet.

Researchers at the University of Washington have devised a way to measure patients' respirations rates and pulse via a computer or smartphone's camera, in an attempt to make telemedicine more accurate and useful.

According to UW News, the system, called MetaPhys, works by using a machine learning algorithm to identify and track changes in how light reflects off a patient's face. Those subtle differences correlate with changes in blood flow, from which pulse and respiration rate can be deduced.

"Machine learning is pretty good at classifying images. If you give it a series of photos of cats and then tell it to find cats in other images, it can do it," Xin Liu, the study's lead author and a Paul G. Allen School of Computer Science & Engineering doctoral student, told UW News. 

"But for machine learning to be helpful in remote health sensing, we need a system that can identify the region of interest in a video that holds the strongest source of physiological information — pulse, for example — and then measure that over time."

The team's original iteration of the system was presented this past December at the Neural Information Processing Systems conference; this second version, which the researchers presented on April 8 at the Association for Computing Machinery's Conference on Health, Interface, and Learning, avoids some of the pitfalls that snared its predecessor.

"There are large individual differences in physiological processes, making designing personalized health sensing algorithms challenging," the researchers wrote in their arXiv-available paper. "Existing machine learning systems struggle to generalize well to unseen subjects or contexts and can often contain problematic biases."

For instance, because the algorithm is looking at reflection of light off the skin, different backgrounds, lighting, and skin color can throw MetaPhys off. 

The first version of the system was trained using a dataset filled with people's faces, as well as their independently measured pulse and respiration rates. But presented with data too different from this dataset, its performance declined.

MetaPhys uses machine learning to calculate respiration rate and pulse based on how light is reflecting off your face.

MetaPhys improves on these issues by creating personalized machine learning models for each patient, picking out places to focus on while under different circumstances, including lighting and skin color. Even still, however, the system needs work. 

"We acknowledge that there is still a trend toward inferior performance when the subject's skin type is darker," Liu said. 

"This is in part because light reflects differently off of darker skin, resulting in a weaker signal for the camera to pick up. Our team is actively developing new methods to solve this limitation."

That issue, which is obviously a cause for concern, slots into the category of what the authors call "individual differences," one of four major challenges facing MetaPhys and other camera-based telehealth measurement systems. Facial hair is something else to overcome.

"Environmental differences" include variations in lighting, while the "sensor differences" between cameras can mean differences in sensitivity. Finally, there are the "contextual differences" in a video, if the patient is performing an action that the algorithm did not encounter in its training data.

Despite the challenges, systems like MetaPhys will likely be needed if telehealth is going to fully live up to the hype. 

We'd love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected]

Up Next

Artificial Intelligence
Deep Learning vs. Machine Learning: Explained
deep learning vs machine learning
Artificial Intelligence
Deep Learning vs. Machine Learning: Explained
Deep learning vs. machine learning AI: what’s the difference?

Deep learning vs. machine learning AI: what’s the difference?

Mental Health
Remote Therapy Is As Effective As Face-to-Face, for Depression
remote therapy
Mental Health
Remote Therapy Is As Effective As Face-to-Face, for Depression
The pandemic has therapists' couches off limits. A new study finds that remote therapy may be as effective for depression as face-to-face, so I gave it a try.

The pandemic has therapists' couches off limits. A new study finds that remote therapy may be as effective for depression as face-to-face, so I gave it a try.

AI
Using Machine Learning to Detect Dementia in Older Drivers
ai predict dementia
AI
Using Machine Learning to Detect Dementia in Older Drivers
Researchers build AI to predict dementia and mild cognitive impairment in older drivers based on driving data.

Researchers build AI to predict dementia and mild cognitive impairment in older drivers based on driving data.

Insects
A French Company Is Building the World’s Biggest Insect Factory Farm
insect farming
Insects
A French Company Is Building the World’s Biggest Insect Factory Farm
Ÿnsect is building the world’s largest insect farming facility in Amiens, France. The mealworms grown there may end up feeding your pets and the salmon on your plate.

Ÿnsect is building the world’s largest insect farming facility in Amiens, France. The mealworms grown there may end up feeding your pets and the salmon on your plate.

Citizen Science
Help Scientists Figure Out Whether Brain Training Apps Work
brain training
Citizen Science
Help Scientists Figure Out Whether Brain Training Apps Work
To figure out how people might benefit from brain training apps, researchers are looking for 30,000 volunteers willing to play brain games science.

To figure out how people might benefit from brain training apps, researchers are looking for 30,000 volunteers willing to play brain games science.

Bionics
The Material That Could Help Humans Become Cyborgs
Cyborgs
Bionics
The Material That Could Help Humans Become Cyborgs
Coating implantable electronics in the polymer PEDOT can extend their life, which could make cyborgs more common in the future.

Coating implantable electronics in the polymer PEDOT can extend their life, which could make cyborgs more common in the future.

Dispatches
FDA Approves AI “Doctor” That Can See Disease in Your Eyes
FDA Approves AI “Doctor” That Can See Disease in Your Eyes
Dispatches
FDA Approves AI “Doctor” That Can See Disease in Your Eyes
How will artificial intelligence transform medicine?