New AI will teach soldiers to dress wounds, fly helicopters in AR

DARPA has demo’d new AI assistants to help soldiers with complex tasks.
Sign up for the Freethink Weekly newsletter!
A collection of our favorite stories straight to your inbox

DARPA (the Defense Advanced Research Projects Agency) is developing AI assistants to help military personnel perform complex tasks — by showing them step-by-step instructions in augmented reality (AR).

AI assistants: Depending on the day, military personnel might be tasked with bandaging battle wounds, repairing fighter jets, or flying them. To make it easier for them to perform such complex duties quickly and correctly, DARPA established the Perceptually-enabled Task Guidance (PTG) program in 2021.

“What we need in the middle is an assistant that can … provide step-by-step guidance.”

Bruce Draper

The idea behind the program is that future soldiers will wear sensors, such as microphones and head-mounted cameras, that collect data on what they’re hearing, seeing, and saying. Instructions from an AI assistant on how to complete tasks could then be displayed to them through AR goggles.

“What we need in the middle is an assistant that can recognize what you are doing as you start a task, has the prerequisite know-how to accomplish that task, can provide step-by-step guidance, and can alert you to any mistakes you’re making,” said Bruce Draper, manager of the PTG program, at the time it was announced.

“The idea is partly for us to see their progress, but also for all of them to work together.”

Bruce Draper

What’s new? Since then, DARPA has awarded PTG contracts and grants to researchers from NYU, Northrop Grumman, and more. Recently, the groups came together at MIT’s Hacker Reactor to demonstrate the first versions of their AI assistants. 

“The idea is partly for us to see their progress, but also for all of them to work together so we get this sort of cross-stimulation of groups working with each other,” said Draper.

 “Cooking is a really good practice domain for all kinds of other highly skilled tasks.”

Bruce Draper

Order up: The first three types of tasks DARPA wants PTG systems to assist with are mechanical repair, battlefield medicine, and pilot guidance.

For this first demo, though, the teams weren’t expected to show how their AI assistants could help fly helicopters — instead, they demonstrated how the systems could help someone follow a cooking recipe.

“[Cooking is] visually quite complex,” said Draper. “There’s specialized terminology. There are specialized devices, and there’s a lot of different ways it can be accomplished, so it’s a really good practice domain for all kinds of other highly skilled tasks.”

“I’m really impressed at how quickly people are making progress toward the goals,” he added.

In addition to showing DARPA their in-development systems, the teams also had a chance to demo them for some of the military members likely to actually use the tech — this allowed the researchers to get feedback on what is or isn’t working from the target end user.

“At the same time, [the military members] hopefully get an idea of where the technology is going so they can bring that back to their leadership with regard to what future possibilities are,” said Draper.

“We believe that our system would not only be useful to DARPA and the military, but also to regular users.”

Ehsan Elhamifar

Over-achieving AIs: It’s not clear when DARPA expects the military to be able to deploy PTG systems, but some of the teams are ahead of schedule, according to Draper, already demonstrating how their AIs could do battlefield medicine tasks.

It’s possible civilians might get a chance to use the AI assistants, too — other tech we use everyday, such as GPS and the internet, was developed for the military before it made its way to the consumer market.

“We believe that our system would not only be useful to DARPA and the military, but also to regular users and people who want to perform new or complex tasks,” said Ehsan Elhamifar, a professor at Northeastern University and one of the PTG grant recipients.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].

Sign up for the Freethink Weekly newsletter!
A collection of our favorite stories straight to your inbox
Related
AI chatbots may ease the world’s loneliness (if they don’t make it worse)
AI chatbots may have certain advantages when roleplaying as our friends. They may also come with downsides that make our loneliness worse.
Will AI supercharge hacking — if it hasn’t already?
The future of hacking is coming at us fast, and it isn’t clear yet whether AI will help attackers and defenders more.
No, LLMs still can’t reason like humans. This simple test reveals why.
Most AI models are incredible at taking tests but easily bamboozled by basic reasoning. “Simple Bench” shows us why.
The future of fertility, from artificial wombs to AI-assisted IVF
A look back at the history of infertility treatments and ahead to the tech that could change everything we thought we knew about reproduction.
“Model collapse” threatens to kill progress on generative AIs
Generative AIs start churning out nonsense when trained on synthetic data — a problem that could put a ceiling on their ability to improve.
Up Next
Subscribe to Freethink for more great stories