The augmented, virtual, human-machine future of surgery is here

Dr. Stephen Murphy had conducted countless hip replacement operations before, but this one was different. In this one, he and his team could see a 3D hologram overlaid on the patient — a digital model of the patient’s body that existed directly in his line of vision.

The effect? The surgical team had a form of X-ray vision with augmented reality.

“We had done a lot of testing on real human specimens, so we knew what it was going to look like, but to see it in a live patient for the first time was just unbelievable,” Murphy said in an interview with Freethink.

It feels to the surgeon as if she has been transported inside of the patient.

Adam Sachs

From 3D bioprinted body parts to NASA-developed remote-controlled robots that enter your body through your belly button, there’s no shortage of sci-fi headlines about medicine’s far-out future. But amid all the sci-fi sounding stuff, it can be easy to miss the very real ways that new technologies are changing medicine in the present.

These innovations aren’t replacing human expertise, but rather enabling new forms of human-machine collaboration. This collaboration is especially key in the high-stakes environment of the operating room. Where startups in other industries might have the luxury of “moving fast and break(ing) stuff,” there’s no margin for error in surgery.

Augmenting Surgery

Dr. Murphy’s operation, performed in February at New England Baptist Hospital in Boston, was in fact the first-ever AR-guided total hip replacement.

Using Microsoft HoloLens 2 mixed reality headsets, the four-person team could see full 3D holograms and other important information overlaid on the patient’s body, accurately positioned in their line of sight.

HipInsight, which Dr. Murphy invented, is the first intraoperative AR guidance platform designed for joint replacement. It received clearance from the U.S. Food and Drug Administration (FDA) in January.

With HipInsight, 3D images of the patient’s anatomy are captured prior to surgery. In the OR, a tracker tells headsets where to project the 3D information relative to the patient’s body.

3D models and visualization tools have been a part of surgery for many years, but the ability to bring them directly into surgeons’ sight lines — and make necessary adjustments in real time — is a novel development. Beyond the cool factor, this technology allows surgeons to be as precise and minimally invasive as possible.

“We had all these holograms locked up in a computer for a decade — taking all that stuff that was sitting there flat on a screen and then putting it right inside the person’s body where they actually are is totally amazing,” Dr. Murphy said. “Everybody can see it all together, and we can project the whole sequence of the operation and what it should look like. If you have a 3D model of someone on a computer 5 feet away, you can see it, but you can’t experience it.”

HipInsight is among a crop of new AR products that are designed to augment surgery in real time.

“Medicine is heading to a place that has never existed before,” Dr. Murphy said. “Everybody can see it; we had 4 HoloLenses running all day today in surgery — and people that aren’t even scrubbed at the field can log on and see what’s going on.”

Some companies, like Proximie, are taking the premise of remote surgery to the next level.

Virtually There

Proximie enables a truly “borderless operating room.” By using AR, the U.K.-based company lets world-class surgeons oversee operations in remote areas of the world.

“We want to bring together the best of human expertise and technology to imagine a world where operating rooms all around the world are connected,” Proximie CEO and surgeon Dr. Nadine Hachach-Haram told Sifted.

Through Proximie, surgeons overseas can view the procedure through multiple different views, and even direct the on-site team using their hands.

Proximie Highlight Reel (Short) from Proximie on Vimeo.

According to the World Health Organization, 5 billion people don’t have access to safe, timely, and affordable surgical care, and nine in ten people in low- and middle-income countries can’t access basic surgical services.

We’re a long way from addressing these structural inequities, but Proximie — initially rolled out in war zones — could erode the barriers posed by physical location. Specialists around the world can work alongside local professionals as need arises.

“Surgery is very visual,” Hachach-Haram said. “This is like I’m virtually scrubbed in with you, working with you in the actual operation.”

Just as important to this effort is ensuring that insights can be saved for future learning; Proximie records surgeries in order to create a database of shareable knowledge for worldwide use.

“The ability to have a network of operating rooms where every interaction is captured, digitized and analyzed changes the paradigm,” Hachach-Haram said. “It makes surgery more democratized and of a higher quality.”

Shrinking Surgeons

Where Proximie is about broadening who can participate in surgeries, Vicarious Surgical is focused on enhancing surgeons’ capabilities within the OR.

The Boston-based company — on path to become the next VR unicorn when it goes public later this year — combines surgical robotics with virtual reality, letting surgeons control surgical machines in the full immersion of VR headsets.

The surgical robot’s arms sport 9 degrees of freedom (9DoF), letting them effectively move anywhere within the abdomen. It can also capture 360 video, giving surgeons a fully immersive and up-close view of whatever they’re working on.

“Legacy surgical robots force surgeons to stare at a screen or peer into a microscope … resulting in a disorienting and limited perspective,” Vicarious Surgical CEO and Cofounder Adam Sachs said in an interview with Freethink. “Our system uses a head-mounted display to create a real and complete immersive view into the patient’s abdomen.”

A fun way to think about it is that Vicarious “shrinks” surgeons so they can see and conduct their work from a new perspective.

“The head-mounted display allows a surgeon to perform the surgery by simply looking where she wants to look without the use of additional foot pedals and hand controllers,” Sachs said. “By allowing the surgeon to control the camera view naturally using her own head, it feels to the surgeon as if she has been transported inside of the patient and is operating from the ideal viewpoint.”

Either Side of the Operating Room

Virtual reality is also finding its way into other realms of surgery, before and after the procedure itself. FundamentalVR, Medical Augmented Intelligence, Osso VR, and Surgical Theater are among a handful of companies that provide medical training in VR.

Aside from the novelty, the clear benefit of training in VR is that surgeons can practice repeatedly using a virtual reconstruction of the patient they’ll be operating on, a la HipInsight. At the Stanford Neurosurgical Simulation Lab, surgeons can practice on a simulated model of the patient’s brain, allowing them to map out the surgery in advance.

“It’s a window into the brain — and a window into the brain of the particular patient we’re going to operate on,” said Dr. Anand Veeravagu, a Stanford professor of neurosurgery and the head of the Neurosurgical Simulation Lab in a statement.

One major benefit to this form of practice — particularly with highly sensitive parts of the body like the brain — is that it establishes a kind of muscle memory before the actual operation.

Previous studies have demonstrated that training with the Osso VR platform leads to a 230- 300% improvement in surgical performance. Now the startup is working with Wake Forest to understand this muscle memory data at a more granular level.

Using specialized motion trackers, the researchers are working to visualize and compare body movements taken in real-world surgeries and in VR.

“As an orthopaedic surgeon, it’s critical to me that our technology is evidence-based,” said Dr. Justin Barad, CEO and cofounder of Osso VR, in a statement. “As we roll out a completely new way to train, we want our users and customers to continue to see this platform as effective and reliable.”

Meanwhile, VR can also help patients understand their own treatment. Back at the Stanford Neurosurgical Simulation Lab, Sandi Rodoni, a patient awaiting her third aneurysm surgery, was “treated to a virtual reality trip inside her own brain” in VR. Her surgeon was able to demonstrate to her, in full immersion, the procedure he would be undertaking.

“Because I had been through this before, I thought I knew it all until I saw this,” Rodoni said. “I felt better knowing it was so clear to the doctor.”

Virtual Pain Relief

According to Dr. Brennan Spiegel, director of Health-Services Research at Cedars-Sinai Medical Center, there are other ways that patients stand to benefit from VR.

Since being introduced to VR in 2014, Spiegel has studied how VR functions as a tool to modify human perception. Where this gets particularly interesting with regard to surgery is how it can be used to reduce anxiety and pain.

“Patients often have stress or anxiety before they go into the room, even while they’re in the room,” Spiegel said. “There’s research that VR can help really put people at ease, both before and even during the procedure — and that patients are much more willing to repeat procedures if they need to in the future, or require less pain medication during the procedure, if they’re using virtual reality as a pain-fighting intervention.”

In 2019, nearly 50,000 people died from opioid overdoses in the United States, part of an epidemic that the CDC estimates costs the U.S. $78.5 billion per year. Spiegel believes that virtual therapeutics could play a role in curbing opioid reliance.

“We know that even one day of opioids confers about a 6% risk of dependency a year later, which is astonishing. All it takes is one day,” Spiegel said. “When people come out of the operating room, there’s pain involved, and the very first thing that they get is an opioid. It’s just standard of care and no one really questions it. So we’ve been interested in using VR postoperatively.”

It’s not as simple as plopping a patient into a headset when they come out of the operating room. VR has to be explained and integrated in advance to help the patients feel comfortable using it.

“We call it ‘prehab,’ instead of rehab,” Spiegel said. “We explain it to them before they go into the operating room — when people are clear of mind and don’t have operative pain yet. The whole idea is if we do that work ahead of time, educate people, then they may be much more willing to interpose VR postoperatively prior to starting to use high doses of opioids, and some may be very willing to stay on virtual reality.”

From Unimaginable to Indispensible

To the general public, VR and AR applications may still feel a bit far-out, but they’re rapidly proving their use in surgery — connecting specialists across the world, providing unprecedented real-time insight, and democratizing the training process.

Unlike the steep learning curves that came with first-generation robot- and computer-assisted surgery, virtual and augmented tools are generally far more intuitive to pick up — and explain to patients.

And for early-adopters like Dr. Murphy, they rapidly become indispensable.

“You have this three-dimensional information that was disconnected, and now it’s completely connected to your eyes, to your brain, and to the person,” Murphy said. “You have this sense as a surgeon of where things are, that are covered up in space, but when you can actually see it in real time, it’s just unbelievable. It makes you feel like if you do surgery without it, that you’re just totally blind.”

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].