Facebook's new augmented reality glasses will turbocharge hearing

Don’t be fooled by Facebook’s augmented reality glasses. Despite being “glasses,” the experience is entirely auditory.

Imagine dining at a cafe, and sounds of clinking silverware disappear. You glance across the table at your friend, and their voice becomes more audible as the chatter coming from the adjacent table fades away. You hear a mountain river and caws of tropical birds — despite sitting in a Starbucks in the middle of the city.

Augmented reality blends the real and virtual worlds by superimposing computer-generated sounds and images in the real world.

Facebook Reality Labs has now released details of how they plan to make augmented reality glasses that can tune out the real world and make computer-generated sounds seem like they’re coming from your actual surroundings. And they are close — but not quite there.

Contextually-Aware Noise Cancellation

The glasses look like regular earbuds, which can be attached to a headset with other devices. Facebook hopes they will enhance face-to-face conversations by erasing extraneous noise in the room and intensifying only the sounds you want to hear. They do this with contextually-aware noise cancellation. Unlike noise cancellation that we are most familiar with, which reduces all outside noise, contextually-aware noise cancellation highlights sounds you want to hear and dims the rest.

They can do this by using multiple tiny microphones that collect sound in your vicinity. Then, by gleaning information from your head movements, it filters out the sounds you aren’t interested in, homing in on those that have your attention.

Audio Presence

The audio team from FRL Research is also creating “Audio Presence,” which would replicate how sounds travel in real space — essentially making virtual sounds coming from the augmented reality glasses indistinguishable from the real world. The goal is that it will be as good as  listening to the world around you, instead of an electronic device, reports Engadget.

“Imagine if you were on a phone call and you forgot that you were separated by distance,” Research Lead Philip Robinson told Road to VR. “That’s the promise of the technology we’re developing.”

Sensors on the headset, combined with AI, can map the space’s geometry and estimate its acoustic properties, giving the headset a general idea of how sound propagates through the room. It can then apply that insight to virtual sounds, making them seem as if they are also coming from somewhere in the area, not the headset.

The crux is in creating a custom “Head-related Transfer Function” (HRTF). It is a digital version of an individual’s ear and how it hears a sound. They are costly and challenging to make, but Facebook hopes to be soon able to generate an HRTF from a photo of an ear, reports Road to VR — yet one more thing Facebook can figure out from a selfie.

The State of Audio AR

Improving the AR audio experience is something that Google, Apple, and other big tech companies are also working on.

Imagine a Zoom meeting where everyone is sitting around a table, and you can hear people to your left, right, and ahead. Apple has a patent for headphones that can do just that. They can identify where sounds come from, enabling users to hear where people are located, even when they are not physically present. And rumors of Apple’s augmented reality glasses or headset leaked earlier this summer. At about the same time, Google announced an AR update that would combine 3D environments and insert virtual objects more realistically, while Microsoft announced new AR tools for its HoloLens.

If all this sounds straight out of a spy movie — smart glasses that pick up a conversation across the room — there is no need to worry yet.

Chief scientist Michael Abrash at FRL Research told The Verge that the technology isn’t powerful enough for long-range spying. But: “If you ever see anyone walking with glasses that are two feet wide… you should be suspicious.”

But, as with any technology once developed, they always seem to get smaller and more inconspicuous.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].