Watch amazing Meta AI translate language in near real-time

Meta has unveiled Seamless, a new language translation AI that can quickly convert speech in one language into another while preserving the original tone and emotion.

State of the art: Back in August 2023 (what feels like ages ago in the field of AI), Meta got a major step closer to its goal of creating a universal language translator, unveiling SeamlessM4T, an AI capable of translating recordings of audio or text in nearly 100 languages into text in any of the languages (or speech in 36 of them).

“In the future, we want to explore how this foundational model can enable new communication capabilities — ultimately bringing us closer to a world where everyone can be understood,” wrote Meta at the time.

What’s new? Meta has now unveiled an updated version of that language translation AI, SeamlessM4T v2, which it says “demonstrates performance improvements for automatic speech recognition, speech-to-speech, speech-to-text, and text-to-speech capabilities.”

It has also shared two new AIs built on the updated model: SeamlessStreaming and SeamlessExpressive.

A typical language translation tool will wait for a speaker to finish a sentence before beginning to translate it. This helps overcome the challenge of different languages relying on different sentence structures — many languages have different subject-verb-object orders, for example — but it can also lead to delays that make conversations feel less natural.

SeamlessStreaming begins translating speech while the speaker is still talking. This means the listener can hear the translation with a delay of just a few seconds.

SeamlessExpressive, meanwhile, focuses less on what the speaker is saying and more on how they’re saying it, helping ensure the translation preserves the rhythm, emotion, and style of the original speech.

Open source: Meta combined SeamlessStreaming and SeamlessExpressive into a single language translation AI, Seamless, and made the code, models, and datasets used to create everything freely available on GitHub.

“By publicly releasing our work, we hope that researchers and developers can expand the impact of our contributions by building technologies aimed at bridging multilingual connections in an increasingly interconnected and interdependent world,” Meta researchers write.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].