Watch amazing Meta AI translate language in near real-time

Seamless also retains the original speaker’s tone and emotion.

Meta has unveiled Seamless, a new language translation AI that can quickly convert speech in one language into another while preserving the original tone and emotion.

State of the art: Back in August 2023 (what feels like ages ago in the field of AI), Meta got a major step closer to its goal of creating a universal language translator, unveiling SeamlessM4T, an AI capable of translating recordings of audio or text in nearly 100 languages into text in any of the languages (or speech in 36 of them).

“In the future, we want to explore how this foundational model can enable new communication capabilities — ultimately bringing us closer to a world where everyone can be understood,” wrote Meta at the time.

What’s new? Meta has now unveiled an updated version of that language translation AI, SeamlessM4T v2, which it says “demonstrates performance improvements for automatic speech recognition, speech-to-speech, speech-to-text, and text-to-speech capabilities.”

It has also shared two new AIs built on the updated model: SeamlessStreaming and SeamlessExpressive.

A typical language translation tool will wait for a speaker to finish a sentence before beginning to translate it. This helps overcome the challenge of different languages relying on different sentence structures — many languages have different subject-verb-object orders, for example — but it can also lead to delays that make conversations feel less natural.

SeamlessStreaming begins translating speech while the speaker is still talking. This means the listener can hear the translation with a delay of just a few seconds.

SeamlessExpressive, meanwhile, focuses less on what the speaker is saying and more on how they’re saying it, helping ensure the translation preserves the rhythm, emotion, and style of the original speech.

Open source: Meta combined SeamlessStreaming and SeamlessExpressive into a single language translation AI, Seamless, and made the code, models, and datasets used to create everything freely available on GitHub.

“By publicly releasing our work, we hope that researchers and developers can expand the impact of our contributions by building technologies aimed at bridging multilingual connections in an increasingly interconnected and interdependent world,” Meta researchers write.

We’d love to hear from you! If you have a comment about this article or if you have a tip for a future Freethink story, please email us at [email protected].

AI for agriculture: How Indian farmers are harvesting innovation
India’s farmers combat climate change, pestilence, and financial burdens, with AI-driven initiatives offering transformative solutions.
4 core strategies as AI changes the “meaning of work”
A 2021 study in Germany revealed that automation can have a serious effect on our sense of meaning at work.
AI startup Magic is building a “superhuman software engineer”
Magic AI is developing an advanced AI software engineer it sees as a milestone along the path to artificial general intelligence (AGI).
Google AI is searching the world for methane leaks from space
Google will provide computing resources to MethaneSAT, a project to identify climate-harming methane leaks from space.
Students use calculators to do math. Let them use ChatGPT to write.
Researchers tasked teachers with rating hundreds of essays, some written by students and others written by ChatGPT. Here’s what they found.
Up Next
A robot is working in a factory with a lot of bottles.
Subscribe to Freethink for more great stories