arXiv · 2023 · Preprint
Seamless Communication et al. (Meta AI) · → Paper · Demo: ? · Code: ✓
SeamlessM4T is a unified multitask, massively multilingual model supporting speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and ASR across up to 100 languages in a single end-to-end system, outperforming strong cascaded baselines on multiple benchmarks.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
SeamlessM4T provides an end-to-end alternative to cascaded speech translation pipelines that traditionally chain ASR, machine translation, and TTS subsystems. Its architecture combines a w2v-BERT 2.0 Conformer speech encoder (approximately 600M parameters, pre-trained on 1 million hours of unlabeled multilingual audio) with a massively multilingual text decoder derived from NLLB, plus a UnitY-style two-pass speech output stage that generates discrete acoustic units subsequently converted to waveforms via multilingual HiFi-GAN vocoders. The system’s text-to-speech translation pathway (text input to speech output) is directly relevant to multilingual TTS, enabling synthesis in 35 target languages from a single model. The accompanying SeamlessAlign corpus (approximately 470,000 hours of automatically mined speech-to-speech and speech-to-text alignments across 37 languages) provides a large-scale training resource for multilingual speech synthesis and translation research.
Wiki Connections
- multilingual-tts — SeamlessM4T introduces a unified text-to-speech translation pathway covering 35 target languages, demonstrating that a single model can replace per-language TTS subsystems in translation cascades.
- self-supervised-speech — The system’s speech encoder (w2v-BERT 2.0) is pre-trained via self-supervised contrastive and masked prediction objectives on 1 million hours of unlabeled speech, and this representation is central to its downstream translation performance.
- speech-to-speech — SeamlessM4T directly targets the speech-to-speech translation task, using a two-pass UnitY framework that generates intermediate discrete units before synthesising target-language waveforms.
- 2210.13438 (EnCodec) — SeamlessM4T’s speech output stage uses discrete acoustic units and HiFi-GAN vocoders rather than EnCodec tokens, but both works address learned discrete speech representations for high-quality synthesis.
- 2212.04356 (Whisper) — SeamlessM4T benchmarks against Whisper-Large-v2 for ASR and speech-to-text translation, demonstrating improvements in WER and robustness across the majority of overlapping languages.
- 2306.12925 (AudioPaLM) — SeamlessM4T compares directly against AudioPaLM-2 on FLEURS speech-to-text translation, outperforming it by 4.2 BLEU points on into-English directions.
- 2301.02111 (VALL-E) — SeamlessM4T and VALL-E both employ discrete acoustic unit representations as an intermediate speech representation, though targeting translation versus zero-shot TTS respectively.
- 2303.03926 (Cross-lingual voice conversion) — Both papers address cross-lingual spoken output generation; SeamlessM4T frames this as speech-to-speech translation, while the cited work targets voice preservation across languages.