arXiv · 2024 · Preprint
Sun et al. (Microsoft Research) · → Paper · Demo: ? · Code: ?
LatentLM introduces a unified multimodal architecture that handles both discrete tokens (text, code) and continuous data (image, audio, video) within a single causal Transformer by replacing the softmax head for continuous outputs with a per-token diffusion process called next-token diffusion, enabling autoregressive generation of latent VAE vectors rather than discrete codes.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
LatentLM provides an alternative to vector-quantized audio tokenizers for autoregressive TTS: instead of compressing speech into discrete codec codes, the system encodes audio with a continuous VAE (sigma-VAE) and generates latent vectors autoregressively through a lightweight diffusion head conditioned on each Transformer hidden state. The sigma-VAE design prevents variance collapse in the latent space, which is critical for robustness to the exposure bias inherent in autoregressive generation. For TTS, the approach achieves compression ratios up to 6400x at frame rates as low as 3.75 Hz, far below the 75 Hz typical of neural codec tokenizers such as EnCodec, allowing one second of speech to be synthesized in 15 or fewer autoregressive steps. At a frame rate of 7.5 Hz, the system outperforms VALL-E 2 on speaker similarity and word error rate on LibriSpeech test-clean while requiring roughly 10x fewer decoding steps, establishing that continuous latent representations can substitute for multi-level residual codebooks in language-model-based TTS pipelines.
Wiki Connections
- autoregressive-codec-tts — LatentLM proposes continuous latent diffusion as an alternative to discrete codec token prediction, achieving comparable or better TTS quality with far fewer autoregressive steps.
- neural-codec — The paper compares its sigma-VAE tokenizer directly against EnCodec, DAC, WavTokenizer, and Mimi, demonstrating competitive reconstruction quality at substantially higher compression ratios.
- zero-shot-tts — LatentLM is evaluated on zero-shot TTS using a reference utterance or 3-second prefix as prompt, outperforming VALL-E 2 on speaker similarity and WER on LibriSpeech test-clean.
- 2406.05370 (VALL-E 2) — LatentLM uses the same Transformer configuration and training corpus (Libriheavy) as VALL-E 2 and benchmarks directly against it, showing improvements in speaker similarity and robustness.
- 2210.13438 (EnCodec) — EnCodec is used as a lower-bound reference tokenizer in reconstruction quality comparisons; sigma-VAE achieves better PESQ and STOI scores at 40x higher compression.
- 2408.16532 (WavTokenizer) — WavTokenizer is one of the high-compression-ratio baselines against which sigma-VAE is compared in Table 5, with sigma-VAE showing better perceptual quality at similar compression levels.
- 2407.08551 (MELLE) — MELLE is included as a baseline in the zero-shot TTS evaluation table; LatentLM at 15 Hz frame rate outperforms it on both speaker similarity and WER.
- 2204.02152 (UTMOS) — UTMOS is among the reconstruction evaluation metrics used to assess sigma-VAE tokenizer quality in Table 5.
- 2210.02747 (Flow Matching) — Flow matching is listed as an alternative to DDPM for the next-token diffusion head, showing LatentLM’s design is agnostic to the specific denoising objective.
- 2301.02111 (VALL-E) — The original VALL-E codec language model is cited as a foundational prior work establishing the discrete codec token paradigm that LatentLM departs from.