Abstract
Autoregressive codec TTS predicts learned speech tokens sequentially, bringing language-model scaling and in-context voice prompting to speech generation. It is established for zero-shot, multilingual, expressive, and conversational synthesis, but tokenization, alignment failures, sequential latency, and stochastic instability remain its defining constraints.
Current State
Codec language modeling established zero-shot voice cloning. Systems beginning with VALL-E show that a short acoustic prompt can condition an unseen voice without speaker-specific fine-tuning. The result generalizes across independent systems, while speaker similarity remains sensitive to prompt quality, duration, and training-speaker coverage.
The tokenizer sets the ceiling. Semantic alignment, bitrate, frame rate, RVQ structure, and reconstruction quality determine what the language model can preserve and how difficult its sequence is to predict. Semantic–acoustic hierarchies improve coherence while retaining waveform detail, but overly strong semantic separation can suppress prosody or identity (AudioLM, SpeechTokenizer).
Alignment—not basic waveform fidelity—is the main robustness problem. Token-by-token systems remain vulnerable to omissions, repetitions, insertions, and runaway loops. Monotonic constraints, duration modeling, phoneme interleaving, and sequence reordering reduce these errors, as in ELLA-V and RALL-E, but often rely on aligners or labels that may fail on spontaneous or low-resource speech.
Autoregression trades flexibility for latency. Causal decoding supports open-ended sampling, streaming, and semantic planning, yet total throughput remains sequential even when time to first audio is low. Hybrid systems increasingly retain autoregressive semantic planning while delegating acoustics to flow, diffusion, or parallel decoders. Masked and parallel alternatives challenge the need for full causality, although matched comparisons remain contested.
Scaling and post-training improve complete systems, not isolated mechanisms. More data and capacity generally improve intelligibility, identity, and expressiveness, while preference or reinforcement learning can improve targeted perceptual dimensions. Data, model, tokenizer, and training recipe often change together, and reward optimization can reduce prosodic diversity or exploit imperfect evaluators.
Automatic scores do not close the quality question. WER and signal-quality predictors miss naturalness, looping, identity drift, and sampling variability. Listener tests and repeated generations remain necessary, especially when tuning decoding or preference rewards.
Method Landscape
- Hierarchical AR plus parallel acoustics predicts a semantic or first-codebook stream autoregressively and fills acoustic detail non-autoregressively.
- Fully autoregressive multicodebook models preserve one causal formulation across token levels, maximizing modeling uniformity at substantial sequence cost.
- Semantic-token plus continuous-renderer hybrids use autoregression for linguistic planning and flow or diffusion for waveform detail; this is the largest recent family.
- Unified speech–text and multistream models share vocabularies across modalities or delay parallel streams to support understanding, generation, streaming, and duplex dialogue.
- Alignment-constrained and preference-tuned variants modify training or decoding around a base codec LM to improve robustness or perceptual quality.
Key Trade-offs
- Semantic coherence versus acoustic fidelity: stronger abstraction improves content while risking identity and prosody.
- Causal flexibility versus throughput: streaming begins early, but sequential generation still limits total speed.
- Stochastic diversity versus determinism: sampling creates prosodic variation and also increases instability and evaluation variance.
- Explicit alignment versus data portability: constraints reduce failures while introducing aligners, duration labels, or language-specific assumptions.
- Unified capability versus specialization: one speech–text backbone broadens tasks but may trail cascades on reasoning or task-specific quality.
Open Questions
- Can alignment failures be eliminated without forced alignment, duration labels, or reranking?
- Which tokenizer properties predict quality and efficiency across unrelated generators?
- Under matched data and compute, when is causality superior to masked or parallel generation?
- How should semantic latency, acoustic latency, interruption, and naturalness be evaluated separately?
- Can preference training improve several perceptual dimensions without reward hacking or diversity collapse?
Go Deeper
Read Autoregressive Codec TTS: In Depth for the evidence on token hierarchies, alignment, scaling, multilingual transfer, duplex systems, preference optimization, and alternatives to full autoregression.
Scope
This synthesis covers 165 papers from 2016 through September 2025, including 117 from 2025. The graph spans TTS, spoken agents, codecs, VC, and evaluation. Fifteen clusters are strongly supported, one is contested, and continuous-token autoregression remains emerging. Results from shared VALL-E, codec, or organizational lineages are not treated as independent replication. System-level gains are interpreted separately from infrastructure, historical, and survey evidence.
This page is generated from wiki/_claims/autoregressive-codec-tts.yaml (digest date: 2026-07-25).
For complete structured claims and paper-level provenance, use the source YAML.