arXiv · 2024 · Preprint
Łajszczak et al. (Amazon AGI) · → Paper · Demo: ✓ · Code: ✗
BASE TTS scales autoregressive codec TTS to 1 billion parameters trained on 100K hours of public-domain speech, introducing novel WavLM-based speaker-disentangled speechcodes with BPE compression and a streamable convolutional decoder, and demonstrating that prosody “emergent abilities” appear at the 500M-parameter / 10K-hour threshold.
Problem
Prior neural TTS systems were trained on a few hundred hours of studio-quality speech, enabling clean enunciation but failing to generalize to textually complex inputs: compound nouns with non-obvious phrasal stress, embedded questions with appropriate intonation, paralinguistic cues like whispers, and foreign-language words. Achieving natural rendering of these phenomena historically required labeled datasets for each specific phenomenon, or explicit syntactic annotations, both of which scale poorly. Discrete speech representations used in earlier large-scale systems (SoundStream, EnCodec) carried residual speaker identity information, making controlled multi-speaker generation harder; and diffusion-based decoders prevented streamable inference, unsuitable for latency-sensitive applications.
Method
BASE TTS (Big Adaptive Streamable TTS with Emergent abilities) frames TTS as next-token prediction across three learned components. A speech tokenizer converts waveforms into a compact discrete sequence; an autoregressive Transformer (“SpeechGPT”) predicts those tokens from text and a reference speech clip; and a convolutional decoder renders the predicted tokens to audio.
The speech tokenizer is the paper’s most architecturally distinctive contribution. Rather than building on an off-the-shelf audio codec, BASE TTS extracts hidden states from a frozen pretrained WavLM model, then passes them through separate content and speaker linear regressors. A contrastive loss and gradient reversal on the speaker branch encourage the vector-quantized content codes to carry phonetic and prosodic information while excluding speaker identity. The resulting codes operate at 50 Hz (one per WavLM frame, i.e. 20ms), which is then compressed roughly 40% by applying Byte-Pair Encoding (BPE) over the discrete vocabulary, reducing the sequence length passed to the Transformer and keeping bitrate at 400 bits/s. A VQ-VAE baseline operating at 325 bits/s is compared against this design.
SpeechGPT is a GPT-2-style decoder-only Transformer trained from scratch. The input sequence concatenates a speaker reference embedding (from a mel-spectrogram encoder), the text token sequence, and the target speechcodes, with separate positional embeddings and prediction heads for text and speech. A small auxiliary text-LM loss (weight 0.01 vs 1.0 for speech) encourages retention of textual structure. Three model sizes are trained: BASE-small (150M, 1K hours), BASE-medium (400M, 10K hours), and BASE-large (980M, 100K hours).
The speechcode decoder replaces the diffusion decoder used in the baseline. It is an end-to-end convolutional system that takes the last hidden state of SpeechGPT (rather than the discrete codes themselves) as input, upsamples via a decoder block, and feeds into a BigVGAN vocoder. This design is streamable: synthesis can begin after only a few decoded speechcodes, achieving first-byte latency of approximately 100ms, compared to diffusion which requires the entire sequence before audio can be produced. Total SpeechGPT+decoder parameter count is approximately 1.13B.
Key Results
In MUSHRA evaluations against publicly available baselines (YourTTS, Bark, TortoiseTTS) in a one-shot setting with 10-second reference clips, BASE TTS achieves scores of 71.7 and 73.7 on two US English speakers respectively, compared to Tortoise’s 68.8 and 58.5, and Bark’s 46.0 and 49.7 (Table 6). Objective metrics on 10 speakers and 307 samples show BASE TTS at WER 6.5% vs Tortoise 8.0%, Bark 19.2%, and YourTTS 16.3%, and speaker similarity (cosine) of 92.7 vs Tortoise 90.3, Bark 91.7, and YourTTS 90.1 (Table 7).
The WavLM-based speechcodes match VQ-VAE on English (MUSHRA 74.7 vs 74.8, non-significant) but outperform it on Spanish despite Spanish comprising only 2% of training data (MUSHRA 74.7 vs 73.3, significant for some speakers), supporting the hypothesis that better representations matter more in lower-resource languages (Table 3). The speechcode decoder matches or improves over the diffusion decoder on 4 of 6 evaluated speakers, while offering a 3x compute reduction and enabling streaming (Table 4).
On the “emergent abilities” linguistic evaluation, BASE-small (150M, 1K hours) fails to recognise emotions and paralinguistics (average score below 1.25 out of 3). BASE-medium (400M, 10K hours) makes a large jump across all categories and never falls below 1.75. BASE-large shows continued but diminishing gains; emotions and paralinguistics remain the hardest categories even at scale (Figure 4, Table 5).
Baseline comparisons (Tortoise, Bark, YourTTS) use publicly available checkpoints not trained on comparable data volumes; the gains partly reflect data scale rather than architecture alone. Evaluations use proprietary test speakers not publicly released, limiting independent verification.
Novelty Assessment
The architecture combines established components, but the combination is not trivial. WavLM-based semantic tokenization with explicit speaker disentanglement via gradient reversal and contrastive losses is genuinely new for the TTS pipeline, distinct from both pure acoustic codecs (EnCodec, SoundStream) and prior semantic-token work (AudioLM, SpeechTokenizer). BPE applied to speechcodes as a sequence-length reduction technique for autoregressive modeling is a direct but effective adaptation from text NLP. The streamable convolutional decoder replacing the diffusion stage is an engineering contribution with clear practical value.
The “emergent abilities” framing and evaluation testset are the paper’s most influential conceptual contribution: they operationalize the LLM notion of emergence for TTS, defining seven challenge categories (compound nouns, emotions, foreign words, paralinguistics, punctuations, questions, syntactic complexities) with a released test set. This benchmark methodology fills a gap that MUSHRA alone cannot address.
The claim of “emergent abilities” in the LLM sense is suggestive rather than rigorously verified: the 500M/10K-hour threshold is observed on one evaluation set and three data-scale points, not established as a phase transition. The paper itself acknowledges the analogy rather than proving it formally.
Field Significance
Tip
High, combining architectural-novelty and empirical-benchmark contributions. BASE TTS introduces a speaker-disentangled SSL-based tokenization scheme and a streamable convolutional decoder that together address two persistent limitations of codec-based autoregressive TTS. More broadly, the emergent abilities test set provides a structured linguistic challenge benchmark that prior scale papers lacked, enabling the field to measure prosody quality on hard textual inputs beyond MUSHRA naturalness scores. The data-and-parameter scaling analysis at 1K/10K/100K hours offers concrete thresholds around which capabilities shift, informing future large-scale TTS training decisions.
Claims
- Scaling autoregressive codec TTS to 500M+ parameters and 10K+ hours of training data produces qualitatively different prosody rendering on linguistically complex inputs compared to smaller models trained on less data. (§4.3, Figure 4, Table 5)
- SSL-based speech representations with explicit speaker disentanglement outperform purely acoustic codec representations for zero-shot TTS, particularly in lower-resource languages. (§4.1, Table 3)
- A streamable convolutional decoder can match or exceed a diffusion-based spectrogram decoder in subjective naturalness while reducing synthesis compute by approximately 3x and enabling low-latency streaming. (§4.2, §4.5, Table 4)
- Applying BPE to discrete speech tokens reduces autoregressive sequence length by approximately 40% without degrading downstream synthesis quality, enabling longer-context training. (§2.2.3)
- Autoregressive TTS trained at scale generalises to a wide range of textual phenomena without any explicit prosody annotation or task-specific supervision. (§4.3, §6)
Limitations and Open Questions
Warning
Model weights are not released, and evaluation uses proprietary test speakers. The MUSHRA baselines (YourTTS, Bark, TortoiseTTS) are not trained on comparable data or compute, making architecture-level conclusions difficult to separate from scale effects.
The speechcode decoder is tightly coupled to a specific frozen SpeechGPT checkpoint via hidden-state conditioning, preventing modular updates and complicating experimentation. The paper identifies this as a limitation requiring future work.
Hallucinations and cutoffs remain an inherent issue of the autoregressive formulation, worsened by misalignment between noisy web audio and ASR-generated transcripts. The authors avoid denoising during training to test robustness but acknowledge this makes the alignment problem harder.
Emotions and paralinguistics remain below ceiling even for BASE-large, suggesting that 100K hours and 980M parameters are not sufficient for reliable rendering of these categories. Formal scaling laws for TTS (analogous to Chinchilla for text LMs) are proposed as future work but not established here.
The “emergent abilities” phenomenon is characterised across only three data-scale points and assessed by a single expert linguist, leaving open whether it is a smooth or discontinuous function of scale and how sensitive it is to tokenization and architecture choices.
Wiki Connections
Concepts: autoregressive-codec-tts · neural-codec · self-supervised-speech · zero-shot-tts · streaming-tts · prosody-control
Related papers: 2301.02111 (VALL-E, similar autoregressive codec LM approach) · 2305.07243 (TortoiseTTS, baseline compared directly) · 2305.09636 (SoundStorm, cited for parallel audio generation) · 2304.09116 (NaturalSpeech 2, cited for latent diffusion comparison) · 1712.05884 (Tacotron 2, prior pipeline contrasted) · 2210.13438 (EnCodec, codec baseline) · 2010.05646 (HiFi-GAN, vocoder component) · 2303.03926 (VALL-E X, cross-lingual extension) · 2308.16692 (SpeechTokenizer, related disentangled codec)
2407.08551 — Autoregressive Speech Synthesis without Vector Quantization