arXiv · 2023 · Preprint
Borsos et al. (Google) · → Paper · Demo: ✓ · Code: ✗
SoundStorm replaces AudioLM’s autoregressive acoustic generation stages with a non-autoregressive, confidence-based parallel decoder operating on residual vector quantization (RVQ) token sequences, achieving two orders of magnitude faster generation at matched quality and improved voice consistency.
Problem
Autoregressive generation of high-quality audio requires long token sequences when using neural audio codecs with RVQ quantisation. The sequence length scales with both audio duration and the number of RVQ levels, making attention-based models computationally prohibitive for generating audio beyond a few seconds. AudioLM’s hierarchical acoustic generation is accurate but slow: generating 30 seconds of audio takes far longer than real time, and the sequential nature of the decoding introduces acoustic drift in long-form outputs. A faster non-autoregressive alternative that respects the hierarchical structure of RVQ tokens was needed.
Method
SoundStorm is a 350M-parameter Conformer trained on the acoustic generation task, taking as conditioning the time-aligned semantic tokens from AudioLM (originating from w2v-BERT quantised with k-means) and producing SoundStream codec tokens (50 frames/second, 12 RVQ levels, codebook size 1024 per level). The model is designed to be a drop-in replacement for AudioLM’s coarse and fine acoustic generation stages.
The key architectural insight is to keep the Conformer’s internal sequence length equal to the number of SoundStream frames rather than the total number of tokens. At the input side, embeddings from all RVQ levels of a single frame are summed together with the conditioning token embedding for that frame, resulting in a sequence length determined solely by audio duration (50 frames/second) independent of the number of RVQ levels. Separate dense output heads per RVQ level then predict the corresponding token targets.
Training uses a masking scheme that extends MaskGIT’s confidence-based parallel decoding to RVQ sequences. In each training step, a prompt delimiter is sampled (tokens before it are never masked, enabling voice prompting), an RVQ level q is sampled uniformly, and tokens at level q are masked with a cosine-schedule masking ratio while all tokens at finer levels q+1,…,Q are fully masked. Cross-entropy loss is computed only on the masked tokens at level q. This scheme explicitly teaches the model the coarse-to-fine conditional structure of RVQ.
At inference, all tokens are initially masked except for the voice prompt. Decoding proceeds RVQ level by level in coarse-to-fine order. Within the first RVQ level, 16 iterations of confidence-based sampling are applied (following MaskGIT’s cosine schedule), with greedy decoding used as the final step of each level. For subsequent levels, greedy decoding in a single pass suffices, exploiting the conditional independence of fine-level tokens given coarser ones. The full decoding of 30 seconds of audio requires only 27 forward passes, yielding a real-time factor of 0.017 on a TPU-v4 (0.5 seconds for acoustic generation, 0.1 seconds for SoundStream waveform decoding).
For dialogue synthesis, a 1.2B-parameter ByT5-large encoder-decoder (frozen encoder, trained decoder) maps byte-level transcripts with speaker-turn markers to semantic tokens, which are then fed to SoundStorm along with short voice prompts. The full pipeline generates 30 seconds of multi-speaker dialogue in 2 seconds.
Key Results
On LibriSpeech test-clean, SoundStorm matches AudioLM’s DNSMOS audio quality (4.01 vs. 3.93 unprompted; both comparable to original recordings at 3.72-3.99) while improving significantly on speech intelligibility and voice consistency. In the prompted short-segment condition, WER drops from 3.77 (AudioLM) to 2.99 (SoundStorm); CER drops from 1.50 to 1.10. For longer segments where AudioLM must rely on a slide-and-prompt chunking strategy, SoundStorm’s coherence advantage grows: acoustic consistency (cosine similarity between prompt embedding and generated embeddings over time) is 0.91 vs. 0.86 for 20-30 second segments. Voice preservation (speaker embedding cosine similarity) improves from 0.48 to 0.59 in the prompted condition.
The runtime gain is two orders of magnitude: SoundStorm generates 30 seconds of audio in 0.5 seconds on a TPU-v4 versus AudioLM’s acoustic stages which scale prohibitively with sequence length. An ablation on decoding iterations (Figure 4) shows that 16 iterations in the first RVQ level provides approximately 0.1-0.2 quality gain over greedy decoding; additional iterations on finer levels yield no statistically significant improvement, validating the greedy-at-fine-levels strategy.
Novelty Assessment
SoundStorm’s primary architectural contribution is the adaptation of MaskGIT’s iterative masked-token decoding to the RVQ hierarchy. The extension is non-trivial: the coarse-to-fine ordering of levels, the frame-level embedding summation, and the masking scheme that mimics inference are all specific to RVQ structure and not directly transferred from the image generation setting. The result is a method that is genuinely faster than any prior acoustic generation approach at comparable quality. The model is not autoregressive in the traditional sense but does rely on autoregressive semantic token conditioning from AudioLM upstream, so “non-autoregressive” applies only to the acoustic stage.
The dialogue synthesis demonstration is largely an engineering integration of SoundStorm with a SPEAR-TTS-style text-to-semantic model, but it provides the first published demonstration of long-form multi-speaker dialogue synthesis at near-real-time speed with controllable speaker turns.
Field Significance
Tip
High — SoundStorm demonstrates that the RVQ hierarchy in neural audio codecs provides a natural factorisation that enables parallel decoding with two-orders-of-magnitude speedup over autoregressive baselines, without quality regression. This result directly informs the design of subsequent codec-based TTS and spoken language model systems that require latency-sensitive or long-form generation. The parallel decoding scheme and the frame-level embedding summation become reference design choices for RVQ-aware acoustic generators.
Claims
- Parallel, iterative masked decoding adapted to RVQ structure enables acoustic token generation two orders of magnitude faster than autoregressive generation at matched perceptual quality. (§4.3, Figure 3)
- Non-autoregressive RVQ-level-by-level decoding maintains better voice and acoustic consistency over long sequences than autoregressive chunk-and-prompt approaches. (§4.2, Table 1, Figure 2)
- Fine-level RVQ tokens are conditionally independent given coarser tokens and can be decoded greedily in a single pass without measurable quality loss. (§3.3, §4.3)
- Confidence-based iterative decoding provides a meaningful quality gain over greedy decoding at the coarsest RVQ level, but additional iterations at finer levels yield no significant improvement for speech. (§4.3, Figure 4)
- Coupling a text-to-semantic token model with an efficient acoustic generator enables real-time synthesis of controllable multi-speaker dialogue at 30-second horizons. (§5)
Limitations and Open Questions
Warning
Evaluation uses a DNSMOS-style estimator rather than human listening tests for audio quality comparisons, and the subjective baseline is carried over from earlier AudioLM papers rather than re-run. Direct perceptual comparisons between SoundStorm and AudioLM on the same conditions by human raters are not reported.
The conditioning mechanism requires time-aligned semantic tokens, restricting drop-in use to AudioLM-family pipelines with compatible semantic tokenisers. Extension to cross-attention conditioning or unconditional generation is left as future work. The 350M acoustic model trained on LibriLight is English-only; no multilingual evaluation is presented. The dialogue synthesis component depends on a proprietary 100k-hour dialogue corpus, limiting reproducibility of that subsystem. The ablation on decoding iterations is limited to speech; the authors hypothesise that multiple fine-level iterations may matter for non-speech audio, but this is not tested.
Wiki Connections
Related concepts: autoregressive-codec-tts, neural-codec, spoken-language-model, zero-shot-tts, self-supervised-speech, evaluation-metrics
In-corpus papers this work builds on: 2209.03143 (AudioLM), 2210.13438 (EnCodec), 2301.02111 (VALL-E), 2301.11325 (MusicLM)
2306.12925 — AudioPaLM: A Large Language Model That Can Speak and Listen 2402.08093 — BASE TTS: Lessons from building a billion-parameter Text-to-Speech model on 100K hours of data 2403.16973 — VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild 2407.05361 — Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation 2407.08551 — Autoregressive Speech Synthesis without Vector Quantization 2411.19842 — Scaling Transformers for Low-Bitrate High-Quality Speech Coding