arXiv · 2023 · Preprint
Rubenstein et al. (Google) · → Paper · Demo: ✓ · Code: ✗
AudioPaLM unifies text and speech in a single decoder-only LLM by expanding a pretrained text model’s vocabulary to include discrete audio tokens, enabling speech recognition, translation, and synthesis within one model that inherits the linguistic and translation capabilities of PaLM-2.
Problem
Prior work had treated text and audio modalities with separate models or encoder-decoder architectures where text was the only output modality. Systems like AudioLM could generate naturalistic speech but had no textual grounding; SPEAR-TTS mapped text to speech but kept the vocabularies disjoint. Initializing speech models from text-pretrained LLMs was known to be beneficial, but no prior system allowed a single decoder-only model to freely interleave text and audio as both inputs and outputs across a unified vocabulary. Cascade speech translation pipelines accumulated compound errors and discarded paralinguistic content such as speaker identity and prosody.
Method
AudioPaLM takes a pretrained text-only decoder (PaLM or PaLM-2, up to 8B parameters) and extends its token embedding matrix to accommodate a new set of discrete audio tokens, producing a joint vocabulary. The modification is minimal: only the embedding matrix is resized; the Transformer layers are unchanged. Audio is discretized into semantic tokens at 25 Hz, either via a multilingual w2v-BERT model or the Universal Speech Model (USM) encoder (up to 2B parameters), with k-means quantization to a vocabulary of 1,024 tokens. The newer USM-v2 tokenizer adds an auxiliary ASR loss, substantially improving multilingual coverage.
Task conditioning uses plain-text prefixes, for example [ASR French] or [S2ST English French], tokenized with the standard SentencePiece tokenizer. No special tokens are introduced. For complex tasks, the model can also be trained on combined tasks, emitting intermediate outputs (transcription, then translation, then translated audio) in a single autoregressive pass, which mirrors chain-of-thought prompting and improves performance relative to direct decoding.
The model outputs either text tokens (for ASR or AST) or semantic audio tokens (for TTS or S2ST). Audio tokens are converted to raw waveforms via two downstream stages: either the AudioLM stages 2 and 3 that generate SoundStream codec tokens and then reconstruct the waveform, or SoundStorm, a non-autoregressive alternative that is two orders of magnitude faster and produces more consistent voice quality. Voice conditioning is provided as a 3-second reference clip encoded as both audio tokens and SoundStream tokens, enabling cross-lingual voice transfer.
Training from a pretrained PaLM-2 checkpoint is crucial: ablations show that finetuning an 8B pretrained checkpoint dramatically outperforms training from scratch at the same scale (BLEU 18.4 vs. 6.9 on CoVoST2 AST; §5.4.2, Table 6). All model parameters are updated during finetuning, not just the new audio embeddings.
Key Results
On the CoVoST2 AST benchmark, AudioPaLM-2 8B achieves 37.8 BLEU, compared to 29.1 for Whisper Large-v2 (1.5B) and 30.7 for USM-M. On CVSS S2ST, the 8B S2ST model achieves 32.5 ASR-BLEU vs. 25.6 for Translatotron 2. On VoxPopuli ASR, AudioPaLM-2 8B achieves 9.8% WER, competitive with mSLAM-CTC (9.1%) and below Whisper (13.6%; Table 2).
For speech quality in S2ST, AudioPaLM substantially outperforms both the CVSS-T ground-truth TTS and Translatotron 2 on subjective audio quality (MOS 4.44 vs. 3.96 vs. 3.88) and voice similarity (SMOS 4.00 vs. 3.51 vs. 3.70; Table 4). The results show that AudioPaLM’s voice transfer quality exceeds what the CVSS-T TTS system achieves despite operating across languages.
Zero-shot AST on FLEURS shows AudioPaLM-2 achieving 20.7 BLEU on language pairs where only ASR data was seen during training, outperforming Whisper (19.6) despite Whisper having seen speech-to-text translation data for all those languages. This demonstrates that translation capability transfers from the PaLM-2 base model to unseen speech-language pairs (§5.2, Table 3).
Ablation over tokenizers shows a large gap: USM-v2 tokens reach 30.5 BLEU on CoVoST2 AST while w2v-BERT tokens yield only 15.2 BLEU at the same model scale, establishing audio tokenizer quality as the primary performance bottleneck (§5.4.3, Table 7).
Novelty Assessment
The core architectural idea, expanding a text LLM’s vocabulary with discrete audio tokens to produce a unified multimodal decoder, is genuine and clean. It differs from encoder-decoder fusion approaches (Whisper, Flamingo-style adapters) because the same sequence model handles both modalities without separate encoders, and the output can be either text or audio. The initialization from PaLM-2 weights is the central empirical finding: it explains most of the performance advantage over from-scratch training and over smaller baselines.
The downstream audio generation (SoundStream, AudioLM stages, SoundStorm) is borrowed directly from prior Google work. The tokenization pipeline builds on w2v-BERT and USM, also preexisting systems. The combined-task prompting scheme is an extension of T5’s text-to-text framing and chain-of-thought prompting to the speech domain. The contribution is therefore a mix of genuine architectural novelty (unified vocabulary, LLM initialization for multimodal speech) and engineering integration of several preexisting systems into one unified training framework.
Field Significance
Tip
High — AudioPaLM demonstrates that a text-pretrained LLM can be converted into a multimodal speech-text system with minimal architectural changes, and that its text-domain linguistic knowledge directly improves zero-shot speech translation. The result establishes a template for initializing speech LMs from text LLM checkpoints that subsequent work builds on, and provides early strong evidence for the hypothesis that the field’s large investment in text LLMs can be leveraged for speech without full retraining.
Claims
- Initializing a speech-text LLM from a pretrained text-only checkpoint substantially outperforms training from scratch at equivalent model scale. (§5.4.2, Table 6)
- Audio tokenizer quality is a primary bottleneck in LLM-based speech generation and understanding: stronger semantic tokenizers yield large downstream gains independent of LM scale. (§5.4.3, Table 7)
- A unified multimodal vocabulary that interleaves text and audio tokens enables zero-shot speech translation to language pairs not seen during speech training, by inheriting translation capability from text pretraining. (§5.2, Table 3)
- Training on combined tasks that decompose complex speech operations into intermediate text steps improves performance over direct end-to-end decoding. (§5.4.4, Table 8)
- Voice identity preservation in cross-lingual speech synthesis can exceed high-quality TTS-based references when an audio LM is conditioned on a short spoken prompt. (§5.3, Table 4)
Limitations and Open Questions
Warning
The entire system depends on the quality of the audio tokenizer, which is not released and requires access to Google-internal USM models. The best-performing configuration (AudioPaLM-2 with USM-v2 tokens) is not reproducible externally; the published ablations use the multilingual w2v-BERT tokenizer as the weakest condition, suggesting that reported performance at USM-v2 quality cannot be independently verified.
Adding S2ST tasks modestly degrades ASR and AST performance, suggesting that sharing model capacity across output modalities introduces trade-offs that are not fully resolved by the training mixture design (§5.4.5, Table 9). The paper evaluates primarily on speech translation tasks; generative speech quality at naturalness is only assessed in the S2ST with voice transfer setting, not for open-ended TTS. The subjective evaluations were conducted on an earlier version of AudioPaLM using AudioLM decoding rather than SoundStorm, meaning the best-performing decoder was not evaluated subjectively. Evaluation benchmarks for generative audio tasks more generally remain underdeveloped relative to text, a limitation the paper explicitly notes.
Wiki Connections
Architecture builds on 2209.03143 (AudioLM) for the hierarchical audio token framework and voice conditioning scheme, and on 2305.09636 (SoundStorm) as the fast non-autoregressive decoder replacing AudioLM’s slow stage-2 and stage-3 inference. The tokenization scheme uses SoundStream from 2210.13438 (EnCodec predecessor) as the acoustic codec underlying AudioLM. The unified vocabulary approach contrasts with VALL-E (2301.02111) which uses a codec-only (EnCodec) token representation and encoder-decoder conditioning, and with SPEAR-TTS which keeps text and audio vocabularies separate. Cross-lingual voice transfer results are compared against 2303.03926.
Relevant concepts: autoregressive-codec-tts, spoken-language-model, neural-codec, self-supervised-speech, multilingual-tts, zero-shot-tts
2310.00704 — UniAudio: An Audio Foundation Model Toward Universal Audio Generation 2507.16632 — Step-Audio 2 Technical Report