Speech Generation Research Wiki

MusicLM: Generating Music From Text

7 min read

arXiv · 2023 · Preprint

Agostinelli et al. (Google) · → Paper · Demo: ✓ · Code: ✗

MusicLM extends AudioLM’s hierarchical autoregressive framework to text-conditioned music generation at 24 kHz by using a joint music-text embedding model (MuLan) as conditioning, enabling training on audio-only data without requiring paired text-music corpora.

Problem

Generating long-form, high-fidelity music from free-text descriptions requires simultaneously solving two hard subproblems: modelling long-term temporal coherence across many seconds of audio, and grounding generation to the semantic content of an unconstrained natural-language prompt. Prior text-conditioned audio models (DiffSound, AudioGen) relied on modest paired audio-text corpora totalling fewer than 5,000 hours, which limited the richness and variety of generated content. Models capable of coherent long-form generation (Jukebox, AudioLM) lacked text conditioning. MusicLM addresses both gaps by decoupling the conditioning problem from the generation problem through a joint music-text embedding space.

Method

MusicLM builds on AudioLM’s hierarchical tokenization and autoregressive generation scheme, extending it with a text conditioning pathway. The system uses three independently pretrained and frozen models: SoundStream (a neural audio codec operating at 24 kHz, 6 kbps, with 12-level RVQ yielding 600 acoustic tokens per second), w2v-BERT (a 600M-parameter self-supervised model whose intermediate representations are discretized into 25 semantic tokens per second via k-means clustering), and MuLan (a contrastive music-text joint embedding model trained on weakly paired music-text data).

The key insight behind the conditioning approach is that MuLan’s shared embedding space allows the training-inference modality gap to be bridged: during training, MuLan audio embeddings (12 tokens computed from 10-second windows) condition the autoregressive models; at inference, MuLan text embeddings replace them. This eliminates the need for paired text-music data during training, allowing the system to be trained on a 280,000-hour proprietary music corpus.

Generation proceeds through three sequential autoregressive stages, each implemented as a 430M-parameter decoder-only Transformer with 24 layers, 16 attention heads, and an embedding dimension of 1024. The semantic modeling stage predicts semantic tokens conditioned on MuLan tokens; the coarse acoustic stage predicts the first four SoundStream RVQ levels conditioned on semantic and MuLan tokens; the fine acoustic stage predicts the remaining eight RVQ levels. Long sequences are generated by advancing with a 15-second stride and using the previous segment as a prefix.

Left: During training we extract the MuLan audio tokens, semantic tokens, and acoustic tokens from the audio-only training set. In the semantic modeling stage, we predict semantic tokens using MuLan audio tokens as conditioning. In the subsequent acoustic modeling stage, we predict acoustic tokens, given both MuLan audio tokens and semantic tokens. Each stage is modeled as a sequence-to-sequence task using decoder-only Transformers. Right: During inference, we use MuLan text tokens computed from the text prompt as conditioning signal and convert the generated audio tokens to waveforms using the SoundStream decoder.

The paper also introduces melody conditioning as an extension: a small ViT trained with semi-hard triplet loss produces melody-invariant-to-acoustics embeddings, which are concatenated with MuLan tokens to enable generation that follows a hummed or whistled melody while adhering to a text style description.

Key Results

On MusicCaps, MusicLM outperforms both baselines across all reported metrics. In audio quality (FAD-VGGish), MusicLM scores 4.0 compared to 9.6 for Mubert and 13.4 for Riffusion. On MuLan Cycle Consistency (MCC, measuring text-audio alignment), MusicLM achieves 0.51 versus 0.34 (Riffusion) and 0.32 (Mubert). In pairwise human listening tests (win-count, 1200 ratings), MusicLM accumulates 312 wins versus 158 for Riffusion and 97 for Mubert, though it still falls behind the MusicCaps ground-truth reference (472 wins). KL divergence of AudioSet class predictions shows MusicLM best captures the acoustic characteristics of the target music.

An ablation removing the semantic modeling stage shows that generating acoustic tokens directly from MuLan tokens preserves FAD scores but degrades KLD (1.01 to 1.05) and MCC (0.51 to 0.49), confirming that the semantic intermediary improves text adherence and long-term structure.

Memorization analysis shows exact token matches below 0.2% even when prompting with 10 seconds of training audio, and approximate matches (based on optimal transport between token histograms) below 1%, with detected approximate matches largely corresponding to low-entropy, repetitive token sequences.

Novelty Assessment

MusicLM’s core architectural idea, hierarchical autoregressive modeling over discrete audio tokens, is inherited directly from AudioLM. The genuine contribution is the conditioning mechanism: using a jointly trained audio-text embedding space (MuLan) to decouple text conditioning from audio generation, allowing training on audio-only corpora while enabling text-driven inference. This avoids the paired-data bottleneck that constrained earlier text-to-audio models. The approach is analogous to how DALL-E 2 uses CLIP embeddings, transposed from the image to the music domain. The MusicCaps dataset is a concrete, independently valuable deliverable. The memorization analysis is methodologically careful and serves as a responsible-AI template for subsequent audio LM work.

Field Significance

Tip

High — MusicLM establishes the paired-data-free training paradigm for text-conditioned audio generation at scale: conditioning on a frozen joint embedding at inference rather than requiring paired supervision during training. This pattern, inherited and scaled from AudioLM, provides a generalisable template for audio language models in domains where text-audio pairs are scarce. The MusicCaps evaluation dataset addresses a genuine gap, giving the community a benchmark for text-to-music generation.

Claims

  • Hierarchical autoregressive modeling over semantic and acoustic tokens enables long-form music generation (several minutes) with temporal coherence at 24 kHz. (§3.2, §6)
  • A joint audio-text embedding space can substitute for paired text-audio supervision at training time, allowing generative models to be trained on audio-only corpora and conditioned on text at inference. (§3.1, §4.2)
  • Semantic token intermediaries improve adherence to text descriptions in hierarchical audio generation beyond what direct acoustic token prediction achieves. (§5, Table 1)
  • For text-conditioned music generation, perceptual audio quality (FAD) and semantic text alignment (MCC, KLD) are complementary evaluation axes that do not always correlate with each other. (§4.4, Table 1)
  • Large autoregressive audio LMs trained on extensive unlabeled corpora memorize only a small fraction of training sequences exactly, but approximate semantic matches affect a higher proportion of generated outputs under targeted prompting. (§5, Figure 3)

Limitations and Open Questions

Warning

MCC, one of the two primary text-adherence metrics, is computed using MuLan itself, the same model used for conditioning. This circularity biases the metric in MusicLM’s favour relative to baselines that do not use MuLan representations.

The system inherits MuLan’s known weaknesses: negations in text prompts are not well handled, and temporal ordering of described events is not captured. Vocal quality and lyrics generation are not supported and are noted as future work. The 280,000-hour training corpus is proprietary and not released, limiting reproducibility. The model weights are not released. Evaluation is conducted against two relatively weak baselines (Mubert, a rule-based tag-to-audio API; Riffusion, a fine-tuned image diffusion model on spectrograms); no contemporary neural music generation system of comparable scale is included in comparisons.

Wiki Connections

  • autoregressive-codec-tts — MusicLM demonstrates that the autoregressive-over-discrete-tokens paradigm extends beyond speech to long-form music generation with text conditioning.
  • neural-codec — SoundStream’s RVQ acoustic tokens are the primary audio representation backbone; the hierarchical coarse-to-fine token structure is central to MusicLM’s generation quality.
  • self-supervised-speech — w2v-BERT provides the semantic token layer whose masked-language-modeling pretraining enables coherent long-term structure in generated sequences.
  • spoken-language-model — MusicLM extends the audio language modelling framework introduced by AudioLM to the text-conditioned, music-specific regime.
  • evaluation-metrics — introduces MuLan Cycle Consistency (MCC) as a proxy metric for text-music alignment and MusicCaps as a reference evaluation set, contributing new evaluation tools to the field.
  • subjective-evaluation — the pairwise win-count listening study and its statistical analysis via Wilcoxon signed-rank test with Bonferroni correction illustrate best-practice human evaluation design for generative audio systems.
  • 2209.03143 (AudioLM) — MusicLM builds directly on AudioLM’s hierarchical semantic-then-acoustic generation framework, extending it with text conditioning and a much larger music training corpus.
  • 2210.13438 (EnCodec) — cited as the concurrent neural audio codec work extending SoundStream to higher bitrates; MusicLM uses SoundStream directly rather than EnCodec.

Backlinks

Graph View