ICML · 2023 · Conference
Liu et al. · → Paper · Demo: ✓ · Code: ✓
AudioLDM applies latent diffusion models to text-to-audio generation, decoupling generative model training from audio-text pairing by conditioning on CLAP audio embeddings during training and text embeddings at inference.
Problem
Prior text-to-audio systems either operated in discrete token space (DiffSound, AudioGen) or required large paired audio-text corpora whose quality was often poor and whose text preprocessing discarded rich semantic detail. Diffusion models applied directly in waveform or full mel-spectrogram space were computationally expensive at inference. No prior work had demonstrated zero-shot text-guided audio manipulation (style transfer, inpainting, super-resolution) from a single general-purpose generative model.
Method
AudioLDM compresses mel-spectrograms into a compact continuous latent space using a convolutional VAE (compression level r=4 by default), then trains a UNet-based latent diffusion model (LDM) within that space. The key architectural decision is the conditioning strategy: during LDM training, the model is conditioned on audio embeddings from a pretrained CLAP model (contrastive language-audio pretraining), rather than on text embeddings. Because CLAP aligns audio and text representations in a shared embedding space, inference can substitute text embeddings for audio embeddings without ever training on audio-text pairs. This decoupling removes the need for supervised language-audio pairs during generative model training and sidesteps the quality and scarcity constraints of available caption data.
Classifier-free guidance (CFG) is applied using the natural language text embedding at sampling time. The UNet backbone follows Stable Diffusion’s architecture, with feature-wise linear modulation (FiLM) for conditioning rather than cross-attention, since the conditioning is a single vector rather than a sequence. A HiFi-GAN vocoder converts reconstructed mel-spectrograms to waveforms. Two model sizes are provided: AudioLDM-S (181M parameters) and AudioLDM-L (739M parameters).
Zero-shot audio manipulation (style transfer, inpainting, super-resolution) is achieved without any task-specific fine-tuning by manipulating the reverse diffusion starting point and observation masks in latent space, as the spatial structure of the VAE latent roughly preserves the time-frequency correspondence of the mel-spectrogram.
Key Results
On the AudioCaps test set, AudioLDM-L-Full achieves a Frechet distance (FD) of 23.31 vs. 47.68 for DiffSound (lower is better). Subjective overall quality (OVL) and text relevance (REL) scores reach 65.91 and 65.97 out of 100, compared to DiffSound’s 45.00 and 43.83. The smaller AudioLDM-S already surpasses both DiffSound and AudioGen on objective metrics, despite being trained on a single RTX 3090 GPU against DiffSound’s 32 V100s and AudioGen’s 64 A100s.
A key ablation (Table 3) demonstrates that conditioning LDMs on audio embeddings during training produces consistently better results than conditioning on text embeddings, even when augmented text data is used. This holds across all model sizes and data scales, confirming that the CLAP-based training strategy is central to the system’s performance advantage.
For zero-shot super-resolution, AudioLDM-L achieves an LSD of 0.98 on VCTK, outperforming the AudioUNet baseline (1.32) though falling short of the specialist NVSR system (0.78). Inference speed on a single A100 is approximately 2x faster than DiffSound for batch generation of 10-second audio (Table G, Figure 8).
Novelty Assessment
The paper’s primary contribution is combining CLAP-conditioned LDM training (conditioning on audio embeddings, inferring from text embeddings) with a mel-spectrogram VAE, directly adapting the Stable Diffusion framework for general audio. The training-inference conditioning asymmetry is genuinely novel and practically significant: it removes the bottleneck of paired data quality without sacrificing text control. The manipulation capabilities (style transfer, inpainting, super-resolution) from a single model are demonstrated convincingly as zero-shot capabilities rather than separate supervised systems.
The LDM backbone and VAE architecture are direct adaptations of Rombach et al.’s image generation work (Stable Diffusion), and HiFi-GAN is a standard vocoder. The novelty is in the domain adaptation and the CLAP-conditioning trick, not in inventing new generative model components. The paper does not cover speech naturalness or speaker control, and operates at 16 kHz, which limits music generation quality.
Field Significance
Tip
High — AudioLDM introduces a practically important training strategy for text-conditioned audio generation: using contrastive audio-text embeddings to bridge the modality gap at training time without requiring paired text supervision for the generative model. This pattern (training on audio embeddings, inferring from text embeddings via a shared contrastive space) enables general audio generation from natural language descriptions at quality levels that far exceed prior discrete-space systems, and does so with an order-of-magnitude reduction in compute. The paper also establishes zero-shot audio manipulation as a tractable capability for latent diffusion systems.
Claims
- Latent diffusion models operating on compressed mel-spectrogram representations achieve significantly better computational efficiency than diffusion models applied in raw waveform or full-resolution spectrogram space, without sacrificing generation quality. (§3.2, §G, Table 4)
- Conditioning generative models on audio embeddings from a contrastive audio-text model during training, then substituting text embeddings at inference, can outperform direct text-conditioned training when paired audio-text data is limited or noisy. (§5.1, Table 3)
- Latent diffusion models trained for audio generation can perform zero-shot manipulation tasks (inpainting, super-resolution, style transfer) by controlling the reverse diffusion starting point, without task-specific fine-tuning. (§4, Table 5)
- Text-guided audio generation quality scales with both model size and training data volume, but data scale provides larger gains than model size at comparable compute. (§5.1, Table 1)
Limitations and Open Questions
Warning
AudioLDM operates at 16 kHz, limiting music generation fidelity. All pipeline modules (VAE, LDM, vocoder) are trained separately, creating potential misalignment; end-to-end training is left as future work. The evaluation covers only AudioCaps and AudioSet benchmarks; how the model generalises to out-of-distribution prompts or professional sound design scenarios is not assessed.
The zero-shot super-resolution capability falls short of specialist models (NVSR), as expected for a general-purpose model trained on noisy in-the-wild data. The human evaluation uses only six raters and 60 samples per model, limiting statistical power. Classifier-free guidance scale interacts with the FD and FAD metrics in non-trivial ways (a guidance scale of 3 is optimal for FD but not FAD), suggesting that the optimal operating point depends on the evaluation metric choice.
Wiki Connections
- diffusion-tts — AudioLDM adapts the latent diffusion paradigm from image generation to audio, demonstrating that operating in compressed latent space rather than raw waveform space enables efficient and high-quality generation.
- evaluation-metrics — The paper uses Frechet Distance, Inception Score, KL divergence, and FAD alongside subjective OVL/REL ratings, and open-sources an evaluation pipeline to facilitate reproducibility.
- 1609.03499 (WaveNet) — WaveNet is cited as the foundational autoregressive waveform generation model that motivated subsequent learned audio representations; AudioLDM moves away from this paradigm toward latent diffusion.