arXiv · 2023 · Preprint
Vyas et al. (Meta FAIR) · → Paper · Demo: ✓ · Code: ?
Audiobox unifies speech and sound generation in a single 330M-parameter flow-matching model that accepts transcripts, natural language style descriptions, audio context prompts, and voice prompts simultaneously, achieving state-of-the-art results on both zero-shot TTS and text-to-audio generation while enabling fine-grained, disentangled control over vocal and acoustic attributes.
Problem
Large-scale speech generative models in 2023 relied almost entirely on example-based (audio prompt) conditioning, leaving natural language style control largely absent. Users could clone a speaker’s voice from a short clip, but they could not specify “a middle-aged woman speaking with a South Asian accent in an echoey cathedral.” Concurrently, sound generation models handled text descriptions but lacked transcript control, so requesting “a person speaking” produced unintelligible mumbling rather than actual words. The two paradigms had evolved separately, with no unified model bridging them. Standard evaluation tools, such as CLAP, were trained on sound-event captions and completely failed to capture fine-grained speech attributes like accent or speaking rate, leaving the description-conditioned speech task without a reliable automatic metric.
Method
Audiobox builds on Voicebox (Le et al., 2023) and SpeechFlow (Liu et al., 2023), both of which use conditional flow-matching over continuous audio representations. The core backbone is a 330M-parameter Transformer with 24 layers, 16 attention heads, 1024/4096 embedding/FFN dimensions, U-Net skip connections, and convolutional positional embeddings. Instead of predicting mel spectrograms as in Voicebox, Audiobox operates in the latent space of EnCodec dense features (prior to residual quantization), which generalises to speech, sound, and music without requiring a separate vocoder for each modality.
The training pipeline has three stages. First, Audiobox SSL is pre-trained on 185K hours of unlabeled audio (speech, music, and sound) using a self-supervised infilling objective: the model learns to reconstruct randomly masked spans of EnCodec features without any transcript or caption supervision. This follows the SpeechFlow pre-training strategy, generalised from speech-only to all audio modalities. Second, the pre-trained model is fine-tuned for specific tasks: Audiobox Speech fine-tunes with transcript-guided speech infilling on 100K hours of transcribed multi-domain speech; Audiobox Sound fine-tunes for text-to-sound generation using a two-stage curriculum of noisy tag-based data followed by high-quality captioned data. Third, unified Audiobox combines both tasks in a single model with four conditioning streams: a frame-aligned character transcript, a T5-based text description embedding, a voice prompt encoder (a 3-layer Transformer over EnCodec features), and a masked audio context. During training, each conditioning channel is randomly dropped with defined probabilities so the model can operate with any subset at inference.
Speech captions are generated automatically using LLaMA-2 7B to convert per-utterance attribute tags (age, gender, pitch, speaking rate, accent, emotion, environment) into natural-language descriptions in varied writing styles. A 500-hour high-quality human-annotated subset complements the automatic captions.
For inference acceleration, the paper integrates Bespoke Solvers (Shaul et al., 2023), which learn a reparameterisation of the ODE trajectory to allow accurate integration in as few as 4 steps (8 NFE), compared to 32 steps for the standard midpoint solver and roughly 280 for the adaptive dopri5 solver, without measurable degradation in speaker similarity or WER.
A new evaluation model, Joint-CLAP, is also introduced: a contrastive audio-text encoder that fine-tunes WavLM-base+ and RoBERTa on both speech-description and sound-caption data. It raises text-to-audio retrieval recall@10 on speech from 2.29 (stock CLAP) to 22.01, and shows a Pearson correlation of 0.727 with human REL scores, compared to 0.028 for stock CLAP.
Key Results
On zero-shot TTS (LibriSpeech test-clean), Audiobox Speech achieves SPK-SIM of 0.745 (resynthesised) and 0.734 (original), surpassing Voicebox (0.696 resynthesised) and UniAudio (0.710). More strikingly, cross-domain generalisation is large: Audiobox Speech improves over Voicebox by 0.096 to 0.156 absolute on CommonVoice, Switchboard, Expresso, and internal Accent datasets. Subjective quality MOS on LibriSpeech reaches 4.11, and style similarity MOS reaches 3.88, both substantially ahead of Voicebox (3.70 / 2.85) and YourTTS (1.89 / 1.67).
On text-to-audio generation (AudioCaps), Audiobox Sound achieves FAD of 0.77, well below AudioLDM2-Full (1.76) and TANGO-full-FT (2.19), while being smaller in parameter count than all audio-LDM2 variants. The unified Audiobox model, handling both speech and sound, achieves FAD of 1.10 on AudioCaps, which places it above all unified baselines and above TANGO-full-FT among sound-only models, only trailing the dedicated Audiobox Sound model (0.77).
Bespoke Solver at 4 steps (8 NFE) matches midpoint at 32 steps (64 NFE) exactly in speaker similarity and WER across five test sets, confirming a 25x speed-up over the adaptive dopri5 baseline without quality loss.
Fairness analysis over 64 accent groups and two gender groups shows WER of approximately 1.5% across all groups, with speaker similarity means within one standard deviation across all groups, providing evidence of equitable performance.
Novelty Assessment
The main contributions are a combination of genuine architectural and framing innovations with strong engineering execution. Extending flow-matching-based infilling from speech to all audio modalities in a single backbone is a meaningful structural step. The disentangled voice-prompt plus description conditioning scheme, with adversarially selected voice prompts during training to de-correlate vocal from non-vocal attributes, is new and not present in prior unified audio models. Joint-CLAP fills a real gap: existing CLAP models scored near-zero on speech-attribute retrieval tasks, making the entire description-conditioned speech evaluation framework effectively unreliable.
The self-supervised pre-training strategy is adapted from SpeechFlow rather than invented here, and the Bespoke Solver is also a separately published technique. The LLM-based caption pipeline, while practical, is incremental (template-fill attribute labelling). The largest single contribution may be empirical: demonstrating that a unified model trained on 185K hours can outperform modality-specific specialists on their own benchmarks, which the community had not previously verified at this scale.
Field Significance
Tip
High significance — Audiobox demonstrates that a unified flow-matching model trained on diverse audio modalities can exceed dedicated modality-specific systems, providing strong evidence against the assumption that specialisation is necessary for competitive performance. It also introduces Joint-CLAP, which enables reliable automatic evaluation of description-conditioned speech generation, a previously unmeasured capability. Together, these contributions expand the evaluation vocabulary and architectural design space for controllable speech generation.
Claims
- A single generative model trained across speech, sound, and music modalities can match or surpass modality-specific models on dedicated benchmarks. (§5.4, §6.4, §7.5, Tables 1, 5, 13, 14)
- Self-supervised pre-training on large-scale unlabeled audio substantially improves multi-domain style generalisation in subsequent supervised fine-tuning, with gains most pronounced on out-of-domain test sets. (§5.5, Table 3)
- General audio-language embedding models trained primarily on sound events fail to capture fine-grained speech attributes, rendering them unreliable for evaluating description-conditioned speech generation. (§7.3.1, Table 8)
- Flow-matching models admit post-training inference optimisation via learned ODE reparameterisation (Bespoke Solvers) that reduces function evaluations by 25x without measurable quality degradation. (§8, Table 15)
- Removing trailing silence from audio context prompts substantially improves speaker similarity in zero-shot TTS, particularly on datasets with long prompt silences. (§5.5, Table 3)
Limitations and Open Questions
Warning
The speech description data is primarily English, and the description-based conditioning pipeline depends on LLM-generated captions derived from categorical attribute labels. The attribute vocabulary (age, gender, pitch, speaking rate, accent, emotion, environment) has limited granularity; fine-grained attributes like specific regional accents or breed-level sound events cannot be reliably generated if no paired training examples exist for those distinctions.
The gap between dedicated Audiobox Sound (FAD 0.77) and unified Audiobox (FAD 1.10) on the AudioCaps benchmark suggests that task-specific models still have an edge in their domain, even at this training scale. The description-based TTS evaluation relies on Joint-CLAP, which is introduced by the same paper and has not been independently validated as a general benchmark metric. The model and training data are not released publicly (no code repository linked), limiting reproducibility. The voice restylization capability, which requires disentangling vocal identity from environment and emotion, is evaluated subjectively on only two internal test sets, and the degree to which the voice prompt de-correlation training objective succeeds across diverse speakers remains unclear.
Wiki Connections
- flow-matching — Audiobox extends the flow-matching infilling paradigm from speech-only (Voicebox/SpeechFlow) to a unified speech-and-sound model, demonstrating the architecture’s generality across audio modalities.
- zero-shot-tts — Audiobox Speech achieves state-of-the-art speaker similarity on LibriSpeech zero-shot TTS and shows substantially improved generalisation to non-audiobook domains over prior flow-matching baselines.
- instruction-conditioned-tts — Audiobox introduces large-scale description-based conditioning for speech generation, allowing fine-grained natural language control over accent, emotion, pitch, environment, and speaking rate without requiring a reference audio clip.
- neural-codec — Audiobox operates in the dense pre-quantization EnCodec feature space rather than mel spectrograms, enabling a single vocoder-free model to generate speech, sound, and music.
- evaluation-metrics — The paper introduces Joint-CLAP, a WavLM-based contrastive embedding model that substantially improves retrieval and human-correlation performance for description-conditioned speech evaluation over standard CLAP.
- 2301.02111 (VALL-E) — Audiobox Speech outperforms VALL-E on LibriSpeech zero-shot TTS and uses a flow-matching rather than autoregressive codec token approach.
- 2304.09116 (NaturalSpeech 2) — Both use latent diffusion/flow-matching over continuous representations for zero-shot TTS; Audiobox extends the paradigm to multi-modal audio.
- 2210.13438 (EnCodec) — Audiobox uses EnCodec’s dense pre-quantization features as its audio representation space for the generative model.
- 2305.09636 (SoundStorm) — Cited as a concurrent unified audio generation approach using autoregressive token modelling; Audiobox takes a flow-matching route instead.
- 2301.12503 (AudioLDM) — Audiobox Sound directly outperforms AudioLDM and AudioLDM2 on AudioCaps text-to-audio generation while being more parameter-efficient.
- 2310.00704 (UniAudio) — Concurrent unified audio model; Audiobox Speech surpasses UniAudio on LibriSpeech speaker similarity and uses richer description conditioning than UniAudio’s attribute-label captions.