arXiv · 2024 · Preprint
He et al. (CUHK-Shenzhen / CAS / Shanghai AI Laboratory) · → Paper · Demo: ✓ · Code: ✓
Emilia introduces Emilia-Pipe, an open-source pipeline for turning raw in-the-wild speech into annotated multilingual training data, and uses it to build Emilia (101k hours) and its expanded successor Emilia-Large (216k+ hours across six languages), then validates both at scale against audiobook-derived corpora on formal, spontaneous, scaling, and multilingual/crosslingual speech generation.
This page synthesizes two versions of the same underlying work. The canonical ID and title (this page, arXiv:2407.05361) correspond to the peer-reviewed SLT 2024 paper, which introduced Emilia at 101k hours and validated it against MLS on English TTS. The content below reflects the fuller, later arXiv version, arXiv:2501.15907 (“Emilia: A Large-Scale, Extensive, Multilingual, and Diverse Dataset for Speech Generation”), which its own authors describe as the full version of this paper. That version expands the dataset to Emilia-Large (216k+ hours) and adds data-scaling-law and multilingual/crosslingual experiments not present in the SLT 2024 short paper. arXiv:2501.15907 is not separately tracked in this wiki (it is marked a duplicate of this page in the corpus record); it is cited here as plain text rather than a wikilink.
Problem
Speech generation models trained on audiobook corpora (LibriTTS, MLS, Libri-Light, Libri-Heavy) produce formal, read-aloud speech because that is what those datasets contain. Real human speech in casual and conversational contexts exhibits breathing, pauses, stuttering, repetitions, speaking-rate variation, and emotional variation that audiobook recordings systematically lack. Existing large-scale in-the-wild corpora (GigaSpeech, WenetSpeech, SeamlessAlign) offer broader speaking styles but degrade generation performance if used without careful preprocessing, and prior automatic preprocessing pipelines for in-the-wild data (AutoPrepWild, WenetSpeech4TTS) are proprietary, restricted to Chinese-only audio, and do not report their computational efficiency. There was no open, multilingual, large-scale corpus of spontaneous, in-the-wild speech paired with a reproducible, efficient preprocessing pipeline suitable for training TTS systems toward human-like naturalness.
Method
The Emilia dataset is constructed via Emilia-Pipe, a six-stage open-source preprocessing pipeline applied to raw speech harvested from video and podcast platforms spanning audiobooks, drama, interviews, talk shows, and commentary.
The pipeline stages are: (1) Standardisation, converting all audio to 24 kHz mono WAV at -20 dBFS with normalized amplitude; (2) Source separation using the UVR-MDX-Net vocal-removal model to strip background music and noise; (3) Speaker diarisation using the pyannote 3.1 pipeline (speaker segmentation, embedding, and clustering) to split long-form audio into single-speaker segments; (4) Fine-grained segmentation via Silero-VAD, concatenating consecutive same-speaker voice-activity chunks into 3-30 second utterances; (5) ASR transcription using Whisper-Medium through the WhisperX/CTranslate2 backend, reusing stage-4 boundaries to skip Whisper’s internal VAD and running batched inference for up to 4x the throughput of the official implementation; (6) Filtering by language-identification confidence (discarding data below 80% confidence or outside the six target languages), DNSMOS P.835 OVRL quality score (threshold 3.0, the ITU-T P.835 “fair” anchor), and average-phone-duration outlier removal via interquartile range.
Running on eight NVIDIA RTX 4090 GPUs, Emilia-Pipe processes source audio at a real-time factor of about 0.006 (roughly 2.5 hours of raw audio per minute of compute). On a 666.94-hour benchmark sample, the pipeline retained 96,117 clips totaling 258.44 hours (38.75% of the input) while lifting mean DNSMOS from 2.52 to 3.26.
Using Emilia-Pipe, the authors first built Emilia: 101,654 hours of 24 kHz speech across six languages (English, Chinese, German, French, Japanese, Korean). They then scaled the pipeline’s output to Emilia-Large, 216,313 hours, primarily by processing the YODAS2 YouTube corpus (Emilia-YODAS) with the same pipeline, using a relaxed DNSMOS threshold of 2.4 for the incremental data to preserve more low-resource-language material. Emilia-Large’s growth over Emilia is concentrated in previously under-represented languages: German expands 4.3x, French 6.0x, and Korean 34.3x.
To validate the datasets, the authors train two established zero-shot TTS baselines, an AR+SoundStorm two-stage model (LLaMA-style autoregressive semantic-token predictor followed by a SoundStorm-based semantic-to-acoustic model) and VoiceBox (a non-autoregressive flow-matching model producing mel-spectrograms), on Emilia, Emilia-Large, and MLS, then compare them on formal (LibriSpeech-Test) and spontaneous (Emilia-Test) evaluation sets, across increasing training-set sizes, and across all six languages plus crosslingual reference/target combinations.
Key Results
On LibriSpeech-Test (formal reading style), models trained on the 46k-hour English subset of Emilia (Emilia-En) and the comparably-sized 45k-hour MLS dataset perform similarly: AR+SoundStorm reaches 8.4% WER on Emilia-En versus 8.9% on MLS, while VoiceBox reaches its best WER (6.1%), speaker similarity (S-SIM 0.625), and naturalness (FSD 16.83) when trained on MLS. On Emilia-Test (spontaneous speech), Emilia-En training wins on every metric for both architectures: AR+SoundStorm improves WER from 7.7% (MLS) to 6.6%, S-SIM from 0.587 to 0.618, and FSD from 20.76 to 12.73; VoiceBox improves WER from 8.2% to 7.4%, S-SIM from 0.528 to 0.601, and SMOS from 3.61 to 3.76. Emilia’s DNSMOS score (3.26) is comparable to LibriTTS (3.25) and MLS (3.33) and well above raw in-the-wild corpora such as GigaSpeech (2.52) and WenetSpeech (2.43), despite Emilia being sourced entirely from unfiltered internet audio.
A data-scaling experiment trains AR+SoundStorm and VoiceBox on progressively larger subsets (5k, 10k, 46k, 100k, 134k hours). WER falls steadily with scale (AR+SoundStorm: 5.2% to 4.2% on LibriSpeech-Test; 5.7% to 4.9% on Emilia-Test), with the steepest gains below 46k hours and diminishing but still-positive returns up to roughly 100k hours, after which improvement largely plateaus.
Training on the full 216k-hour, six-language Emilia-Large produces functional multilingual TTS across all six languages (AR+SoundStorm WER ranging from 3.6% on Zh-Zh to 6.3% on Ko-Ko) and workable, if degraded, crosslingual generation (AR+SoundStorm WER rising from 6.3% on Ko-Ko to 8.3% on En-Ko; VoiceBox S-SIM falling from 0.588 on En-En to 0.386 on En-Zh). Multilingual training also costs a small amount of in-language quality: the Emilia-Large-trained multilingual AR+SoundStorm model scores WER 4.9%, S-SIM 0.636, and FSD 15.31 on English, slightly worse across all three metrics than an English-only 134k-hour model (WER 4.5%, S-SIM 0.65, FSD 14.8).
Novelty Assessment
The dataset itself, at 216k+ hours in Emilia-Large, is the paper’s central artifact: open, multilingual, spontaneous speech at a scale exceeding all comparable open corpora except SeamlessAlign (270k hours, but uncurated). Emilia-Pipe is an engineering integration of established components (UVR-MDX-Net separation, pyannote diarisation, Silero-VAD, WhisperX ASR, DNSMOS filtering); none of the individual stages are novel, and the contribution is the combination, the filtering design, and the decision to open-source both pipeline and data. Beyond the dataset itself, this version adds a genuine empirical contribution absent from the shorter published paper: a controlled data-scaling-law study isolating the effect of training-set size at fixed model capacity, and a systematic multilingual/crosslingual validation across all six languages using the full Emilia-Large scale. These experiments do not introduce new architectures (AR+SoundStorm and VoiceBox are both prior work) but they substantially deepen the evidence base for what the dataset enables, going beyond the original paper’s narrower English-vs-audiobook comparison.
Field Significance
Tip
High — Emilia fills a structural gap in the training-data ecosystem: a large-scale, multilingual, spontaneous-speech corpus that is open-access, paired with a reproducible and efficient preprocessing pipeline, and validated at scale across formal, spontaneous, and multilingual/crosslingual generation settings. The expanded experiments demonstrate that gains from in-the-wild training data continue to compound with scale up to roughly 100k hours per language and hold up across six languages, extending the evidence beyond the original short paper’s single-language comparison.
Claims
- supports: Training speech generation models on large-scale in-the-wild data, after automated quality filtering, can match audiobook-derived training data on formal-domain benchmarks while improving performance on spontaneous speech generation.
Evidence: AR+SoundStorm and VoiceBox trained on Emilia-En (46k hours) match MLS-trained counterparts on LibriSpeech-Test WER/S-SIM/FSD, but improve WER, S-SIM, and FSD on Emilia-Test (spontaneous speech) for both architectures. (§V.A, Table V)
- supports: Increasing in-the-wild training-set size continues to improve zero-shot TTS quality at fixed model capacity, with the largest gains at smaller scales and diminishing but still-positive returns as scale grows.
Evidence: Scaling training data from 5k to 134k hours steadily reduces AR+SoundStorm WER (5.2% to 4.2% on LibriSpeech-Test; 5.7% to 4.9% on Emilia-Test), with gains concentrated below 46k hours and convergence emerging around 100k hours. (§V.B, Figure 5)
- complicates: Training a single model on pooled multilingual in-the-wild data enables crosslingual generation but trades off same-language quality relative to a monolingual model trained on comparable per-language data volume.
Evidence: The Emilia-Large-trained (216k hours, 6 languages) multilingual AR+SoundStorm model scores WER 4.9%, S-SIM 0.636, and FSD 15.31 on English, all slightly worse than an English-only 134k-hour model (WER 4.5%, S-SIM 0.65, FSD 14.8). (§V.C)
- complicates: Automated quality-filtering pipelines for in-the-wild speech data necessarily discard a large majority of raw source audio to reach a training-quality bar.
Evidence: On a 666.94-hour benchmark sample, Emilia-Pipe’s six stages retain only 38.75% of the input (258.44 hours, 96,117 clips) after DNSMOS, language-confidence, and duration-outlier filtering. (§III, Table 2)
- complicates: Crosslingual voice transfer in zero-shot TTS remains substantially harder than same-language transfer even when the training corpus itself is multilingual and balanced across the target languages.
Evidence: Crosslingual WER and speaker similarity degrade relative to matched-language pairs across all six languages (e.g., VoiceBox S-SIM falls from 0.588 En-En to 0.386 En-Zh; AR+SoundStorm WER rises from 6.3% Ko-Ko to 8.3% En-Ko). (§V.C, Table VI)
Limitations and Open Questions
Warning
Subjective evaluation in the audiobook-vs-in-the-wild comparison (§V.A) uses only sixteen samples (eight per evaluation set) and twelve English-speaking evaluators, limiting the statistical reliability of the reported CMOS and SMOS differences. No confidence intervals or significance tests are reported for any subjective metric.
The authors themselves note three further limitations. The speaker diarisation model is imperfect and can leave a small proportion of segments containing overlapping speakers, which may degrade downstream generation quality. Emilia-Pipe constrains segments to 3-30 seconds, and generating speech outside that range is observed to produce unexpected outcomes, meaning pipeline hyperparameters may need adjustment for other use cases. Emilia-Large covers only six languages due to resource constraints, and the authors explicitly invite community contributions to extend language coverage. Additionally, all multilingual and crosslingual evaluation (§V.C) relies entirely on objective metrics (WER, S-SIM, FSD) without subjective naturalness assessment, and Emilia-Test is drawn from the same distribution as Emilia’s training data, which may favour Emilia-trained models’ apparent generalisation relative to MLS-trained ones.
Wiki Connections
- Multilingual TTS — Emilia-Large is explicitly constructed and validated for training multilingual and crosslingual zero-shot TTS models across six languages, with dedicated experiments quantifying same-language versus crosslingual degradation.
- Evaluation Metrics — the paper defines a multi-axis objective evaluation protocol (WER for intelligibility, S-SIM for speaker similarity, FSD via emotion2vec features for naturalness) used throughout its scaling and multilingual experiments.
- Subjective Evaluation — CMOS and SMOS listening tests with twelve human evaluators are used to corroborate the objective naturalness and similarity gains from in-the-wild training data.
- Zero-Shot TTS — Emilia and Emilia-Large are built specifically as training corpora for zero-shot speech generation models, and are validated using two established zero-shot TTS baselines (AR+SoundStorm, VoiceBox).
- SoundStorm — used as the acoustic half of the AR+SoundStorm baseline model trained on Emilia, Emilia-Large, and MLS for all validation experiments.
- VALL-E — cited as an early example of large-scale, codec-based zero-shot TTS whose data requirements motivate the need for a larger, more diverse open training corpus.
- BaseTTS — cited for its finding that TTS models can exhibit emergent prosodic abilities as training data scales past 100k hours, a comparison point for Emilia’s own data-scaling-law experiment.
- Seed-TTS — cited alongside BaseTTS as evidence that state-of-the-art speech generation systems have moved to training corpora exceeding 100k hours, the scale Emilia-Large is designed to match and exceed.
- A Survey on Neural Speech Synthesis — cited for broader context on the evolution of speech generation model architectures that Emilia’s datasets are intended to train.
- SeamlessM4T — its SeamlessAlign corpus (270k hours, 76 languages) is the closest existing in-the-wild dataset by scale, used as a direct comparison point in the dataset landscape table.