COLING · 2025 · Conference
Liu et al. (Zhejiang University) · → Paper · Demo: ✓ · Code: ✓
VoxpopuliTTS refines the VoxPopuli corpus into a 30,000-hour multilingual TTS dataset covering English, French, and Spanish, using ASR confidence-based filtering and semantic-aware segment merging to improve transcript quality and naturalness.
Problem
Large-scale multilingual TTS training data is scarce. Existing corpora either concentrate on English (LibriTTS, GigaSpeech) or heavily favour one or two languages even when nominally multilingual (Emilia provides less than 5,000 hours for languages other than English and Chinese). Audiobook-derived corpora such as LibriTTS also exhibit a formal reading style that lacks the natural prosodic variation, breath patterns, and pausing that real conversational speech contains, limiting trained models’ ability to produce naturally flowing output. This gap restricts multilingual and cross-lingual TTS research to a small set of resource-rich languages.
Method
VoxpopuliTTS is derived from VoxPopuli, a corpus of European Parliament recordings spanning multiple languages and many speakers. The authors apply a five-stage processing pipeline: voice activity detection (VAD) using a dedicated VAD model to segment long audio recordings; multilingual transcription via FasterWhisper, including language identification and filtering of misidentified language segments; word-level forced alignment via wav2vec 2.0 to obtain precise timestamps; speaker diarization using pyannote.audio to group segments by speaker; and a semantic-completeness merging step that uses sentence-final punctuation as a proxy for utterance completeness, concatenating adjacent segments when a transcript ends with a comma or other non-terminal punctuation.
The key processing innovations are (1) confidence-based filtering: FasterWhisper’s per-word logits are used to compute three confidence metrics (average logits of first three words, last three words, and all words), and any transcript below a threshold of 0.7 is discarded to reduce hallucinated transcriptions; and (2) segment merging: adjacent segments from the same speaker are joined when the current segment’s transcript ends without terminal punctuation, ensuring downstream TTS models receive semantically complete input sequences rather than mid-sentence fragments.
After processing, DNSMOS is applied to stratify the corpus into small (approx. 1,000 hours per language), medium (approx. 5,000 hours), and large (approx. 10,000 hours) subsets, enabling researchers to select the quality-scale trade-off appropriate for their application. The final dataset totals approximately 30,000 hours across English, French, and Spanish, with 100 hours per language held out for validation and test.
Dataset effectiveness is validated by training XTTS_v2 on the large subset and evaluating zero-shot cross-lingual synthesis using WER (via Whisper), speaker similarity (cosine distance of speaker embeddings from 3D-speaker toolkit), and UTMOS alongside human MOS.
Key Results
XTTS_v2 trained on the VoxpopuliTTS large subset achieves WER of 8.25% (EN), 8.98% (FR), and 9.20% (ES) on the respective test sets. Speaker similarity (SIM) is 67.02% for EN, 67.45% for FR, and 71.69% for ES. UTMOS scores are 3.43 (EN), 3.62 (FR), and 3.54 (ES), with human MOS values of 3.37, 3.52, and 3.31 respectively (Table 2). The authors note that the relatively high WER reflects the naturalness and conversational style of VoxpopuliTTS, which increases training complexity compared to clean studio recordings. No direct comparison is made against models trained on alternative large multilingual corpora such as Emilia, so the relative benefit of VoxpopuliTTS over existing options is not quantified.
Novelty Assessment
The paper’s contribution is the dataset itself and its processing pipeline, not a new model architecture or training method. The component techniques — VAD, Whisper-based transcription, wav2vec forced alignment, pyannote diarization, and DNSMOS filtering — are all established tools applied in a new combination. The notable engineering contribution is the confidence-based hallucination filter using per-word logits and the punctuation-based segment merging heuristic, both of which address practical data quality issues in noisy multi-speaker recordings. The scale (30,000 hours across three European languages) is the primary value over prior art; however, the absence of a controlled comparison against Emilia or WenetSpeech4TTS makes it difficult to assess whether the processing pipeline yields measurably superior data quality. The evaluation uses a single model family (XTTS_v2) and does not include ablations on the filtering steps.
Field Significance
Moderate — VoxpopuliTTS addresses a real gap in multilingual TTS resources by providing a large, naturally-spoken, multi-speaker corpus for three languages at a scale not previously available from curated sources. Its utility for training cross-lingual zero-shot TTS models is demonstrated, though the advantage over alternatives such as Emilia is not empirically quantified, limiting how strongly the claim of superiority can be made.
Claims
-
supports: Large-scale, naturally-spoken multilingual corpora enable cross-lingual zero-shot TTS capabilities in models that lack them when trained on single-language or audiobook-derived data.
Evidence: XTTS_v2 trained on the VoxpopuliTTS large subset (10,000h per language) achieves cross-lingual synthesis in EN, FR, and ES with MOS scores of 3.37, 3.52, and 3.31, respectively, demonstrating the dataset’s utility for zero-shot multilingual TTS. (§4, Table 2)
-
complicates: Naturalness and conversational style in training data trades off against intelligibility metrics during TTS model training.
Evidence: The authors attribute relatively high WER values (8.25–9.20%) to the naturalness and conversational complexity of VoxpopuliTTS, suggesting that models trained on naturalistic in-the-wild data face greater challenges in transcript fidelity than those trained on clean studio recordings. (§4)
-
supports: ASR confidence scores provide an effective proxy for filtering hallucinated transcripts in automatic speech processing pipelines for TTS data curation.
Evidence: Per-word logit thresholding (confidence < 0.7 discarded) is used to remove FasterWhisper hallucinations, and sentence-final punctuation is used to identify semantically incomplete segments for merging, both validated as effective in the pipeline design. (§2.2)
-
complicates: Quality-stratified corpus subsets introduce a trade-off between data scale and audio quality, requiring practitioners to select the appropriate tier for their task.
Evidence: VoxpopuliTTS is split into small (~1,000h, DNSMOS ~3.87), medium (~5,000h, DNSMOS ~3.71), and large (~10,000h, DNSMOS ~3.39) subsets per language, with quality decreasing as scale increases. (§3, Table 1)
Limitations and Open Questions
The paper does not compare VoxpopuliTTS against Emilia or WenetSpeech4TTS in a controlled training experiment, making it impossible to assess whether the processing pipeline or the dataset composition offers measurable advantages over alternatives. Coverage is limited to three European languages (English, French, Spanish), so the approach’s scalability to morphologically complex or lower-resource languages is untested. The evaluation relies on a single model family (XTTS_v2) and does not include ablations that isolate the contribution of the confidence filter or the segment merging step. Human MOS evaluations are reported with confidence intervals but the number of raters, the test items, and the evaluation protocol are not described.
Wiki Connections
- Multilingual TTS — VoxpopuliTTS directly targets the training data gap for multilingual TTS, providing 30,000 hours across English, French, and Spanish for zero-shot cross-lingual synthesis research.
- Zero-Shot TTS — the dataset is designed and validated for zero-shot speech generation scenarios, as demonstrated by training XTTS_v2 for zero-shot cross-lingual TTS.
- Evaluation Metrics — the paper employs DNSMOS for dataset quality stratification and uses WER, speaker cosine similarity, UTMOS, and human MOS to validate the dataset’s utility as a training resource.