arXiv · 2025 · Preprint
Zongyang Du et al. · → Paper · Demo: ✗ · Code: ✓
Releases NaturalVoices, a 5,049-hour, richly annotated corpus of spontaneous, emotionally expressive podcast speech built specifically for voice conversion and emotional voice conversion research, together with an open-source annotation pipeline and baseline VC/EVC benchmarks.
Problem
Voice conversion (VC) and emotional voice conversion (EVC) research has been built almost entirely on studio-recorded, scripted, or acted corpora: VCTK, CMU-Arctic, the VCC series, LibriTTS, and the widely used Emotional Speech Dataset (ESD, 15-30 hours of exaggerated, acted emotion from 10 speakers). Larger in-the-wild speech corpora exist (Emilia, GigaSpeech, LibriLight), but the paper argues they either lack segment-level annotations needed for VC filtering, lack emotion labels, or strip out the acoustic variability that VC models need to learn to handle. The result is a structural mismatch: VC models are designed and benchmarked on simplified, clean, emotionally narrow data, so they are never tested on the spontaneity, disfluency, and emotional nuance of real-world speech. This paper targets that gap directly for VC and EVC.
Method
NaturalVoices is built from 6,790 Creative-Commons podcast episodes (about 45 minutes each on average), stored at their original sampling rates (77.29% at 44.1 kHz) in addition to a 16 kHz downsampled version. The dataset shares its underlying recordings with the MSP-Podcast corpus (a speech-emotion-recognition resource with 409 manually annotated hours), but NaturalVoices extends automatic annotation to all speaking turns across the full episode set, yielding 5,049 hours versus MSP-Podcast’s 409.
The contribution is an automatic, open-sourced, four-stage data-sourcing pipeline (data collection, document-level annotation, segment-level annotation, filtering/extraction) that layers pretrained models onto the raw audio to produce per-segment metadata: Faster-Whisper transcripts and ASR confidence, PyAnnote speaker diarization merged with MSP-Podcast’s human-annotated global speaker identities (via a two-stage mapping and mapping+prediction process), speech-quality scores (PESQ, STOI, SI-SDR, MOS via Torchaudio-Squim; SNR via WADA-SNR; DNSMOS Pro), gender and age estimates, categorical emotion labels (anger, sadness, happiness, neutral, from a WavLM+LoRA PEFT-SER classifier), continuous emotion attributes (valence, arousal, dominance, from a regression WavLM model), and sound-event tags from a pretrained AST model over 527 classes. All annotations are exposed through a flexible filtering interface so downstream users can build task-specific subsets (e.g., single-speaker, high-SNR, emotion-balanced).
For the VC and EVC validation experiments, the authors filter to a 870.26-hour subset (speech-only, single-speaker, 1-20s duration, DNSMOS >= 2.6, SNR >= 30 dB, ASR confidence >= 0.7) and train existing any-to-any VC baselines (TriAAN-VC, ConsistencyVC, DDDM-VC) on 10%/50%/100% splits of this subset, plus a separate emotion-balanced 340-hour subset (85 hours per emotion category) used to train DISSC, an EVC baseline whose speaker lookup table is replaced with d-vector embeddings to enable any-to-any conversion. No new architecture is proposed; the pipeline and the resulting corpus are the paper’s contribution, and the VC/EVC experiments serve to validate that the dataset supports training and to characterize how existing architectures behave on its scale and spontaneity.
Key Results
On the 100%-data setting, TriAAN-VC achieves the best intelligibility on the NaturalVoices in-domain test set (WER 0.255, CER 0.162 averaged across Whisper and wav2vec 2.0 ASR scorers), while ConsistencyVC attains the highest speaker similarity (SV accuracy 0.979, SECS 0.715) and the highest subjective speaker-similarity MOS (3.88 ± 0.27) and speech-quality MOS (3.73 ± 0.40) among the three baselines (Tables VI, VIII). Scaling from 10% to 100% of training data improves intelligibility for TriAAN-VC consistently but produces inconsistent or even degrading results for DDDM-VC, whose out-of-domain (ESD) speaker similarity drops from SV accuracy 0.933 at 50% to 0.768 at 100% (§V.C.5, Table VI).
A listening test comparing ground-truth reference audio from NaturalVoices against ESD reference audio finds NaturalVoices rated as more natural (MOS 4.51 ± 0.18 vs. 4.39 ± 0.17), despite being spontaneous, in-the-wild speech rather than studio-recorded (Table VII). In the EVC experiment, DISSC trained on the 340-hour emotion-balanced subset achieves higher emotion category accuracy (0.617) and emotion embedding cosine similarity (0.724) on the in-domain NaturalVoices test set than on the out-of-domain ESD test set (0.255 and 0.286 respectively), but with substantially worse intelligibility on NaturalVoices (WER 0.454 vs. 0.112 on ESD) (§V.D.4, Table IX).
Novelty Assessment
The contribution is a dataset and an annotation pipeline, not a new VC architecture: all VC and EVC systems evaluated (TriAAN-VC, DDDM-VC, ConsistencyVC, DISSC) are prior published baselines applied unchanged to the new corpus. The genuine novelty is in scale and annotation depth for VC-specific use: at 5,049 hours with segment-level speech-quality, emotion, and speaker-identity annotations and an open filtering pipeline, NaturalVoices is substantially larger than any existing emotion-labeled, spontaneous-speech VC resource in the paper’s own comparison table, and larger in coverage (though smaller in emotion-focused breadth) than the general-purpose in-the-wild corpus Emilia, which lacks segment-level annotation and emotion labels entirely. The paper explicitly builds on a smaller predecessor release (NaturalVoices-v0, 3,846 hours, neutral-only), so the incremental step here is expanding scale and adding emotion-specific annotation and EVC validation on top of that earlier release.
Field Significance
High
The dataset fills a documented and specific gap: voice conversion and emotional voice conversion research has had no large-scale, spontaneous, emotion-annotated corpus, forcing reliance on acted resources such as ESD (15-30 hours, 10 speakers) that structurally cannot capture the variability of real-world speech. At 5,049 hours with an open, reusable annotation pipeline and demonstrated (not merely claimed) validation across four distinct SOTA VC/EVC baselines, NaturalVoices provides both a training resource and a benchmark that surfaces a genuine, reproducible finding: several current VC architectures do not scale cleanly with additional spontaneous training data, and some degrade on out-of-domain speaker similarity as training data grows.
The paper’s own experiments substantiate its claims rather than merely asserting them: the 100%-data DDDM-VC out-of-domain speaker-similarity degradation and TriAAN-VC’s cleaner scaling curve are concrete, reproducible findings from this paper’s own tables, not projections.
Claims
- supports: A voice conversion dataset that reflects real spontaneous and emotional variability, rather than acted or scripted speech, can serve as reference audio of comparable or higher perceived naturalness than a widely used acted emotional-speech benchmark.
Evidence: Ground-truth listening tests rate NaturalVoices reference audio as more natural than ESD reference audio (MOS 4.51 ± 0.18 vs. 4.39 ± 0.17), despite NaturalVoices being unscripted, in-the-wild podcast speech. (§V.C.5, Table VII)
- complicates: Increasing the amount of spontaneous, in-the-wild training data does not uniformly improve voice conversion model performance; some architectures degrade rather than benefit from scale.
Evidence: DDDM-VC’s out-of-domain speaker-verification accuracy on ESD falls from 0.933 (50% training data) to 0.768 (100% training data), and its objective WER/CER also worsen at 100% relative to 50%, while TriAAN-VC and ConsistencyVC scale more consistently. (§V.C.5, Table VI)
- complicates: Models trained for emotional voice conversion on spontaneous, naturalistic emotional speech transfer emotion more faithfully in-domain but at a substantial cost to intelligibility on that same in-domain data, compared to conversion evaluated on acted, cleaner emotional speech.
Evidence: DISSC trained on the emotion-balanced NaturalVoices subset achieves higher emotion category accuracy (0.617 vs. 0.255) and emotion embedding cosine similarity (0.724 vs. 0.286) on NaturalVoices than on ESD, but WER is far worse on NaturalVoices (0.454) than on ESD (0.112). (§V.D.4, Table IX)
- supports: Segment-level, multi-dimensional automatic annotation (speech quality, emotion category and attributes, speaker identity, sound events) applied at scale to in-the-wild recordings can substitute for costly manual annotation while still supporting flexible, task-specific subset construction for voice conversion.
Evidence: The four-stage pipeline (ASR/diarization, speaker mapping against MSP-Podcast human labels, multi-metric speech-quality scoring, emotion category/attribute prediction, sound-event detection) produces the filtered 870.26-hour VC subset and 340-hour emotion-balanced subset used directly to train and validate four different baseline VC/EVC systems. (§III, §V.B, Table V)
Limitations and Open Questions
No new architecture is proposed or trained by the authors; all VC/EVC systems evaluated are prior published baselines, so the paper's claims about NaturalVoices's utility rest entirely on how well existing architectures happen to transfer to this data, not on any model designed for it.
Speaker identity resolution is a hybrid of human-annotated global labels (from MSP-Podcast, 409 hours) and automatic diarization-based mapping for the remainder of the corpus, so speaker-identity accuracy is not uniform across the full 5,049 hours. The emotion labels (categorical and dimensional) are themselves the output of pretrained classifiers rather than human raters, so systematic classifier bias could propagate into any model trained on these labels. The paper’s own results also show that intelligibility on NaturalVoices lags noticeably behind the cleaner ESD test set for every model and setting tested, which the authors attribute to the dataset’s spontaneity and acoustic variability, but this also means absolute WER/CER numbers on this benchmark should not be compared directly to numbers reported on studio-recorded VC benchmarks.
Wiki Connections
- Voice Conversion — introduces a large-scale, spontaneous, emotion-annotated training and evaluation resource specifically to address the acted-data bottleneck in VC and EVC research, validated with four existing any-to-any VC/EVC baselines.
- Emotion Synthesis — provides both categorical and continuous (valence/arousal/dominance) automatic emotion annotation at scale and an emotion-balanced subset used to train and evaluate an emotional voice conversion baseline.
- Evaluation Metrics — reports a broad battery of objective VC/EVC metrics (WER, CER, speaker verification accuracy, SECS, emotion category accuracy, emotion embedding cosine similarity) alongside per-segment automatic speech-quality metrics (PESQ, STOI, SI-SDR, DNSMOS, SNR) used both to characterize the dataset and to filter it.
- Subjective Evaluation — runs MOS listening tests with twelve human raters comparing converted speech and ground-truth reference audio for both speech quality and speaker/emotion similarity across NaturalVoices and ESD.
- Speaker Adaptation — benchmarks any-to-any VC and EVC baselines that must generalize to unseen target speakers drawn from a corpus spanning thousands of speakers, substantially broader speaker coverage than prior VC training data.