arXiv · 2025 · Preprint
Seki et al. (University of Tokyo) · → Paper · Demo: ? · Code: ?
An active learning framework for TTS corpus construction that iteratively selects informative web-crawled speech data, achieving higher synthesizable speaker coverage than static data selection methods at equal corpus size.
Problem
Large-scale web-crawled speech corpora contain substantial redundancy: data quality filtering can remove low-fidelity samples, but model-agnostic filtering ignores whether the TTS model has already learned to synthesize a given speaker. Prior core-set selection methods for TTS maximize diversity or eliminate redundant samples without consulting the training model, meaning they cannot identify which data would produce the most learning gain for the current model state. Active learning has been shown to outperform static core-set selection in image recognition, but no prior work applied this principle to TTS corpus construction.
Method
The framework targets multi-speaker TTS conditioned on x-vectors (512-dimensional speaker embeddings), with the goal of increasing the number of synthesizable speakers from web-crawled data. The procedure has two phases.
In the initial corpus construction, the first 10% of a shuffled YouTube video ID list is downloaded. Audio-text pairs pass a two-stage screen: CTC-based text-audio alignment filtering removes low-alignment utterances, and intra-video x-vector variance filtering removes videos with inconsistent speaker identity. A data quality estimator, trained to predict synthesis quality of a TTS model conditioned on candidate data, selects utterances expected to exceed a quality threshold. This threshold is calibrated against a studio-recorded multi-speaker corpus (JVS), setting the minimum observed synthesis quality from that corpus as the target.
Subsequent data acquisition introduces an informativeness criterion. For each new data segment, the system first applies the same quality filter, then evaluates informativeness via zero-shot synthesis: if the current TTS model can already synthesize a given speaker above the quality threshold, that speaker’s data is deemed redundant and excluded. Only samples satisfying both criteria (quality above threshold, and current model synthesis below threshold) are added to the corpus. This ensures new data targets capability gaps rather than reinforcing speakers already handled well.
The TTS backbone is FastSpeech 2 with HiFi-GAN (UNIVERSAL V1) as the vocoder. Speaker conditioning is via x-vector averaging over all utterances per speaker, added to the encoder output through a linear projection.
The paper additionally proposes a speaker generation module for inference over unseen speakers. A diffusion model is trained on the principal components of the collected x-vector set (PCA retaining 99% variance, reducing 512 dimensions to 28), allowing synthesis for speakers outside the training set. The residual dimensions are modeled as unit Gaussian.
Key Results
With a corpus of approximately 3,943 utterances, the proposed method achieves a 71.0% ratio of high-quality speakers (real speakers) versus 66.6% for the static selection baseline at the same size (Table I). For generated speakers evaluated via the speaker generation module, the proposed method achieves 69.2% versus 62.8% for the baseline. Using all 60K unfiltered utterances yields only 42.6% for real speakers, confirming the value of selective data collection even at lower data volumes.
The data quality estimator trained on 10% of the candidate pool shows strong correlation with the full-data estimator, suggesting the active learning loop does not require pre-processing the entire candidate dataset (Figure 7). The diffusion-based speaker generation model achieves lower Wasserstein distance to real speaker distributions compared to GMMs across all evaluated cluster counts (Figure 5).
One limitation is visible in the pseudo-MOS histogram: the proposed method performs worse than the static baseline for speakers with pseudo-MOS above 2.6, because speakers already above threshold are explicitly excluded from additional data collection. The method optimizes coverage breadth over the threshold boundary rather than lifting top-quality speakers further.
Novelty Assessment
The primary contribution is applying an established active learning paradigm to TTS data selection, adapted with a TTS-specific informativeness criterion (zero-shot synthesis as a signal for data redundancy). The individual components are each established: active learning, evaluation-in-the-loop quality estimation (from the authors’ prior work), and x-vector speaker modeling. The diffusion-based speaker embedding generation is the most technically distinct element, using PCA decomposition to handle the small sample size relative to embedding dimensionality, but the novelty is modest. The comparison is primarily against the authors’ own prior static selection method; broader comparison with other active learning or data selection approaches is absent. Experiments cover Japanese-only data, limiting generalizability claims.
Field Significance
Moderate — the active learning framing for TTS corpus construction addresses a real practical challenge as web-crawled data scales to sizes where downloading entire candidate sets upfront becomes impractical. The informativeness criterion, using zero-shot synthesis quality to identify redundant speakers, is a principled adaptation of model-feedback data selection to the TTS domain. The experimental scope is narrow (one language, two iterations), and the gains over the baseline are real but modest, so the paper provides useful evidence for the approach rather than establishing it as a general technique.
Claims
-
supports: Active learning with model-informed informativeness criteria produces more data-efficient TTS corpora than model-agnostic static data selection at equal corpus size.
Evidence: The proposed method achieves 71.0% synthesizable speaker ratio at 3,943 utterances versus 66.6% for the static baseline of the same size, measured by pseudo-MOS exceedance rate. (§III.B, Table I)
-
complicates: Data selection methods that optimize synthesizable speaker coverage may reduce synthesis quality for speakers already near the quality threshold.
Evidence: The proposed method underperforms the static baseline for speakers with pseudo-MOS above 2.6, because the informativeness criterion explicitly excludes speakers the current model can already synthesize well. (§III.B, Figure 3)
-
supports: TTS corpus quality estimators remain effective when trained on a small fraction of candidate data rather than requiring full candidate set pre-processing.
Evidence: The data quality estimator trained on 10% of the YouTube candidate pool shows a strong correlation with the estimator trained on 100% of the data, with no significant change in selection behavior. (§III.B, Figure 7)
-
supports: Diffusion models better capture complex speaker embedding distributions than Gaussian mixture models for speaker generation tasks with limited training samples.
Evidence: The diffusion model achieves Wasserstein distance to held-out test speakers that is more than twice the standard deviation lower than GMMs across all cluster sizes (M=1..10). (§III.B, Figure 5)
Limitations and Open Questions
Warning
All experiments are conducted on Japanese YouTube data only. Generalizability to other languages, recording conditions, or web data sources is untested, and the quality threshold calibration requires a language-matched studio corpus.
The active learning loop is evaluated with only two iterations (K=2). Whether additional iterations continue to improve coverage or reach diminishing returns is not explored. The main baseline comparison is the authors’ own prior static selection method; no comparison against other active learning or subset selection approaches from the computer vision or NLP literature is included. The evaluation uses pseudo-MOS (an automatic quality predictor) rather than subjective listening tests, so the absolute synthesis quality of the final corpora is not independently verified.
Wiki Connections
- Evaluation Metrics — pseudo-MOS quality prediction is the core criterion driving data selection decisions in both the initial and iterative corpus construction phases.
- Speaker Adaptation — the framework’s primary objective is broadening the set of synthesizable speakers, making speaker coverage the central measure of success.
- Zero-Shot TTS — zero-shot synthesis quality with the current model serves as the informativeness signal for deciding whether new speaker data adds learning value.
- GAN Vocoder — HiFi-GAN is used as the vocoder backbone for all synthesis quality evaluations throughout the corpus construction loop.
- J-CHAT — Japanese large-scale spoken dialogue corpus from the same research group, cited as a related web-crawled corpus collection effort.