ICNLSP 2025 · 2025 · Workshop
Zhuravlov and Sydorskyi (National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”) · → Paper · Demo: ? · Code: ✓
A two-stage fine-tuning pipeline that applies Direct Preference Optimization to an LM-based TTS model using only ASR-transcribed and LLM-augmented text, matching the quality of fine-tuning on human-labeled data without requiring manual annotation.
Problem
High-quality TTS fine-tuning conventionally relies on manually annotated speech data, which is expensive and time-consuming to produce. ASR-based auto-transcription offers a cheaper alternative but introduces transcription errors that degrade synthesis quality. Prior work on preference alignment for TTS (Tian et al., 2024; Hussain et al., 2025) required either labeled preference pairs or additional feedback mechanisms. The paper asks whether DPO can be applied to LM-based TTS using entirely unlabeled audio without sacrificing perceptual quality.
Method
The proposed pipeline fine-tunes xTTSv2, a multilingual zero-shot AR LM-based TTS system, in two sequential stages. In the first stage, the base model is fine-tuned on ASR-transcribed audio using standard cross-entropy loss over both audio and text tokens. In the second stage, a DPO preference dataset is constructed automatically: for each (reference audio, text) pair, 10 candidate audio token sequences are generated, ranked by a harmonic mean of three automatic metrics (WER via Whisper-Medium, speaker similarity via ECAPA2, and proxy MOS via UTMOS), and the second-best and second-worst samples are selected as the preferred and rejected pair. This avoids selecting the most extreme samples to ensure non-trivial preference signal.
The DPO loss augments the standard audio-level DPO objective with an additional text cross-entropy term, reflecting the practice in xTTSv2 training of jointly optimising over text and audio token streams. To increase text diversity beyond what ASR transcripts provide, 15,000 texts are generated by Llama 3.2 3B using an attribute combinator that controls length, topic, domain, and complexity. These generated texts are optionally combined with the ASR-derived transcripts to form the DPO training set.
Six experiments are run across two datasets: LJSpeech (single-speaker audiobook) and VCTK (multi-speaker, 108 speakers). The critical design finding is that DPO should be applied from the base xTTSv2 checkpoint rather than from the ASR-fine-tuned checkpoint, because standard fine-tuning narrows generation diversity, which in turn limits the preference signal available to DPO.
Key Results
On LJSpeech, DPO applied directly from the base model on ASR-transcribed texts (L3) achieves WER 0.041 and SS 0.445, outperforming classical fine-tuning on human-labeled data (L1: WER 0.056, SS 0.481) on intelligibility while trailing on speaker similarity. Adding LLM-generated texts (L4) improves UTMOS to 3.959 versus L1’s 3.816, at the cost of slightly higher WER (0.064).
On VCTK, the best automatic-metric result comes from DPO plus LLM-generated texts from the base checkpoint (V4): WER 0.013, SS 0.482, UTMOS 4.108. This surpasses both the human-labeled fine-tuning baseline (V1: WER 0.041, UTMOS 3.685) and the xTTSv2 base (WER 0.020) on all three metrics.
CMOS human evaluation compares four experimental pairs across 11 speakers spanning 5 accents. For the (human-labeled FT vs. V4) and (ASR FT vs. V4) comparisons, neither speaker similarity nor combined naturalness/intelligibility scores reach statistical significance (all CM p-values exceed 0.30), indicating perceptual equivalence. In contrast, DPO initialised from the ASR checkpoint (Experiments 5 and 6) shows significant degradation in speaker similarity (p < 0.001) relative to both baselines, confirming that the checkpoint initialisation choice is the dominant factor in outcome quality.
Novelty Assessment
The contribution is engineering integration: DPO is a well-established NLP alignment technique and xTTSv2 is a pre-existing production-grade TTS system. What is new is the specific combination of ASR auto-transcription, LLM text augmentation, and automatic metric-based preference pair construction to create a fully unsupervised DPO training pipeline for LM-based TTS. The preference pair construction procedure (generating 10 samples, ranking by harmonic mean of WER/SS/UTMOS, selecting second-best and second-worst) is a practical design decision rather than a theoretical advance. The finding that initialising DPO from the base rather than the ASR-fine-tuned checkpoint is the critical variable is a genuinely useful engineering insight. Evaluation is limited to xTTSv2, two well-studied datasets, and English speech; generalisation to other architectures and languages is unconfirmed. The CMOS evaluation methodology is sound but sample sizes within accent subgroups are small.
Field Significance
Moderate — This paper demonstrates that DPO can be applied to LM-based TTS using automatically constructed preference pairs from unlabeled audio, reducing the data annotation barrier without measurable perceptual quality loss. The result is primarily confirmatory of the DPO-for-TTS direction established by earlier work, with the new element being fully unsupervised preference pair construction. Its practical value lies in making preference alignment accessible to practitioners who lack human-labeled TTS corpora.
Claims
-
supports: Preference alignment via DPO can match the perceptual quality of supervised fine-tuning on human-labeled TTS data when preference pairs are constructed automatically from model-generated samples.
Evidence: On VCTK, DPO fine-tuning with LLM-augmented texts (V4) achieves WER 0.013, SS 0.482, and UTMOS 4.108 from the base xTTSv2 checkpoint, outperforming classical fine-tuning on human-labeled data (V1: WER 0.041, SS 0.500, UTMOS 3.685) on intelligibility and naturalness. CMOS evaluation confirms no statistically significant perceptual difference (all CM p > 0.30). (§3.5, Table 1, Table 2, Table 5)
-
complicates: The initialisation point before DPO fine-tuning critically determines outcome quality, with checkpoint diversity being a more important factor than label quality of the preceding supervised stage.
Evidence: DPO applied from the ASR-fine-tuned checkpoint (L5/V5) consistently underperforms DPO from the base model (L3/V3), with V5 reaching WER 0.273 versus V3’s WER 0.014. The explanation offered is that standard fine-tuning narrows generation diversity, leaving insufficient variation for DPO preference signal. (§3.6, Table 1, Table 2)
-
supports: Augmenting unlabeled TTS training data with LLM-generated texts improves synthesis naturalness when the dataset’s own texts are structurally mismatched with natural language.
Evidence: On VCTK, which contains newspaper sentences well-aligned with LLM output, adding 15,000 LLM-generated texts (V4 vs. V3) raises UTMOS from 4.009 to 4.108. On LJSpeech, which contains audiobook fragments (partial sentences), the improvement is smaller and WER increases, attributed to domain mismatch. (§3.5, Table 1, Table 2)
-
complicates: Speaker similarity is the most difficult quality dimension to preserve when fine-tuning TTS on ASR-transcribed or automatically constructed data, even when naturalness and intelligibility are maintained.
Evidence: Human-labeled fine-tuning (L1/V1) achieves the highest SS scores across both datasets (SS 0.481 on LJSpeech, 0.500 on VCTK). DPO from the base model recovers intelligibility and naturalness but trails on SS. Statistical significance testing shows SS degradation is significant for checkpoint-initialised DPO (Exp 6 vs. Exp 1/2, p < 0.001), while naturalness differences are not significant. (§3.5, §3.7, Table 4, Table 5)
Limitations and Open Questions
Warning
All experiments use xTTSv2 as the sole base model; it is unknown whether the same training recipe and checkpoint-initialisation finding generalise to other LM-based TTS architectures. Evaluation covers English only despite xTTSv2 being multilingual.
The comparison between fine-tuning from scratch on ASR-labeled versus human-labeled data is not performed; the human-labeled baseline starts from the same pre-trained xTTSv2 checkpoint, not a randomly initialised model, which limits conclusions about the true cost of label-free training. CMOS subgroup sample sizes (as few as 5 per accent in some comparisons) are too small for reliable accent-level conclusions. The bias implications of using ASR and LLM components for underrepresented speaker groups are acknowledged but not empirically quantified.
Wiki Connections
- Autoregressive Codec TTS — the paper fine-tunes xTTSv2, an LM-based TTS system that generates audio codec tokens autoregressively, making DPO applicability contingent on this architecture class.
- RLHF Speech — the paper applies DPO (a reward-free alternative to RLHF) to TTS, demonstrating that preference alignment can be achieved without human-annotated preference pairs.
- Zero-Shot TTS — xTTSv2 provides zero-shot speaker conditioning via a reference audio prompt, and the paper evaluates whether DPO fine-tuning preserves this capability across seen and unseen speakers.
- Evaluation Metrics — WER, speaker similarity, and UTMOS (proxy MOS) are used both to construct the DPO preference dataset and to evaluate fine-tuned models, creating a closed-loop relationship between the training signal and evaluation signal.
- Subjective Evaluation — the paper conducts CMOS evaluation across 11 speakers and 5 accent groups with statistical significance testing, providing a methodologically detailed human evaluation framework for comparing TTS fine-tuning strategies.