arXiv · 2026 · Preprint

Yizhong Geng et al. (Beijing University of Posts and Telecommunications / Beijing Logic Intelligence Technology / Eastern Institute of Technology, Ningbo) · → Paper · Demo: ✓ · Code: ?

Shows that scaling flat synthetic speech in low-resource TTS fine-tuning improves intelligibility but collapses prosodic diversity beyond roughly a 1:1 synthetic-to-real ratio, and recovers expressivity with two annotation-free DPO recipes for Thai (style-toggled self-contrastive pairs) and Lao (multi-temperature self-critique with ASR filtering).

Problem

Autoregressive speech-token TTS models (which the authors call “Spoken Language Models”) remove the need for a grapheme-to-phoneme front end, which is attractive for languages such as Thai (five lexical tones, tone sandhi, no word boundaries) and Lao (very little public transcribed speech). In such settings the practical way to get enough phonetic supervision is to synthesize training data with existing, prosodically flat TTS systems. The paper asks how the synthetic share of the training data affects the expressivity of the fine-tuned model, and how to retain the phonetic stability that synthetic data provides without inheriting its low prosodic variability. Prior synthetic-data and model-collapse studies concern recursive self-consumption in text and images; the paper argues the speech case differs because erosion comes from the low-entropy bias of external TTS outputs.

Method

All systems are fine-tuned from CosyVoice 2: a 500M Qwen2.5-0.5B text-speech LM predicts S3Tokenizer tokens (FSQ, 25 Hz, 6,561 entries), and a 300M causal conditional flow-matching decoder with a HiFi-GAN vocoder renders 24 kHz audio, conditioned on a timbre embedding from a reference clip. Synthetic data are text from mC4 rendered by off-the-shelf TTS systems (MMS-TTS, SeamlessM4T-v2, Typhoon2-Audio for Thai; MMS-TTS plus cross-lingual transfer from Thai models for Lao), filtered by ASR WER, duration ratio and repetition.

Synthetic-ratio scaling. With real Thai speech nominally fixed at 300h (Common Voice, ASR-filtered), the synthetic ratio α is swept from 3% to 100%. As a cheap training-time diagnostic the paper proposes prosodic entropy H_p, the Shannon entropy of the empirical speech-token distribution over generated utterances, motivated by the claim that in this architecture the AR tokens carry content and prosody while the flow-matching decoder carries timbre. A mixture-entropy argument (strict concavity of entropy over p_α = (1−α)p_real + αp_syn) is offered as qualitative intuition for a single entropy peak α*.

Disentanglement-Guided Self-Alignment (DGSA). For each text with a real recording, the frozen SFT model generates two outputs with the same timbre embedding: one with a style-token prefix extracted from real speech (expressive but more phonetic errors) and one without (stable but flat). Two DPO preference sets are built, each with the real recording as the preferred sample: real over expressive (stability objective) and real over stable (expressivity objective). The losses are weighted by a schedule tied to α: λ_e = max(0, (α−α*)/(1−α*)), λ_s = 1−λ_e, with α* fixed at 0.5. Training is strictly sequential (SFT, frozen-model generation, DPO from the same SFT checkpoint), with β = 0.1 over 50k preference pairs.

Temperature-Driven Self-Critique (TDSC). For Lao, where no real training speech is used, each input text yields 12 candidates (4 per temperature at T = 0.7, 1.0 and an expanding T_max). A rule-based judge accepts candidates with ASR WER below 40%, repetition rate below 10% (fraction of positions with five identical consecutive tokens) and length within 0.5x to 2.0x of the text length. Each iteration first runs SFT on accepted samples, then DPO with the lowest-WER accepted sample as winner and a length- and repetition-passing but high-WER candidate as loser, so the preference signal targets phonetic accuracy rather than duration artifacts. T_max grows by 0.1 per iteration over 5 iterations. WER is scored with Whisper-large-v3 for Thai and Dolphin-small for Lao.

Figure 3. Temperature-Driven Self-Critique (TDSC) For each input, the model generates candidates at temperatures T ∈ { low, mid, high } , spanning conservative (stable) to exploratory (expressive) outputs. The Judge Model filters candidates by WER, length, and repetition criteria, yielding accepted ( G ) and rejected ( R ) sets for preference-based refinement.

Key Results

Scaling (Thai, Table 1). WER falls monotonically from 75.0% (α = 3%) to 36.2% (α = 100%). NMOS rises from 3.82 to a peak of 4.51 at α = 50% and then falls to 3.61 at α = 80% and 3.08 at α = 100%; SMOS peaks at 4.63 (α = 50%) and falls to 3.01. Token repetition rate stays near 2% up to α = 67%, then jumps to 6.51% (α = 80%) and 9.83% (α = 100%). H_p tracks NMOS (peak 10.506 bits at α = 50%). SMOS starts degrading earlier than NMOS (α = 60%: 4.35 vs 4.45). In a controlled 2,000-pair study with matched WER, high-H_p samples show larger F0 range and standard deviation, higher F0 correlation with reference recordings, and higher expressivity MOS (4.2 vs 3.7) (Table 2).

DGSA (Thai, α = 80%, Table 3). DGSA keeps the SFT WER (38.9%) while raising NMOS from 3.61 to 4.42 and SMOS from 3.54 to 4.53, and cutting repetition from 6.51% to 2.82%. Single-objective DPO raises NMOS only to 3.92 and degrades WER to 45.2%; rejection sampling gives small gains (NMOS 3.75, WER 40.5%). Ablations (Table 7) attribute the largest drop to removing the expressivity objective (NMOS 4.4 to 3.7); dropping the stability objective raises WER to 46.5%; random rather than identity-consistent pairing gives WER 42.8% and NMOS 3.9.

TDSC (Lao, Table 4). From an all-synthetic SFT baseline, TDSC lowers WER from 38.5% to 29.8%, repetition from 7.62% to 4.15% and raises NMOS from 3.12 to 3.94, against self-training (35.2%, 3.31) and rejection sampling (36.8%, 3.24). Removing the DPO step costs the most (NMOS 3.9 to 3.4, Table 8). Accepted samples come 42% / 35% / 23% from T = 0.7 / 1.0 / 1.3 (Table 9).

System comparison (Tables 5-6). On Thai TSynC-2, DGSA reports NMOS 4.51 vs ElevenLabs-v3 4.21, Gemini Pro 4.05 and Azure 4.01; Azure has lower WER (36.5% vs 38.9%). In Thai zero-shot cloning, DGSA reports SIM 0.84 vs 0.78 and SMOS 4.51 vs 4.23 against ElevenLabs-v3. On Lao Common Voice, TDSC reports WER 29.8% and NMOS 4.53 vs Gemini Flash 34.2% and 4.12, and zero-shot cloning SIM 0.81 / SMOS 4.32 with no competing system. Human ratings come from 20 native speakers per language in a double-blind randomized design with 95% CIs and paired t-tests. Absolute WERs are high for every system (roughly 30-50%), so ASR quality on these languages bounds what the WER figures can resolve.

Novelty Assessment

The most transferable contribution is empirical: a dense sweep of synthetic-to-real ratio for an AR speech-token TTS model, showing a non-monotonic expressivity curve and a sharp repetition-loop onset beyond a critical ratio. DGSA’s specific idea, toggling the style prefix while holding the timbre embedding fixed to produce two complementary negatives against a shared real positive, is a genuinely new way to build DPO pairs without annotation, and the dual-objective weighting is a small but sensible addition. TDSC is closer to existing practice (multi-sample generation, ASR-based filtering, SFT plus DPO iterations, akin to rejection-sampling and self-rewarding pipelines); the temperature curriculum and the length-matched loser selection are the incremental pieces. H_p as a prosody proxy is plausible in this architecture but its justification rests on an assumed content/prosody vs timbre split rather than an independent measurement. The base model, tokenizer and DPO loss are all reused unchanged.

Field Significance

Moderate — The paper provides a concrete, quantified warning that synthetic data used to bootstrap low-resource TTS trades away prosodic and speaker-similarity quality beyond a dataset-specific ratio, and demonstrates two annotation-free preference-alignment recipes that recover much of that loss on Thai and Lao. Its value lies more in the scaling characterization and the pair-construction idea than in architectural change, and the reported numbers carry internal inconsistencies that limit how far individual figures can be relied on.

Claims

  • complicates: Increasing the share of synthetic TTS output in low-resource TTS training improves intelligibility monotonically but degrades naturalness, speaker similarity and token diversity beyond a critical ratio.

    Evidence: Sweeping α from 3% to 100% on Thai, WER falls from 75.0% to 36.2% while NMOS peaks at 4.51 (α = 50%) and falls to 3.08, SMOS falls from 4.63 to 3.01, and repetition rate rises from about 2% to 9.83%. (§5.2, Table 1, Figure 1)

  • supports: Preference optimization can recover expressivity lost to flat synthetic training data without human preference annotation, using real recordings as positives and self-generated outputs as negatives.

    Evidence: DGSA at α = 80% keeps SFT WER at 38.9% while raising NMOS from 3.61 to 4.42 and reducing repetition from 6.51% to 2.82%. (§5.3, Table 3)

  • refines: Single-objective preference alignment for TTS expressivity can trade away intelligibility, and pairing it with an explicit stability objective is needed to improve both.

    Evidence: Standard DPO on an expressivity objective raises NMOS to 3.92 but worsens WER from 38.9% to 45.2%; removing DGSA’s stability objective gives WER 46.5%, and removing its expressivity objective gives flat output (NMOS 3.7). (§5.3, Table 3; §Appendix C.1, Table 7)

  • supports: Iterative self-improvement with ASR-filtered multi-temperature sampling and DPO can stabilize autoregressive speech-token TTS trained entirely on synthetic data.

    Evidence: For Lao trained on 1,500h synthetic speech, TDSC lowers WER from 38.5% to 29.8% and repetition from 7.62% to 4.15% and raises NMOS from 3.12 to 3.94, outperforming self-training and rejection sampling; removing the DPO step costs 0.5 NMOS. (§5.4, Table 4; §Appendix C.2, Table 8)

  • supports: The entropy of generated discrete speech tokens can serve as a training-time proxy for prosodic diversity in AR token TTS whose decoder handles timbre separately.

    Evidence: H_p follows the same peak-then-decline trend as NMOS across synthetic ratios, and WER-matched high-H_p samples show larger F0 range (128.4 vs 109.7 Hz), higher F0 correlation (0.68 vs 0.52) and higher expressivity MOS (4.2 vs 3.7) than low-H_p samples. (§5.2, Table 2, Figure 1)

Limitations and Open Questions

Warning

Some reporting details are inconsistent or underspecified, so individual figures should be read with care: Table 14 lists TSynC-2 within the 300h real Thai training source although §5.5 evaluates on “held-out TSynC-2”; the α = 100% scaling row is described as 0h real data although the sweep is described as fixed 300h real speech (§5.2); and Figure 2 labels the style-token path as the stable output, opposite to the text. NMOS for the same system also differs by setting (Lao TDSC 3.94 in Tables 4 and 6 vs 4.53 in the standard-TTS Table 5; Thai DGSA 4.42 in Tables 3 and 6 vs 4.51 in Table 5), and the paper does not state the test conditions for Tables 3 and 4, so these values are not directly comparable across tables.

  • The framework assumes a usable ASR system for the target language for both data filtering and the TDSC judge; the authors note this excludes most of the world’s languages. (§Limitations)
  • Only two Southeast Asian tonal languages are tested; generality to typologically different languages is unvalidated.
  • The critical ratio α* = 0.5 is a fixed heuristic taken from the Thai sweep and reused for scheduling; whether the peak location transfers to other data mixes is not tested.
  • Commercial baselines (ElevenLabs-v3, Gemini, Azure) are API snapshots, and the Lao zero-shot cloning result has no comparison system; “first voice cloning for Lao” is a claim about the authors’ survey of available systems.
  • WER levels of 30-50% for every system, including commercial ones, suggest the ASR models limit measurement resolution; small WER differences may not be meaningful.
  • H_p is validated only through correlation with MOS and F0 statistics within this setup; it is a corpus-level token statistic and could rise with noise as well as with genuine prosodic variation.
  • Code is not released; only a demo page is provided. TDSC costs about 200-300 GPU-hours on 8x RTX 4090.

Wiki Connections

  • Autoregressive Codec TTS — Characterizes how synthetic training data induces repetition loops and token-diversity collapse in an AR speech-token TTS model and proposes alignment-based fixes.
  • Flow Matching — Relies on the flow-matching decoder carrying timbre separately from the AR token stream to build identity-consistent preference pairs.
  • RLHF Speech — Contributes two annotation-free DPO recipes (dual-objective self-contrastive pairs and ASR-filtered multi-temperature self-critique) for low-resource TTS.
  • Zero-Shot TTS — Reports zero-shot voice cloning for Thai and an all-synthetic-data Lao system.
  • Subjective Evaluation — Uses double-blind NMOS and SMOS tests with 20 native raters per language, 95% CIs and paired t-tests.
  • Evaluation Metrics — Proposes speech-token entropy as a training-time proxy for prosodic diversity, validated against F0 statistics and MOS.
  • CosyVoice 2 — Provides the base architecture (Qwen2.5 text-speech LM, S3Tokenizer, flow-matching decoder) fine-tuned for Thai and Lao.
  • Vevo — Cited alongside CosyVoice as evidence that AR tokens carry content and prosody while the acoustic decoder carries timbre, the premise behind DGSA and H_p.
  • Scaling Under-Resourced TTS for Thai — Prior Thai TTS work by overlapping authors that this paper extends to synthetic-data scaling and preference alignment.
  • DPO for Speech Autoregressive Diffusion Models — Related preference-optimization work for speech generation that the paper positions DGSA against.
  • Lao-English Code-Switched Speech Synthesis — Cited as evidence of the scarcity of Lao speech resources that motivates the all-synthetic TDSC setting.
  • Stochastic Prosody Modeling — Cited for the stability-diversity trade-off of sampling temperature that TDSC exploits.
  • VALL-E — Cited as a foundation of the AR codec-token TTS paradigm the paper adapts to low-resource languages.
  • SeamlessM4T — Used both as a synthetic-data generator for Thai and as an open-source baseline.