arXiv · 2026 · Preprint
Chenlin Liu et al. (Harbin Institute of Technology / Zhejiang University) · → Paper · Demo: ? · Code: ?
A training-free decoding rule for LM-based TTS adds a positive-only, adaptively scaled bonus to tokens that the full text-conditioned prediction favours over a text-ablated prediction from the same model, cutting WER/CER on four frozen TTS models.
Problem
LM-based TTS still produces hallucinations: mispronunciations, substitutions, omissions, repetitions and unintended continuations that stay fluent and speech-like while departing from the target text. Prior mitigation changes the architecture (monotonic alignment, phoneme interleaving, transducers), the training objective, or post-training, or manipulates model-specific attention heads. The decoding step itself is mostly inherited from text generation (top-k, nucleus sampling, temperature, repetition penalties). These samplers cannot tell whether a candidate token’s probability comes from the text or from the acoustic history and learned speech prior, so they can select a locally plausible but text-inconsistent token. Once that token enters the history, the error can persist.
Method
The paper frames each next-token prediction as combining two sources. Alignment information comes from the prompt transcript and target text. Experience information comes from the acoustic prompt, the generated history and the learned speech prior. To make this operational, the same speech LM is run twice per step on the same teacher-forced history: once with all conditions (the expert, p_E) and once with both text conditions removed (the amateur, p_A). The authors stress that this is a within-model intervention, not an exact decomposition.
Two analysis measures follow from the contrast. Distribution-level alignment influence I_i is the forward KL divergence D_KL(p_E ‖ p_A). Decision-level alignment gain G_i is log p_E − log p_A evaluated at the selected token. The hypothesis is that an important class of hallucinations starts as an onset failure: near a vulnerable linguistic transition, the selected token receives too little relative alignment support. Later propagation failures then follow, because renewed textual guidance is read through an already deviated acoustic history.
Experience-Calibrated Contrastive Decoding (ECCD) starts from conventional contrastive decoding (CD), which restricts candidates to the expert’s top-k set and scores them by the log-ratio log(p_E/p_A). The authors argue CD is wrong for speech: it discards the expert score and penalises tokens that the text-ablated branch supports, even though that support carries pronunciation, duration and continuity cues. ECCD makes three changes:
- Expert anchor: the score stays log p_E, and the contrast is added as a correction instead of replacing it. Tokens outside the top-k set keep their expert scores and are not masked.
- Positive-only enhancement: within the top-k set, only tokens with p_E > p_A receive a bonus, [log(p_E/p_A)]₊. Tokens preferred by the amateur are not penalised.
- Experience Compatibility Coefficient (ECC): the bonus is scaled by α(1 − C_i), where C_i is the probability mass p_A assigns to the expert’s top-k set. When the two predictions agree (large C_i), the correction shrinks. When they diverge, it grows.
The modified scores are applied before each model’s native filtering and penalty operations, so ECCD sits on top of the existing sampler. Defaults are α = 1 and k = 25 for all experiments. The method needs no training. Each decoding step runs a second, text-ablated forward pass; the paper reports no latency or real-time-factor overhead. The four evaluated models (CosyVoice 2, CosyVoice 3, Llasa, GLM-TTS) are all autoregressive speech-token LMs and stay frozen. ECCD acts only on their AR token decoding. Model size and tokenizer are those of each base model and are not discussed.
Key Results
SeedTTS-Eval (Table 1). ECCD lowers WER/CER on all twelve model-subset combinations (four models by test-zh, test-en, test-hard):
| Model | test-zh CER | test-en WER | test-hard CER |
|---|---|---|---|
| CosyVoice2 → +ECCD | 1.34 → 0.95 | 2.98 → 2.08 | 8.10 → 6.91 |
| CosyVoice3 → +ECCD | 1.19 → 0.92 | 3.01 → 1.82 | 6.92 → 6.19 |
| Llasa → +ECCD | 8.66 → 3.95 | 6.83 → 3.03 | 26.26 → 14.27 |
| GLM-TTS → +ECCD | 1.01 → 0.97 | 2.12 → 1.94 | 11.11 → 9.70 |
The largest relative reduction is Llasa on test-en (55.6%, the abstract’s “up to” figure). The smallest is GLM-TTS on test-zh (about 4%). The weakest base model gains most. Speaker similarity (CAM++) and UTMOS fall slightly for CosyVoice and GLM-TTS: the largest SS drop is 0.019 (GLM-TTS test-hard, 0.858 → 0.839), and the largest UTMOS drop is 0.083 (GLM-TTS test-hard, 2.687 → 2.604). Both metrics rise for Llasa on every subset.
Lower-temperature baseline (Table 1, CosyVoice2). Sampling at τ = 0.75 or 0.875 helps on some subsets but is inconsistent: at τ = 0.75, test-en WER rises from 2.98 to 4.35. ECCD gives the lowest WER/CER of the CosyVoice2 variants on all three subsets (test-zh 0.95 vs. 1.11 and 1.18; test-en 2.08 vs. 4.35 and 2.76; test-hard 6.91 vs. 7.28 and 7.60). This argues that ECCD’s gains are not just distribution sharpening.
CV3-Eval zero-shot track (Tables 2 and 3). ECCD lowers WER/CER in 24 of 25 supported model-language subsets, including every hard-zh and hard-en case. The largest drop is Llasa hard-en (39.20 → 24.16). The single exception is CosyVoice2 on Japanese (9.46 → 9.71), while CosyVoice3 on Japanese improves (12.13 → 10.29). The authors attribute the reversal to CosyVoice2’s handling of overlapping Chinese and Japanese characters but do not test this. Only CosyVoice 3 covers German, Spanish, French, Italian and Russian, so those five gains come from one model.
Listening test (Table 4). Twenty-five native listeners rated 30 test-hard utterances per method from CosyVoice2. CMOS (−3 to 3, target text shown, relative to native decoding) is +0.644 ± 0.091 for ECCD, against +0.021 and +0.037 for the two lower-temperature settings. SMOS is 3.811 for ECCD: above native decoding (3.678) but below both lower-temperature settings (3.831 at τ = 0.75, 3.962 at τ = 0.875). The 95% intervals of the ECCD and native SMOS scores overlap.
Ablation (Table 7, CosyVoice2 test-hard). Conventional CD raises CER from 8.10 to 8.72, cuts SS to 0.738, and shrinks utterance token count and average character duration to 0.66 and 0.76 of native decoding. Adding the expert anchor drops CER to 6.86. Positive-only enhancement gives the lowest CER of all variants (6.43). Adding ECC raises CER back to 6.91 but restores SS to 0.811 and the duration ratios to 0.87/0.93, the closest of any contrastive variant to native decoding. Using I_i in place of 1 − C_i as the scale gives similar CER (6.90) but lower SS (0.799) and weaker temporal preservation.
Hyperparameters (Figure 2). Larger α generally lowers CER and gradually lowers SS; α = 4 degrades both. Smaller k tends to give lower CER. α = 1, k = 15 beats the default on both CER and SS in the sweep.
Onset analysis (Tables 5 and 6, Figure 1). On native CosyVoice2 test-hard outputs, with character regions located through ASR alignment and timestamps, I_i and G_i rise near character boundaries, peak around the second token of a region, and fall during continuation. Gap regions average less than half the character-region values. At the first ASR-detected error, both measures are lower than at the matched preceding correct boundary. The shortfall is larger for G_i (0.237 over the {−1} window) than for I_i (0.073). Complete error regions show slightly higher averages than correct ones (I_i 0.934 vs. 0.885), which the authors read as textual influence returning without restoring the correct mapping.
Novelty Assessment
Contrastive decoding with context-ablated amateurs is established in text LLMs, vision-language models and audio-language models. The paper’s claim to be the first to adapt it to autoregressive acoustic-token generation is plausible from its related work. The genuinely new parts are the speech-specific changes: the expert anchor, positive-only enhancement and the ECC-scaled strength. The ablation shows that each change matters, and that plain CD hurts TTS. The contribution is a new inference procedure, not a new model.
The evaluation is broad for a decoding paper: four base models, two benchmarks and nine languages, plus a listening test. However, hallucination is measured only through ASR WER/CER, the listening test covers only CosyVoice2, and the only competing decoder tested is lower-temperature sampling. Attention-constrained inference and beam search are discussed but not compared. The information-theoretic analysis is descriptive and the authors state that it is correlational: it locates a shortfall near onset but does not show that ECCD fixes those specific decisions.
Field Significance
Moderate — The paper shows that a training-free decoding change can cut content errors across several frozen LM-based TTS models, with the biggest gains on the least robust model. It also demonstrates that contrastive decoding borrowed unchanged from text generation harms speech, because the text-ablated branch carries useful duration and continuity information. The gains over strong base models are small on easy subsets, and the comparison set of decoding baselines is narrow.
Claims
- supports: Contrasting a speech LM’s text-conditioned and text-ablated next-token predictions at decoding time reduces content errors in autoregressive TTS without any training.
Evidence: ECCD lowers WER/CER on all four models and all three SeedTTS-Eval subsets (e.g. Llasa test-hard CER 26.26 → 14.27) and in 24 of 25 CV3-Eval model-language subsets. (§Main Results, Tables 1-3)
- complicates: Treating text-ablated predictions purely as negative evidence degrades speech generation, because those predictions carry duration, prosody and continuity information.
Evidence: Conventional contrastive decoding on CosyVoice2 raises test-hard CER from 8.10 to 8.72, lowers speaker similarity from 0.822 to 0.738, and compresses token count and character duration to 0.66 and 0.76 of native decoding; adding the expert anchor and positive-only enhancement reverses the CER degradation. (§Ablation Studies, Table 7)
- refines: The content-fidelity gains of alignment-enhanced decoding are not reproduced by sharpening the sampling distribution.
Evidence: On CosyVoice2, lower-temperature sampling is inconsistent (test-en WER rises to 4.35 at τ = 0.75) and earns CMOS of only +0.021 and +0.037, against +0.644 for ECCD with the target text shown to listeners. (§Main Results, Tables 1 and 4)
- complicates: In LM-based TTS decoding, stronger enforcement of text alignment trades against speaker similarity and natural temporal structure.
Evidence: Positive-only enhancement reaches the lowest CER (6.43) but keeps SS at 0.795 and token count at 0.78 of native; adding the compatibility scaling raises CER to 6.91 while restoring SS to 0.811 and duration ratios to 0.87/0.93, and α = 4 degrades both CER and SS. (§Ablation Studies, Table 7, Figure 2)
- supports: Hallucination onset in autoregressive TTS coincides with a localised shortfall in the text’s influence on the selected token, rather than uniformly weak alignment across the error span.
Evidence: At the first ASR-detected error boundary in native CosyVoice2 outputs, the decision-level gain is 0.237 lower than at the matched correct boundary over the {−1} window (0.073 for the distribution-level measure), while complete error regions average slightly higher than correct regions. (§Information-Theoretic Analysis of Hallucination, Tables 5-6)
Limitations and Open Questions
- Hallucination is measured only through ASR-based WER/CER, which mixes hallucinations with ordinary mispronunciations and ASR errors. No hallucination-type breakdown (omission, repetition, continuation) is given.
- The listening test covers one model (CosyVoice2) on one subset. ECCD’s SMOS sits below both lower-temperature baselines, and its interval overlaps native decoding.
- Every step needs a second, text-ablated forward pass. The paper does not report the latency or memory cost, which matters for the streaming models it tests.
- The only decoding baseline is lower-temperature sampling. Attention-constrained inference, beam search for LM-based TTS and training-based fixes are discussed but not compared.
- The default α = 1, k = 25 is not the best setting in the paper’s own sweep (α = 1, k = 15 improves both CER and SS), and the sweep covers only CosyVoice2 test-hard.
- The onset analysis depends on approximate ASR alignment and token-rate mapping, and the authors state it is consistent with, but does not causally establish, the alignment-shortfall hypothesis.
- Non-Chinese, non-English language results come from a single model (CosyVoice 3), and the CosyVoice2 Japanese reversal is unexplained.
- The paper is marked as work in progress; code and demo availability are not stated.
Wiki Connections
- Autoregressive Codec TTS — ECCD modifies the next-token scores of autoregressive speech-token LMs and targets the hallucinations this family is prone to.
- Zero-Shot TTS — All evaluation is prompt-based voice cloning on SeedTTS-Eval and the CV3-Eval zero-shot track, with speaker similarity tracked as the main trade-off.
- Subjective Evaluation — A 25-listener CMOS/SMOS test with the target text shown checks whether the content-error reductions are perceived.
- CosyVoice 2 — Main frozen base model for the baseline comparisons, listening test, ablations and onset analysis.
- CosyVoice 3 — Frozen base model and source of CV3-Eval; it supplies all the results for the additional European languages.
- Llasa — Frozen base model with the highest native error rates and the largest ECCD gains.
- GLM-TTS — Frozen base model with the smallest gains and the largest speaker-similarity and UTMOS drops.
- Seed-TTS — Source of the SeedTTS-Eval test-zh, test-en and test-hard subsets used as the main benchmark.
- GOAT (GFlowNet hallucination mitigation) — Same group’s earlier post-training approach to LM-based TTS hallucination, which this paper positions as a training-based alternative to decoding-time control.
- ELLA-V — Architectural alignment fix (phoneme-acoustic interleaving) cited as the kind of model-specific intervention ECCD avoids.
- VALL-E R — Monotonic-alignment approach to robust zero-shot TTS, cited for the hallucination taxonomy ECCD addresses.