arXiv · 2025 · Preprint
Yizhou Peng et al. (Alibaba, Nanyang Technological University) · → Paper · Demo: ? · Code: ?
Introduces a classifier-free guidance variant and a distillation recipe that specifically target the case where a TTS system’s target emotion instruction conflicts with the semantic content of the text being spoken.
Problem
Instruction-following TTS systems increasingly let users specify emotion via natural-language prompts rather than fixed labels or reference audio, but the target emotion can conflict with the semantic content of the text itself (e.g., asking for a “surprised” reading of “It isn’t a happy memory”). The paper shows this cross-modal conflict is not a benign edge case: ground-truth recordings retain high emotion-recognition accuracy regardless of text-rendered emotion consistency, but autoregressive TTS baselines (CosyVoice2, HierSpeech++) suffer large accuracy drops as the conflict grows more severe. Standard classifier-free guidance (CFG), the common workaround for weak style adherence, does not resolve this: it merely drops the condition to an unconditional baseline, which is an insufficiently strong contrast to overcome the text’s competing semantic pull, and pushing the guidance scale higher to compensate introduces audible artifacts.
Method
The proposed system operates on an autoregressive TTS backbone (CosyVoice2). Given a target text and a user-supplied “Rendered-Emo” natural-language emotion prompt, an external LLM (used as judge) extracts the “Text-Emo” implied by the text itself and classifies the pair into one of three inconsistency profiles: Identical, Inconsistent, or Highly Inconsistent.
Cross-modal Consistency Guided CFG (CCG-CFG) modifies standard CFG by replacing the unconditional dropout branch with the Text-Emo condition whenever inconsistency is detected, explicitly widening the contrast between the two forward passes instead of contrasting against an uninformative null condition. Dynamic-Scale CCG-CFG (DS-CCG-CFG) further maps the three inconsistency profiles to three fixed guidance scales (1.0, 2.5, 3.0) rather than using one scale for all inputs, since the paper’s own scale sweep shows that guidance helps Inconsistent samples but harms Identical ones and destabilizes intelligibility at high scales regardless of profile.
To remove the two-pass decoding overhead and the residual artifacts of CFG at inference time, the DS-CCG-CFG guidance signal is distilled directly into the TTS model’s weights via Direct Preference Optimization (DPO). Preference pairs are constructed with a hard-sample mining strategy: for each text in an external 20k-utterance text-only corpus (drawn from VCTK), a contrasting target emotion is paired with the text to synthetically induce cross-modal inconsistency, 25 candidate speech samples are generated per text (5 seeds × 5 guidance scales), and each candidate is scored with an objective function balancing (1 − WER) and emotion-confidence (with a penalty when the predicted emotion disagrees with the target). The highest- and lowest-scoring candidates form the DPO preference pair. The full pipeline (SFT, then DPO on the mined preference pairs) is trained on a single H20-96GB GPU with a constant learning rate of 1e-5 for 4 epochs.
Key Results
Using a rendered-emotion reference audio clip with CosyVoice2’s zero-shot cloning inflates apparent emotion control (65.11% EmoACC) relative to the more realistic neutral-reference setting the paper adopts as its main baseline (50.63% EmoACC), because the reference leaks style information that bypasses genuine natural-language emotion control. Against that neutral-reference baseline, CCG-CFG and DS-CCG-CFG substantially improve emotion-recognition accuracy over both the unmodified baseline and standard CFG (DS-CCG-CFG reaches 64.83% EmoACC and the best Model-as-Judge score of 58.4 among training-free configurations), but at the cost of raising WER from 4.61% to 7.86% and lowering human-rated naturalness (NMOS 3.31) and quality (MOS 3.91) relative to the baseline.
The distilled model (DS-CCG-CFG-DPO+EXT.) trades some of that peak expressiveness for much better robustness: it lowers WER to 3.76% (below the unguided baseline), achieves the best human evaluation scores of all configurations tested (MOS 4.33, NMOS 3.94, EMOS 3.67), and still improves EmoACC to 59.55% versus the 50.63% neutral-reference baseline, a 12% absolute EmoACC gain reported in the paper’s headline comparison. Under the paper’s own inconsistency-stratified breakdown (Table 4), the largest expressiveness gains of the distilled model occur specifically on the inconsistent and highly-inconsistent samples rather than uniformly across all inputs. The distilled model does not match the peak EmoACC of the training-free DS-CCG-CFG (59.55% vs. 64.83%), reflecting the WER/expressiveness trade-off the distillation is designed to resolve rather than eliminate.
Novelty Assessment
The contribution is a targeted modification to an existing inference-time technique (CFG) plus a training recipe (DPO with LLM-judged hard-sample mining) applied to an existing AR TTS backbone (CosyVoice2); it does not propose a new model architecture. The core idea, replacing the null condition with a specific, semantically meaningful competing condition, is a reasonably novel and mechanistically motivated change to CFG’s guidance formulation, and the dynamic per-profile scale mapping plus the LLM-based inconsistency classifier are engineering choices built entirely on existing tools (an off-the-shelf LLM judge, standard DPO). The evaluation is a genuine strength: the paper isolates the cross-modal inconsistency problem with a dedicated stratified analysis (Table 1, Table 4) rather than only reporting aggregate numbers, and it compares against real baselines (HierSpeech++, Qwen3-TTS, unmodified CosyVoice2) under a carefully chosen realistic condition (neutral reference) rather than the more favorable rendered-reference setting. The method’s inference-time variant (DS-CCG-CFG) is also honestly shown to degrade naturalness and intelligibility, and the paper does not overstate the distilled model as strictly dominant, it explicitly reports the residual EmoACC gap versus the guided (non-distilled) variant.
Field Significance
Moderate — this paper isolates and quantifies a specific, previously underexplored failure mode in autoregressive TTS emotion control (conflict between textual semantics and instructed emotion) and provides a concrete guidance-and-distillation recipe that reduces it without a full architectural change. Its contribution is primarily a training-free inference technique plus a distillation recipe applied to a single existing backbone; the evaluation methodology (stratifying results by inconsistency severity) is a useful analytical tool independent of the specific guidance formulation proposed.
Claims
- supports: In autoregressive TTS models, classifier-free guidance can be made more effective at resolving competing conditioning signals by contrasting the target condition against the specific opposing signal rather than against an uninformative unconditional baseline.
Evidence: CCG-CFG, which substitutes the CFG dropout branch with the text-implied emotion when inconsistency is detected, raises EmoACC over CosyVoice2-CFG at the same nominal guidance-scale setting (e.g., CCG-CFG (H) reaches 64.79% EmoACC vs. 56.40% for CosyVoice2-CFG (H)) while WER is comparable. (§2.2.1, §4.1, Table 3)
- complicates: Increasing emotional expressiveness in autoregressive TTS via inference-time guidance amplification tends to trade off against intelligibility and perceived naturalness.
Evidence: DS-CCG-CFG raises EmoACC to 64.83% over the 50.63% neutral-reference baseline but nearly doubles WER (from 4.61% to 7.86%) and lowers human-rated naturalness (NMOS 3.31) and quality (MOS 3.91) below the unguided baseline (NMOS 3.6, MOS 4.2). (§4.1, §4.2, Table 3, Table 5)
- supports: Distilling an inference-time guidance signal into a TTS model’s weights via preference optimization can recover much of the guidance’s expressiveness benefit while removing its synthesis-artifact cost and its two-pass decoding overhead.
Evidence: DS-CCG-CFG-DPO+EXT. improves EmoACC to 59.55% over the 50.63% baseline while lowering WER to 3.76% (better than the unguided baseline’s 4.61%) and achieving the highest human MOS (4.33), NMOS (3.94), and EMOS (3.67) scores of any configuration tested, without requiring inference-time two-pass CFG decoding. (§4.1, §4.2, Table 3, Table 5)
- complicates: Providing a target-emotion reference audio clip to a zero-shot voice cloning TTS system, rather than relying on a text-only emotion instruction, can leak style information that inflates apparent natural-language emotion-control performance.
Evidence: CosyVoice2 with Natural Language Emotion Control achieves 65.11% EmoACC when given a rendered-emotion reference clip, dropping to 50.63% when the reference is replaced with a neutral-emotion clip, leading the authors to adopt the neutral-reference condition as the realistic baseline for all subsequent experiments. (§4.1, Table 3)
Limitations and Open Questions
Warning
The training-free guidance variants (CCG-CFG, DS-CCG-CFG) and the hard-sample mining pipeline both depend on calls to external, proprietary LLMs (a GPT-family model for text-emotion/inconsistency judging, a Gemini model for the Model-as-Judge subjective proxy). This makes the inference-time methods costly and non-reproducible with fixed, versioned tooling, and it means the paper’s own automated “subjective” metric (MaJ) is generated by a proprietary system rather than an open or fully documented one.
All experiments are conducted on a single AR TTS backbone (CosyVoice2); the paper does not report whether the CCG-CFG formulation or the DPO distillation recipe transfers to other autoregressive or non-autoregressive TTS architectures. Human evaluation is limited to a 14-sample subset (2 speakers × 7 emotions), which is a small basis for the subjective naturalness and emotion-similarity comparisons. The distilled model does not fully close the gap to the training-free DS-CCG-CFG’s peak emotion-recognition accuracy, indicating the trade-off between expressiveness and robustness is reduced but not eliminated by distillation.
Wiki Connections
- Emotion Synthesis — proposes a guidance mechanism and a preference-distillation recipe specifically aimed at improving emotion control in autoregressive TTS under text-emotion conflict.
- Instruction-Conditioned TTS — targets the Natural Language Emotion Control setting where emotion is specified via a free-text instruction rather than a fixed label or reference clip.
- Zero-Shot TTS — builds on a zero-shot voice-cloning AR TTS backbone and shows that the choice of reference audio (rendered vs. neutral emotion) materially affects measured emotion-control performance.
- Autoregressive Codec TTS — modifies the guidance and training procedure of an autoregressive speech-token TTS model rather than its token or codec representation.
- RLHF Speech — uses Direct Preference Optimization with a hard-sample mining strategy to distill an inference-time guidance signal into model weights.
- Subjective Evaluation — reports human listening-test ratings (EMOS, NMOS, MOS) alongside a Model-as-Judge proxy to assess emotional expressiveness, naturalness, and quality.
- CosyVoice2 — used as the AR TTS backbone that CCG-CFG, DS-CCG-CFG, and the DPO distillation are fine-tuned and evaluated on, and as the unmodified baseline in all comparisons.
- Qwen3-TTS — used as a baseline LLM-based TTS system, evaluated both with its built-in emotion tags and with a rendered-emotion reference clip.
- LibriTTS — one of the seven corpora combined to build the paper’s training, validation, and test splits used to evaluate emotion control and intelligibility.
- emotion2vec — used as the pre-trained speech emotion recognition model (emo2vec-plus-large) that computes the EmoACC metric reported throughout the paper.