Speech Generation Research Wiki

Selective Classifier-free Guidance for Zero-shot Text-to-speech

5 min read

arXiv · 2025 · Preprint

John Zheng et al. (University of Calgary) · → Paper · Demo: ? · Code: ?

This paper evaluates a range of classifier-free guidance strategies adapted from image generation for zero-shot TTS, finding that most image-generation CFG techniques fail to transfer, but a timestep-selective approach — standard CFG for early timesteps switching to speaker-emphasized CFG for later ones — can improve speaker similarity on F5-TTS while limiting WER degradation, though its effectiveness is model- and language-dependent.

Problem

Zero-shot TTS systems based on flow matching (F5-TTS, CosyVoice 2, Voicebox) use classifier-free guidance (CFG) to condition generation on both text and speaker reference. A trade-off exists: higher CFG weights improve speaker similarity but degrade text adherence (raise WER). Techniques developed for image generation — weight schedules, perpendicular re-weighting, zero-init, and separated-condition CFG — have been largely unexplored for speech synthesis. Prior work on separated-condition CFG (VoiceLDM, DualSpeech, MegaTTS 3) showed speaker similarity improvements but required additional model evaluations per step or still had WER side-effects.

Method

The paper is a systematic evaluation study rather than a new training recipe. No new model is trained; all experiments apply inference-time CFG modifications to existing pre-trained models.

Models tested: F5-TTS (336M params, Diffusion Transformer, OT flow matching, trained on Emilia dataset) and CosyVoice 2 (LLM-generated semantic tokens + 71M flow-matching model). Both use a cosine timestep scheduler, with F5-TTS defaulting to 32 steps and CosyVoice 2 to 10 steps.

CFG strategies evaluated:

  1. Standard CFG (baseline): amplify conditioned vs. unconditioned prediction.
  2. Weight schedules (linearly increasing/clamped): from image generation literature.
  3. Perpendicular re-weighting (CFG-Zero*): use orthogonal component of conditioned prediction.
  4. Zero-init (CFG-Zero*): start from a later timestep.
  5. Input text condition: replace the unconditioned prediction with a text-only prediction — effectively sets λ_text = 1 in separated-condition CFG.
  6. Input audio condition: replace unconditioned prediction with audio-only prediction.
  7. Def text (proposed): apply standard CFG during early timesteps (t < 0.08) and switch to the input-text separated CFG for later timesteps. The rationale is that text content is established within the first ~6 of 32 steps; CFG for text adherence is redundant afterward.

Evaluation: LibriSpeech (1,127 pairs per F5-TTS protocol) and English/Mandarin subsets of Seed-TTS-eval. Metrics: SIM (WavLM Large cosine similarity) and WER (Hubert-Large ASR).

Key Results

Weight schedules, perpendicular re-weighting, and zero-init from image generation all fail to improve F5-TTS — some degrade both SIM and WER substantially. High CFG weights early in generation are particularly harmful.

The proposed def text strategy for F5-TTS:

  • LibriSpeech: SIM 0.675 → 0.682 (+0.007), WER 0.020 → 0.022 (minimal increase).
  • Seed-TTS-eval English: SIM 0.679 → 0.690, WER unchanged at 0.018.
  • Seed-TTS-eval Mandarin: No SIM improvement; WER increases. The strategy is ineffective for Mandarin with F5-TTS.

For CosyVoice 2:

  • Input text condition achieves SIM 0.661 → 0.671 with no WER degradation on LibriSpeech.
  • No language gap observed between English and Mandarin.

Compared to closed-source SOTA: Minimax-Speech (SIM 0.738, English Seed-TTS-eval) and Seed-TTS (SIM 0.762) remain substantially ahead. The paper’s methods are purely inference-time and do not close this gap.

Novelty Assessment

The contribution is primarily empirical. The def text strategy is a minor adaptation of timestep-conditional CFG from negative prompting literature applied to speech. The value of the paper lies in the systematic negative result: image-generation CFG techniques do not transfer well to speech synthesis. The observation that CFG strategy effectiveness is language- and text-representation-dependent is a novel empirical finding with practical implications for multilingual TTS deployment. The paper is conference short-paper scale (5 pages).

Field Significance

Moderate — This paper provides useful empirical evidence that CFG strategies from image generation do not transfer to flow-matching TTS, serving as a cautionary data point for practitioners who might otherwise import these techniques. The inference-time timestep-selective approach offers a practical, training-free SIM improvement for English zero-shot TTS, but the language dependence limits its general applicability. The paper’s primary contribution is the systematic negative result rather than a new architectural direction.

Claims

  • CFG strategies effective in image generation (weight schedules, perpendicular re-weighting, zero-init) do not improve zero-shot TTS quality and can degrade it. (§5, Figure 1, Figure 2)
  • Applying standard CFG only in early timesteps and switching to speaker-emphasized CFG for later timesteps can improve speaker similarity in zero-shot TTS without substantially increasing WER. (§4, §5, Figure 4)
  • The effectiveness of separated-condition CFG in zero-shot TTS is dependent on the text representation architecture of the underlying model, with language differences producing divergent outcomes for the same CFG strategy. (§5, §6, Figure 6)
  • Inference-time CFG tuning alone is insufficient to close the speaker similarity gap between open-weight and closed-source zero-shot TTS systems. (§6, Table 1)

Limitations and Open Questions

  • Only two base models (F5-TTS, CosyVoice 2) are tested; generalizability to other flow-matching TTS architectures (e.g., Voicebox, E2-TTS variants) is unverified.
  • The root cause of the Mandarin failure for F5-TTS is hypothesized (small ConvNeXt text encoder creates different conditioning for Chinese vs. English) but not verified experimentally.
  • CosyVoice 2’s immunity to the language gap is attributed to its 506M-parameter LLM text encoder but is not ablated.
  • The gap to closed-source models remains large; CFG tuning alone is unlikely to be sufficient.
  • No subjective evaluation (MOS) is reported; only objective metrics.

Wiki Connections

  • zero-shot-tts — the paper directly addresses inference-time CFG strategies for flow-matching zero-shot TTS systems.
  • flow-matching — both evaluated models (F5-TTS and CosyVoice 2) use OT flow matching as their generation backbone.
  • evaluation-metrics — the evaluation follows the SIM/WER protocol established by Seed-TTS, using WavLM Large for speaker similarity.
  • MegaTTS 3 and MiniMax-Speech appear in Table 1 as closed-source SOTA reference points that the proposed CFG variants do not reach.
  • E2 TTS is the architectural predecessor of F5-TTS, relevant to understanding the text conditioning design that causes the language-dependent CFG behaviour.

Backlinks

Graph View