arXiv · 2026 · Preprint

Runwu Shi et al. (Institute of Science Tokyo / Wuhan University / Beijing Institute of Technology / Ant Group) · → Paper · Demo: ? · Code: ?

Decomposes the classifier-free guidance (CFG) direction of flow-matching zero-shot TTS into a text residual, a speaker-prompt residual, and a joint text-speaker residual using four masked branches, then reweights the speaker and joint residuals at inference time on top of standard CFG; applied training-free to F5-TTS and CosyVoice 2, it raises speaker similarity with text error rates roughly unchanged.

Problem

In flow-matching zero-shot TTS, the vector-field predictor is conditioned on two signals, the target text and a speech prompt, and CFG extrapolates from a less-conditioned prediction toward the fully conditioned one to strengthen conditioning. Standard CFG amplifies the full-minus-null direction with a single strength, so it cannot tell text-driven guidance from speaker-driven guidance. Condition-specific alternatives (separated or dual CFG, as in DualSpeech and VoiceLDM, and speaker-selective CFG from Selective CFG for Zero-shot TTS) take a difference between two branches, for example full minus text-only, to emphasise the speech prompt. The authors argue that such branch differences still treat each difference as a single direction and tend to trade text correctness against speaker similarity: strengthening the speaker direction raises similarity but can hurt text adherence. Their specific diagnosis is that the speaker-selective direction (full minus text-only) silently carries a component that exists only when text and prompt are given together, and that this joint component is amplified with the same coefficient as the speaker residual.

Method

The method operates purely at inference time on an existing flow-matching acoustic model; neither backbone is retrained. With a velocity predictor , masking neither, either, or both conditions gives four branches: , (speaker prompt masked), (text masked), and . Using as a common baseline, the paper defines single-condition residuals and , and a joint residual

so that the standard CFG direction splits exactly as (§III-A). In this view, standard CFG scales all three residuals by .

The four-branch notation also serves as a taxonomy (Table I). Any guidance rule can be written as weights on the three residuals : standard CFG gives ; separated CFG (DualSpeech, VoiceLDM) gives , leaving the joint residual fixed; speaker-selective CFG gives , tying the speaker and joint residuals together. The paper also places MegaTTS 3-style text-conditioned speaker guidance in this last group.

The proposed general rule adds independently weighted residuals on top of standard CFG, . The paper uses the speaker-oriented variant (), described as the best performer in the authors’ design-space exploration, which corresponds to residual weights (§III-B, Table I). Computing needs four network evaluations per sampling step instead of two; the number of steps and the schedule are unchanged, and the four branches share and so they can be batched.

Settings (§V-A): on F5-TTS, 32 sampling steps with the original CFG scale 2.0, plus times the speaker residual and times the joint residual. On CosyVoice 2, the method acts on the flow-matching acoustic generator only (the text-speech language model is untouched): the text condition is the flow encoder output , the speaker condition is the global speaker embedding together with the prompt acoustic features, with the official CFG strength 0.7, 10 steps, plus times the speaker residual and times the joint residual. The weights are fixed per backbone.

Key Results

Evaluation is zero-shot voice cloning on LibriSpeech-test and on the English and Chinese Seed-TTS-Eval sets, using pretrained F5-TTS and CosyVoice 2 checkpoints. Metrics are automatic only: SIM from a WavLM-large speaker-verification model, WER from Whisper large-v3 (English), and CER from Paraformer-zh (Chinese). No human listening test is reported. Error rates are reported as fractions (0.0196 = 1.96%).

CosyVoice 2 (Table II). Against both official-CFG rows (strength 0.7 and 1.0), the method raises SIM and lowers the error rate on all three sets: LibriSpeech-test SIM 0.669 vs 0.6561 / 0.6585 with WER 0.0211 vs 0.0212 / 0.0219; SEED-EN SIM 0.6706 vs 0.6586 / 0.662 with WER 0.0194 vs 0.0205 / 0.0198; SEED-ZH SIM 0.7631 vs 0.7531 / 0.7547 with CER 0.0139 vs 0.0153 / 0.0144. The WER changes on LibriSpeech-test are very small (0.0001 against the 0.7 baseline).

F5-TTS (Table II). Against CFG strength 2.0 (the default), SIM rises on LibriSpeech-test (0.6819 vs 0.6745, WER 0.0196 vs 0.0197) and SEED-EN (0.6875 vs 0.6811), but SEED-EN WER is higher (0.0146 vs 0.0136; equal to the CFG 1.5 row). On SEED-ZH, CER drops slightly (0.0153 vs 0.0158) while SIM is marginally lower (0.7630 vs 0.7636).

Selective CFG reference (Table II). The Selective CFG rows are quoted from Selective CFG for Zero-shot TTS, and the authors note the evaluation setup may not match theirs, so these comparisons are indicative only. On F5-TTS, Selective CFG reports SIM equal to or slightly above the proposed method (0.682 vs 0.6819 on LibriSpeech-test, 0.690 vs 0.6875 on SEED-EN, 0.764 vs 0.7630 on SEED-ZH) with higher error rates (WER 0.022 vs 0.0196 and 0.018 vs 0.0146; CER 0.019 vs 0.0153). On CosyVoice 2, Selective CFG reports higher SIM on LibriSpeech-test (0.671 vs 0.669), lower SIM on SEED-EN (0.666 vs 0.6706), essentially the same SIM on SEED-ZH (0.763 vs 0.7631), and higher error rates on all three sets (0.025, 0.026, 0.018 vs 0.0211, 0.0194, 0.0139). The authors frame this as Selective CFG trading text correctness for similarity while their method keeps error rates lower at comparable similarity.

Ablation (Table III, F5-TTS, CFG scale 2.0). Adding speaker plus joint residuals at equal weight raises SIM from 0.6745 to 0.6788 (WER 0.0197 to 0.0196); increasing the joint weight to 2.5 raises SIM to 0.6819 at the same WER. A control adding speaker plus text residuals lowers WER to 0.0180 but drops SIM to 0.6621. The ablation caption names LibriSpeech-PC; its baseline and selected rows carry the same values as the F5-TTS LibriSpeech-test rows in Table II.

A qualitative CosyVoice 2 example on one LibriSpeech utterance (Figure 2) has zero WER under both samplers and speaker similarity rising from 0.57 to 0.71; this is a single case study, not an aggregate result.

Novelty Assessment

The contribution is an inference-time guidance rule plus an analytical lens, not a new model. The four-branch decomposition is a short algebraic step (an inclusion-exclusion style interaction term between the two conditions), but making it explicit is useful: it shows in one table (Table I) how standard, separated, and speaker-selective CFG differ only in how they weight three residuals, and it identifies the joint residual as the degree of freedom that separated CFG freezes and speaker-selective CFG ties to the speaker residual. The proposed sampler then simply adds independent weights for the speaker and joint residuals. Empirical support is modest: two backbones, three automatic metrics, gains in SIM of roughly 0.006 to 0.013 absolute, a single hand-picked weight setting per backbone, no human evaluation, no variance or significance reporting, and the closest competing method compared only through numbers quoted from another paper. The cost is a doubling of network evaluations per step.

Field Significance

Low. The paper offers a compact residual taxonomy of CFG variants for two-condition zero-shot TTS and shows that reweighting the joint text-speaker residual is a usable extra control knob, with consistent but small speaker-similarity gains on CosyVoice 2 and mixed text-accuracy effects on F5-TTS. It is primarily an incremental sampler variant and an analysis framing; the evidence is limited to automatic metrics and fixed hyperparameters, and the method doubles per-step inference cost.

Claims

  • refines: In two-condition (text plus speech prompt) flow-matching TTS, the CFG direction can be split into text, speaker, and joint interaction residuals, and existing condition-specific guidance rules differ mainly in how they weight the joint residual.

    Evidence: With four masked branches, ; standard CFG weights the residuals , separated CFG , and speaker-selective CFG , so the latter binds the joint residual to the speaker residual. (§III-A, §IV, Table I)

  • supports: Independently amplifying the speaker and joint text-speaker components of the guidance field at inference can raise speaker similarity of zero-shot TTS without retraining and with little change in text error rate.

    Evidence: On CosyVoice 2, adding 0.5 speaker and 0.25 joint residual on top of official CFG raised SIM on LibriSpeech-test, SEED-EN, and SEED-ZH (e.g. 0.6561 to 0.669) while WER/CER also decreased on all three; on F5-TTS, SIM rose on both English sets, but SEED-EN WER rose from 0.0136 to 0.0146 and SEED-ZH SIM fell marginally. (§V-B, Table II)

  • supports: The joint interaction between text and speaker-prompt conditioning carries speaker-relevant information beyond the speaker-only residual.

    Evidence: In the F5-TTS ablation, raising the joint-residual weight from 1.0 to 2.5 (with the speaker residual fixed at 1.0) increased SIM from 0.6788 to 0.6819 at an unchanged WER of 0.0196. (§V-C, Table III)

  • complicates: Adding text-oriented guidance components improves intelligibility at a clear cost in speaker similarity, so guidance components cannot be combined arbitrarily.

    Evidence: Adding speaker plus text residuals to F5-TTS CFG lowered WER from 0.0197 to 0.0180 but reduced SIM from 0.6745 to 0.6621, below the CFG baseline. (§V-C, Table III)

  • complicates: Fine-grained multi-branch guidance in flow-matching TTS increases per-step inference cost, since each additional residual requires an extra masked forward pass.

    Evidence: The joint-residual sampler needs four branch evaluations per step versus two for standard CFG; the authors describe it as a design-space and analysis formulation rather than a free improvement and leave interval-limited application or distillation to future work. (§VI)

Limitations and Open Questions

  • Inference cost: four forward evaluations per step instead of two; batching hides latency only when memory allows.
  • Automatic metrics only: no MOS, SMOS, or preference test, so perceived naturalness and speaker similarity are not measured; the “more expressive acoustic pattern” in Figure 2 rests on one spectrogram example.
  • Weak comparison to the closest baseline: Selective CFG numbers are quoted from its paper with a possibly different evaluation setup, so the claimed better trade-off is not a controlled comparison.
  • Fixed, backbone-specific weights: one hand-picked setting per backbone; sensitivity beyond the small Table III ablation, and the search procedure, are not reported. No variance across seeds is given.
  • Small and mixed gains on F5-TTS: SEED-EN WER rises against the default CFG, and SEED-ZH SIM dips marginally.
  • Backbone-dependent masking: what counts as the text and speaker condition differs between F5-TTS and CosyVoice 2 (where the speaker condition bundles a global embedding and prompt acoustics), which matters for faithful reimplementation.
  • The general rule with a text-oriented variant () is described but not evaluated beyond the S + T control.
  • The method is applied to two pretrained bilingual (English and Chinese) backbones; no new cross-lingual capability is claimed or tested.

Wiki Connections

  • Flow Matching — Proposes a four-branch guidance rule for the velocity field of flow-matching TTS samplers, applied without retraining to F5-TTS and the CosyVoice 2 acoustic generator.
  • Zero-Shot TTS — Targets the speaker-similarity versus text-accuracy trade-off of CFG in prompt-based cloning, evaluated on LibriSpeech-test and Seed-TTS-Eval with pretrained models.
  • F5-TTS — Backbone; its default CFG (strength 2.0) is the main baseline and the residual reweighting is added on top without retraining.
  • CosyVoice 2 — Second backbone; the method acts on its chunk-aware flow-matching acoustic generator with the official CFG as the base sampler.
  • Selective CFG for Zero-shot TTS — Closest baseline; recast as speaker-selective guidance that ties the speaker and joint residuals, with its results quoted for comparison.
  • Seed-TTS — Source of the SEED-EN and SEED-ZH zero-shot evaluation sets.
  • MegaTTS 3 — Its text-conditioned speaker guidance is placed in the same residual family as speaker-selective CFG.
  • E2 TTS — Cited as a flow-matching zero-shot TTS system motivating the guidance study.
  • DMP-TTS — Cited as a TTS-specific guidance method (chained guidance) that exposes the limits of single-scale CFG.
  • ReStyle-TTS — Cited as related TTS guidance work on relative style control.