arXiv · 2026 · Preprint

Ho-Lam Chung et al. (National Taiwan University / Amazon) · → Paper · Demo: ✓ · Code: ✓

Turns the Fréchet distance, normally an offline evaluation statistic, into a differentiable fine-tuning loss on speech sampled with the deployment-time four-step sampler, so that a few-step flow-matching TTS model’s output distribution is pulled toward reference statistics in frozen Whisper and CTC content spaces; four-step VoxCPM2 then reaches lower Seed-TTS English WER than its own ten-step sampler.

Problem

Few-step flow-matching and diffusion TTS systems are trained with local objectives: a per-frame velocity loss, frame reconstruction, and stop prediction, all computed under teacher forcing. None of these losses looks at what the compressed sampler actually produces at inference time, so a model can show healthy training loss while its few-step outputs drift away from the distribution of its own full-step outputs. On the tokenizer-free autoregressive TTS model VoxCPM2, cutting the flow-matching decoder from ten to four Euler steps raises Seed-TTS English WER from 1.74% to 2.23%, so the four-step sampler is not just a faster copy of the ten-step one. Distillation methods (progressive distillation, consistency models) shrink the trajectory gap but do not constrain the population distribution of complete utterances, and distribution-matching distillation from image generation needs teacher score networks or auxiliary fake-score models.

Method

The backbone is VoxCPM2, a 2B-parameter tokenizer-free TTS model: an autoregressive language model produces continuous AudioVAE latents through a diffusion-transformer acoustic decoder trained with conditional flow matching, and the latents are decoded to a waveform. The base fine-tuning objective combines the flow-matching loss, a stop-prediction loss, and three small auxiliary losses inherited from the underlying recipe (teacher-endpoint, preference-feature and Whisper-text), which the authors state are not their contribution. SR-FD (Speech Representation Fréchet Distance loss) adds one weighted term on top.

During each update the model synthesizes a complete short utterance with the same four-step Euler sampler used at deployment, keeping the computation differentiable. Frozen extractors map each generated utterance to a pooled feature vector: a Whisper large-v3 encoder for semantic content (960 dimensions) and a wav2vec 2.0 CTC model whose posteriors are summarized by mean and standard-deviation pooling over non-blank content plus blank statistics (72 dimensions). The generated mean and covariance are estimated over a 50,000-vector feature queue plus the current mini-batch; only the current batch carries gradients, which makes a population-scale covariance affordable at the memory cost of one batch.

Fig. 1. SR-FD training pipeline. The model samples speech with the deployment-time four-step sampler; frozen Whisper and CTC extractors map it to features whose mean and covariance are matched to offline reference moments via a Fréchet distance. Gradients flow only into the LoRA weights (dashed). Blue: trainable; gray: frozen; orange: loss.

These generated moments are matched to three reference targets computed offline, all describing speech content because content drift dominates the four-step failures: a low-step Whisper anchor (statistics of original four-step generations whose ASR transcripts match the target text), a teacher CTC target (from ten-step VoxCPM2 generations, transferring the stronger sampler’s content behaviour), and a real-speech CTC target (from real LibriTTS voice-cloning speech, so the CTC branch does not simply copy teacher artifacts). Each Fréchet term is divided by its own detached value so that terms of different scale contribute balanced gradients; the Whisper and CTC branches are weighted equally, with the two CTC targets splitting the CTC half. A length gate admits a sample only when its generated-to-target duration ratio lies in [0.92, 1.08], since truncated or runaway outputs would inject noise into the moments. The Whisper covariance is rank-deficient (numerical rank about 240 from 1,000 utterances), so it is regularized with a small diagonal term.

Training updates only LoRA adapters (rank 32 on the q, k, v and o projections of the language model and the DiT) in two stages: a supervised LoRA adaptation with the base loss, then 1,600 further steps with SR-FD enabled on a 767-row LibriTTS-derived manifest that up-weights hard lexical examples. The second stage is a small residual refinement (relative LoRA weight change of about 7.9 × 10⁻⁵). At test time the extractors, queues, reference moments and Fréchet computation are removed: the deployed model is a plain four-step VoxCPM2 with LoRA adapters, with no added parameters or inference computation.

Key Results

All systems are evaluated on Seed-TTS test-en (1,088 prompts, 11,805 reference words) with the upstream Seed-TTS scorer and a frozen, shared decoding protocol (§IV-A, §IV-C).

  • WER: four-step SR-FD fine-tuning reaches 1.41% (167 word errors), against 2.23% (263) for the original four-step VoxCPM2 and 1.74% (205) for the original ten-step VoxCPM2. These are 36.5% and 18.5% relative reductions; under an utterance-level paired bootstrap the absolute reductions are 0.81 points (95% CI [0.57, 1.06], p < 10⁻⁴) and 0.32 points (95% CI [0.14, 0.51], p = 0.0004) (Table II, §V-A).
  • Reported references: the SR-FD figure is below the reported 32-step results for ARCHI-TTS (1.47%) and F5-TTS (1.83%), but these are quoted from the original papers, not re-decoded, so only WER is comparable. CosyVoice 3 (1.45%) and Qwen3-TTS (1.24%) are mentioned in the text under the same protocol; Qwen3-TTS’s reported WER is lower than SR-FD’s (Table II, §IV-B).
  • Speaker and quality proxies: speaker similarity rises from 0.74 to 0.76, equal to the ten-step baseline. UTMOS recovers from 3.30 to 3.76 and DNSMOS OVRL/P808 from 2.90/3.53 to 3.07/3.65, close to but still slightly below the ten-step values (3.81, 3.09/3.67) (Table II).
  • Listening test: in a blinded pairwise test of four-step SR-FD against the ten-step baseline (13 listeners, 229 judgments, 128 decisive), SR-FD wins 61 to 67 (47.7% preference, binomial p = 0.659), with 42.8% ties. The Wilson 90% CI (40.5% to 54.9%) lies inside a pre-specified ±10-point equivalence margin, so TOST supports practical equivalence. There is no listening test against the four-step baseline (Table III, §V-C).
  • Cost: real-time factor is 0.2285 against 0.2501 for the ten-step baseline, about 8.6% lower; the autoregressive component does not scale with flow steps, so the authors claim zero added inference cost rather than a large speedup (§V-B).
  • Error sources: substitutions fall from 203 (four-step) and 168 (ten-step) to 142, deletions from 43 and 30 to 21, insertions from 18 and 8 to 5 (Table IV). SR-FD is below both baselines in every length bucket, with the largest gain over the four-step baseline on short prompts (≤10 words: 2.42% to 1.17%, against 2.62% to 2.01% for medium and 1.77% to 1.18% for long) (Table V). SR-FD also introduces a few regressions on prompts both baselines got right (§V-D).
  • Target ablation: removing any one target worsens full-set WER: without the low-step Whisper anchor 1.54% (the largest degradation), without the real-speech CTC target 1.49%, without the teacher CTC target 1.48%, against 1.41% for all three; all ablated variants tie at 23/2070 on the gate subset, so the ranking only appears on the full set. UTMOS and DNSMOS stay within 0.005 of the full model throughout (Table VI, §V-E).
  • FD as a diagnostic: across 16 checkpoints, raw FD correlates weakly with gate WER (strongest Spearman ρ = 0.383, p = 0.143, for the real-speech CTC FD; ρ = -0.021 for the Whisper anchor). In a separate CTC feature pass, the ten-step baseline is closer to real speech in FD (0.00517) than SR-FD (0.00681) despite its higher WER (§V-F).

Novelty Assessment

The new element is the training objective: a Fréchet distance between Gaussian moments of frozen speech-encoder features, computed on speech sampled with the deployment-time few-step sampler and backpropagated into the TTS model. FID/FAD-style distances are usually offline metrics and feature-matching or MMD objectives usually need a discriminator or learned kernel; here the reference side is just stored first- and second-order moments, with a feature queue to make covariance estimation tractable. The multi-target design (a successful-low-step anchor, a stronger-sampler teacher, and real speech, spread across two content spaces) is a specific recipe whose components are each shown to matter. The architecture is unchanged VoxCPM2 with standard LoRA adapters, the base recipe including its auxiliary losses is inherited, and the result covers one backbone, one language and one benchmark. This is a training-objective contribution with a careful statistical evaluation, not a new model.

Field Significance

Moderate. The paper shows that a cheap, discriminator-free distributional loss on sampled outputs can do more than close the few-step gap: on this backbone it reverses the usual step-count trade-off, with the four-step model beating the ten-step one on intelligibility while listeners cannot reliably tell them apart. Its frank negative finding, that raw representation FD is a poor checkpoint or model selector even though minimizing it during training helps, is a useful caution for anyone treating Fréchet-style distances as quality targets. The evidence is limited to a single model and English benchmark, and the method is explicitly framed as an intelligibility regularizer, not a general quality objective.

Claims

  • supports: Matching the distribution of speech sampled with the deployment-time few-step sampler, rather than only supervising teacher-forced frames, can reduce content errors in few-step flow-matching TTS without adding inference cost.

    Evidence: Four-step VoxCPM2 fine-tuned with SR-FD reaches 1.41% Seed-TTS test-en WER against 2.23% for the original four-step and 1.74% for the original ten-step sampler, both reductions bootstrap-significant; the deployed model is a plain four-step VoxCPM2 with LoRA adapters. (§V-A, Table II; §III-E)

  • supports: Few-step sampling compression in flow-matching TTS degrades primarily content fidelity (word substitutions), and a content-space distributional regularizer recovers it across utterance lengths.

    Evidence: Reducing VoxCPM2 from ten to four steps raises substitutions from 168 to 203 and deletions from 30 to 43; SR-FD lowers them to 142 and 21 and is below both baselines in every length bucket, with the largest gain on prompts of 10 words or fewer. (§V-D, Tables IV and V)

  • complicates: A smaller Fréchet distance in a speech-representation space does not reliably indicate a more intelligible TTS system, so representation FD cannot replace external content metrics for checkpoint or model selection.

    Evidence: Across 16 checkpoints, raw FD correlates weakly with gate WER (best Spearman ρ = 0.383, p = 0.143; Whisper-anchor ρ = -0.021), and the ten-step baseline sits closer to real speech in CTC FD (0.00517) than the SR-FD model (0.00681) despite its higher WER. (§V-F)

  • refines: For distribution-matching losses on TTS output, combining reference targets from different sources (successful low-step outputs, a stronger sampler, and real speech) is more effective than any two-target subset.

    Evidence: Leave-one-out ablation raises full-set WER from 1.41% to 1.54% (no Whisper anchor), 1.49% (no real-speech CTC) and 1.48% (no teacher CTC), while UTMOS and DNSMOS stay within 0.005, indicating the targets act on ASR-recoverable content rather than proxy quality scores. (§V-E, Table VI)

  • complicates: Objective intelligibility gains from few-step regularization need not translate into a perceptual advantage, and proxy quality scores may lag slightly behind the slower sampler.

    Evidence: Against the ten-step baseline, listeners split 61 to 67 on decisive votes (47.7% SR-FD preference, TOST equivalence within ±10 points), and UTMOS (3.76 vs 3.81) and DNSMOS OVRL (3.07 vs 3.09) remain slightly below ten-step levels. (§V-B, §V-C, Tables II and III)

Limitations and Open Questions

The headline gain is not cleanly isolated from the training recipe

The reported SR-FD model is the step-1600 checkpoint chosen on the full Seed-TTS test-en WER frontier, the same set used for the headline result; only the target ablations select on the 200-prompt gate subset. The paper also reports no result for the first-stage supervised LoRA model (base loss and inherited auxiliary losses, including a Whisper-text loss) without SR-FD, so how much of the gain over the original four-step model comes from SR-FD itself is not measured.

  • Single backbone, single language: all results are on VoxCPM2 and Seed-TTS English; the authors do not test other few-step flow-matching or diffusion TTS models, Chinese or hard test sets, or other step counts.
  • Small, close margins: the strongest systems differ by a few dozen word errors out of 11,805, and target ablation differences are a few errors (175 vs 176); the ablation ranking is invisible on the gate subset.
  • Content-only regularizer: all three targets live in ASR-oriented spaces; the paper frames SR-FD as an intelligibility regularizer and does not claim prosody, expressiveness or speaker gains beyond recovering the ten-step similarity.
  • Limited perceptual evidence: the listening test is a 13-listener preference test against the ten-step baseline only, establishing equivalence within a ±10-point margin; there is no MOS and no comparison with the four-step baseline.
  • Prompt-level regressions: SR-FD introduces some new errors on prompts both baselines transcribe correctly, so it does not monotonically improve text fidelity.
  • Whisper covariance bias: the Whisper branch’s covariance is rank-deficient, so its absolute FD is biased and not usable for selection.
  • Open question: whether the same loss composes with consistency or distillation-based step reduction, or extends to one- or two-step samplers, is not tested.

Wiki Connections

  • Flow Matching — Adds a sampled-output distributional loss to the conditional flow-matching objective of VoxCPM2’s DiT decoder to recover intelligibility under four-step Euler sampling.
  • Zero-Shot TTS — Evaluates prompt-based voice cloning on Seed-TTS test-en and shows speaker similarity is preserved at the ten-step level.
  • Self-Supervised Speech — Uses a frozen wav2vec 2.0 CTC model as one of two feature extractors inside the training loss.
  • Subjective Evaluation — Uses a blinded pairwise preference test with a pre-specified TOST equivalence margin to check that the objective gain has no perceptual cost.
  • VoxCPM2 — The backbone that SR-FD fine-tunes with LoRA; its original four- and ten-step samplers are the main baselines.
  • Seed-TTS — Provides the test-en benchmark and upstream WER and speaker-similarity scoring protocol.
  • ARCHI-TTS — Closest reported flow-matching reference (1.47% WER at 32 steps), which four-step SR-FD undercuts.
  • F5-TTS — Reported 32-step flow-matching reference on Seed-TTS test-en (1.83% WER).
  • Qwen3-TTS — Large post-trained system cited at 1.24% WER on the same protocol, still below SR-FD.
  • CosyVoice 3 — Large post-trained system cited at 1.45% WER on the same protocol.
  • Whisper — Its large-v3 encoder is the frozen semantic extractor for the low-step anchor target.
  • UTMOS — Objective quality proxy used to show quality recovers alongside WER.