Speech Generation Research Wiki

OZSpeech: One-step Zero-shot Speech Synthesis with Learned-Prior-Conditioned Flow Matching

5 min read

ACL · 2025 · Conference

Nghia Huynh Nguyen Hieu et al. (FPT Software AI Center) · → Paper · Demo: ✓ · Code: ✓

OZSpeech reformulates optimal transport conditional flow matching to start from a learned prior (rather than Gaussian noise) that closely approximates the target distribution, enabling single-step zero-shot TTS that is 2.7–6.5x faster than SOTA while achieving multi-fold WER improvement and stable intelligibility under noisy prompts.

Problem

Flow-matching TTS systems (E2 TTS, F5-TTS) require many inference steps (32–200) because they must traverse the full path from Gaussian noise to the target speech distribution. This makes them slow at inference and impractical for real-time applications. Additionally, traditional OT-CFM approaches require large training datasets to be robust (F5-TTS needs 95,000 hours), making them unsuitable for low-resource languages. A related problem is that most zero-shot TTS systems degrade substantially when the reference audio prompt is noisy.

Method

OZSpeech consists of three components:

FACodec (frozen): NaturalSpeech 3’s factorized neural codec decomposes speech into 6 discrete token sequences encoding prosody (1 quantizer), content (2 quantizers), and acoustic details (3 quantizers), plus a separate timbre embedding.

Prior Codes Generator: A cascaded feed-forward transformer that generates prior codes x_pr hierarchically: the first code sequence conditioned on phoneme embeddings, each subsequent sequence conditioned on the previous one. A Duration Predictor (FastSpeech-style) aligns phoneme embeddings to codec token length. A Prior Loss minimizes the negative log joint probability, pushing content codes close to their targets while acoustic codes converge to mean representations.

Vector Field Estimator (OT-CFM): Rather than regressing vectors from Gaussian noise x_0 to data x_1, the estimator regresses from the learned prior x_pr to x_1. The prior time τ is learned implicitly (not fixed), reducing both the number of steps and step size. A folding mechanism models all 6 quantizer sequences simultaneously (rather than sequentially like VALL-E) by concatenating them along the hidden dimension, reducing computation. A quantizer encoding (learned per-quantizer identifier embeddings) prevents cross-quantizer confusion. Anchor Loss regularizes embedding collapse. The acoustic prompt (prosody + acoustic codes, content masked) is concatenated with x_pr as the conditioning signal. Training uses arbitrary-segment prompt selection (not always the beginning) for better generalization.

Overview of OZSpeech: (a) The overall architecture: The text prompt is converted to phonemes and then into prior codes via the Prior Codes Generator. Simultaneously, the audio prompt is encoded into codes using the FACodec Encoder. These codes are concatenated along the sequence dimension and fed into the OT-CFM Vector Field Estimator, which generates codes preserving the text content and acoustic attributes. Finally, the FACodec Decoder converts them into output speech. (b) The Prior Codes Generator produces sequences of phoneme-aligned codes. (c) The Vector Field Estimator refines these codes with the prosody and acoustic details from the acoustic prompt.

Key Results

On LibriSpeech test-clean with 3-second prompts (Table 1):

ModelWERUTMOSSIM-ONFE
F5-TTS (95,000h data)0.243.760.5332
VoiceCraft (9,000h)0.183.550.51—
VALL-E (500h)0.193.680.40—
OZSpeech (500h)0.053.150.401

OZSpeech achieves the best WER by a large margin (44–58% relative improvement over the next-best method), consistent across 1s/3s/5s prompt lengths. UTMOS is lower than F5-TTS and VALL-E (3.15 vs. 3.7+) due to codec trade-offs (FACodec balances acoustic and semantic representations). Model size is 145M parameters — only 29–71% of competitors.

Under noisy prompts (Table 4), all baselines degrade dramatically. VALL-E reaches 93% WER at 0 dB SNR. OZSpeech maintains WER at 0.05–0.06 across all SNR levels, which is essentially unchanged from clean conditions. Fine-tuning with noisy prompts further improves acoustic quality metrics.

Novelty Assessment

The reformulation of OT-CFM with a learned prior (replacing Gaussian initialization) is a genuine algorithmic contribution. The single-step paradigm without distillation stages is novel in the flow-matching TTS literature. The noise robustness finding is practically important and underexplored in the field. The trade-off — significantly better intelligibility at the cost of lower perceptual naturalness (UTMOS) — is honest and well-characterized. The codec dependency on FACodec is a limitation for adaptation.

Field Significance

Tip

High — OZSpeech introduces a meaningful algorithmic shift in flow-matching TTS by replacing Gaussian initialization with a learned prior, enabling single-step sampling without distillation. The WER improvement over comparable systems (trained on 190x less data than F5-TTS) and the first systematic study of noisy-prompt robustness in zero-shot TTS open two concrete new directions for the field.

Claims

  • Replacing Gaussian noise with a learned prior as the starting point for flow matching reduces the number of required inference steps to one without needing a separate distillation stage. (§3.1, §3.3)
  • Flow-matching TTS systems trained on traditional OT-CFM are data-hungry and fail to generalise when retrained on limited data, while neural codec-based systems remain effective with as few as 500 hours. (§4.2, Table 1)
  • Zero-shot TTS systems trained exclusively on clean prompts degrade substantially in intelligibility when given noisy reference audio, with autoregressive codec models being especially vulnerable. (§4.4, Table 4)
  • Factorised codec representations that balance acoustic and semantic attributes trade perceptual naturalness (UTMOS) for intelligibility (WER) relative to codecs that prioritise acoustic fidelity. (§4.2)
  • Fine-tuning a zero-shot TTS model on noise-augmented prompts preserves intelligibility and substantially recovers acoustic quality metrics under low-SNR conditions. (§4.4, Table 4)

Limitations and Open Questions

  • UTMOS and speaker similarity lag behind F5-TTS and VoiceCraft; the prior-based approach trades acoustic naturalness for intelligibility and speed.
  • Duration predictor rounding errors (integer quantization of phoneme durations) introduce temporal domain artifacts.
  • FACodec dependency means reproduction requires NaturalSpeech 3’s codec infrastructure.
  • Noise robustness fine-tuning improves non-WER metrics but was validated only on the QUT-NOISE database.
  • Future work: multilingual extension, adaptive noise filtering integration.

Wiki Connections

This paper is a significant advance for flow-matching TTS, demonstrating that the “start from learned prior” paradigm can replace multi-step sampling. It directly competes with autoregressive systems on zero-shot-tts while being much faster. The use of FACodec for disentangled factorized speech tokens connects to neural-codec and disentanglement. Comparison to VALL-E 2301.02111 positions it relative to autoregressive-codec-tts. The noise robustness analysis opens a new dimension of evaluation not typically covered by evaluation-metrics standards.

OZSpeech is directly compared to F5-TTS in Table 1 — OZSpeech achieves WER 0.05% vs. F5-TTS’s 0.24% on LibriSpeech test-clean using 190× less training data (500h vs. 95K h) and a single inference step (NFE=1 vs. 32). F5-TTS outperforms OZSpeech on naturalness (UTMOS 3.76 vs. 3.15), highlighting the WER/naturalness trade-off between the two approaches. The two papers represent distinct strategies within the flow-matching paradigm: F5-TTS optimizes the ODE at inference via Sway Sampling; OZSpeech shrinks the integration path via a learned prior.

Backlinks

Graph View