DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Factorized Discrete Flow Matching

arXiv · 2025 · Preprint

Ngoc-Son Nguyen et al. (FPT Software AI Center) · → Paper · Demo: ✓ · Code: ?

DiFlow-TTS applies discrete flow matching directly over factorized codec token sequences for zero-shot TTS, achieving state-of-the-art naturalness and prosody with a 164M-parameter model that is up to 11.7x smaller and 34x faster than comparable baselines.

Problem

Most zero-shot TTS systems based on flow matching operate in continuous spaces — either over mel-spectrograms or continuous representations derived from discrete tokens — rather than directly over discrete probability distributions. Autoregressive codec language models achieve strong quality but suffer from slow inference and require large-scale training data. Diffusion-based discrete methods couple the training and sampling processes, making it costly to adjust noise schedules or sampling configurations post-training. There was no established baseline for applying discrete flow matching (DFM) specifically in the speech domain, leaving an open question about whether generative modelling directly in discrete token space could match continuous-space approaches in quality while offering efficiency advantages.

Method

DiFlow-TTS builds on FACodec (from NaturalSpeech 3; Ju et al., 2024), which factorizes speech into three disentangled token sequences — prosody (1 codebook), content (2 codebooks), and acoustic details (3 codebooks) — each with vocabulary size 1024 — plus a speaker embedding extracted from Conformer blocks with temporal pooling.

The system has three modules:

Speech Tokenizer. FACodec encodes a reference prompt into factorized prosody, content, and acoustic tokens, plus a 256-dimensional speaker embedding. This codec is frozen during training.

Phoneme-Content Mapper (PCM). A text input is converted to phonemes, encoded by a phoneme encoder (hidden size 256), and aligned to speech token length using a Duration Predictor (MSE loss on log-scale durations) and a Length Regulator. The aligned sequence is passed through a stack of Feed-Forward Transformer (FFT) blocks to produce content embeddings (768-dim) and content token logits trained with cross-entropy against FACodec content tokens.

Factorized Discrete Flow Denoiser (FDFD). The generative backbone applies DFM over a factorized target: the joint distribution over prosody and acoustic tokens. The source distribution is all-mask tokens; a cubic scheduler ${\kappa }_t=t^2$ linearly interpolates from source to target. At each timestep, the corrupted input $x_t$ (prosody + acoustic tokens concatenated) is split and embedded; reference prosody and acoustic tokens from the prompt provide in-context conditioning. Attribute-type embeddings ($g_p$, $g_c$, $g_a$, each 768-dim) are added before concatenating all attribute embeddings into a unified tensor. A 12-layer DiT (Diffusion Transformer; Peebles and Xie, 2023) with hidden size 768, 12 attention heads, and RoPE positional encoding processes this tensor. A global conditioning vector formed by summing projected speaker embedding and timestep embedding modulates DiT features via Adaptive Layer Normalization (AdaLN). A long skip connection wraps the DiT stack. The final representation is split into prosody ($h^p$) and acoustic ($h^a$) parts processed by dedicated prediction heads (factorized multi-head prediction), each outputting logits over the full vocabulary. The full loss is a weighted sum: ${\lambda }_{dur}{\mathcal{L}}_{dur}+{\lambda }_c{\mathcal{L}}_c+{\lambda }_{FDFD}{\mathcal{L}}_{FDFD}$, with ${\lambda }_{dur}=0.5$, ${\lambda }_c=1.0$, ${\lambda }_{FDFD}=1.0$.

A smaller variant, DiFlow-TTS-Small (122M params), reduces attention heads to 8 and DiT layers to 8. The model is trained on 4×A100 GPUs for 315K steps with AdamW (lr=1e-4, weight decay=0.01, 200K warm-up steps), batch size 16.

At inference, sampling runs for a configurable number of function evaluations (NFE), with 128 NFE yielding peak quality and 4–16 NFE enabling very low latency (RTF 0.03–0.07 on a single A100 80 GB).

Key Results

Objective evaluation (LibriSpeech test-clean, 3s prompt, Table 1 and Table 3):

• UTMOS: 3.98 (DiFlow-TTS) vs. 3.83 (MaskGCT, 100K h) and 3.76 (F5-TTS) — best among all systems
• WER: 0.05, matching OZSpeech — best among baselines
• SIM-O (speaker similarity): 0.45, behind MaskGCT (0.67) and F5-TTS (0.52) — a clear weakness
• F0-RMSE: 7.97, best by a wide margin (MaskGCT 14.33, F5-TTS 13.78)
• Energy RMSE: 0.007, matching MaskGCT

Subjective evaluation (MOS, Table 2, 30 listeners):

• Naturalness MOS: 4.18 ± 0.16 vs. 3.97 (F5-TTS), 3.97 (MaskGCT), 3.94 (VoiceCraft) — best
• Intelligibility MOS: 4.41 ± 0.13 — best
• Similarity MOS: 4.42 ± 0.12 — best (notably higher than SIM-O ranking suggests; authors attribute difference to perceptual identity cues not captured by embedding cosine similarity)

Efficiency (Table 3):

• DiFlow-TTS (164M, 4 NFE): RTF 0.03 — matches OZSpeech’s 1-NFE RTF
• DiFlow-TTS-Small (122M, 16 NFE): RTF 0.05, 5.2×–34× faster than continuous-space baselines (VoiceCraft 1.70, NaturalSpeech 2 1.66), 1.2×–11.7× smaller

Comparisons are made under heterogeneous training data conditions (DiFlow-TTS: 470h; MaskGCT: 100K h; VoiceCraft: 9K h), which should be noted when interpreting quality differences.

Novelty Assessment

The primary contribution is architectural: applying discrete flow matching directly over a structured, factorized discrete space (prosody + acoustic tokens jointly), rather than over a single homogeneous discrete sequence or over continuous representations derived from discrete tokens. The factorized multi-head prediction — separate probability velocity fields for prosody and acoustic subspaces within one DiT — is claimed to be a first in the DFM-for-speech literature. Using FACodec’s disentangled representation as the target space also meaningfully differentiates this work from prior DFM work (Gat et al., 2024), which operates on monolithic sequences.

The work is positioned explicitly as an entry point and baseline study rather than a finished production system. The contribution is therefore primarily architectural/methodological rather than data-scale or engineering. Model size and latency improvements are secondary benefits that follow naturally from the non-autoregressive DFM design with a small DiT backbone.

Field Significance

Moderate — DiFlow-TTS establishes the first baseline for discrete flow matching in zero-shot TTS, demonstrating that probability flows defined directly over factorized discrete token sequences can match or surpass continuous-space systems on naturalness and prosody while achieving competitive latency. The factorized multi-head velocity prediction design provides a concrete architectural template for future discrete generative speech models. Speaker similarity remains the main gap relative to continuous-space approaches, and the work’s self-described role as an entry point limits its immediate impact compared to higher-data production systems.

Claims

• Discrete flow matching defined directly over factorized speech token subspaces achieves competitive naturalness and superior prosody reconstruction compared to continuous-space flow and diffusion TTS baselines trained on comparable data. (§4.2, Table 1, Table 2)
• Factorized multi-head velocity prediction — separate prediction heads for distinct speech attribute subspaces within a single discrete flow model — improves prosody and acoustic fidelity over a single-head alternative. (§4.3, Table 4)
• Global speaker embedding conditioning via AdaLN is insufficient for reliable zero-shot speaker similarity in discrete token-space TTS; perceptual similarity judgements diverge from embedding-based automatic metrics (SIM-O) in this setting. (§4.2, Table 1, Table 2)
• Non-autoregressive discrete flow models with compact DiT backbones can match the inference latency of single-step flow matching systems while operating at higher NFE, without compromising data efficiency. (§4.2, Table 3)

Limitations and Open Questions

The model’s weakest dimension is automatic speaker similarity (SIM-O): simple global AdaLN speaker conditioning is insufficient for reliable timbre reproduction, especially in zero-shot settings. The authors acknowledge this and suggest cross-attention over local timbre embeddings as future work.

Because FACodec separates speaker identity via a timbre embedding external to the discrete token streams, integrating speaker conditioning into a DFM framework is non-trivial; alternative codecs such as EnCodec implicitly embed speaker information within VQ codebooks, potentially enabling stronger speaker adaptation in future discrete-space models.

The evaluation uses a single language (English) and a moderately sized training set (470h), leaving open questions about multilingual scalability and behavior at larger data scales. The gap between SIM-O ranking (third among five) and perceptual similarity MOS ranking (best) also points to known deficiencies in embedding-based automatic speaker similarity metrics, and a better evaluation framework for zero-shot speaker identity remains an open research direction.

Wiki Connections

• flow-matching — DiFlow-TTS extends this paradigm from continuous to discrete probability space; distinct from continuous-space flow TTS systems
• zero-shot-tts — in-context learning from a short reference prompt; same paradigm as VALL-E and OZSpeech from the same lab
• neural-codec — the system depends on FACodec from NaturalSpeech 3 for factorized discrete representations
• speaker-adaptation — global AdaLN speaker conditioning is identified as the system’s primary weakness; contrasted with approaches using cross-attention over local timbre embeddings
• evaluation-metrics — concrete divergence between SIM-O (automatic speaker embedding cosine similarity) and perceptual MOS similarity rankings; illustrates limits of embedding-based speaker identity evaluation
• F5-TTS — continuous-space flow baseline; DiFlow-TTS matches or exceeds on naturalness and prosody but trails on SIM-O
• MaskGCT — masked generative codec baseline trained on 100K h; outperforms on SIM-O but uses 200x more training data
• VoiceCraft — autoregressive baseline used for size and latency comparison; 830M params vs 164M for DiFlow-TTS