Speech Generation Research Wiki

Flamed-TTS: Flow Matching Attention-Free Models for Efficient Generating and Dynamic Pacing Zero-shot Text-to-Speech

6 min read

arXiv · 2025 · Preprint

Hieu-Nghia Huynh-Nguyen et al. (FPT Software AI Center, Vietnam) · → Paper · Demo: ✓ · Code: ✓

Flamed-TTS replaces self-attention in the flow-matching denoiser with ConvNeXt modules — enabled by using semantically-enriched discrete codec codes as the prior distribution — and introduces probabilistic duration and silence generators, achieving best-in-class WER (4%) with 10x lower RTF than F5-TTS and 4-5x higher temporal diversity (pauses) than deterministic NAR baselines.

Problem

Existing flow-matching TTS systems rely on self-attention in the iterative denoiser, which introduces quadratic O(L²·d) computational complexity and inference latency. AR TTS models produce temporally diverse speech with natural pauses, but NAR models with deterministic duration predictors generate fixed durations per phoneme, resulting in unnatural, monotone pacing. NAR models also sometimes suffer from token repetition or content leakage from the reference prompt. There is a gap between the temporal naturalness of AR and NAR systems that has not been specifically addressed.

Method

Flamed-TTS is a zero-shot TTS framework with two main components: a Code Generator and an attention-free Denoiser.

Code Generator. Given input phonemes, a Phoneme Encoder produces hidden representations. A Probabilistic Duration Generator samples phoneme durations from noise via flow matching (OT-CFM objective), replacing the standard deterministic MSE regressor. This allows different durations to be sampled across runs, producing temporal variability. A Silence Generator (identical design) samples optional silence durations after each phoneme — a special [SIL] token prepended to each phoneme enables flexible silent pause insertion with zero-duration possible when no pause is appropriate. After duration expansion and silence insertion, a Code Decoder (hierarchical FFT-based architecture, following the authors’ prior OZSpeech work) generates 6 code sequences (1 prosody + 2 content + 3 acoustic) from FACodec, conditioned on phonemes and the speech prompt. The speech prompt is integrated at each hierarchical level, conditioning acoustic prediction at the appropriate code layer.

Overview of Flamed-TTS. The input speech prompt is first processed by the Codec Encoder, which produces six latent codes: one for prosody, two for content, and three for acoustic details. These encoded representations are then duplicated based on the durations predicted by the Duration Generator, while the Silence Generator inserts silences after each phoneme. The Code Decoder then generates predicted codes for the text prompt, conditioned on both the encoded phonemes and the latent representation of the reference speech. These predicted codes are converted into embeddings and merged before being passed through the Denoiser, where flow matching is performed. Finally, the output embeddings are fed into the Codec Decoder to synthesize the final speech waveform.

Attention-Free Denoiser. The 6 code sequences are encoded and folded into a single embedding of shape B×L×D, passed through CNN layers, and used as the semantically enriched prior x_pr for the flow-matching process. The initial input is defined as x_0’ = x_pr + τ·ε, where ε ~ N(0,I) and τ is a noise scaling hyperparameter (optimal τ=0.3). The Denoiser is a DiT where all multi-head self-attention modules are replaced with lightweight ConvNeXt modules. The hypothesis is: since the prior distribution already contains semantic information (captured in the code sequences), the Denoiser only needs to refine acoustic details, eliminating the need for global attention. The computational complexity reduces from O(L²·d) to O(L·k²·d), where k is the convolutional kernel size.

Code Decoder architecture. With the encoded phonemes, the codes are generated gradually by corresponding FFT Blocks with condition of prior code of speech prompt. These synthesized codes are combined by a CNN module. This embedding is finally finetuned by the Denoiser with flow matching algorithm.

The total training loss combines: L_prior (code decoder NLL), L_dur (duration flow matching), L_sil (silence flow matching), L_CFM (denoiser flow matching), and L_anchor (an auxiliary self-consistency loss that directly predicts x_1 from x_t to stabilize training).

The final waveform is produced by passing the Denoiser output through the FACodec decoder.

Model sizes. Flamed-TTS-Base: 143M trainable parameters. Flamed-TTS-Small: 76M parameters (with small UTMOS degradation but equivalent WER). Both use the same 102M frozen FACodec.

Training. LibriTTS 500 hours. No data augmentation beyond the existing corpus. Trained on NVIDIA A100.

Key Results

Table 1 (LibriSpeech test-clean, 1s/3s/5s prompts). At 5s prompt, Flamed-TTS achieves WER 0.04 (best among all baselines), UTMOS 3.87 (second to SparkTTS 4.33, which uses 100k training hours), SIM-O 0.51 (comparable to F5-TTS 0.57 on same training data), SIM-R 0.59. Flamed-TTS uses the same 500h LibriTTS as VALL-E and F5-TTS baselines. Compared to OZSpeech (same dataset, WER 0.05), Flamed-TTS improves UTMOS 3.87 vs 3.15 while maintaining comparable WER.

Table 2 (latency). RTF 0.016 at 16 NFE vs F5-TTS RTF 0.26 at 32 NFE — a 10x speedup. Comparable RTF to OZSpeech (0.013, single step), but with substantially higher UTMOS (3.72 vs 3.15).

Table 3 (temporal diversity, 5s prompt). Flamed-TTS generates 4.47 pauses per utterance (mean) vs OZSpeech 1.18 and NaturalSpeech2 1.20 (#Pauses); mean pause duration 0.149s vs 0.030–0.032s for deterministic NAR baselines — 4-5x more pauses and longer pauses. Speech rate 4.51 ± 0.76 (more natural variability than OZSpeech 5.61 ± 0.55) and approaches AR VALL-E (4.02 ± 1.85).

Table 4 (NFE ablation). WER plateaus at NFE=4, UTMOS improves monotonically. Good quality at NFE=16 (UTMOS 3.79) vs diminishing returns beyond NFE=64.

Table 5 (noise scaling τ). Optimal τ=0.3; performance degrades symmetrically on both sides.

Novelty Assessment

The attention-free DiT denoiser enabled by semantically-enriched priors is the primary architectural novelty. The hypothesis — that self-attention is unnecessary when the input already encodes semantic content — is well-supported empirically and has clear computational motivation. The probabilistic silence generator is a simple but underexplored mechanism for temporal diversity in NAR TTS; prior NAR systems (FastSpeech, OZSpeech) do not model silences probabilistically. The combination of probabilistic duration + silence modeling provides temporal diversity comparable to AR systems while maintaining the efficiency advantages of NAR. The baseline comparisons are generally fair (same training data). The main limitation is that evaluations use only LibriTTS 500h and LibriSpeech test-clean; broader generalization is not assessed.

Field Significance

Moderate — Flamed-TTS introduces an empirically supported hypothesis: that self-attention in flow-matching denoisers becomes redundant when the prior distribution carries sufficient semantic content, enabling a switch to convolutional modules with order-of-magnitude latency gains. The probabilistic silence generator provides a concrete, reusable mechanism for closing the temporal diversity gap between AR and NAR TTS that prior work left unaddressed. Both contributions are validated on a single English benchmark, so the scope of the evidence remains limited.

Claims

  • Replacing self-attention with convolutional modules in a flow-matching TTS denoiser preserves competitive naturalness and intelligibility when the prior distribution is semantically enriched with discrete codec representations. (§ Flow Matching Attention-Free Models, Table 1)
  • Probabilistic duration and silence generators allow non-autoregressive TTS models to achieve temporal diversity (pause rate, pause duration, speech rate variability) approaching that of autoregressive systems. (§ Probabilistic Duration and Silence Generation, Table 3)
  • Attention-free flow-matching denoisers achieve substantially lower inference latency than self-attention-based counterparts without a proportional reduction in synthesis quality. (§ Experiments, Table 2)
  • In zero-shot TTS trained on equal data, explicit phoneme-to-speech alignment mechanisms yield better intelligibility than purely data-driven alignment, independent of the generative backbone architecture. (§ Main Results, Table 1)

Limitations and Open Questions

  • Evaluated only on English LibriSpeech; multilingual and zero-shot cross-lingual performance is not assessed.
  • Training data (500h LibriTTS) is relatively small; performance gap with SparkTTS (100k hours) in naturalness is expected.
  • The attention-free hypothesis is validated empirically but not theoretically analyzed — it is unclear whether this would hold for content-less (pure noise) priors.
  • The convolutional kernel size k may limit modeling of long-range dependencies in very long utterances.
  • Flamed-TTS-Small shows UTMOS degradation (~5-6%) at half the model size, suggesting UTMOS is more sensitive to capacity than WER or speaker similarity.
  • No streaming deployment or real-time application evaluation.

Wiki Connections

Flamed-TTS is most directly related to OZSpeech, which uses a single-step flow-matching paradigm and the same FACodec prior initialization. The attention-free design contrasts with F5-TTS which retains self-attention at the same step count. The probabilistic duration modeling extends work on NAR TTS with stochastic duration, connecting to prosody-control. The use of FACodec relates to the neural-codec concept page. The flow-matching training paradigm is documented in flow-matching. The zero-shot capability and LibriSpeech benchmark situate the paper in the zero-shot-tts literature.

Backlinks

Graph View