arXiv · 2025 · Preprint
Yanqing Liu et al. (Microsoft) · → Paper · Demo: ? · Code: ?
Dragon-FM unifies autoregressive next-token prediction and flow-matching within a single model, processing audio in 2-second chunks at a codec frame rate of 12.5 tokens per second — enabling KV-cache efficiency across chunks while using bidirectional attention within each chunk to accelerate denoising.
Problem
Prior TTS architectures face a fundamental tension between two leading generative paradigms. Autoregressive (AR) models predict tokens sequentially using causal attention, enabling KV-cache reuse for low first-byte latency, but cannot exploit future context and accumulate slow step-by-step inference costs as sequence length grows. Diffusion and flow-matching models generate tokens in parallel with bidirectional attention and produce high-quality continuous representations, but are incompatible with KV-caching, leading to high latency on short segments. Additionally, discrete codec token sequences for 48 kHz audio at standard frame rates (e.g., 75–150 Hz) create long sequences that strain both AR and diffusion architectures. No prior single-model design simultaneously addressed all three issues: KV-cache compatibility, bidirectional within-chunk context, and low frame-rate codec quality.
Method
Dragon-FM processes speech as a sequence of 2-second chunks, each containing 25 discrete FSQ (finite scalar quantizer) tokens at 12.5 tokens per second. Across chunks the model operates autoregressively — each chunk is conditioned on previously generated chunks using a KV cache, providing global coherence and incremental generation that approaches streaming. Within each chunk, flow-matching denoising runs in parallel with bidirectional self-attention, exploiting full chunk context to refine all 25 tokens simultaneously. The number of flow-matching denoising steps per chunk is further reduced through mean flow optimisation, bringing per-chunk inference to as few as 2 NFE (neural function evaluations) without substantial quality loss.
The codec is a custom-designed asymmetric architecture: the encoder uses full-attention transformers to compress 48 kHz waveforms into 12.5 Hz latent representations, which are then quantised using a finite scalar quantiser (FSQ) into compact discrete token sequences. The decoder is fully causal — upsampling FSQ tokens through causal-attention transformers and reconstructing 48 kHz audio via inverse STFT — enabling streaming synthesis. Two discriminators (Multi-Period Discriminator and Multi-Scale STFT discriminator) drive reconstruction quality.
A notable design choice is treating FSQ token embeddings as continuous vectors for the flow-matching denoising objective. Rather than predicting one-hot discrete logits or mel spectrograms, the model denoises continuous representations of the FSQ codebook entries. This unification of discrete and continuous feature modeling avoids the instability of low-frame-rate mel spectrograms and simplifies downstream training without requiring a separate discrete language model head.
The full system is trained end-to-end on 60,000 hours of diverse English podcast audio, with speaker prompts drawn from the same speaker and paragraph as the target utterance to maximise speaker similarity.
Key Results
Codec ablations (Table 4) show that CodecB at 12.5 Hz achieves SPK-SIM of 0.916 and WER of 2.74, outperforming mel-spectrogram vocoders at 16 Hz (SIM 0.904, WER 3.01) and approaching performance of mel vocoders at 25 Hz (SIM 0.937, WER 2.39). This demonstrates that FSQ-based discrete tokens can match or exceed STFT-based reconstruction at rates where the stationarity assumption underlying STFT begins to break down.
For diversity, Dragon-FM variants achieve Fréchet Audio Distance (FAD) scores of 1.9–2.6 across chunk size and flow-matching step configurations (Table 2), with shorter chunks (1s) and more FM steps generally yielding lower FAD. The model achieves a Total Number of Function Evaluations (TNFE) of 2–4 for a 2-second segment, versus 150 for VALL-E and 32 for E2 TTS (Table 3), demonstrating substantially lower computational cost per second of audio in the short-segment regime.
Note
Evaluations are conducted on an internal podcast dataset and compared against internal mel-spectrogram vocoder baselines. No comparison against published zero-shot TTS systems (e.g., VALL-E 2, E2 TTS, NaturalSpeech 3) on a standard held-out benchmark is included. Diversity metrics (FAD) are used as the primary quality proxy; MOS listening tests are absent.
Novelty Assessment
The chunk-wise AR + within-chunk flow-matching design is the paper’s genuinely novel contribution. The core insight — that autoregressive generation across fixed-length blocks can be composed with parallel denoising inside each block — is architecturally clean and addresses a real limitation of both paradigms. The use of continuous FSQ embeddings as the denoising target, effectively predicting discrete tokens via continuous flow, is a novel and practically motivated design choice that simplifies training compared to cascaded AR-then-diffusion pipelines.
The codec itself is an engineering contribution rather than an architectural novelty: FSQ-based codecs at compact frame rates have appeared in prior work (e.g., Parker et al. 2024), and the asymmetric encoder/decoder design is a known pattern. The use of mean flow for step reduction is borrowed directly from Geng et al. 2025. The evaluation is limited to internal data and ablations; the absence of MOS comparisons against published baselines on shared benchmarks makes it difficult to situate quality claims precisely. The system is also Microsoft-internal, which limits reproducibility.
Field Significance
Moderate — Dragon-FM advances the hybrid AR-diffusion design space for TTS by demonstrating that chunk-wise autoregressive generation and within-chunk flow-matching can coexist in a single jointly-trained model, simultaneously achieving KV-cache compatibility and bidirectional chunk context. The unification of discrete token prediction and continuous flow-matching objectives is a contribution with potential relevance beyond speech. The paper’s impact is limited by the absence of MOS evaluations and publicly reproducible comparisons against peer systems.
Claims
- Composing autoregressive generation across fixed-length token blocks with parallel flow-matching denoising within each block can simultaneously provide KV-cache efficiency and bidirectional contextual refinement. (§3.1, Table 3)
- Neural codecs using finite scalar quantisation can preserve speaker similarity and intelligibility at frame rates (12.5 Hz) where STFT-based vocoders suffer significant quality degradation. (§4.3.3, Table 4)
- Continuous denoising models can implicitly classify discrete token targets through appropriate embedding design, without requiring a separate discrete language model head. (§3.1)
- Reducing the token frame rate is a more tractable path to low-latency hybrid AR-diffusion TTS than increasing diffusion step efficiency alone, given the quadratic scaling of self-attention with sequence length. (§3.1, §4.3.2)
Limitations and Open Questions
Warning
No MOS or SMOS listening test results are reported. All quality comparisons use SPK-SIM, WER, and FAD on an internal podcast dataset. The absence of subjective evaluation and fair comparison against published baselines (VALL-E 2, E2 TTS, NaturalSpeech 3) on a public benchmark makes it impossible to independently verify naturalness claims.
The model is trained and evaluated on English podcast data only. Generalisation to other languages, controlled studio-quality TTS, and expressive speech domains is untested. The podcast use-case naturally emphasises diversity and disfluency tolerance over precise prosody control, so the evaluation protocol may not transfer to production TTS settings.
The codec and acoustic model are proprietary (Microsoft internal), with no public code or demo reported. Reproducibility relies entirely on the architectural description in the paper.
Mean flow optimisation for step reduction is borrowed from concurrent work (Geng et al. 2025); the sensitivity of Dragon-FM quality to NFE count at scale is not fully characterised — ablations cover only 2, 4, 6, 12, and 24 steps.
Wiki Connections
- flow-matching — Dragon-FM uses flow-matching as the within-chunk denoising objective, applying mean flow for step reduction.
- autoregressive-codec-tts — the chunk-wise AR backbone is the global coherence mechanism; the design explicitly positions itself relative to VALL-E-family discrete AR models.
- neural-codec — a custom FSQ-based 48 kHz codec at 12.5 Hz is a core contribution; comparison to mel-spectrogram baselines at varying frame rates.
- zero-shot-tts — the system generates speech conditioned on a speaker prompt without per-speaker fine-tuning, evaluated on diverse podcast speakers.
- streaming-tts — the causal codec decoder and chunk-wise AR design approach streaming behaviour; first-byte latency with 2 NFE is highlighted as a system advantage.
- DiTAR — related hybrid autoregressive-diffusion approach for speech generation, cited as prior work.
- FELLE — token-wise coarse-to-fine flow matching for AR speech synthesis, also from the Microsoft group.
- PALLE — pseudo-autoregressive codec language models for zero-shot TTS, related in-corpus reference.