arXiv · 2026 · Preprint
Detai Xin et al. (Meituan) · → Paper · Demo: ? · Code: ✓
LongCat-AudioDiT is a non-autoregressive, flow-matching TTS system from Meituan that replaces mel-spectrogram intermediates with a custom waveform VAE latent space, achieving state-of-the-art zero-shot speaker similarity on the Seed benchmark with a 3.5B parameter model trained entirely without human-annotated data.
Problem
Diffusion-based non-autoregressive TTS systems have historically relied on mel-spectrograms as an intermediate acoustic representation, requiring a separate neural vocoder to reconstruct waveforms. This two-stage cascade introduces compounding errors: inaccuracies in mel prediction are amplified by the vocoder, and the fine-grained high-frequency detail most critical for speaker identity is easily lost. While hybrid autoregressive+diffusion systems have come to dominate benchmarks, the design space for high-performance pure diffusion TTS remained underexplored due to the non-disclosure of technically capable systems like Seed-DiT. A secondary problem is a training-inference mismatch in flow-matching TTS that had gone unaddressed in prior work: during inference, the noisy latent corresponding to the conditioning prompt drifts off its theoretical trajectory because the flow-matching objective never supervises that region.
Method
LongCat-AudioDiT is built from two components: a waveform variational autoencoder (Wav-VAE) and a flow-matching Diffusion Transformer (DiT) backbone.
Wav-VAE. The Wav-VAE is a fully convolutional encoder-decoder that compresses raw 24 kHz waveforms into a compact continuous latent at 11.72 Hz frame rate with 64-dimensional latents. The encoder uses cascaded Oobleck blocks with dilated residual units and Snake activations, and incorporates parameter-free shortcut paths (space-to-channel reshaping) to stabilize training under aggressive downsampling. The bottleneck applies the reparameterization trick to sample z from predicted mean and log-variance. The decoder mirrors the encoder in reverse with channel-to-space shortcuts. Training uses a combined loss: multi-resolution STFT loss, multi-scale mel loss, time-domain L1 loss, KL divergence regularization, adversarial loss, and feature matching loss against a multi-scale STFT discriminator, with adversarial terms held off during a warmup phase. The Wav-VAE has 157M parameters and was trained on 200K hours of Chinese and English speech.
DiT backbone. The CFM (conditional flow matching) backbone uses the rectified flow formulation, training a velocity field network to transport Gaussian noise to waveform latents. The network follows the DiT architecture with Adaptive Layer Normalization (AdaLN) for timestep injection, QK-Norm for attention stability, and Rotary Positional Embeddings (RoPE) on all attention layers. Cross-attention handles implicit text-to-speech alignment without an explicit duration predictor, following the alignment-free approach of E2-TTS and F5-TTS. Long-skip connections and a global AdaLN formulation (from GenTron) reduce parameter count while preserving quality. Representation Alignment (REPA) with a pretrained mHuBERT model accelerates convergence by grounding DiT layer 8 representations to multilingual speech features.
Text encoder. Rather than training from scratch or using ByT5 (which produces prohibitively long sequences for Chinese), the system uses UMT5-base (supporting 107 languages) as the text encoder. A key innovation is combining both the final hidden state (rich in semantic information) and the raw word embeddings (rich in phonetic/lexical detail) via layer-normalized addition. The combined representation is refined by a lightweight ConvNeXt V2 module. This dual-embedding strategy substantially improves intelligibility compared to using only the final hidden state.
Inference improvements. Two fixes address degraded generation quality. First, the training-inference mismatch fix: during inference, the noisy latent for the conditioning prompt region is overwritten at each step with its ground-truth value under the rectified flow interpolation formula, preventing drift. The unconditional velocity for classifier-free guidance is also computed with the prompt latent fully dropped. Second, Adaptive Projection Guidance (APG) replaces standard CFG by decomposing the guidance residual into components parallel and orthogonal to the conditional prediction, dampening the parallel term (which causes oversaturation) while preserving the orthogonal term. A negative reverse-momentum term is applied to focus guidance on the current update direction.
Two model sizes are released: 1B and 3.5B parameters. The 3.5B variant was trained on 1 million hours of Chinese and English speech.
Key Results
On the Seed benchmark (Table 1), LongCat-AudioDiT-3.5B achieves SIM scores of 0.818 (Seed-ZH), 0.797 (Seed-Hard), and 0.786 (Seed-EN), establishing new state-of-the-art for diffusion-based non-autoregressive models on Seed-ZH and Seed-Hard. It surpasses the previous best diffusion system, Seed-DiT (SIM 0.809 on Seed-ZH), and all open-source baselines including F5-TTS (0.741), MaskGCT (0.774), and ZipVoice (0.751). It also outperforms many AR and hybrid systems including CosyVoice2 (0.748) and even narrows the gap to CosyVoice3.5 (0.797) on Seed-Hard. On Seed-EN, it achieves WER 1.50%, competitive with the best proprietary systems.
The 1B model ablation (Table 4) confirms: (1) fixing the training-inference mismatch raises SIM from 0.769 to 0.812 and UTMOS from 2.83 to 3.16 on Seed-ZH; (2) APG over standard CFG improves UTMOS from 3.06 to 3.16 and DNSMOS from 3.38 to 3.40, with no intelligibility trade-off.
Wav-VAE vs. Mel-VAE ablation (Table 3) shows Wav-VAE outperforms a Mel-VAE baseline on all Seed metrics, with particularly dramatic improvements in SIM (0.812 vs. 0.706 on Seed-ZH), validating the core hypothesis about compounding errors in mel-based pipelines.
The Wav-VAE standalone evaluation (Table 2) shows the 11.72 Hz variant achieves PESQ 3.237 and STOI 0.967 on LibriTTS test-clean, exceeding most discrete codecs at much lower frame rates.
Notably, the paper’s VAE ablation reveals a counterintuitive result: increasing VAE reconstruction fidelity (via higher latent dimensionality or higher frame rate) does not improve downstream TTS quality — a higher-dimensional latent imposes excessive modeling burden on the diffusion backbone.
Novelty Assessment
The primary contributions are the shift to waveform latent space for a flow-matching TTS backbone and the empirical validation of the resulting benefits for zero-shot voice cloning. The Wav-VAE architecture adapts Oobleck building blocks from audio compression. The inference mismatch correction is a genuine and previously overlooked issue in flow-matching TTS that affects Voicebox and F5-TTS style training; the fix is technically simple but its impact is large (SIM gain of ~0.04). APG is imported from the image diffusion literature (Sadat et al., ICLR 2024) and adapted to the flow-matching setting — this is a clean transfer but not an original algorithm. The dual text-embedding approach (final hidden state + raw word embeddings from UMT5) is a practical multilingual design contribution. Scalability to 3.5B on 1M hours of data is an engineering and resource contribution. The VAE–TTS interplay analysis (counterintuitive trade-off between reconstruction fidelity and generation quality) is a useful empirical insight for future system design.
Field Significance
Tip
High — LongCat-AudioDiT demonstrates that pure diffusion-based TTS operating in waveform latent space can match or exceed hybrid AR+NAR systems on the Seed benchmark, a result that challenges the prevailing assumption that hybrid pipelines are necessary to achieve top-tier zero-shot speaker similarity. The training-inference mismatch identification provides an actionable correction applicable to any Voicebox-style flow-matching system, and the VAE-TTS fidelity trade-off analysis offers a counterintuitive empirical principle that can inform latent representation design across the field.
Claims
- Generating speech directly in waveform latent space yields substantially higher zero-shot speaker similarity than equivalent pipelines operating through mel-spectrogram intermediates, primarily because high-frequency acoustic detail critical for voice identity is preserved end-to-end. (§5.3.1, Table 3)
- Voicebox-style flow-matching TTS training introduces a systematic training-inference mismatch in the prompt region, and correcting it by overwriting the noisy prompt latent with its ground-truth interpolated value at each inference step yields large improvements in speaker similarity and naturalness. (§4.3, §5.3.3, Table 4)
- In waveform latent space diffusion TTS, increasing VAE reconstruction fidelity through higher latent dimensionality degrades downstream generation quality rather than improving it, because the diffusion backbone cannot efficiently model the increased complexity. (§5.3.2, Figure 3)
- Decomposing classifier-free guidance into parallel and orthogonal components and dampening the parallel term (Adaptive Projection Guidance) eliminates oversaturation artifacts from high-scale CFG without sacrificing intelligibility or speaker similarity. (§4.4, §5.3.3, Table 4)
- Combining the final hidden state and raw word embeddings from a multilingual language model improves TTS intelligibility over using the final hidden state alone, because the latter abstracts away low-level phonetic cues needed for acoustic mapping. (§4.2)
Limitations and Open Questions
The system is evaluated only on the Seed benchmark (Chinese and English); its performance on other languages supported by UMT5 is not reported. Naturalness is assessed via automated metrics (UTMOS, DNSMOS) rather than human listening tests, limiting direct comparability with systems reporting MOS. The organization is listed as Meituan but this is inferred from author affiliations and team name, not stated in the metadata. Training data is fully proprietary (200K + 1M hours internal), making reproduction difficult. The paper’s ablation leaves open the question of whether the mismatch fix is equally beneficial at 3.5B scale; Table 1 reports only final metrics, not the contribution of individual components at that scale. Future work explicitly mentions RLHF and distillation as next steps, neither of which is explored here.
Wiki Connections
This paper advances the flow-matching and diffusion-tts paradigms by demonstrating that operating in a continuous waveform latent space rather than mel-spectrogram space yields substantial improvements in zero-shot speaker similarity. It connects directly to zero-shot-tts as the primary capability target, using audio prompt masking inherited from Voicebox (not in corpus) and achieving SOTA on the Seed benchmark defined in Seed-TTS.
The alignment-free design builds on the E2-TTS / F5-TTS line (F5-TTS); the paper treats F5-TTS as the primary baseline for NAR flow-matching TTS. It benchmarks against a broad set of in-corpus systems: VALL-E (autoregressive codec baseline), CosyVoice, CosyVoice 2, DiTAR, IndexTTS2, Qwen3-TTS, MOSS-TTS.
The Wav-VAE comparison draws on VibeVoice (the closest prior continuous VAE for TTS) and CLEAR (contemporaneous work exploring continuous latent representations). The counterintuitive VAE–TTS fidelity trade-off is relevant to evaluation-metrics and to anyone using multilingual-tts representations for generative modeling.