Interspeech · 2025 · Conference
Hyungchan Yoon et al. (LG AI Research) · → Paper · Demo: ? · Code: ?
APTTS introduces adversarial post-training (AP) on a latent flow matching TTS framework that reduces inference to 4 ODE steps while achieving state-of-the-art intelligibility (WER 1.73%) and real-time synthesis (RTF 0.052) on zero-shot TTS, using only ~900 hours of training data.
Problem
Flow matching TTS models achieve high-quality synthesis but require many ODE solver steps (typically 20–64), making real-time deployment difficult. Existing acceleration methods (consistency distillation, reflow, distribution matching distillation) address mel-spectrogram-domain models. Prior work has not explored latent flow matching for TTS — operating over a learned continuous latent space rather than fixed mel-spectrograms — nor applied adversarial post-training to the intermediate representation domain of TTS. The training-inference mismatch in few-step ODE solvers also degrades quality.
Method
APTTS is trained in three stages.
Stage 1 — VAE pre-training: A VAE based on PeriodVITS encodes mel-spectrograms into a 64-dimensional continuous latent space. Explicit F0 and voicing flags are extracted separately and fed to the VAE decoder to improve pitch reconstruction. A multi-period discriminator (MPD) and multi-resolution discriminator (MRD) with truncated pointwise relativistic loss improve audio fidelity.
Stage 2 — Flow Matching Decoder + Pitch Predictor: A Diffusion Transformer (DiT) with 24 layers, 16 attention heads, 1024/2048 dimensions, and RoPE embeddings is trained as the FM decoder. A transformer-based text encoder with SC-CNN style injection and a NeXt-TDNN style encoder extract conditioning from text and speech prompt. Zero-shot TTS is framed as speech infilling: a binary mask covers 70–100% of the latent, and the decoder reconstructs masked frames. Dual classifier-free guidance (CFG) drops context embeddings and prompt independently at 5/10/10% rates during training.
Stage 3 — Adversarial Post-training (AP): The trained FM decoder is repurposed as a 4-step generator using a fixed Euler ODE with time steps [0, 0.08, 0.29, 0.62] (emphasizing earlier steps). A joint conditional-unconditional (JCU) discriminator, conditioned on a time-averaged prompt embedding and using dilated convolutions, provides adversarial and feature-matching losses. The AP loss combines L1 reconstruction loss, JCU adversarial loss, and feature matching loss, applied only to masked frames. A hybrid CFG technique addresses the training-inference mismatch: during AP the decoder is always fully conditioned, while at inference the unconditional terms are derived from the pre-AP base model rather than the few-step generator, enabling stable guidance without the error accumulation from a fixed large guidance scale.
The system is trained on ~918 hours from LibriTTS, VCTK, Hi-Fi TTS, and Expresso. Inference uses 22050 Hz audio, 80-band mel-spectrograms.
Key Results
On 39-sample LibriSpeech test-clean evaluation against public models: APTTS (4 steps) achieves MOS 3.58, SMOS 3.65, WER 2.08% vs. F5-TTS (32 steps) at MOS 3.66, SMOS 3.60, WER 2.52%. RTF is 0.052 vs. F5-TTS’s 0.217, a 4× speedup. On the full LibriSpeech test-clean (4–10s), APTTS achieves WER 1.73% and CER 0.40% — the lowest of all baselines including models trained on 60–100K hours (VALL-E at WER 5.9%, Voicebox at WER 1.9%, FlashSpeech at WER 2.7%). SIM-o is 0.55, below Voicebox (0.66) and F5-TTS (0.64), attributed to the 100× smaller training set. AP reduces inference by more than 5× vs. the 32-step base model with minimal quality loss.
Scalability is validated on Matcha-TTS (VCTK, multi-speaker): AP at 2 steps surpasses MAT-10 in both MOS (3.96 vs. 3.71) and WER (4.8% vs. 6.81%), confirming the strategy generalizes to mel-spectrogram-domain FM models.
Novelty Assessment
The primary architectural novelty is the combination of latent flow matching with adversarial post-training for TTS, specifically the prompt-conditioned JCU discriminator applied in the latent domain and the hybrid CFG technique. The APTTS system achieves competitive zero-shot quality with dramatically less training data (0.9K vs. 60–100K hours for comparisons), which is a practically important result. The AP strategy’s generalization to Matcha-TTS is a meaningful scalability demonstration. The distillation-like AP procedure is analogous to adversarial score distillation in image generation, adapted here with novel masking and discriminator design choices for TTS.
Field Significance
Moderate — APTTS introduces adversarial post-training as a post-hoc acceleration technique for flow matching TTS, applicable to both latent and mel-spectrogram-domain models without retraining from scratch. The demonstration that competitive zero-shot intelligibility is achievable with roughly 1K hours — against systems trained on 60–100K hours — provides useful evidence that training data scale is not the only lever for FM-based zero-shot TTS quality. The contribution is primarily architectural; the components (VAE latent spaces, adversarial training, CFG) are individually established, but their combination in the latent domain for few-step TTS is new.
Claims
- Adversarial post-training applied to a pre-trained flow matching decoder can reduce the required ODE sampling steps by 5–8x while maintaining competitive naturalness and improving intelligibility relative to the full-step baseline. (§3.3, Table 1, Table 2)
- Latent flow matching TTS trained on approximately 1K hours can match or surpass the intelligibility of zero-shot systems trained on 60–100K hours, though speaker similarity remains lower at this data scale. (§4.2, Table 2)
- Framing zero-shot TTS as a speech infilling task within a latent space enables prompt-conditioned generation without requiring a separate speaker encoder or d-vector lookup. (§3.2)
- Hybrid classifier-free guidance — decoupling the unconditional branch to a frozen base model checkpoint — reduces error accumulation in few-step generators compared to applying a fixed large guidance scale during adversarial post-training. (§3.3)
- Adversarial post-training generalises across flow matching representation domains: the same post-training procedure improves both latent-domain (APTTS) and mel-spectrogram-domain (Matcha-TTS) models. (§4.3, Table 3)
Limitations and Open Questions
SIM-o scores lag behind large-scale trained models, suggesting prompt fidelity would improve with more data. The hybrid CFG requires inference calls to both the post-trained and base FM models, adding complexity. The fixed ODE time steps (empirically chosen) may not be optimal across all input lengths. The system is English-only; multilingual extension is unexplored.
Wiki Connections
APTTS directly advances flow-matching by introducing latent-domain FM and adversarial acceleration, and zero-shot-tts by demonstrating competitive zero-shot quality with only 918 hours. The VAE component connects to disentanglement (pitch separation). The adversarial training at inference time connects to patterns from gan-vocoder literature.
Related few-step FM acceleration work from Interspeech 2025: RapFlow-TTS takes a complementary approach via consistency flow matching on straight ODE trajectories, achieving 2-NFE synthesis on mel-spectrograms where APTTS uses 4-step adversarial post-training in latent space. Both papers converge on the conclusion that flow-matching’s straight trajectories enable aggressive inference step reduction, but via different mechanisms: consistency constraints (RapFlow-TTS) vs. adversarial discrimination (APTTS).