arXiv · 2025 · Preprint
Chun Yat Wu et al. (CUHK; Huawei Hong Kong Research Center) · → Paper · Demo: ✓ · Code: ?
CLEAR replaces discrete audio tokens in autoregressive TTS with continuous VAE latents decoded via a per-token MLP rectified flow head, achieving state-of-the-art WER and lower inference latency than both discrete-token and continuous mel-spectrogram AR baselines.
Problem
Autoregressive TTS systems that operate on discrete neural codec tokens face two fundamental issues: lossy tokenization limits reconstruction fidelity, and the long token sequences required at practical bitrates drive up inference latency and complicate sequence modeling. Cascaded two-stage systems (language model + diffusion refinement, as in CosyVoice and Seed-TTS) partially address the quality ceiling but introduce error accumulation and increased computational cost. Alternative continuous AR approaches like MELLE use Gaussian distribution assumptions that are too simple to capture complex speech patterns, while those adopting Diffusion Transformer (DiT) backbones (ARDiT, DiTAR) incur heavy computational overhead and cannot produce streaming output efficiently because the DiT must wait for all AR hidden states before denoising.
Method
CLEAR formulates zero-shot TTS as next-token prediction over continuous latent vectors produced by a custom waveform VAE (wav-VAE). The system has three components trained end-to-end in a single stage:
Wav-VAE. A convolutional encoder-decoder with a downsampling ratio of 2048 compresses 16 kHz audio into roughly 7.8 latent frames per second (versus 100 frames/s for mel spectrograms). To maintain reconstruction quality under this aggressive compression, parameter-free shortcut connections perform space-to-channel reshaping with channel averaging in each downsampling block and the reverse in each upsampling block. The VAE is trained with a multi-task criterion combining multi-resolution STFT loss, feature matching loss from a multi-scale STFT discriminator, adversarial loss, and KL divergence. A causal decoder variant enables streaming synthesis.
Autoregressive language model. A unidirectional Transformer decoder (Pre-Norm, RMSNorm, RoPE, SwiGLU) processes the concatenated phoneme and audio latent sequence [S, x, T, y]. At each step k it produces a conditioning vector h_k. CLEAR-Base uses 24 layers with embedding dim 1024 (403M params); CLEAR-Large scales to embedding dim 1280 (630M params).
MLP rectified flow head. A lightweight MLP (6 ResBlock1D layers, 1024 dims; 36M params for Base) receives h_k and denoises a sample drawn from a standard normal to produce the next continuous latent y_k. The rectified flow loss defines straight-line trajectories from noise to data; an auxiliary directional loss based on cosine similarity between predicted and ground-truth vector fields accelerates convergence. Importantly, the MLP operates independently on each conditioning vector, so denoising can start as soon as the first AR hidden state is available — enabling streaming without the full-sequence wait imposed by DiT-based heads. Only 10 denoising function evaluations (NFEs) are used at inference.
Classifier-free guidance is applied by randomly zeroing text embeddings with probability 20% during training, allowing guided inference with scale w=2.5.
For zero-shot synthesis, a short audio prompt and its transcription are prepended as context: [S, x_prompt, x_target, T, y_prompt]; the model then autoregressively completes y_target. Training data is LibriHeavy (50,000h). The VAE is pre-trained on LibriTTS (585h).
Key Results
On LibriSpeech-PC test-clean Subset-B (1,127 samples, F5-TTS evaluation protocol):
- CLEAR-Large: WER 1.88%, SPK-SIM 0.59, UTMOS 4.22
- CLEAR-Base: WER 2.21%, SPK-SIM 0.59, UTMOS 4.22
- Compared to DiTAR (600M, Emilia-100k): CLEAR-Large achieves 0.51% absolute WER reduction (21.3% relative) at lower parameter count and with far fewer AR decoding steps
On LibriSpeech test-clean Subset-A (40 samples, NaturalSpeech3 protocol):
- CLEAR-Large: WER 1.74%, SPK-SIM 0.56, UTMOS 4.26
- CLEAR-Base: WER 1.83%, SPK-SIM 0.55, UTMOS 4.21
- Outperforms VALL-E (WER 6.11%), MegaTTS (WER 2.32%), NaturalSpeech2 (WER 1.94%)
Inference efficiency (Table 2, 10-second generation on single GPU):
- CLEAR-Base: RTF 0.18, avg. 78 AR decoding steps (vs. VALL-E 750 steps / RTF 1.03; MELLE 620 steps / RTF 0.55; F5-TTS RTF 0.31)
- CLEAR-Large: RTF 0.29
Subjective evaluation on Subset-B vs. F5-TTS and CosyVoice-2:
- CLEAR-Base: N-MOS 4.09, Q-MOS 4.14, S-MOS 4.02, CMOS +0.04 (outperforms ground truth on N-MOS and Q-MOS)
Streaming synthesis (Table 4, chunk size 4): first-frame latency 96ms, WER 2.34%, UTMOS 4.27 — largely matching non-streaming performance.
Comparisons are partially confounded by different training data: CLEAR uses Libri-50k whereas DiTAR and F5-TTS use Emilia-100k; direct WER comparisons must be read with this in mind.
Novelty Assessment
The core novelty is architectural: coupling a high-compression-ratio VAE (not a standard codec) with an MLP-based rectified flow head that decodes each AR hidden state independently. This is distinct from DiTAR/FELLE which use transformer-based diffusion backbones requiring full context before denoising, and from MELLE which models mel spectrograms with simple Gaussian assumptions. The single-stage joint training (LM + flow head, no separate pre-training of the generative model on top of the LM) is a meaningful simplification over two-stage cascaded systems. The shortcut-connection VAE achieving stable training at 2048x downsampling is a secondary but non-trivial contribution. The design is primarily architectural and engineering in character: no new training paradigm, no new data, and the rectified flow formulation itself is transferred from image generation.
Field Significance
Tip
High — CLEAR demonstrates that a high-compression-ratio VAE combined with a per-token MLP flow head can match or exceed the quality of discrete-codec AR systems while significantly reducing inference latency, offering a concrete alternative architecture for streaming zero-shot TTS. The single-stage training design and the independence of the MLP denoising step from full AR context are engineering decisions that may influence subsequent continuous AR work.
Claims
- Replacing discrete neural codec tokens with continuous VAE latents in an autoregressive TTS framework reduces the number of AR decoding steps and lowers inference RTF without sacrificing speech quality. (§5.1, Table 2)
- A per-token MLP rectified flow head that operates independently on each AR hidden state enables streaming speech synthesis with latency competitive with non-streaming systems. (§5.3, Table 4)
- High-compression-ratio VAE latents (2048x downsampling) can achieve reconstruction quality comparable to ground truth on standard benchmarks while producing sequences short enough to benefit AR sequence modeling. (§3.4, Appendix C.1)
- Continuous AR TTS systems trained with a joint language model and flow-matching head in a single stage can match or exceed the WER of two-stage cascaded discrete AR systems trained on comparable data. (§5.1, Table 1)
- Objective speaker similarity metrics underestimate the perceptual voice fidelity of AR systems that rely entirely on in-context learning from an audio prompt, as shown by divergence between WavLM-TDNN scores and subjective S-MOS ratings. (§5.2, Table 3, Appendix D.2)
Limitations and Open Questions
The paper explicitly acknowledges: (1) objective speaker similarity (WavLM-TDNN) lags behind models that inject speaker embeddings directly into the generator (e.g., CosyVoice, NaturalSpeech3 with FACodec); CLEAR relies entirely on in-context learning from the audio prompt with no explicit speaker conditioning, which is a design choice that limits speaker fidelity under high compression. (2) Evaluation is English-only; multilingual generalization is untested. (3) Sequential single-token AR generation; multi-token prediction could improve throughput further. Additionally, training data for VAE (LibriTTS, 585h) is much smaller than TTS training data (LibriHeavy, 50kh), and the interaction between VAE compression ratio and downstream TTS quality could be investigated more rigorously in full-scale settings.
Wiki Connections
This paper is most directly relevant to autoregressive-codec-tts (replacing discrete tokens with continuous latents), flow-matching (MLP rectified flow head for per-token distribution modeling), zero-shot-tts (in-context learning from audio prompt), and streaming-tts (causal VAE + per-token diffusion enabling 96ms first-frame latency).
In-corpus references: FELLE — a competing continuous AR approach using coarse-to-fine flow matching on mel spectrograms, cited as a two-stage system with cumbersome training that CLEAR explicitly improves upon; MELLE — continuous AR baseline using Gaussian mel-spectrogram modeling; ARDiT — DiT-based continuous AR system cited as too computationally heavy for streaming; DiTAR — the primary competitor on both evaluation subsets; F5-TTS and CosyVoice 2 — baselines in the subjective evaluation; VALL-E — the original discrete codec AR baseline.