arXiv · 2026 · Preprint
Tianhui Su et al. (Universiti Sains Malaysia / Avignon University / Le Mans University) · → Paper · Demo: ✗ · Code: ✗
A non-autoregressive end-to-end streaming TTS system that replaces continuous Mel-spectrogram regression and neural vocoders with depth-wise sequential decoding over 32 Mimi codec RVQ layers, achieving 303× real-time synthesis and 48.99 ms average time-to-first-byte.
Problem
Conventional two-stage TTS pipelines — a non-autoregressive acoustic model producing Mel-spectrograms followed by a neural vocoder — face two interlocking bottlenecks for streaming deployment. The vocoder requires dense transposed convolution over overlapping frames to reconstruct phase information, which is incompatible with strict chunk-level streaming. Simultaneously, MSE-based continuous regression forces the model toward the statistical mean of a multi-modal acoustic distribution, producing spectral over-smoothing: high-frequency textures and transient phonetic details are lost.
Autoregressive codec language models such as VALL-E eliminate the vocoder but introduce prohibitive temporal latency by flattening multi-layer RVQ codes into a single sequential prediction chain. Masked iterative models still require multiple refinement passes. A fully feed-forward path from text to discrete codec tokens — without temporal autoregression or iterative sampling — was not available in a form suitable for real-time single-speaker streaming.
Method
The architecture combines a modified FastSpeech 2 non-autoregressive backbone with the pre-trained Mimi neural audio codec, which operates at 12.5 Hz using 32 RVQ codebooks of 2048 codes each. Three modules compose the pipeline.
The first module is the phoneme encoder and variance adaptor, which follows FastSpeech 2 closely. Feed-forward transformer blocks build phoneme representations; a variance adaptor predicts log-duration, log-F0, and frame-wise energy via 1-D convolutional sub-networks. A length regulator expands the hidden sequence to match the 12.5 Hz Mimi frame rate. A dummy token mechanism injects placeholder frames for sub-frame transient phonemes (those spanning fewer than 80 ms), preventing alignment collapse in the length regulator.
The second module is the depth-wise cascaded discrete decoder. Rather than predicting all 32 quantizer layers in parallel — which collapses because coarse semantic and fine acoustic layers are interdependent — the decoder proceeds sequentially across quantization depth within each individual temporal frame. At frame t, the conditioned state for the i-th quantizer accumulates learned embeddings of all previously predicted layers: each layer’s output is embedded and summed into the running hidden state before the next softmax classification. Crucially, this recurrence runs within a single frame’s depth dimension, not across the time axis, so it introduces no temporal autoregressive delay.
The third module is the frozen pre-trained Mimi neural decoder, which reconstructs 24 kHz waveforms directly from the predicted discrete indices.
Training uses a hybrid multi-task loss: cross-entropy over all 32 codebook heads with staged loss weighting (1.0 for layers 1–4, 0.5 for 5–16, 0.1 for 17–32), MSE for duration, pitch, and energy predictors, and an auxiliary Mel-spectrogram L1 reconstruction loss (weight 10.0) via a parallel PostNet branch that is discarded at inference. Mel features extracted at a 10 ms hop are average-pooled by a factor of 8 to synchronise with the 80 ms Mimi hop. The English model trains for 200,000 steps and the Malay model for 90,000 steps, both on a single RTX 4090.
Key Results
On inference speed (Table 7), the proposed system achieves an RTF of approximately 0.0033 (303× real-time) on LJSpeech, versus roughly 0.020 for VITS and 0.025 for FS2+HiFi-GAN. Against the slowest cascaded baseline (FS2+Parallel WaveGAN at RTF ~0.035), the speedup reaches 10.6×. Average time-to-first-byte is 48.99 ms, well below the 200 ms perceptual threshold for conversational systems.
On acoustic fidelity (Table 4), the proposed system records an MCD of 10.20 dB versus 7.31 dB for VITS and 8.24 dB for FS2+HiFi-GAN — a notable quality cost from aggressive quantization. The Voiced/Unvoiced Error of 2.67% outperforms FS2+HiFi-GAN (3.62%) and approximately matches VITS (2.82%), indicating the depth-wise conditioning preserves fundamental voicing well. WER on LJSpeech (Table 5) is 8.89% versus 1.64% for VITS and 5.19% for FS2+HiFi-GAN.
The English MOS (2.51 ± 0.11) falls below the continuous baseline (2.81 ± 0.13 for FS2+Parallel WaveGAN), but the Malay MOS (4.31 ± 0.11) approaches ground truth (4.65 ± 0.08) with no baseline for comparison. The listener pools differ — non-native English listeners versus native Malay speakers — limiting direct cross-language conclusions.
Ablation results (Tables 9–10) confirm two design choices: (1) using 32 codebooks rather than 16 yields lower WER (8.89% vs. 9.07%) and MCD (10.20 vs. 10.38 dB); (2) depth-wise sequential decoding versus naive parallel prediction reduces WER from 14.37% to 8.89% and MCD from 12.49 to 10.20 dB, validating that inter-layer conditioning is the key mechanism. Subword aggregation via byte-pair encoding caused catastrophic output failure, establishing phoneme-level granularity as a hard constraint.
Novelty Assessment
The contribution is primarily an engineering integration. FastSpeech 2 and Mimi are existing components; the genuinely novel element is the depth-wise sequential decoding strategy, which conditions each RVQ layer on the accumulated embeddings of preceding layers within a single frame. This is a focused, well-motivated solution to a real problem — naive parallel prediction of deep RVQ codes collapses — and the ablation validates it cleanly. The codec choice at 12.5 Hz is also architecturally motivated: the 8× lower frame rate versus standard 100 Hz Mel-spectrograms substantially reduces transformer attention cost, enabling the backbone to remain lightweight enough for streaming.
The paper is honest about trade-offs: the English MOS gap relative to the vocoder baseline (2.51 vs. 2.81) is acknowledged and attributed to 12.5 Hz quantization rather than masked. The evaluation is limited by a small listener pool (11 people), single-speaker scope, and the absence of a direct streaming baseline comparison at the protocol level. The Malay results are promising but lack a baseline system.
Field Significance
Moderate — this paper demonstrates that non-autoregressive discrete codec decoding is viable for ultra-low latency streaming TTS without temporal autoregression or iterative refinement passes. The depth-wise sequential conditioning mechanism addresses a concrete structural failure of naive parallel RVQ prediction and is directly reproducible. The work is incremental relative to the broader shift toward codec-based TTS pipelines, but the explicit latency characterisation (48.99 ms TTFB, RTF 0.0033) and the ablation isolating the conditioning mechanism provide useful engineering reference points for practitioners targeting conversational streaming applications.
Claims
- Replacing continuous Mel-spectrogram generation with discrete neural codec decoding can eliminate the neural vocoder bottleneck and achieve sub-100 ms time-to-first-byte in non-autoregressive TTS. (§4.2.4)
- Naive parallel prediction of multi-layer RVQ codes in non-autoregressive TTS degrades intelligibility; sequential conditioning across quantizer depth within individual frames recovers acoustic quality without introducing temporal autoregressive delay. (§3.3.2, §4.4.2, Table 10)
- The quality-latency trade-off for aggressive low-frame-rate codec quantization (12.5 Hz) is language-dependent: phonetically transparent languages tolerate the compression better than deep-orthography languages with complex prosodic variation. (§4.3, Table 8)
- Ultra-low frame-rate codecs reduce quadratic transformer attention cost and are a practical enabler of lightweight non-autoregressive streaming architectures. (§3.1, §4.2.2)
- Subword-level input aggregation is incompatible with non-autoregressive depth-wise codec decoding, as it breaks the one-to-one alignment required between linguistic units and codec frames. (§4.4.3)
Limitations and Open Questions
Warning
English MOS (2.51) falls below the continuous FS2+vocoder baseline (2.81), and the Malay evaluation uses only 11 listeners with no baseline system for comparison. The MOS gap and the disparate listener pools mean the naturalness claims should be treated cautiously.
The system is single-speaker; multi-speaker and zero-shot capabilities are stated as future work but are absent here, which is a significant gap for practical deployment. Code and models are not publicly available — access requires contacting the corresponding author.
No streaming baseline is evaluated under identical real-time streaming conditions: because neural vocoders require ahead-of-time frame buffering, the paper cannot report a head-to-head millisecond TTFB comparison. The 10.6× speedup is a system-level RTF figure measured offline. The authors note that high prosodic variance can trigger minor quantization artifacts, and propose lightweight flow-matching within the discrete latent space as a future refinement direction.
Wiki Connections
This paper develops a streaming non-autoregressive path within streaming-tts that bypasses the vocoder bottleneck by operating directly in the neural-codec latent space. The backbone belongs to the transformer-enc-dec-tts family (FastSpeech 2), contrasting architecturally with the autoregressive-codec-tts approach exemplified by VALL-E (VALL-E), which achieves zero-shot quality at the cost of temporal latency. The companion paper 2512.00937 on Arabic TTS with FastPitch addresses spectral over-smoothing in the non-autoregressive continuous setting, the same degradation this paper targets through discrete classification.