Continuous Speech Tokenizer in Text To Speech

NAACL · 2025 · Conference

Yixing Li et al. (The Chinese University of Hong Kong / Tencent) · → Paper · Demo: ✗ · Code: ✓

Cont-SPT replaces residual vector quantization (RVQ) in codec-based TTS with a continuous speech tokenizer that retains the encoder output directly, eliminating quantization information loss especially at high frequencies, and integrating with autoregressive LM generation and flow-matching decoding.

Problem

All leading autoregressive TTS systems (VALL-E, CosyVoice, MusicLM, etc.) rely on discrete speech tokens produced by RVQ codebooks. RVQ discretization introduces information loss: codebook size limits the representable space, and the loss is disproportionately severe at high frequencies (8kHz range). Narrow-band and varied-sampling-rate speech further exacerbates this. The authors ask whether replacing discrete tokens with continuous speech representations can improve TTS quality and robustness.

Method

Continuous Speech Tokenizer (Cont-SPT). The encoder architecture mirrors EnCodec-style encoders operating at 24kHz. Instead of passing encoder outputs through RVQ, Cont-SPT uses a learned embedding module that produces continuous speech token representations directly. These continuous tokens are used as both the input to and generation target of the autoregressive language model.

TTS framework. Following the VALL-E design:

• Text is tokenized and embedded.
• Text embeddings and continuous speech tokens are concatenated as LM input during autoregressive generation.
• The LM predicts continuous speech tokens via MSE loss (L_LM = MSE(O, M) where O is LM output and M is the continuous speech token of the target).
• An audio decoder (codec decoder + OT-CFM flow matching + denoising module) converts predicted continuous tokens to mel-spectrogram and then waveform.

Two-stage training:

1. Pre-training the continuous tokenizer: encoder connected to decoder, trained VAE-style with similarity loss (reconstruction) + CTC loss (ASR consistency).
2. Joint training: decoder frozen, LM and tokenizer trained jointly with different learning rates (tokenizer LR = 0.05 × LM LR).

Transfer function analysis. The information retention of continuous vs. discrete tokenizers is measured as average attention rate (transfer function analogue) across frequency bands:

• 2kHz: continuous 0.94 vs discrete 0.95 (comparable)
• 5kHz: continuous 0.81 vs discrete 0.78
• 8kHz: continuous 0.55 vs discrete 0.34 (large gap)

The continuous tokenizer maintains substantially better high-frequency information.

Sampling rate robustness. Continuous tokenizer is more robust than discrete when window length ratios are modified (simulating mixed narrow/wideband data).

Key Results

On LibriSpeech test-clean (Tables 1 and 2):

Model	WER	SIM	EMoS	STOI	Continuity
VALL-E	12.73%	0.53	0.83	0.20	1.80
MELLE	7.26%	0.67	—	—	—
Cont-SPT	6.59%	0.73	1.32	0.42	3.61

The proposed model outperforms VALL-E on all metrics and MELLE on WER and SIM. The continuity quality score (NISQA-based) nearly doubles VALL-E (3.61 vs 1.80), consistent with the claim that continuous tokens reduce temporal discontinuities. Ablation confirms that both pre-training the tokenizer and joint training are necessary; removing either significantly hurts WER and SIM.

Novelty Assessment

The core idea — removing RVQ from a codec-based TTS pipeline and keeping representations continuous — is simple and clearly articulated. It parallels the concurrent MELLE paper (which removes the tokenizer entirely and predicts mel-spectrograms), though Cont-SPT takes a different route by retaining the tokenizer concept while making it continuous. The frequency-domain analysis of information retention is a useful diagnostic contribution. The overall novelty is moderate: the architecture is largely VALL-E with the RVQ swapped, and the gains are empirically validated on a single benchmark (LibriSpeech). The paper is stronger as a conceptual critique of discrete tokenization than as an architectural advance.

Field Significance

Moderate — This paper provides early empirical evidence that eliminating RVQ discretization from codec-based TTS pipelines can improve intelligibility, speaker similarity, and perceived continuity, opening a fork between “improve quantization” and “abandon it” as competing design strategies. Its contribution is primarily conceptual and engineering: the architecture closely follows VALL-E, and evaluation is limited to a single English benchmark, but the frequency-domain analysis offers a reusable diagnostic lens for comparing tokenizer designs.

Claims

• Continuous speech token representations preserve high-frequency information more effectively than discrete RVQ tokens, with the gap largest in the 8kHz band. (§3.3, Table 3)
• Replacing RVQ with a continuous tokenizer in an autoregressive TTS framework reduces word error rate and improves speaker similarity compared to a discrete-token baseline. (§3.2, Table 1)
• Continuous speech tokenizers exhibit greater robustness to sampling rate variation than discrete tokenizers, particularly when window length ratios are modified. (§3.4)
• Two-stage training (tokenizer pre-training followed by joint LM-tokenizer training) is necessary for continuous-token TTS; removing either stage degrades both intelligibility and speaker similarity. (§2.4.3, Appendix A.1, Table 4)
• Autoregressive generation over continuous speech tokens reduces temporal discontinuity artefacts relative to RVQ-based generation, as measured by NISQA continuity scores. (§3.2, Table 2)

Limitations and Open Questions

• Evaluation is English-only on LibriSpeech test-clean, a relatively clean/read-speech benchmark; generalization to spontaneous or multilingual speech is untested.
• Model size is not reported, making cost comparisons with VALL-E/MELLE impossible.
• The continuous LM output requires MSE regression rather than cross-entropy classification; training stability and mode averaging in the continuous space are not thoroughly discussed.
• No subjective MOS evaluation from human raters; EMoS (automated MOS) is used instead.
• The relationship to MELLE (which also drops discrete tokens, via mel-spectrogram prediction) deserves more analysis — they represent two distinct approaches to the same problem.
• How the continuous tokenizer interacts with multi-speaker generalization is not tested.

Wiki Connections

This paper challenges a core assumption of neural-codec and autoregressive-codec-tts: that RVQ discretization is necessary or desirable. It extends the design space of autoregressive-codec-tts toward continuous-domain LM generation. The frequency-domain information retention analysis informs the evaluation-metrics discussion around objective speech quality. The use of CTC loss during tokenizer pre-training draws on self-supervised-speech ideas for representation quality.

This paper directly benchmarks against VALL-E on LibriSpeech test-clean, showing WER improvements from 12.73% to 6.59% and SPK-SIM from 0.53 to 0.73. It is complementary to Discrete Representation Inconsistency (DRI), which takes the opposite approach — improving RVQ consistency rather than removing discretization — and achieves WER 3.51% (consistency-trained VALL-E, 960h). The two papers together define a key fork in codec-based TTS research: improve quantization vs. abandon it.