arXiv · 2025 · Preprint
Zheng et al. · → Paper · Demo: ✓ · Code: ✓
VARSTok introduces a variable-frame-rate speech tokenizer that uses temporal-aware density-peak clustering and implicit duration coding to produce compact token sequences fully compatible with autoregressive speech language models, achieving better reconstruction quality than a 40 Hz fixed-rate baseline while using up to 23% fewer tokens.
Problem
Fixed-rate speech tokenizers allocate a uniform number of tokens per second regardless of how much information a given segment carries. Silence and steady vowels are oversampled, while rapid articulatory transitions and expressive prosody receive no preferential treatment. This mismatch between token density and signal information density leads to inefficient compression and, for downstream speech language models, hinders learning of natural prosody and rhythm. Prior variable-rate approaches in the semantic domain (SD-HuBERT, syllable-aligned SSL representations) discard fine-grained acoustic information and cannot reconstruct waveforms. The closest acoustic work (TFC) discretises only among three predefined frame rates and omits duration modeling, limiting its utility for autoregressive speech LMs.
Method
VARSTok comprises four components: a speech encoder, a temporal-aware density-peak clustering module, a single-codebook VQ module, and a speech decoder. The encoder is a 1D convolutional stack followed by a two-layer bidirectional LSTM that produces frame-level embeddings at a base frame rate. The clustering module then segments these embeddings into variable-length temporal units using a greedy density-peak algorithm: each frame receives a peak score equal to the product of its local embedding density and its distance from any denser frame, and high-scoring frames seed new clusters. Clusters expand bidirectionally from their seed, adding adjacent frames that satisfy a cosine-similarity threshold relative to the seed while enforcing strict temporal contiguity. A maximum span constraint (S_max) bounds cluster length. Each cluster is summarised by mean-pooling its constituent frame embeddings, then quantised with a single codebook of K = 4096 entries via L2-nearest-neighbour lookup and exponential moving average updates.
The key innovation for downstream compatibility is implicit duration coding. Rather than pairing each token with a separate duration integer, VARSTok encodes both content and duration into a single expanded token ID: D = (d - 1) * K + k, where k is the VQ codebook index and d is the cluster span in frames. This expands the effective vocabulary to K * S_max entries. During decoding the original VQ index and duration are recovered by simple integer division and modulo operations, allowing the speech LM to predict one unified token per cluster rather than a content token plus a duration token. The decoder applies attention layers, ConvNextV2 blocks, and an inverse Fourier transform upsampling to reconstruct the waveform from expanded quantised embeddings.
VARSTok is initialised from the pretrained WavTokenizer (75 Hz) checkpoint and trained on the 585-hour LibriTTS corpus with a combined mel-spectrogram reconstruction loss, commitment loss, adversarial hinge loss, and feature-matching loss, matching the WavTokenizer training recipe. No auxiliary predictors or additional codebooks are added.
Key Results
On LibriTTS test-clean speech reconstruction, VARSTok at 30.95 Hz (tau=0.7, S_max=4) achieves UTMOS 3.8949 and PESQ 1.7095, outperforming the 40 Hz WavTokenizer (UTMOS 3.6107, PESQ 1.7075) with 23% fewer tokens. The more aggressive VARSTok configuration at 36.81 Hz (tau=0.8) reaches UTMOS 4.000, nearly matching 75 Hz WavTokenizer (4.025) while using less than half the tokens (Table 1). Below 40 Hz, VARSTok consistently leads all baselines including BigCodec and SQCodec across UTMOS, PESQ, STOI, and V/UV F1.
On the ARCH benchmark semantic classification tasks (emotion recognition, digit recognition, intent classification), all VARSTok configurations outperform the 40 Hz WavTokenizer despite lower average frame rates, suggesting that density-adaptive token boundaries preserve more semantically discriminative structure than uniform ones (Table 2).
In zero-shot TTS using a 12-layer Transformer decoder conditioned on text and a 3-second speech prompt, VARSTok (tau=0.8, 36.81 Hz) achieves MOS 4.053 and WER 6.787%, compared to the 40 Hz WavTokenizer baseline at MOS 3.983 and WER 7.481%. Subjective SMOS scores for both are statistically indistinguishable (3.946 vs 3.918, Table 5). There is a modest drop in objective speaker similarity (SIM 0.899 vs 0.918 for tau=0.8) that does not translate to perceivable difference in human evaluation. On TTS inference speed, VARSTok (tau=0.6) achieves RTF 0.487 versus 0.766 for the baseline, a 36% speedup, because shorter token sequences more than offset the cost of the expanded vocabulary projection layer (Appendix J).
Codebook utilisation analysis reveals that K = 4096 with S_max = 4 (effective vocabulary 16,384) achieves 99.16% utilisation of the expanded space, while larger codebooks (K = 16384) suffer severe collapse (40.17% utilisation), making K = 4096 the preferred setting (Table 6).
Novelty Assessment
The temporal-aware density-peak clustering algorithm is genuinely novel applied to acoustic tokenisation. The technique adapts a 2D image-patch clustering method to sequential speech by enforcing temporal contiguity constraints, which is a meaningful adaptation rather than a direct transplant. The implicit duration coding mechanism is elegant: it solves the duration management problem without any learned predictor by algebraic expansion of the codebook index space, at zero parameter cost. Both contributions are clearly motivated by the practical requirement of seamless integration into autoregressive speech LMs.
The encoder-decoder backbone and training objective are inherited from WavTokenizer (the paper uses its pretrained weights as initialisation and its loss formulation unchanged). The primary claim is therefore in the tokenisation mechanism, not in the acoustic modelling. Comparisons are mostly against WavTokenizer variants and do not include the full range of recent single-codebook codecs, and SQCodec and XCodec2.0 comparisons are noted as not directly equivalent due to different training data and sampling rates. The work is strongest as a demonstration that fully dynamic variable-rate tokenisation is feasible for reconstruction and downstream TTS, rather than as a claim of state-of-the-art reconstruction quality across all settings.
Field Significance
Moderate — VARSTok provides principled evidence that content-adaptive tokenisation can outperform fixed-rate acoustic codecs at equivalent or lower token budgets, and introduces a practical mechanism (implicit duration coding) for making variable-rate tokens usable in standard autoregressive LMs. The contribution is relevant primarily to the neural codec and speech LM integration sub-field. Whether the density-peak clustering approach or its specific implementation proves broadly reusable depends on comparisons against a wider baseline set that this paper does not provide.
Claims
-
supports: Content-adaptive token allocation in acoustic tokenisers achieves better reconstruction quality than fixed-rate designs at equal or lower token budgets.
Evidence: VARSTok at 30.95 Hz achieves UTMOS 3.8949 on LibriTTS test-clean, surpassing the 40 Hz WavTokenizer (3.6107) while using 23% fewer tokens; the 36.81 Hz configuration (UTMOS 4.000) nearly matches the 75 Hz WavTokenizer (4.025) with fewer than half the tokens. (§4.2, Table 1)
-
supports: Dynamically segmented speech tokens carry more semantically discriminative information than uniformly sampled tokens at the same average rate.
Evidence: All VARSTok configurations outperform the 40 Hz WavTokenizer on all four ARCH benchmark classification tasks (emotion, digit recognition, intent), despite operating at lower average frame rates. (§4.3, Table 2)
-
supports: Encoding token duration implicitly in the VQ codebook index eliminates the need for auxiliary duration predictors and preserves compatibility with autoregressive speech language models.
Evidence: The implicit duration coding scheme maps each cluster’s content index k and duration d to a single token ID D = (d-1)*K + k, enabling a standard cross-entropy autoregressive model to generate variable-rate token sequences without modification; MOS and WER improve over the fixed-rate baseline in zero-shot TTS. (§3.4, §4.4, Table 3)
-
complicates: More aggressive temporal compression in variable-rate tokenisers trades reconstruction quality for token efficiency beyond a practical compression threshold.
Evidence: Increasing S_max from 2 to 8 reduces the average frame rate from 46.5 Hz to 22.38 Hz but degrades UTMOS from 4.038 to 3.647 and PESQ from 2.069 to 1.453 on LibriTTS test-clean; the optimal configuration (tau=0.7, S_max=4 at 30.95 Hz) sits at the knee of this trade-off curve. (§4.2, Table 1)
-
refines: Inference speed in autoregressive speech LMs depends primarily on sequence length rather than vocabulary size, so variable-rate tokenisers with expanded vocabularies still accelerate decoding.
Evidence: VARSTok (tau=0.6) achieves RTF 0.487 versus 0.766 for the 40 Hz WavTokenizer baseline (36% speedup) despite expanding the token vocabulary from K to K*S_max = 16,384 entries, because shorter sequences reduce the dominant cost of attention computation over more function evaluations. (§J, Table 5)
Limitations and Open Questions
Warning
Evaluation is restricted to English (LibriTTS). The clustering algorithm relies on cosine similarity in a WavTokenizer embedding space trained on English read speech; whether the density-peak boundaries remain meaningful for other languages, spontaneous speech, or emotionally expressive styles is untested.
Speaker similarity under more aggressive compression (tau=0.6, 26.29 Hz) does show a statistically modest decline in objective SIM (0.880 vs 0.918 for the baseline), and while subjective SMOS remains comparable, the long-tail impact on voices far from the training distribution is unknown. Codebook collapse becomes severe for K above 4096 in the expanded index space, suggesting that very large vocabulary configurations require dedicated regularisation strategies not addressed here. The clustering algorithm is not differentiable, so joint end-to-end training with a downstream TTS model is not straightforward.
Wiki Connections
- Neural Audio Codec — VARSTok proposes a variable-frame-rate variant of the single-codebook acoustic tokeniser paradigm, extending the neural codec design space with a content-adaptive compression mechanism.
- Autoregressive Codec TTS — The paper demonstrates VARSTok’s integration into a VALL-E-style decoder-only autoregressive TTS model, showing improved naturalness (MOS 4.053) and WER (6.787%) over the fixed-rate WavTokenizer baseline.
- Spoken Language Model — VARSTok is explicitly motivated by and evaluated for compatibility with autoregressive speech LMs, with the implicit duration coding scheme designed to avoid architectural modifications to the LM.
- Zero-Shot TTS — The downstream TTS evaluation follows the zero-shot paradigm (text plus a 3-second reference prompt), demonstrating that VARSTok tokens support speaker adaptation without speaker-specific fine-tuning.
- Subjective Evaluation — Human MOS and SMOS judgements on Amazon Mechanical Turk (35 native English speakers, 25 samples) confirm that VARSTok’s perceptual gains are not artefacts of UTMOS and that objective speaker similarity (SIM) diverges from perceived speaker similarity at lower token rates.