arXiv · 2025 · Preprint
Zuo et al. · → Paper · Demo: ? · Code: ?
An entropy-based dynamic aggregation framework that compresses HuBERT semantic tokens from 50 Hz down to 7–24 Hz by using a lightweight autoregressive LM’s predictive uncertainty to identify segment boundaries, then fusing each segment via cross-attention — achieving comparable or better downstream performance than dense token sequences at significantly reduced sequence length.
Problem
SSL-derived semantic speech tokens (HuBERT, w2v-BERT) are typically produced at 25–50 tokens per second, far exceeding the 2–5 words per second that speech actually conveys. This mismatch introduces substantial redundancy: consecutive tokens in low-entropy regions convey little new information, yet downstream models — ASR encoders, voice conversion systems, speech LMs — must process every one of them. Fixed-length downsampling (pooling over uniform windows) discards boundaries indiscriminately, degrading task performance. The field lacked a principled, adaptive mechanism to compress semantic token sequences while preserving linguistically meaningful boundaries.
Method
The framework has three stages. First, input speech is processed by a pretrained HuBERT model and discretized via k-means clustering (K=500 or K=2000) into frame-level tokens at 50 Hz. Second, a lightweight autoregressive LM — a LLaMA-style transformer with ~0.1B parameters (8 layers, model dimension 1024, 16 attention heads) trained on the English MLS corpus (~20k hours) via next-token prediction — computes token-wise conditional entropy. Where entropy exceeds a global threshold θ_g (or shows a sharp relative increase θ_r), a segment boundary is placed; adjacent low-entropy tokens are merged into a group. The global threshold alone (Method 1) proves sufficient and is adopted as the default. This produces variable-length groups at effective rates of 7 Hz (coarse), 15 Hz (moderate), or 24 Hz (fine), controlled by adjusting θ_g. Third, a Cross-Attentive Local Encoder (CALE) — 4 stacked cross-attention layers — takes each group’s tokens as keys/values and a max-pooled group embedding as the initial query, iteratively refining a compact group-level representation. The CALE is not merely a pooling step: ablation studies show it contributes meaningfully over max, average, and attention pooling baselines by enabling global contextual awareness across the original sequence.
For downstream evaluation, ASR and speech-to-text translation use a Whisper-style encoder-decoder (~104M total parameters); voice conversion uses a Diffusion Transformer (Seed-VC style) with HiFi-GAN synthesis. These are standard architectures adapted to accept the compressed representations rather than novel contributions.
Key Results
At 15 Hz (moderate compression), the entropy-guided method achieves WER 5.6% and CER 2.9% on LibriSpeech test-clean — outperforming both HuBERT original (50 Hz, WER 6.2%) and HuBERT deduplicated (26 Hz, WER 5.9%), while reducing sequence length by 70%. For speech-to-text translation (CVSS-C), entropy-guided 15 Hz reaches BLEU 31.5 (EN-CN), 30.9 (EN-FR), 31.6 (EN-ES), beating HuBERT original (28.2/28.5/29.3) and fixed-length pooling approaches. The 7 Hz high-compression setting still outperforms fixed-length pooling at 12.5 Hz on ASR (WER 10.5% vs 6.8%), demonstrating that adaptive boundaries preserve more linguistic structure than uniform windows at the same compression level.
For voice conversion, the picture is less favorable: entropy-guided 15 Hz achieves Q-MOS 3.85 and S-MOS 3.78, compared to HuBERT deduplicated at 4.12/3.95. Coarser compression degrades quality progressively, with 7 Hz dropping to Q-MOS 3.33 and WER 9.1% on the VC task. This confirms that generation tasks are less tolerant of compression than understanding tasks.
Decoding latency at 15 Hz (120 ms) falls between HuBERT original (172 ms) and deduplicated (148 ms), validating the efficiency-quality trade-off without being the fastest option.
Note
Voice conversion MOS values are human ratings from Amazon Mechanical Turk (50 utterances, ≥10 listeners each), reported as Q-MOS and S-MOS. ASR/ST comparisons are on standard public benchmarks with identical conditions — the comparisons are fair.
Novelty Assessment
The core novelty is the use of a pre-trained autoregressive LM’s predictive entropy as an adaptive segmentation signal for semantic token compression. Using entropy as an information-theoretic proxy for redundancy is conceptually clean and well-motivated. The CALE module adds a meaningful refinement step over simple pooling. However, neither component is architecturally transformative: entropy-based segmentation has been applied to text tokenization, and cross-attention aggregation is standard. The contribution is primarily a well-executed combination applied to a real gap in speech tokenization — producing a framework that is practical, controllable, and empirically validated across three task families. The finding that 15 Hz is near-optimal for recognition tasks (matching typical phoneme rates) is a useful calibration result, though it confirms intuition rather than surprising the field.
Field Significance
Moderate — This paper addresses a genuine efficiency bottleneck in the SSL speech tokenization pipeline and provides the first systematic evidence that entropy-based adaptive compression outperforms fixed-length downsampling at matched token rates. Its practical value lies in the flexibility of the threshold-controlled granularity and the empirical characterization of the compression-task trade-off across understanding and generation settings. It is primarily an engineering contribution with well-grounded analysis, positioned to inform codec and speech LM design choices rather than to redirect the field’s approach.
Claims
- Adaptive entropy-based segmentation of discrete speech tokens preserves more task-relevant linguistic information than fixed-length downsampling at equivalent compression ratios. (§6.1, Table 3; §6.2, Table 4)
- Optimal token granularity differs systematically between understanding and generation tasks: recognition-oriented tasks (ASR, ST) benefit from moderate compression near phoneme rate, while voice conversion requires finer token density to maintain acoustic fidelity. (§5.1, Table 1; §6.3, Table 5)
- SSL-derived semantic tokens at standard rates (50 Hz) contain substantial redundancy that can be removed without degrading — and occasionally improving — downstream task performance. (§6.1, Table 3)
- Entropy boundaries in compressed token sequences align with linguistically meaningful units: 15 Hz compression achieves 83.2% phoneme boundary alignment, while 7 Hz aligns primarily with word boundaries (89.7%). (Appendix C.3, Table 11)
Limitations and Open Questions
Warning
Voice conversion quality degrades noticeably with compression: entropy-guided 15 Hz already falls below HuBERT deduplicated (Q-MOS 3.85 vs 4.12), and the gap widens at higher compression. For generation tasks, this framework does not improve on simply using deduplicated tokens — only for understanding tasks does compression help.
The framework is evaluated exclusively on HuBERT-derived tokens; whether the entropy-based approach transfers to other SSL features (WavLM, w2v-BERT) or supervised tokenizers (S3, FACodec) is untested. The entropy LLM requires pre-training on 20k hours of speech, adding a pipeline step beyond vanilla k-means clustering. The method is tested on English only, and its behaviour on morphologically complex or tonal languages — where token redundancy patterns may differ — remains unknown. All evaluations use the English MLS training corpus, so generalisation across domains and recording conditions is unexplored.
Wiki Connections
Concept pages this paper most directly informs: neural-codec (on the question of how many tokens per second are actually needed for semantic content), self-supervised-speech (HuBERT-derived representations as the foundation), voice-conversion (entropy-compressed tokens as VC conditioning), disentanglement (compression as implicit disentanglement of redundant from informative tokens).