Interspeech · 2025 · Conference
Shunsuke Kando et al. · → Paper · Demo: ✗ · Code: ✓
A systematic 64-configuration grid search over segmentation width (N ∈ {20,…,280} ms) and K-means vocabulary size (K ∈ {128,…,16384}) for HuBERT-based speech tokenization demonstrates that moderately coarse segmentation (N=80 ms) paired with large vocabulary (K=16384) outperforms GSLM-style original-resolution tokenization while reducing training data by 50% and training runtime by 70%.
Problem
Speech language models (SLMs) depend critically on their discrete unit vocabulary, yet the design space of segmentation width and cluster size has not been systematically explored. Prior work either uses original-resolution HuBERT features with small K (GSLM: K=50–200) or very coarse segmentation with large K (syllable-level), but does not compare these jointly. The consequence is that practitioners lack guidance on the optimal tokenization tradeoff between sequence length (computational cost), vocabulary expressiveness, and downstream SLU performance.
Method
The pipeline is: (1) extract HuBERT layer-9 representations; (2) segment by fixed width N ms and mean-pool within segments; (3) apply K-means with vocabulary size K; (4) deduplicate consecutive identical units. This yields 8×8 = 64 tokenization configurations.
The SLM is an OPT-style decoder-only Transformer (12 layers, 16 attention heads, hidden dim 1024, FFN 4096), trained on LibriSpeech 960h (or smaller after pooling). Training follows GSLM hyperparameters, with sequences chunked to 2048 tokens, batch size 16, up to 50k steps with early stopping, on a single A100. Results are averages across 3 random seeds.
Evaluation uses five zero-shot spoken language understanding benchmarks:
- sBLIMP: grammatical knowledge (12 phenomena)
- sWUGGY: lexical knowledge (real vs. nonce word)
- pros-syntax / pros-lexical: prosodic phrase/word boundary perception (ProsAudit)
- tSC (TopicStoryCloze): commonsense/semantic coherence
Variable-width segmentation (phoneme via UnsupSeg, syllable via Sylber, word via GradSeg) is also evaluated against fixed-width baselines of matched median segment duration.
Key Results
Best fixed-width configuration: (N=80, K=16384), average accuracy 0.67 across 5 tasks. vs. (N=20, K=128) GSLM-style baseline: average accuracy 0.65, training runtime 12.4h, 87M tokens. Best (N=80, K=16384): training runtime 8.3h, 42M tokens — 50% data reduction, 70% runtime reduction.
Task-specific patterns:
- pros-syntax achieves very high accuracy even at large N, because the 400 ms inserted pauses greatly exceed N.
- sBLIMP hovers near chance (0.5) for all configurations, consistent with prior literature on SLM grammatical limitations.
- Some sBLIMP subtasks (ellipsis, quantifiers) show above-chance accuracy with unique optimal settings, suggesting multi-token ensembling could help.
- Variable-width segmentation (phoneme, syllable, word levels) generally matches but does not exceed fixed-width, with syllable showing slight gains on sBLIMP and pros-syntax at the cost of added unsupervised segmentation compute.
Qualitative analysis: at (80, 128) — large N, small K — the model cannot distinguish phonetically similar tokens because the vocabulary is too small to reflect the increased acoustic diversity at 80ms resolution. Increasing K to 16384 restores discriminability (illustrated with the yonder/zonder sWUGGY example).
Novelty Assessment
The contribution is empirical and systematic rather than architectural: a comprehensive ablation over a large tokenization grid that had not been done before. The finding that moderately coarse (80ms) fixed-width tokenization with large K beats fine-grained (20ms) tokenization is practically important for SLM efficiency without hurting accuracy. The phoneme-morpheme analogy (combining fewer, broader segments requires more codewords) is an intuitive explanation for the observed tradeoff. The study does not address speech synthesis (generation quality) — only spoken language understanding — which limits its relevance to TTS directly, though the tokenization findings apply to codec-based TTS architectures using HuBERT-like semantic tokens.
Field Significance
Moderate — This paper provides the most comprehensive publicly available sweep over segmentation width and vocabulary size for HuBERT-based SLM tokenization, filling a gap that prior work left open by evaluating only narrow regions of this design space. Its primary value is practical: SLM practitioners gain concrete guidance on a tokenization configuration that reduces training cost by roughly two-thirds without sacrificing SLU accuracy. The finding that variable-width linguistic segmentation does not systematically outperform fixed-width tokenization is a useful negative result for the subfield.
Claims
- Moderately coarse fixed-width segmentation (around 80 ms) combined with a large K-means vocabulary outperforms fine-grained original-resolution tokenization on zero-shot spoken language understanding tasks without sacrificing accuracy. (§4, Table 3)
- Larger segmentation widths require proportionally larger vocabularies to preserve phonetic discriminability, analogous to the phoneme-morpheme relationship in linguistics. (§5.1)
- Variable-width segmentation based on linguistic units (phoneme, syllable, word boundaries) does not consistently outperform fixed-width segmentation of matched median duration, and incurs additional computational cost. (§5.3)
- Optimal tokenization settings vary across spoken language understanding benchmarks, suggesting that ensembling multiple tokenization schemes may be necessary for broad SLU capability. (§5.2)
Limitations and Open Questions
- Evaluation is exclusively on SLU (understanding) tasks; no speech generation quality assessment.
- Training data (LibriSpeech 960h) is small by current SLM standards; findings may differ at larger scale (LibriLight 60k).
- Only HuBERT layer-9 as the SSL backbone; other models (WavLM, wav2vec 2.0) may yield different tradeoff curves.
- Variable-width segmentation using predicted (rather than oracle) boundaries introduces inaccuracy; learned segment representations (as in Sylber) may be better than raw pooling.
- Optimal tokenization varies per benchmark, motivating multi-token ensemble strategies not explored here.
Wiki Connections
- neural-codec — this paper characterises the design space of K-means-based discrete speech representations built on top of HuBERT features
- self-supervised-speech — HuBERT layer-9 representations are the direct input to all 64 tokenization configurations evaluated
- spoken-language-model — the SLM training and SLU evaluation framework is the central experimental vehicle
- evaluation-metrics — the paper evaluates across five zero-shot SLU benchmarks (sBLIMP, sWUGGY, ProsAudit, TopicStoryCloze), illuminating their sensitivity to tokenization choices