arXiv · 2026 · Preprint
Osakuade et al. (University of Edinburgh) · → Paper · Demo: ? · Code: ?
Probing experiments on Mandarin and Yorùbá show that quantising HuBERT latents into discrete speech units preserves phone identity almost perfectly but loses a large share of lexical tone information, and that multi-level quantisers (residual VQ, or K-means on residuals after a phone-level pass) recover much of the loss.
Problem
Discrete speech units (DSUs), obtained by quantising self-supervised speech representations, are a common interface for TTS, speech-to-speech translation, and spoken language modelling because they let speech share a token sequence with text. Prior work (ToneUnit, and the authors’ own earlier study) found that lexical tone is present in SSL latents but poorly preserved after K-means quantisation, which matters for tone languages where tone is lexically contrastive. Two questions remain open: why quantisation degrades tone more than segmental content, and whether a better quantiser at a fixed, language-model-friendly vocabulary size can mitigate the loss. Simply raising K is not a practical answer: moving from 1,000 to 5,000 clusters lifts Mandarin tone F1 only from 0.80 to 0.87, and tone F1 still plateaus below the latent baseline at 10,000 clusters (§1.2, §4.1, Figure 1).
Method
The study is a controlled probing analysis in which the quantiser is the only experimental variable: data are fixed and the SSL encoders are frozen (§3). Frame-level latents come from MandarinHuBERT for Mandarin (AISHELL-1) and AfriHuBERT, an mHuBERT-147 extension, for Yorùbá (BibleTTS, single speaker). Because tone in both languages is realised mainly on the vowel nucleus, the analysis uses vowel segments from Montreal Forced Aligner alignments, with a tone-aware Yorùbá G2P supplying the lexicon (§2.1, §2.2).
Probes are deliberately lightweight. Frame sequences per vowel go to a one-layer LSTM (hidden size 128) with separate linear heads for phone and tone, trained with class-weighted cross-entropy. Mean-pooled segment vectors go to a softmax logistic-regression classifier. Performance is weighted F1. Probes read the codebook vectors (e.g. the K-means centroids), not the integer codes (§2.3, §2.4). Mandarin tone labels are lexical tone only, since AISHELL-1 does not annotate sandhi.
Five quantiser families are compared at a matched budget of 500 codes (§3.1 to §3.4):
- classic frame-level K-means (K=500), the standard HuBERT-unit recipe;
- neural VQ trained with a reconstruction objective following RepCodec (one 500-entry codebook), plus residual VQ variants (250x2 and 125x4);
- mean-pooled phone-segment K-means;
- segmentation-variant codebooks (SVC), which average a frame-level centroid and a phone-level centroid;
- the proposed residual K-means: a coarse K=50 pass on mean-pooled phone segments (roughly the phoneme inventory size) produces a “phone-identity” codebook, each latent’s phone centroid is subtracted, and the residuals are clustered with K=450. It comes in frame-level and segmental variants.
Key Results
All numbers are weighted F1 from Table 1 (§4).
- Latents encode both phone and tone. Continuous latents reach 0.99 phone / 0.94 tone on Mandarin and 0.97 / 0.92 on Yorùbá.
- Classic K-means keeps phones and loses tone. At K=500, Mandarin phone F1 stays at 0.99 while tone falls to 0.70. On Yorùbá, phone is 0.95 and tone 0.77 (§4.1). The authors read this as phonetic variance dominating the latent space, so any variance-driven quantiser allocates codes to segmental distinctions first.
- Single-codebook neural VQ gives mixed results. It beats K-means on Mandarin tone (0.78 vs 0.70) but is worse on Yorùbá (0.66 vs 0.77), and it also drops Yorùbá phone F1 to 0.85 (§4.2).
- Residual VQ helps. RVQ 125x4 gives the best quantised Mandarin tone score (0.82), with Yorùbá at 0.76. The 250x2 variant is slightly lower on both (§4.3).
- Pooling hurts tone. Mean-pooled K-means is worst on Mandarin tone (0.57), which the authors attribute to discarding within-segment pitch trajectories. SVC sits between pooled and frame-level K-means (0.62 Mandarin, 0.78 Yorùbá) (§4.4).
- Residual K-means is competitive with RVQ. The segmental variant reaches 0.79 on Mandarin and the best Yorùbá tone score (0.83). Figure 2 shows the first-level codebook carries phone identity and the residual level carries most of the recovered tone information (§4.5).
- Language differences. Mandarin tone drops more under K-means than Yorùbá tone (0.94 to 0.70 vs 0.92 to 0.77). The best multi-level method also differs by language: deeper RVQ for Mandarin contour tones, segmental residual K-means for Yorùbá level tones (§4.6).
The comparison is internally fair: same encoder, same probes, same code budget. However, no downstream task is run (no synthesis, resynthesis, or language modelling), so the results stand in for downstream utility rather than measuring it. The paper also reports no variance or significance testing, so small gaps such as RVQ 250x2 vs 125x4 should be read cautiously.
Novelty Assessment
The main finding, that K-means DSUs under-represent tone relative to SSL latents, confirms prior work (ToneUnit and the authors’ 2024 preprint). The new parts are a broader quantiser comparison at a matched budget (neural VQ, RVQ, SVC, pooled K-means), a second typologically different tone language with level tones, and a variance-based explanation for why segmental structure dominates. The residual K-means scheme is a simple two-stage clustering idea, not a new learned architecture. It relies on forced alignments during quantisation, which the authors flag (§5). Overall the contribution is an empirical analysis with a modest methodological suggestion. The scope is short: two languages, one SSL encoder per language, probing only. The paper is reported as accepted at Speech Prosody 2026.
Field Significance
Moderate. The paper gives controlled evidence that the discretisation step itself, not the SSL encoder, causes the loss of suprasegmental information in discrete-unit pipelines, and that multi-level quantisation can partly recover it at a fixed vocabulary size. For TTS and speech LMs that use HuBERT-style units in tone languages, this points to the tokenizer as a likely source of tone errors. The extension to prosody more generally is the authors’ hypothesis and is not tested here.
Claims
- supports: Quantising self-supervised speech representations into a small discrete vocabulary preserves segmental (phonetic) information much better than suprasegmental information such as lexical tone.
Evidence: At 500 K-means codes, Mandarin phone F1 stays at 0.99 while tone F1 falls from 0.94 (continuous HuBERT latents) to 0.70. Yorùbá tone falls from 0.92 to 0.77 while phone stays at 0.95. (§4.1, Table 1)
- complicates: Enlarging a flat discrete-unit codebook gives quickly diminishing returns for recovering suprasegmental information and conflicts with keeping the token vocabulary small for language modelling.
Evidence: Raising K from 1,000 to 5,000 lifts Mandarin tone F1 only from 0.80 to 0.87, and tone F1 plateaus below the latent baseline even at 10,000 clusters. (§1.2, §4.1, Figure 1)
- supports: Multi-level (residual or hierarchical) quantisation of speech representations preserves more suprasegmental information than single-codebook quantisation at the same total code budget.
Evidence: At 500 total codes, RVQ 125x4 raises Mandarin tone F1 to 0.82 (vs 0.70 for K-means and 0.78 for single-codebook VQ). Residual K-means on phone-centroid residuals reaches 0.83 on Yorùbá, and its second level carries most of the recovered tone information. (§4.3, §4.5, Table 1, Figure 2)
- complicates: Reconstruction-trained neural vector quantisers do not consistently preserve suprasegmental information better than distance-based clustering of self-supervised features.
Evidence: RepCodec-style single-codebook VQ improves Mandarin tone F1 over K-means (0.78 vs 0.70) but is worse on Yorùbá (0.66 vs 0.77) and also lowers Yorùbá phone F1 to 0.85. (§4.2, Table 1)
- refines: Which discretisation strategy best preserves lexical tone depends on how the tone system is realised: contour tones benefit from deeper residual quantisation, and vowel-aligned level tones benefit from segment-aware residual clustering.
Evidence: The best quantised Mandarin (contour tone) result comes from RVQ 125x4 (0.82). The best Yorùbá (level tone) result comes from segmental residual K-means (0.83). Mean pooling alone hurts Mandarin tone badly (0.57). (§4.4, §4.6, Table 1)
Limitations and Open Questions
Warning
All conclusions rest on linear and LSTM probes of codebook vectors. No TTS, resynthesis, or speech-LM experiment is run, so the claimed impact on synthesised tone accuracy or naturalness is untested.
- The best non-neural method (segmental residual K-means) needs forced phone alignments at quantisation time, which may be unavailable in low-resource settings. The authors suggest fixed-duration pooling, automatic unit discovery, or neural RVQ as alternatives (§5).
- Each language uses one SSL encoder (MandarinHuBERT, AfriHuBERT), and the Yorùbá data is single-speaker read Bible speech, so speaker and style variability in Yorùbá is barely tested.
- Mandarin tone labels ignore tone sandhi, and only vowel segments are probed.
- The paper asserts but does not test that the findings extend to prosody (phrasing, prominence, rhythm).
- No confidence intervals or repeated runs are reported, so small F1 differences between quantisers cannot be judged for significance.
- Model sizes, codebook training details, and code release are not reported.
Wiki Connections
- Self-Supervised Speech — Probes frozen HuBERT-family latents and shows that tone is well encoded before quantisation, then lost mainly at the discretisation step.
- Neural Audio Codec — Compares single-codebook VQ, residual VQ, and K-means speech tokenizers at a matched 500-code budget, with residual multi-level quantisation retaining the most tone.
- Evaluation Metrics — Uses phone and tone probing classifiers (weighted F1) on codebook vectors as a proxy for how well discrete units suit downstream tasks.
- Segmentation-Variant Codebooks — Re-implemented as the SVC quantiser baseline. It improves on mean-pooled clustering but trails RVQ and residual K-means for tone.
- Speech Resynthesis from Discrete SSL Units — The source of the classic frame-level K-means HuBERT unit recipe that serves as the main baseline.
- VALL-E — Cited as an example of TTS built on discrete speech tokens, the kind of downstream application whose tone fidelity the paper’s findings concern.
- Speech Language Model Survey — Cited as background on spoken language models that rely on discrete speech units.