ASRU · 2025 · Conference
Onda et al. (The University of Tokyo / AIST) · → Paper · Demo: ? · Code: ?
A systematic benchmark of how well SSL-derived discrete speech tokens encode prosodic information, using artificially modified speech to disentangle linguistic, paralinguistic, and non-linguistic dimensions.
Problem
Discrete semantic tokens obtained by k-means clustering of self-supervised learning (SSL) features are central to speech language models, yet the choices governing tokenisation — which SSL model, how many clusters, which training data for k-means, and whether to apply feature preprocessing — are typically made heuristically. Prior work has benchmarked these tokens mainly for linguistic content and acoustic robustness, but not for prosody. The open question of whether prosodic features should be retained or suppressed in semantic tokens has been disputed without direct, controlled experiments. Existing analyses relying on downstream-task performance conflate token quality with decoder capability.
Method
The paper constructs a controlled experimental protocol using the TIMIT corpus and the WORLD vocoder to apply surgical, independently verifiable modifications to speech prosody and speaker identity. Pitch and intensity are modified at two granularities: word-level (emphasising a single word while leaving the rest unchanged) and utterance-level (scaling the dynamic range of the entire utterance). Speaker identity is simulated via vocal-tract-length modification of the spectral envelope.
Token sequences before and after each modification are compared using Token Error Rate (TER), the edit distance between the discrete token sequences of original and modified speech. This directly measures sensitivity without involving any downstream model.
Four SSL models are compared — HuBERT, ContentVec, data2vec, and emotion2vec — each representing a different pretraining objective. k-means cluster sizes of 100, 500, and 2000 are tested. The study also evaluates two additional design choices: (1) whether training k-means on emotional speech (MEAD dataset) versus neutral read speech (LibriSpeech) changes prosodic sensitivity, and (2) whether applying a moving average to SSL features before clustering, at varying window sizes, affects the trade-off between prosodic sensitivity and speaker invariance.
Key Results
SSL models trained with frame-wise masked prediction (HuBERT, ContentVec, data2vec) consistently respond more strongly to word-level prosodic emphasis than to utterance-level uniform scaling, for both pitch and intensity. This demonstrates that these models encode relative prosodic contours rather than absolute acoustic magnitudes. Utterance-level intensity scaling causes near-zero TER change for these three models, attributable to waveform normalisation in the CNN front-ends inherited from wav2vec 2.0. emotion2vec, which adds utterance-level supervision via temporal mean pooling, shows notably higher sensitivity to global prosodic changes.
Regarding cluster size and SSL pretraining objective: models trained to predict discrete labels (HuBERT, ContentVec) capture phoneme-like structure even at 100 clusters and show monotonically increasing prosodic sensitivity with cluster size. Models trained to predict continuous features (data2vec, emotion2vec) exhibit degraded prosodic sensitivity at small cluster sizes at the final transformer layers, but their intermediate layers (around layers 9–10) behave similarly to HuBERT and ContentVec. Phone normalised mutual information (PNMI) confirms this layer-specific divergence.
Training k-means on emotional speech (MEAD) significantly increases sensitivity to prosodic variation in most conditions (Table I, p < 0.05), at the cost of moderately increased sensitivity to real speaker variation. An exception is found at the final layer of data2vec and emotion2vec with small cluster sizes, where the ‘clustering-unfriendly’ nature of the representations causes the emotionally trained centroids to produce unbalanced cluster distributions that reduce sensitivity. Applying a moving average to SSL features can simultaneously increase prosodic sensitivity and reduce speaker sensitivity at an intermediate window size — for HuBERT’s final layer with 2000 clusters, window sizes of 9–11 outperform no smoothing on prosody while reducing speaker sensitivity.
Novelty Assessment
The contribution is primarily a methodological and empirical one. The specific combination of WORLD-based prosody manipulation, TER-based direct token comparison, and systematic factorial evaluation across SSL model type, cluster size, training data, and feature preprocessing is new and fills a genuine gap. The individual techniques (WORLD vocoder, k-means clustering, TER) are all established; the value is in the controlled design and the breadth of coverage. The findings provide directly actionable guidance for practitioners choosing tokenisation configurations, which is not available from prior downstream-task-based studies. No new model or training objective is proposed.
Field Significance
Moderate — This paper fills a specific analytical gap by providing a controlled prosody-focused benchmark of SSL-derived semantic tokens, a type of evaluation absent from prior work on discrete speech units. The practical guidance — that emotional k-means training data and appropriate moving average window sizes can tune the prosody/speaker trade-off without changing the upstream SSL model — is immediately applicable to speech language model design. The contribution is empirical and methodological rather than architectural.
Claims
- SSL models using frame-wise masked prediction capture relative prosodic contours within an utterance rather than absolute acoustic magnitudes, making them insensitive to global intensity rescaling. (§V-A, Fig. 1)
- Models pretrained to predict discrete targets encode phoneme-like structure effectively at small cluster sizes, while models pretrained on continuous targets require larger cluster sizes to approach the same phonemic alignment. (§V-A, Fig. 3)
- Training k-means clustering on emotionally expressive speech increases the prosodic sensitivity of resulting tokens for most SSL model and layer combinations. (§V-B, Table I)
- Applying a temporal moving average to SSL features before k-means clustering provides an adjustable trade-off between prosodic sensitivity and speaker invariance, with intermediate window sizes improving both simultaneously. (§V-C, Fig. 5)
- Differences between SSL pretraining objectives in their token-level linguistic and prosodic encoding are concentrated in the final transformer layers, while intermediate layers exhibit largely similar behaviour across model families. (§V-A, §V-B)
Limitations and Open Questions
The analysis measures sensitivity via TER — a proxy for how much token sequences change in response to acoustic manipulation — rather than directly probing what information is decodable from the tokens. Whether the observed sensitivity differences translate to actual gains in downstream prosody-related tasks (prosody-conditional TTS, emphasis transfer, emotion recognition from discrete tokens) remains untested. The evaluation uses a single corpus (TIMIT) of read speech by native English speakers, which may limit generalisability to spontaneous, conversational, or multilingual speech. Duration was excluded from prosody analysis because deduplication collapses durational information; this omission means the benchmark does not cover the full prosody space. The study does not test acoustic tokens (neural codec outputs), focusing exclusively on semantic tokens.
Wiki Connections
Relates to the design of discrete tokens for spoken-language-model systems, particularly the question of what paralinguistic information they should carry. Connects to self-supervised-speech as the upstream representations being discretised, prosody-control as the target capability being benchmarked, and disentanglement as the conceptual frame for separating speaker, linguistic, and paralinguistic information in tokens. Also relevant to neural-codec in contrasting semantic versus acoustic token design choices, and to evaluation-metrics for its TER-based methodology for token property analysis.