arXiv · 2026 · Preprint
Vigneaux et al. (Northwestern University) · → Paper · Demo: ? · Code: ?
Trains linear and nonlinear probes on the unquantized latents and codebook representations of the Mimi neural audio codec to test whether English nuclear (phrase-final) intonational tunes are recoverable, finding that pitch information is distributed across codebooks rather than confined to the “semantic” codebook.
Problem
State-of-the-art spoken dialogue models such as Moshi and Sesame’s conversational model rely on a neural audio codec (Mimi) to tokenize speech into a hierarchy of discrete codebooks before a transformer predicts these tokens autoregressively. One codebook (Codebook 0) is explicitly distilled from WavLM and labeled “semantic,” implying it should carry linguistically meaningful content, while the remaining residual-VQ codebooks are treated as “acoustic.” Intonation, and specifically the phrase-final (“nuclear”) pitch tunes of English, conveys pragmatic meaning such as the distinction between questions and assertions, and between focused and given information. Prior work has not established whether a codec’s discrete tokenization preserves these pragmatically relevant pitch patterns, or in which part of the codebook hierarchy such information would reside if it is preserved at all.
Method
The authors use labeled audio data from an existing production/perception study (Cole et al. 2023): 4,656 short utterances realized with one of eight phonologically defined nuclear tunes (combinations of monotonal pitch accent, phrase accent, and boundary tone), plus the five tune clusters that study found to be robustly distinguished by human listeners and speakers. Utterances are aligned with a forced aligner to isolate the final accented word, then passed through the Mimi codec, yielding both unquantized 512-dimensional latent vectors and 256-dimensional codewords from Codebook 0 (VQ, WavLM-distilled) and Codebooks 1-7 (residual VQ).
For each utterance, the sequence of frame-level vectors over the accented word is aggregated into a single vector via a weighted average whose forward- and backward-decay hyperparameters control how much weight is placed on the pitch-accent region versus the edge-tone region. The aggregated vector is dimensionality-reduced with PCA (fit on the training split) and then classified with either a linear softmax classifier or a small nonlinear classifier (linear layer, LayerNorm, leaky ReLU, second linear layer, softmax). Hyperparameters (decay rates, PCA dimension, batch size, learning rate, epochs) are tuned with Optuna on the unquantized embeddings and then reused for probes trained on each codebook’s codewords. Five classification problems are evaluated: the full 8-way tune distinction, the 5-cluster distinction, and three binary distinctions (high-rising vs. low-falling; pitch-accent only; edge-tone only, i.e. question vs. assertion framing). Data is split 70/15/15 into train/dev/test, preserving label proportions, and each reported accuracy is averaged over three independently trained probes.
Key Results
Every probe beats the ZeroR (majority-class) baseline, with the largest relative gains on the hardest problems: at least 33% relative improvement for the pitch-accent-only binary task and 126% for the full 8-way task. Top average test accuracy (TATA) is 0.31 for the 8-way tune distinction and 0.45 for the 5-cluster distinction using linear probes; the binary rising-vs-falling distinctions reach 0.74-0.89 TATA, with the highest accuracy (about 0.90) on the coarsest contrast (high-rising vs. low-falling). Unquantized embeddings match or exceed every codebook’s codewords, and among the codebooks, Codebook 1 (the first residual-VQ codebook, not the WavLM-distilled Codebook 0) consistently outperforms Codebook 0. Nonlinear probes improve over linear ones only slightly for the binary tasks (up to 4.8% relative gain) but substantially for the 5-cluster problem (24.4% relative gain), though even the best nonlinear probes remain far below the near-ceiling accuracy humans show in the source production/perception study. The confusion matrix for the 5-cluster problem mirrors a confusion pattern already present in the original human-imitation data, where the lhh cluster overlaps with tokens of hhh/hhl.
Novelty Assessment
The contribution is empirical and diagnostic rather than architectural: the paper applies an established interpretability technique (linear/nonlinear probing, following Alain and Bengio 2016) to a new question, distributed intonational encoding, on a specific production codec (Mimi) using an existing, independently validated labeled dataset. No new model, training method, or dataset is introduced. The genuine contribution is the finding itself: that the codebook explicitly aligned to a “semantic” self-supervised model (WavLM) is not where pitch-tune information is best represented, which complicates the semantic/acoustic codebook dichotomy that is often assumed in the neural-codec and spoken-language-model literature.
Field Significance
moderate — This paper provides a targeted, negative-leaning data point about a widely used production codec (Mimi, underlying Moshi and related speech-LM systems): it shows that the codebook explicitly distilled from a “semantic” self-supervised model does not best capture a pragmatically important prosodic distinction, and that discrimination of fine-grained tune categories remains well below human performance. Its scope is narrow (one codec, one language, a small scripted/imitation corpus), so its main contribution is as a cautionary probe result and a call for prosody-aware benchmarks rather than a generalizable finding across codecs or languages.
Claims
- supports: Intonational pitch-pattern information in a neural audio codec’s discrete representation is distributed across multiple codebooks rather than isolated to a single one.
Evidence: Probes trained on Mimi’s Codebook 1 through Codebook 2 (and beyond) recover nuclear-tune information above chance, and Codebook 1 consistently outperforms Codebook 0 across all five classification problems tested. (§3, Figure 1)
- contradicts: A codec codebook explicitly distilled from a self-supervised “semantic” model is not necessarily the codebook that best encodes prosodically and pragmatically relevant information.
Evidence: Mimi’s Codebook 0, distilled via alignment to WavLM and labeled “semantic” in the Mimi/Moshi literature, is outperformed by Codebook 1 (an ordinary residual-VQ codebook with no such alignment) on every nuclear-tune classification problem tested. (§4)
- complicates: Discrete neural codec tokenizations approximate but substantially underperform human-level discrimination of fine-grained intonational categories, even though coarse pitch distinctions are well preserved.
Evidence: TATA for the 8-class and 5-class nuclear-tune problems is only 0.31 and 0.45 respectively (versus near-ceiling human performance reported in the source production/perception study), while the coarse high-rising vs. low-falling binary distinction reaches 0.74-0.89 TATA. (§3, §4, Figure 3)
- complicates: Standard self-supervised speech representation benchmarks may not adequately validate whether a model’s representations capture pragmatically relevant prosodic distinctions, even when the model performs well on paralinguistic tasks generally.
Evidence: WavLM, the model Mimi’s “semantic” Codebook 0 is distilled from, was evaluated on SUPERB, which includes only emotion recognition among paralinguistic tasks, and emotion-recognition performance there is reported to correlate strongly with speaker identification/verification performance rather than with pragmatic, speaker-independent prosody. (§4)
Limitations and Open Questions
The evidence base is narrow: a single codec (Mimi), a single language (American English), and a small, largely scripted or imitation-elicited corpus (48 pitch-resynthesized base recordings plus 4,608 imitations of those recordings, spanning only six model sentences total). Results may not generalize to other codecs, languages, or spontaneous conversational speech.
The paper itself notes that the better performance on edge-tone distinctions versus pitch-accent distinctions could suggest that Mimi-based spoken dialogue models are less precise at encoding pragmatic distinctions tied to pitch accent (e.g., focus, givenness), but the authors state this conclusion is difficult to confirm given the absence of benchmarks that test pragmatic reasoning grounded in intonation. The authors also note they do not know how well represented the five robust tune clusters were in Mimi’s original training data, leaving open whether the accuracy gap versus human performance reflects a representational limitation of the codec architecture or simply a data-coverage gap.
Wiki Connections
- Neural Audio Codec — directly probes what the Mimi codec’s discrete codebook hierarchy encodes, providing evidence that its “semantic” (WavLM-distilled) codebook is not the primary locus of pitch-tune information.
- Spoken Language Model — analyzes a tokenization component (Mimi) that underlies production spoken dialogue models such as Moshi and Sesame’s conversational model, with implications for their prosodic and pragmatic fidelity.
- Evaluation Metrics — argues that existing self-supervised speech benchmarks (SUPERB, SUPERB-SG) lack tasks that would validate prosody-based pragmatic encoding, and calls for extending such benchmarks.