arXiv · 2026 · Preprint
Xuan Shi et al. (University of Southern California / Dolby Laboratories) · → Paper · Demo: ? · Code: ?
Systematically probes what four widely-used speech codecs (EnCodec, DAC, MIMI, MIMO) actually encode, finding that their “semantic” tokens capture phonetic (speech-production) structure far more than lexical-semantic meaning.
Problem
Speech tokenizers are the interface between raw audio and large language models in multimodal systems, and the field has converged on a two-stream framing in which a “semantic” codebook is meant to carry linguistic content while remaining acoustic layers carry residual detail. However, the paper argues that the term “semantic” as used in this literature is borrowed loosely from self-supervised speech representation work without verifying that it actually reflects lexical meaning rather than phonetic (speech-production) similarity. If codec tokens described as “semantic” are in fact phonetic, the resulting mismatch with text-derived semantic tokens could explain degraded performance in multimodal LLMs on speech understanding tasks. No prior work had systematically probed this distinction across multiple speech codec architectures using both functional and physiological evidence.
Method
The paper first draws an explicit conceptual distinction between semantic (lexical meaning, e.g. “big” and “large” are close) and phonetic (speech production similarity, e.g. “accept” and “except” are close) information, then probes four representative codecs spanning different design philosophies: EnCodec (pure reconstruction, RVQ), DAC (reconstruction with codebook and activation improvements), MIMI (the Moshi dialogue system’s codec, whose first RVQ layer is distilled from WavLM to carry a “semantic” signal), and MIMO (the MiMo-Audio tokenizer, trained jointly on reconstruction and ASR alignment).
Three complementary probing analyses are conducted per codec layer. First, a functional probing task extends a prior SSL probing method to codecs: word segments are extracted from LibriSpeech, paired into synonym sets (via WordNet) and near-homophone sets (via CMU Pronouncing Dictionary phonemes and Levenshtein distance), and the Euclidean distance between accumulated per-layer codec features is compared across pair types, with larger synonym-pair distances relative to homophone-pair distances indicating more semantic (rather than phonetic) content. Second, an articulatory phonetic probing analysis correlates codec features with Vocal Tract Distance (VTD) features extracted from real-time MRI recordings of the vocal tract (75-Speaker dataset and its Annot-16 annotated subset) using Projection Weighted Canonical Correlation Analysis (PWCCA), providing a physiologically grounded, non-proxy measure of phonetic content. Third, a cross-modal semantic alignment analysis measures Centered Kernel Alignment (CKA) between decoded speech-token and text-token latent representations for MIMI and MIMO (chosen because both target conversational-AI use and have accessible compatible LLMs), with a random-permutation baseline used to control for spurious geometric alignment.
Key Results
Across all four codecs, feature distances between synonym pairs collapse toward (or below) the random baseline as codebook depth increases, while phonetic (near-homophone) and speaker-related distances remain elevated, indicating that codec representations preserve substantially more phonetic than semantic information. EnCodec and DAC show this “fading” of distinguishable semantic content in an drastic and gradual pattern respectively, consistent with their purely reconstruction-based training objectives. MIMI and MIMO instead show phonetic, semantic, and speaker information accumulating with depth, but MIMI converges earlier and skews more strongly phonetic, which the paper attributes to its first codebook layer being distilled from WavLM. The articulatory (VTD-correlation) analysis cross-validates this pattern: EnCodec and DAC show declining phonetic correlation with depth while MIMI and MIMO show accumulation, confirming that the phonetic dominance reflects genuine speech-production structure rather than an artifact of the proxy word-pair task. A separate analysis of MIMI’s first codebook layer shows that removing the WavLM-distilled layer still leaves the remaining acoustic layers accumulating substantial phonetic information, and that the WavLM distillation itself injects phonetic rather than semantic knowledge. In the cross-modal alignment analysis, MIMI and MIMO both yield CKA scores well below 1.0 (0.329 and 0.122 respectively) against text-token representations, and the improvement over a random-permutation baseline is modest in both cases (Δ = 0.087 for MIMI, Δ = 0.054 for MIMO), indicating weak structural alignment between speech-codec and text-token spaces; the paper notes MIMI’s higher raw CKA is likely inflated by the low effective dimensionality inherited from its WavLM-distilled layer, making the baseline-corrected Δ the fairer comparison.
Novelty Assessment
The contribution is not architectural: no new codec, tokenizer, or model is proposed. The novelty is methodological and conceptual — extending a prior SSL-probing methodology to neural speech codecs, introducing a physiologically grounded articulatory probe (VTD correlation via rt-MRI) as an independent cross-check on the functional word-pair probe, and applying cross-modal CKA alignment with a random-permutation control to speech-text token spaces. The clearest conceptual contribution is terminological: showing that codecs’ “semantic” codebooks, including one distilled directly from an SSL model, encode predominantly phonetic rather than lexical-semantic structure, which challenges how the term “semantic tokens” is used across the speech-LLM tokenizer literature. The empirical scope is narrow (four codecs, English-only word pairs, two codecs for the CKA analysis), so the findings are best read as a systematic diagnostic study rather than a general theory of codec representation.
Field Significance
moderate — the paper does not introduce new methods but provides converging, physiologically grounded evidence for a specific and consequential claim: that codec tokens labeled “semantic” in the speech-LLM literature primarily encode phonetic rather than lexical-semantic structure, even when explicitly distilled from an SSL model intended to carry semantic content. This diagnostic result and its accompanying probing methodology (functional word-pair probing plus articulatory rt-MRI correlation plus cross-modal CKA) can inform how future speech tokenizers are evaluated and designed, particularly for use in multimodal LLMs.
Claims
- complicates: Speech codec representations labeled “semantic” (including those explicitly
distilled from self-supervised models) may primarily encode phonetic/speech-production structure
rather than lexical-semantic meaning, complicating claims that such codebooks capture linguistic
content suitable for text-aligned semantic modeling.
Evidence: Across EnCodec, DAC, MIMI, and MIMO, synonym-pair feature distances fade toward the random baseline with codebook depth while near-homophone and speaker-related distances remain elevated, and MIMI’s WavLM-distilled first layer is shown to inject phonetic rather than semantic knowledge when isolated from the remaining layers. (§3.1, §3.2, Figures 2-4)
- supports: Physiologically grounded articulatory measurements can independently corroborate
functional (proxy-task) evidence about what information a learned speech representation encodes,
strengthening claims that would otherwise rest on indirect distance-based probes alone.
Evidence: Vocal Tract Distance features extracted from real-time MRI, correlated with codec features via PWCCA, reproduce the same per-codec pattern (fading phonetic correlation for EnCodec/DAC, accumulating correlation for MIMI/MIMO) found in the word-pair distance probing. (§2.4, §3.2, Figure 3)
- complicates: Cross-modal structural alignment between speech-codec and text-token
representation spaces, as measured by representational similarity metrics, remains weak even in
codecs explicitly designed for joint speech-text LLM integration.
Evidence: CKA between MIMI/MIMO speech-token representations and text-token representations is well below 1.0 (0.329 and 0.122 respectively) with only modest gains over a random-permutation baseline (Δ = 0.087 and 0.054), and MIMI’s higher raw score is attributed to the low effective dimensionality of its WavLM-distilled layer rather than genuine alignment. (§3.3)
- refines: The common two-stream codec design that separates a “semantic” codebook layer
(typically distilled from an SSL model) from residual “acoustic” layers should be understood as
separating phonetic/production information from acoustic detail, not semantic/lexical
information from acoustic detail.
Evidence: Isolating MIMI’s WavLM-distilled first codebook layer and comparing its VTD correlation to the remaining acoustic layers shows the distilled layer injects significant phonetic-related knowledge, not lexical-semantic knowledge. (§3.2, Figure 4)
Limitations and Open Questions
The probing methodology is restricted to English word pairs and only four codec
architectures, and the cross-modal semantic alignment analysis (CKA) is run on only two codecs (MIMI, MIMO) whose compatible LLMs were accessible; results for the remaining codecs on that axis are not established.
The paper relies on indirect proxy tasks (word-pair distances, CKA scores) for two of its three analyses, with the articulatory rt-MRI probe as the sole physiologically direct measurement, and that probe itself is drawn from existing datasets (75-Speaker, Annot-16) rather than newly collected data spanning the same word-pair vocabulary used in the functional probing task. The paper also does not propose or test any concrete architectural remedy; it outlines two directions (distilling from text-semantic models such as LLM embeddings, or adding explicit semantic constraints to the training objective) without implementing or evaluating either.
Wiki Connections
- Neural Audio Codec — directly probes the internal representations of four neural speech codecs (EnCodec, DAC, MIMI, MIMO) to characterize what information their codebook layers actually encode.
- Spoken Language Model — motivates the analysis by arguing that a phonetic-versus-semantic mismatch in codec tokens can degrade multimodal LLM performance on speech understanding tasks.
- Self-Supervised Speech — traces MIMI’s “semantic” first codebook layer to WavLM distillation and shows that this distillation injects phonetic rather than semantic structure.
- Disentanglement — evaluates whether codecs’ semantic-acoustic codebook separation (as in MIMI) actually achieves the intended semantic/phonetic/acoustic distinction it claims to.
- Evaluation Metrics — introduces a probing methodology combining word-pair distance analysis, articulatory (rt-MRI) correlation, and CKA alignment as diagnostic tools for codec representation quality.
- High Fidelity Neural Audio Compression — one of the four codecs directly probed; used as the pure-reconstruction baseline with no explicit semantic modeling.
- Moshi — the dialogue system whose codec (MIMI) is analyzed in detail, including an isolated study of its WavLM-distilled first codebook layer.
- CosyVoice — cited as part of the broader trend toward codecs built on “supervised semantic tokens,” the framing this paper’s findings complicate.