arXiv · 2026 · Preprint
Shih-Heng Wang et al. (University of Southern California) · → Paper · Demo: ? · Code: ✗
Applies sparse autoencoders to decompose neural audio codec representations into interpretable sparse activations, using accent classification as a case study to quantify and compare how interpretably four widely used codecs encode paralinguistic information.
Problem
Neural audio codecs (NACs) have become a standard component of modern speech systems, serving as compact discrete representations for text-to-speech and speech foundation models, and prior work has benchmarked their performance on downstream speech understanding and generation tasks and probed their robustness and generalization. However, how NAC representations internally encode linguistic and paralinguistic information remains largely unexamined. This gap in mechanistic understanding limits the ability to deploy NACs responsibly in sensitive domains such as healthcare and assistive technology, where interpretability of what a representation encodes is important for trustworthy decision-making. The paper targets accent as a first case study because it is a paralinguistic attribute shaped by an entangled mix of speaker characteristics, phonetic realization, and contextual variation, making it a demanding test of whether codec representations can be decomposed into interpretable, attribute-specific structure.
Method
The framework has three stages. First, for a given NAC, the waveform is encoded and quantized, then the quantized codes are decoded back to a continuous representation, which is mean-pooled over time to obtain a single utterance-level vector (Sec. 2.1). Second, a Top-K sparse autoencoder (SAE) with a linear encoder/decoder and no bias in either layer is trained on these utterance-level vectors with a reconstruction (MSE) objective, projecting into a higher-dimensional latent space and retaining only the k largest activations (Sec. 2.2). Third, the resulting sparse activation is decomposed into two complementary views for analysis: a position feature that binarizes which latent dimensions are active, and a magnitude feature that keeps only the sorted activation strengths of the active dimensions while discarding their identity (Sec. 2.3). Logistic regression classifiers are trained separately on the full sparse activation, the position-only feature, and the magnitude-only feature to perform binary accent classification, letting the authors attribute accent-relevant information to either “where” or “how strongly” a codec’s sparse code activates.
To make interpretability comparable across codecs whose raw representations carry different amounts of accent information to begin with, the authors define a relative performance index, delta-F1, as the difference between a configuration’s classification F1 and a codec-specific reference F1 obtained by training a classifier directly on the un-decomposed representation (Sec. 2.4). A higher (less negative) delta-F1 indicates that accent information survives sparse decomposition with little loss, which the authors interpret as higher interpretability; a larger drop indicates the information is smeared across many dimensions and resists sparse decomposition.
The framework is applied to four off-the-shelf, frozen NACs treated purely as feature extractors: EnCodec (24 kHz, tested at 1.5/6/12 kbps), DAC (24 kHz), SpeechTokenizer (16 kHz), and Mimi (24 kHz). EnCodec and DAC are characterized as acoustic-oriented codecs, while SpeechTokenizer and Mimi are characterized as phonetic-oriented because they are distilled from self-supervised speech models (HuBERT and WavLM respectively) during pretraining. Each codec is evaluated under 16 SAE configurations spanning four latent-size ratios (2, 5, 10, 20 times the codec’s native dimensionality) and four sparsity levels (50%, 25%, 10%, 5% of dimensions active), using data from the Vox-Profile benchmark’s self-reported English accent labels, split into two binary tasks (US vs. UK, US vs. Non-US-UK) with speaker-disjoint train/validation/test splits inherited from Vox-Profile.
Key Results
Across the 16 SAE configurations, DAC achieves the highest interpretability in the US vs. UK setting (ranking first in 13 of 16 configurations), while SpeechTokenizer leads in the US vs. Non-US-UK setting (ranking first in 14 of 16 configurations); Mimi and EnCodec trail both settings. This ranking is not predicted by raw information content: Table 1 shows the phonetic-oriented codecs (Mimi, SpeechTokenizer) retain more accent information in their un-decomposed representations (reference F1 up to 92.44% for SpeechTokenizer in US vs. UK) than the acoustic-oriented codecs, yet that does not translate into higher interpretability after sparse decomposition.
The position-versus-magnitude decomposition (Table 2, at 5% sparsity) shows a consistent split by codec type: for the acoustic-oriented codecs (EnCodec, DAC), delta-F1 computed from magnitude-only features is less negative than from position-only features, meaning accent information is carried more by activation strength. For the phonetic-oriented codecs (Mimi, SpeechTokenizer), the pattern reverses: position-only features outperform magnitude-only features, meaning accent information is carried more by which dimensions activate. EnCodec at 6 kbps shows the largest degradation from position-only features (down to -45.06 at q=20), which the authors trace to a highly non-uniform activation density across the latent space (Fig. 4), indicating the SAE fails to spread information evenly across dimensions for this configuration.
A separate bitrate analysis of EnCodec (Table 3) finds that although the three bitrate variants (1.5/6/12 kbps) achieve similar reference F1 (Table 1), interpretability declines as bitrate increases: the 1.5 kbps variant shows the smallest delta-F1 drop (under 10%), while the 6 and 12 kbps variants drop by roughly 15-24%.
Novelty Assessment
The architectural machinery, Top-K SAEs trained with a reconstruction objective, is adopted directly from prior SAE literature (OpenAI’s Top-K SAE implementation) rather than proposed here; the contribution is not a new model. The genuine novelty is methodological: a relative performance index (delta-F1) that normalizes for each codec’s differing baseline information content, paired with a position/magnitude decomposition that turns a single interpretability score into a diagnostic of how information is encoded. This is best characterized as an evaluation-contribution (a new interpretability measurement protocol) combined with an empirical-benchmark study (systematic comparison across 4 codecs x 16 SAE configurations x 2 classification tasks). The scope is narrow by design: a single case study (English accent, binary classification) on a single institution’s compute budget, explicitly framed by the authors as a first step toward a broader interpretability framework for other paralinguistic attributes.
Field Significance
moderate — This paper contributes a normalized, comparable methodology for measuring how interpretably neural audio codec representations encode a paralinguistic attribute, and demonstrates that raw information content (reference classification accuracy) and interpretability (sparse decomposability) are distinct properties that can diverge across codecs. It also provides a concrete mechanistic hypothesis, that acoustic-oriented codecs encode accent in activation magnitude while phonetically distilled codecs encode it in activation position, that is falsifiable and could be tested on other paralinguistic attributes or codec families.
Claims
- supports: How much paralinguistic information a codec’s representation carries and how interpretably (sparsely) that information can be decomposed are distinct properties, and a codec can rank highly on one while trailing on the other.
Evidence: Phonetic-oriented codecs (Mimi, SpeechTokenizer) retain more raw accent information than acoustic-oriented codecs (EnCodec, DAC) per reference F1 in Table 1, yet DAC and SpeechTokenizer, not Mimi, achieve the highest sparse-decomposition interpretability across configurations (§4.1, Table 1).
- supports: Codecs trained with different objectives (acoustic reconstruction vs. distillation from self-supervised phonetic representations) encode a given paralinguistic attribute through different structural mechanisms within their representation space.
Evidence: At 5% sparsity, magnitude-only features outperform position-only features for the acoustic-oriented codecs (EnCodec, DAC), while position-only features outperform magnitude-only features for the phonetic-oriented codecs (Mimi, SpeechTokenizer) (§4.2, Table 2).
- complicates: Lowering a neural codec’s bitrate can improve the interpretability of paralinguistic information in its representation even when it has little effect on how much of that information is present.
Evidence: Across the three EnCodec bitrate variants, reference F1 (raw information content) is similar (Table 1), but delta-F1 (interpretability) is markedly better at 1.5 kbps (drop under 10%) than at 6 or 12 kbps (drops of roughly 15-24%) (§4.3, Table 3).
- complicates: Sparse-autoencoder-based interpretability measurements can be confounded by a codec-specific failure to decompose information evenly across latent dimensions, independent of how much task-relevant information the representation actually contains.
Evidence: EnCodec at 6 kbps shows a markedly non-uniform activation density across its 16 grouped latent dimensions compared to other codecs, coinciding with its largest position-only delta-F1 degradation (§4.2, Figure 4).
Limitations and Open Questions
The case study is limited to a single paralinguistic attribute (English accent, in two binary settings) using self-reported labels from one benchmark (Vox-Profile), and evaluates only four codecs at fixed configurations; the authors themselves frame this as a first step rather than a validated general-purpose interpretability protocol.
The study does not test whether the position/magnitude encoding hypothesis generalizes to other paralinguistic attributes (emotion, speaking style) or to non-English accents. Source code was not available at the time of the preprint (promised for a camera-ready release), limiting independent verification of the SAE training and evaluation pipeline. The bitrate analysis is conducted only for EnCodec and is not replicated across the other three codecs, leaving open whether the bitrate-interpretability relationship is specific to EnCodec’s architecture or a broader phenomenon.
Wiki Connections
- Neural Audio Codec — proposes an interpretability measurement framework applied directly to four widely used neural audio codecs, probing what their discrete/continuous representations encode rather than how well they compress or reconstruct audio.
- Evaluation Metrics — introduces the relative performance index (delta-F1) as a normalized metric for comparing interpretability across representations with differing baseline information content.
- Self-Supervised Speech — centers its core hypothesis on the distinction between codecs distilled from self-supervised phonetic models (SpeechTokenizer from HuBERT, Mimi from WavLM) and purely acoustic codecs, finding this distinction predicts how accent information is structurally encoded.
- Do Neural Codecs Generalize? A Controlled Study Across Unseen Languages and Non-Speech Tasks — a companion study by the same first author probing codec generalization across languages and domains; this paper extends the group’s broader codec-analysis agenda into representation interpretability.
- Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis — a related paper by a co-author studying accent-related phenomena in speech synthesis, complementing this paper’s focus on how accent is encoded in codec representations rather than generated.
- Vox-Profile — supplies the English accent-labeled dataset and speaker-disjoint splits used for both SAE training and the downstream accent classification tasks.
- Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers — cited as an example of the downstream neural codec language models that motivate the need for interpretable codec representations.
- Discrete Audio Tokens: More Than a Survey! — cited as prior benchmarking work on discrete audio tokens that the paper positions its interpretability framework against, noting such work characterizes codec performance without explaining internal encoding mechanisms.
- DASB - Discrete Audio and Speech Benchmark — cited alongside other codec benchmarking efforts as prior work evaluating NAC representations on downstream tasks without addressing their internal interpretability.