LREC 2026 · 2026 · Conference
Naber et al. (University of Stuttgart) · → Paper · Demo: ✗ · Code: ✓
Introduces a three-probe framework (token-distribution divergence, bag-of-token classifiers, token-phoneme co-occurrence analysis) for reading speaker and paralinguistic information directly off discrete speech tokens, and applies it to tokenizer outputs and LM-generated tokens of CosyVoice2 and Spark-TTS, finding that gender is strongly recoverable in both but organized differently.
Problem
Discrete speech tokens are the interface through which speech information enters and leaves LLM-based speech generation pipelines, yet prior analyses either studied a single tokenization stage in isolation (phonetic analysis of HuBERT units, interpretability of acoustic codec tokens) or measured token quality indirectly through downstream tasks such as phone classification or speaker verification. The authors argue that this leaves two gaps: no direct comparison of token representations across pipeline stages (tokenizer output versus tokens the LM itself generates), and little evidence on how stable attribute-related token patterns are across datasets and across architectures with different speaker-conditioning designs. This matters for controllable generation, where speaker information may be spread across semantic tokens, reference prompts, and separate global embeddings.
Method
The study probes two LLM-based TTS pipelines that the authors call spoken language models. In CosyVoice2, a supervised semantic tokenizer produces 25 tokens/s, a Qwen2.5-0.5B text-speech LM predicts semantic tokens (in zero-shot mode conditioned on prompt text and prompt speech tokens from a reference utterance), and a causal flow-matching model plus vocoder render audio using a reference speaker embedding. In Spark-TTS, the BiCodec tokenizer emits 50 semantic tokens/s plus a fixed-length set of global tokens; the LM is conditioned on target text and global tokens from the reference, and the BiCodec decoder reconstructs audio from semantic and global tokens (§3.1, §3.2).
For each utterance three token sets are extracted: tokenizer tokens (the tokenizer applied to the original audio), zero-shot tokens (LM generation from the utterance text with the same utterance as reference), and text-only tokens (LM generation without reference audio) (§5.1). Gender labels come from the datasets; valence, arousal, and dominance (VAD) labels are produced automatically by a wav2vec2-based dimensional emotion model and split into high/low groups at the 0.2 and 0.8 quantiles (§5.2, §6.2).
Three probes are applied (§4):
- Distributional divergence: Jensen-Shannon divergence (JSD) between token-frequency distributions of the two attribute groups, compared against a 20-shuffle label-permutation baseline, with per-token frequency differences identifying marker tokens (§4.2).
- Token-based classifiers: logistic regression on bag-of-token features in three modes (raw counts BOW, L1-normalized shares SHARE, binary presence SET), evaluated with balanced accuracy on 80/20 speaker-disjoint splits; coefficients identify predictive tokens (§4.3).
- Attribute-conditioned representation analysis: tokenizer tokens are aligned to Montreal Forced Aligner phones by center-time sampling, and the per-token phone distributions of the two attribute groups are compared by cosine similarity and a signed bias score over the most divergent phone (§4.1.1, §4.4). This probe is limited to tokenizer tokens because only they have an audio-derived alignment.
Follow-up controls rerun the probes on shared-text subsets (matched content), remove silence-aligned positions, and run a small token/reference swap generation probe in which one expert listener labels perceived gender for 25 samples per setting (§7.2, §7.2.1, §7.3).
Key Results
Gender is recoverable from tokenizer and zero-shot tokens in both systems, and absent from text-only tokens. CosyVoice2 reaches balanced accuracy of 0.909 to 0.977 across all three feature modes and both datasets for tokenizer and zero-shot tokens (Table 1). Spark-TTS reaches 0.789 to 0.894 with BOW and SET, but SHARE is lower (0.717 and 0.842 on LibriTTS-R) and at chance on VCTK (0.500 tokenizer, 0.529 zero-shot) (Table 2). Text-only tokens fall to chance on VCTK for both models (0.49 to 0.50) and show JSD equal to the shuffled baseline (§6.1.1, §6.1.2), which the authors read as evidence that the gender signal enters through speaker conditioning rather than text.
Stage stability differs by model. Cosine similarity of SHARE classifier weights between tokenizer and zero-shot tokens is 0.8940 (LibriTTS-R) and 0.9625 (VCTK) for CosyVoice2, versus 0.7745 and 0.5223 for Spark-TTS. CosyVoice2 shows the same top marker tokens (for example 1950 and 4137 for male, 4299 and 2112 for female) across datasets and stages; Spark-TTS has no stable markers on VCTK (§6.1.2, Tables 1 and 2). In Spark-TTS, zero-shot tokens give stronger gender separation than tokenizer tokens (JSD 0.0162 vs. 0.0079 on LibriTTS-R; 0.0227 vs. 0.0107 on VCTK); CosyVoice2 zero-shot JSD is also higher on LibriTTS-R (0.0614 vs. 0.0385) but slightly lower on VCTK (0.1080 vs. 0.1183) (§6.1.1).
Controls change the Spark-TTS picture more than CosyVoice2’s. On VCTK shared-text subsets, CosyVoice2 is nearly unchanged (tokenizer 0.977 / 0.946 / 0.969), while Spark-TTS SHARE rises from 0.500 to 0.806 for tokenizer and from 0.529 to 0.703 for zero-shot tokens, suggesting lexical variability had masked a normalized-share code. On the LibriTTS-R shared-text subset (2,476 entries) Spark-TTS drops to 0.589 / 0.574 / 0.587 for tokenizer tokens; the CosyVoice2 LibriTTS-R subset has only 314 entries and the authors caution against reading it (Table 4). Removing silence-aligned positions leaves CosyVoice2 LibriTTS-R almost unchanged and keeps VCTK well above chance (BOW 0.977 to 0.941), whereas Spark-TTS LibriTTS-R SHARE collapses from 0.717 to 0.500 while BOW and SET barely move (Table 5). The phone-alignment probe finds that the most gender-biased Spark-TTS tokens are dominated by the pause phone /sp/ on both datasets; CosyVoice2 retains segmental male-biased associations (/AO1/, /Z/, /DH/, /AA1/) on LibriTTS-R (§6.1.3, §7.2.1).
VAD signals are weak. Best balanced accuracy is mostly between 0.50 and 0.59; the clearest exception is dominance on VCTK (CosyVoice2 0.635 / 0.637, Spark-TTS 0.688 / 0.587 for tokenizer / zero-shot), and CosyVoice2 text-only dominance keeps a small residual 0.549 (Table 3). Tokenizer-to-zero-shot SHARE weight similarity for VAD ranges 0.411 to 0.695 (CosyVoice2) and 0.231 to 0.516 (Spark-TTS) (§6.2).
Generation swap probe. With mismatched token sequence and reference embedding, CosyVoice2 output gender is consistently determined by the reference embedding, while Spark-TTS outputs often mix male and female characteristics (§7.3). No counts or agreement statistics are reported for this probe, and it uses a single expert listener.
Novelty Assessment
The individual probes are standard: JSD with a permutation baseline, bag-of-token logistic regression, and a phone co-occurrence analysis adapted from prior property-neuron work on self-supervised speech transformers. The contribution is their combination into a coarse-to-fine framework and, more distinctively, the side-by-side comparison of tokenizer tokens against LM-generated tokens under reference and no-reference conditioning, which isolates what the conditioning path injects. The empirical scope is narrow (two systems, two English read-speech corpora, gender plus model-labelled VAD), and all three probes use unigram token statistics, so contextual or sequential encodings are not tested. The architectural interpretation (temporal resolution and the BiCodec global-token factorization explaining Spark-TTS’s diffuse code) is offered as plausible rather than tested.
Field Significance
Moderate. The paper provides a reusable, interpretable recipe for comparing what discrete speech tokens carry across pipeline stages and across tokenizers, and demonstrates that a tokenizer designed to factor speaker identity into separate global tokens still leaks recoverable gender into its semantic stream, while a tokenizer trained only with ASR supervision carries a strong, stable gender code that the downstream decoder then overrides with the reference embedding. Its evidence is correlational and limited to two systems, so it serves mainly as a measurement tool and a cautionary data point on speaker leakage in semantic tokens.
Claims
- complicates: Supervised semantic speech tokens intended to carry linguistic content retain strongly recoverable speaker-gender information.
Evidence: Bag-of-token logistic regression on CosyVoice2 tokenizer tokens (trained with ASR supervision) reaches 0.909 to 0.977 balanced accuracy for gender on speaker-disjoint splits of LibriTTS-R and VCTK, and the signal survives text balancing and silence removal on VCTK. (§6.1.2, §7.2, §7.2.1, Tables 1, 4, 5)
- complicates: Factorizing speaker attributes into separate global tokens does not fully remove speaker information from the semantic token stream of a speech codec.
Evidence: Spark-TTS BiCodec semantic tokens still yield 0.789 to 0.894 balanced accuracy for gender with count and presence features, and in token/reference swap generation the outputs mix male and female characteristics, indicating both streams influence realized gender. (§6.1.2, §7.3, §7.4, Table 2)
- supports: In LLM-based zero-shot TTS, speaker-related information in generated speech tokens is introduced by the reference speech conditioning rather than by the text.
Evidence: For both CosyVoice2 and Spark-TTS, zero-shot tokens show gender JSD far above the shuffled baseline and high classifier accuracy, while text-only generations have JSD equal to baseline and balanced accuracy at chance (0.49 to 0.50 on VCTK). (§6.1.1, §6.1.2, Table 4)
- refines: Speaker attributes can be encoded in discrete speech tokens in qualitatively different forms, ranging from a stable set of marker tokens to diffuse pause- and prosody-aligned count cues, so a single probe type can understate what tokens carry.
Evidence: Spark-TTS gender recoverability is near chance under normalized token-share features on VCTK (0.500) but 0.789 under raw counts; its most biased tokens are /sp/-dominated, and removing silence-aligned positions collapses LibriTTS-R SHARE accuracy from 0.717 to 0.500 while leaving BOW and SET nearly unchanged, unlike CosyVoice2. (§6.1.3, §7.2.1, Tables 2 and 5)
- complicates: Information that is recoverable from speech tokens is not necessarily what controls the corresponding attribute in synthesized speech.
Evidence: Although CosyVoice2 tokens carry a strong gender code, swapping reference embeddings yields output gender that is consistently determined by the reference embedding in both mismatched settings, as judged by an expert listener. (§7.3)
Limitations and Open Questions
Warning
All probes are correlational and use unigram token statistics; the only interventional evidence is a qualitative 25-sample-per-setting swap test judged by a single expert listener with no reported counts, so the paper cannot establish which tokens the models actually use to encode or control an attribute.
- Only two systems are probed, both built on the same Qwen2.5-0.5B LM backbone family, and only English read speech (LibriTTS-R, VCTK); generalization to acoustic codecs, multi-codebook tokenizers, or full-duplex speech LMs is untested (§Limitations).
- Gender is treated as a binary dataset label, and VAD labels come from an automatic emotion model, which the authors note can introduce labeling and speaker confounds (§6.2, §Limitations, §Ethical Statement).
- The phone-alignment probe applies only to tokenizer tokens, since generated tokens have no audio-derived alignment (§5.1).
- The LibriTTS-R shared-text control for CosyVoice2 rests on 314 entries, which the authors flag as unreliable (Table 4).
- Zero-shot tokens use the same utterance as reference and target, so it is unclear how the conditioning effect would look with a different reference utterance.
Wiki Connections
- Neural Audio Codec — provides a probing framework for what speaker and paralinguistic attributes discrete speech tokenizers (CosyVoice2’s semantic tokenizer, Spark-TTS’s BiCodec) retain.
- Autoregressive Codec TTS — compares tokenizer outputs with tokens generated by LLM-based autoregressive TTS under reference and no-reference conditioning.
- CosyVoice 2 — one of the two probed pipelines; its supervised semantic tokens show a strong, stage-stable gender code.
- Spark-TTS — the second probed pipeline; its BiCodec semantic tokens retain gender in a weaker, pause-dominated form despite global-token factorization.
- LibriTTS — the restored LibriTTS-R variant of this corpus is one of the two probing datasets.
- Bringing Interpretability to Neural Audio Codecs — prior interpretability analysis of acoustic codec tokens that this work extends to semantic tokens and generated token streams.
- DC-Spin — cited as a speaker-invariant tokenizer evaluated through downstream tasks rather than direct token inspection.
- Phonological Tokenizer — cited as a speaker-discarding, prosody-aware tokenizer whose information content was assessed indirectly via downstream tasks.
- Moshi — cited for the view of discrete tokens as an information bottleneck in spoken language model pipelines.