arXiv · 2026 · Preprint
Hankun Wang et al. · → Paper · Demo: ? · Code: ?
By training identical 1.1B-parameter LMs on the same data under five progressively speech-like modalities (text-BPE → phone-BPE → phone-raw → phone-repeat → speech-HuBERT), this paper isolates that paralinguistic variability in speech tokens (Factor C) is the dominant cause of SLM semantic incoherence — larger than the effect of token phonetic vs. semantic content (Factor A) or sequence length (Factor B).
Problem
End-to-end speech language models (SLMs) that generate speech without text transcription guidance still perform far below text LLMs on semantic coherence tasks. The community lacks a principled understanding of why: three candidate explanations exist — (A) speech tokens are more phonetic than semantic, (B) speech sequences are much longer than text (50 tokens/s vs. ~4.5 tokens/s), and (C) speech tokens retain paralinguistic variation (prosody, timbre, accent) that creates combinatorial explosion in the lexical space. Without isolating these factors, improvements to SLMs remain empirical.
Method
The paper uses a controlled factorial study design. A single architecture (TinyLlama: 22 transformer layers, 32 heads, GQA, 1.1B parameters) is trained from scratch on the same audio corpus (LibriHeavy-large, ~50 k hours) under five different modality representations:
- Text-BPE: 2048-unit SentencePiece subword tokenizer (~4.45 tokens/s). Serves as the topline.
- Text-Raw: Character-level tokenizer (~80 types, ~14.53 tokens/s). Baseline comparison.
- Phone-BPE: BPE over phones with 2048 units (~4.04 tokens/s). Adds Factor A (phonetic).
- Phone-Raw: Raw phone tokens (~80 types, ~9.97 tokens/s). Adds Factor A at finer resolution.
- Phone-Repeat: Phone tokens duration-repeated to 50 Hz (~7.74B tokens, ~50 tokens/s). Adds Factor B (length).
- Speech-HuBERT: Discrete HuBERT-Large clusters (2048 units at 50 Hz, ~7.74B tokens). Adds Factor C (paralinguistics).
Evaluation on four tasks in zero-shot setting:
- sWUGGY (lexical modeling): real vs. pseudo-word likelihood comparison.
- sBLIMP (syntactic modeling): grammatical vs. ungrammatical sentence comparison.
- Topic-StoryCloze (semantic modeling): plausible vs. implausible story continuation.
- Continuation task (generation coherence): free autoregressive generation, scored by Llama-3.1-8B perplexity after Whisper transcription.
Training data per modality is the same underlying speech; only the tokenization changes. All models trained to convergence on validation loss.
Key Results
| Modality | sWUGGY Acc% | sBLIMP Acc% | Topic-SC Acc% | Continuation PPL |
|---|---|---|---|---|
| Text-BPE | 85.1 | 74.9 | 73.6 | 51.3 |
| Text-Raw | 85.6 | 73.3 | 66.0 | 54.6 |
| Phone-BPE | 85.0 | 75.0 | 70.9 | 59.1 |
| Phone-Raw | 85.8 | 74.5 | 66.6 | 69.1 |
| Phone-Repeat | 85.5 | 66.2 | 58.3 | 130.1 |
| Speech-HuBERT | 50.8 | 57.3 | 52.9 | 313.2 |
Key differentials (relative to the prior modality in the chain):
- Factor A (Text → Phone): Near-zero impact on lexical and syntactic tasks; −3.7% on Topic-SC for BPE variant; +26.6% PPL increase for Raw.
- Factor B (Phone-Raw → Phone-Repeat): −11.1% on sBLIMP, −12.5% on Topic-SC, +88.3% PPL. Sequence length severely challenges syntactic and semantic modeling.
- Factor C (Phone-Repeat → Speech-HuBERT): −40.6% on sWUGGY (to near-random), −13.4% on sBLIMP, −9.3% on Topic-SC, +140.7% PPL. Paralinguistic variability is the dominant driver — it creates combinatorial explosion in the lexical space, preventing stable lexical grounding.
Data scaling analysis shows Speech-HuBERT’s sWUGGY slope is essentially zero (models can scale compute but still fail to establish lexical representations), whereas all other modalities show positive scaling.
Layer-wise analysis reveals that BPE-based modalities establish lexical representations in early layers; speech tokens create combinatorial explosion at the lexical level that propagates through syntax and semantics.
Novelty Assessment
The controlled modality-evolving framework is the key methodological contribution: using identical architecture and training data across modalities isolates causal factors cleanly. The finding that Factor C (paralinguistic variability) dominates over Factor A (phonetic vs. semantic representation) is a somewhat counterintuitive result — much prior work has focused on building more semantic speech tokens (reducing Factor A) while this paper shows Factor C is more pressing. The paper is ICASSP 2026 short paper (5 pages), so depth is limited, but the core experimental design and finding are well-executed.
Field Significance
Tip
High — This paper provides the first systematic causal decomposition of why end-to-end speech language models underperform text LLMs on semantic coherence, answering a question the field has approached only empirically until now. The finding that paralinguistic variability (Factor C) is the dominant bottleneck — not the phonetic nature of speech tokens (Factor A) as widely assumed — redirects research priorities toward better paralinguistic suppression in codec and tokenizer design rather than toward more semantic discrete representations. It can serve as a diagnostic reference for any future SLM architecture or tokenization strategy.
Claims
- Paralinguistic variability in speech tokens is the dominant obstacle to lexical modeling in speech language models, causing near-random lexical discrimination accuracy even after training to convergence on 50k hours of audio. (§5.1, Table 4)
- Extending token sequences to match speech frame rates significantly degrades syntactic and semantic modeling in autoregressive LMs, even when the underlying phonetic content remains unchanged. (§5.1, Table 3)
- The phonetic versus semantic nature of tokens has negligible impact on lexical and syntactic modeling when comparing phone-based to text-based representations at equivalent vocabulary sizes. (§5.1, Table 3)
- Speech LMs trained on discrete acoustic tokens show near-zero improvement on lexical tasks as compute scales, whereas all other token representations exhibit positive scaling behavior. (§5.2, Figure 1)
- Stable lexical grounding in early transformer layers is a prerequisite for higher-level syntactic and semantic modeling; speech tokens that create combinatorial explosion at the lexical level propagate failure upward through the layer stack. (§5.3, Figure 3)
Limitations and Open Questions
- Only HuBERT-based discrete speech tokens are tested as the speech modality; codec tokens with decoupled semantic/acoustic streams (e.g., EnCodec RVQ-1, Mimi) are not included, making it unclear whether more semantic-focused codec tokens change the Factor A/C balance.
- The study uses 50 k hours of audiobook speech (LibriHeavy); the results may differ for conversational or in-the-wild speech with more extreme paralinguistic variation.
- The proposed future directions (variable-length tokens, explicit semantic supervision) are stated but not experimentally validated.
- No correction for the fact that speech-modality LMs effectively train on more distinct tokens (Factor B creates longer sequences from the same audio), which could confound the scaling analysis.
Wiki Connections
This paper is a foundational diagnostic for spoken-language-model research, explaining why SLMs underperform text LLMs. It directly informs the design of neural-codec tokens: the result that Factor C dominates motivates designing codecs that better suppress paralinguistic variability in first-layer semantic tokens. The distillation approach used by Mimi and DualCodec (semantic decoupling) is the direction this paper’s findings support. The paper also connects to self-supervised-speech since HuBERT is the speech tokenizer tested. Related to autoregressive-codec-tts because both VALL-E-family models and SLMs use autoregressive LMs over codec tokens and face the same Factor B/C challenges.
Follow-on work building on or citing this paper: FlexiCodec directly addresses the Factor B (sequence length) problem by achieving 6.25 Hz AR tokens; Discrete Representation Inconsistency (DRI) addresses a closely related phenomenon (DRI/context-dependence in RVQ encoding) from the codec training angle; VocalNet addresses speech LLM coherence via MTP rather than tokenizer design; SpeechLM survey synthesizes the broader SpeechLM landscape including the coherence problems diagnosed here; interspeech-2025-1289 addresses Factor B by reducing sequence length at the codec level through entropy-guided temporal allocation.
2502.17239 — Baichuan-Audio: A Unified Framework for End-to-End Speech Interaction