arXiv · 2026 · Preprint
Xiaoran Fan et al. (Fudan University) · → Paper · Demo: ✓ · Code: ✓
A controlled comparison of coupled, semi-decoupled, and fully decoupled speech tokenizers inside LLM-centric speech-language models, finding that decoupling improves cross-modal alignment, plus a multi-token prediction scheme for speech heads and a new speaker-aware role-playing knowledge QA benchmark.
Problem
Prior comparisons of speech tokenizer designs (coupled, semi-decoupled, fully decoupled) are conducted on speech reconstruction tasks in isolation from the downstream speech-language model (SLM) that will actually consume the tokens. This leaves it unclear which tokenizer family is best suited to LLM-centric SLM training, where the tokenizer must support cross-modal alignment with text rather than just accurate audio reconstruction. Separately, speech token sequences are far denser than text token sequences (hundreds of tokens per second of speech vs. roughly 20 text tokens’ worth of information), which the paper argues hampers cross-modal alignment and slows inference. Finally, existing SLMs largely operate with a fixed role and voice, with no systematic benchmark for jointly evaluating knowledge retention and speaker-timbre consistency in role-conditioned spoken QA.
Method
The paper builds a fair SLM comparison framework: a decoder-only Transformer backbone (initialized from Qwen2.5-0.5B-Instruct) is extended with a speech tokenizer, a language head, and a speech head, trained on interleaved text-speech sequences for both TTS and ASR. Tokenizers are grouped into three families: coupled (e.g., Encodec, WavTokenizer, BigCodec, StableCodec), which jointly encode semantic and acoustic content in one stream; semi-decoupled (SpeechTokenizer), which distills semantic content via HuBERT while retaining partial coupling; and fully decoupled (FACodec, from NaturalSpeech 3), which separates speech into independent semantic, prosodic, and timbre subspaces. Each tokenizer is trained into the same SLM shell under matched conditions on LibriTTS to isolate the effect of tokenizer design from confounds like data or model scale.
To address the information-density mismatch between text and speech, the paper introduces multi-token prediction (MTP) into the speech head: rather than predicting one speech token per hidden state (next-token prediction, NTP), each hidden state decodes a group of g adjacent speech tokens (coupled or decoupled) via an extended speech-head weight tensor and a fusion network (MLP) that compresses grouped token embeddings back into a single input vector. Compression ratios from 1x (NTP) to 12x are evaluated.
For speaker-aware generation, a pretrained timbre extractor produces a speaker representation that is prepended to the speech-language context, conditioning generation on speaker identity throughout decoding. Building on this, the paper introduces RoleTriviaQA, a role-playing knowledge QA dataset: TriviaQA text answers are synthesized into speech for 15 Genshin Impact character voices using CosyVoice 2 (10 seen roles for training, 5 held out for out-of-domain evaluation), producing 138,384 training samples and combined in-domain/out-of-domain test sets. Training proceeds in two stages: cross-modal alignment pretraining on a DNSMOS-filtered subset of Emilia merged with LibriTTS (2,495 hours, “Emilia-3.5”) on TTS and ASR objectives, followed by supervised fine-tuning on RoleTriviaQA for joint knowledge and role-voice generation.
Key Results
On LibriTTS test-clean, decoupled (FACodec) and semi-decoupled (SpeechTokenizer) tokenizers substantially outperform coupled tokenizers when trained inside the SLM: most coupled tokenizers achieve synthesis success rates at or below 70%, while FACodec and SpeechTokenizer reach 82-99%. FACodec-based SLMs achieve markedly higher speaker similarity than any coupled or semi-decoupled baseline (SIM 0.47-0.50 vs. 0.06-0.18), despite BigCodec (coupled) achieving better standalone reconstruction quality in isolation.
Increasing MTP compression from 1x (NTP) to 12x on the decoupled tokenizer improves WER from 6.07 to 3.01 (a 48% relative improvement from 3x to 12x alone) while also delivering up to 12x faster decoding, with only a slight UTMOS decline (3.93 to 3.67) and stable speaker similarity. Cross-modal alignment metrics (Table 4) confirm the mechanism: cross-modal cosine similarity between text and speech hidden states rises from 0.967 (3x) to 0.986 (12x) as compression increases, while intra-modal similarities stay stable.
Adding speaker-aware conditioning to the decoupled+MTP configuration improves WER further (e.g., 7.04 to 2.94 at 12x compression with vs. without speaker embedding) and raises SIM substantially (0.47 to 0.60). On RoleTriviaQA, the decoupled SLM reaches 12.0 EM / 23.8 F1 in-domain and 9.0 EM / 17.9 F1 out-of-domain, compared to 5.1 EM / 12.5 F1 (WavTokenizer-v2, coupled) and 3.4 EM / 9.1 F1 (SpeechTokenizer, semi-decoupled) in-domain, and outperforms the text-only Qwen2.5-0.5B-Instruct backbone on out-of-domain EM (9.0 vs. 1.1) while producing spoken output.
Novelty Assessment
The core contribution is empirical and diagnostic rather than a new architecture: a controlled, fair comparison isolating tokenizer coupling degree as the variable of interest inside SLM training, revealing that reconstruction-task performance does not predict SLM-training compatibility. The MTP-for-speech-heads mechanism adapts an established LLM-inference technique (multi-token prediction) to a new context, extending it to both coupled and decoupled token groupings; this is an engineering adaptation of prior work rather than a new prediction paradigm. RoleTriviaQA is a genuine dataset contribution, filling a previously identified gap (no open dataset jointly assessing knowledge correctness and speaker-timbre consistency in spoken QA), though it is synthetically voiced (CosyVoice 2 over game-character reference audio) rather than drawn from real human speakers.
Field Significance
Field Significance
high — This paper provides a systematic, controlled answer to a design question (which tokenizer coupling strategy is best for LLM-centric SLMs) that prior work had only addressed indirectly through reconstruction benchmarks, and demonstrates that the answer differs from what reconstruction-only evaluation would suggest. It packages this finding with a directly reusable technique (MTP for speech heads) and a new benchmark (RoleTriviaQA) for a task category (role-conditioned spoken knowledge QA) with no prior open dataset.
Claims
- supports: The degree to which a speech tokenizer decouples semantic and acoustic information determines its compatibility with LLM-centric speech-language model training, independent of its standalone reconstruction quality.
Evidence: The coupled BigCodec achieves better reconstruction quality than FACodec in isolation, yet when trained jointly inside the SLM, FACodec (fully decoupled) yields far higher synthesis success rate, WER, and speaker similarity (§2.1, Table 1).
- supports: Grouping multiple speech tokens per hidden state (multi-token prediction) can compress the information-density mismatch between speech and text representations, improving both decoding speed and cross-modal alignment.
Evidence: Increasing MTP compression from 1x to 12x on the decoupled tokenizer cuts WER from 6.07 to 3.01, delivers up to 12x faster decoding, and raises cross-modal cosine similarity (STSim) from 0.967 to 0.986 (§4.2, §4.6, Table 1, Table 4).
- supports: Injecting explicit speaker-identity embeddings into the speech-language context improves both speaker consistency and intelligibility in SLM-generated speech.
Evidence: Adding speaker-aware conditioning to the MTP-12H decoupled configuration improves WER from 7.04 to 2.94 and speaker similarity from 0.47 to 0.60 relative to the non-speaker-aware variant (§4.3, Table 2).
- complicates: Speech-tokenizer quality comparisons conducted only on reconstruction tasks can mispredict a tokenizer’s suitability for LLM-centric SLM training.
Evidence: Coupled tokenizers exhibit severe convergence difficulties during joint SLM training (synthesis success rate ≤70% for most coupled baselines) despite performing comparably to decoupled tokenizers on standalone reconstruction metrics (STOI, PESQ, UTMOS) (§4.1, Table 1, Table 5).
Limitations and Open Questions
All quality and intelligibility metrics reported (UTMOS, WER, SIM) are automatic; the paper includes no human listening test (MOS/SMOS) to validate the naturalness or speaker-similarity findings that motivate its core tokenizer-comparison claims.
The comparison is conducted at a single, small backbone scale (Qwen2.5-0.5B-Instruct); whether decoupled tokenizers retain their advantage as SLM backbone size increases is not tested. Training and evaluation are English-only (LibriTTS, Emilia-filtered, TriviaQA-derived), so multilingual generalization of the tokenizer-coupling findings is untested. RoleTriviaQA’s 15 speaker identities are synthetic game-character voices generated by CosyVoice 2 rather than recordings of real individuals, which may not reflect the acoustic diversity or naturalness of real-world speaker identities the paradigm would need to handle in deployment.
Wiki Connections
- Neural Audio Codecs — provides a controlled, SLM-internal comparison of coupled, semi-decoupled, and fully decoupled tokenizer designs, showing standalone reconstruction quality does not predict downstream SLM compatibility.
- Spoken Language Models — proposes a fair evaluation framework for LLM-centric SLM design choices and a multi-token prediction mechanism to address the speech-text information-density mismatch in the speech head.
- Speaker Adaptation — introduces a speaker-aware generation paradigm that injects timbre-extractor embeddings into the SLM context, improving both WER and speaker similarity over non-speaker-aware training.
- Zero-Shot Text-to-Speech — evaluates speaker similarity on held-out unseen speaker identities (5 unseen roles) in addition to seen ones, testing generalization of the speaker-aware conditioning mechanism.
- NaturalSpeech 3 — supplies FACodec, the fully decoupled tokenizer this paper’s entire empirical study is built around and finds most SLM-compatible.
- SpeechTokenizer — used as the semi-decoupled tokenizer baseline, directly compared against coupled and fully decoupled designs.
- WavTokenizer — used as a coupled tokenizer baseline compared unfavorably against decoupled tokenizers in SLM training.
- BigCodec — used as a coupled tokenizer baseline; achieves strong standalone reconstruction but weaker SLM-training compatibility than FACodec.
- CosyVoice 2 — used as the TTS tool to synthesize the RoleTriviaQA dataset’s role-voiced speech answers.
- Moshi — cited as a representative LLM-centric duplex SLM baseline that this work’s speaker/role-conditioning paradigm extends beyond a fixed role and voice.