arXiv · 2026 · Preprint

Yuhan Song et al. (Peking University / Tencent WeChat AI) · → Paper · Demo: ? · Code: ✓

A single-codebook, Whisper-initialized speech tokenizer that is trained to predict structured descriptions of content, voice attributes and the acoustic scene instead of plain transcripts. A gated shallow-to-deep feature fusion is added to the encoder. On speech, the tokens give lower reconstruction WER than four single-codebook baselines and lower TTS WER than CosyVoice 2 tokens, and they also cluster non-speech sound events.

Problem

Semantic speech tokenizers built by quantizing an ASR encoder (the CosyVoice 2 tokenizer, GLM-4-Voice-Tokenizer, StableToken) are a common input/output interface for speech LMs. They use one codebook, produce short sequences and are aligned with text. The authors argue that ASR supervision makes them “acoustically blind”: deep ASR layers suppress voice attributes and background events as noise, so the tokens carry little about who is speaking, how, or in what environment. Acoustic single-codebook codecs such as WavTokenizer keep these details but get no semantic guidance, so acoustically similar but semantically different sounds overlap. The paper asks whether a semantic tokenizer can gain this acoustic information without losing linguistic alignment or speech generation quality.

Method

UniAudio-Token is a Whisper-style encoder-decoder (initialized from Whisper-large-v3) with a vector-quantization (VQ) bottleneck inside the encoder. It has a single codebook of 8,192 entries at a 25Hz token rate, which is 325 bits per second. The encoder has two additions.

Semantic-Acoustic Primitives (SAP) replace the ASR transcript as the decoder target. Each clip is described as a JSON object with three layers: (1) linguistic content (the verbatim transcript); (2) six vocal-attribute fields (age, gender, emotion, accent, prosody, timbre); and (3) an auditory-scene summary plus a list of transient and persistent sound events. Fields that do not apply, such as the speech fields for instrumental music, are set to null. The labels come from an automatic pipeline. Qwen3-Omni-Captioner writes free-form acoustic captions, and Qwen3-30B-A3B-Instruct merges each caption with the ground-truth transcript into the SAP schema. Ontology, consistency and duration checks then filter the outputs. Three expert annotators audited 500 samples and found most fields above 95% accuracy, with emotion and the event lists lowest (Appendix B, Figure 8). An instruction-style variant, SAP-Instruct (direct QA, multiple-choice and true/false items generated by Qwen3-235B-A22B), is mixed into training.

Semantic-Acoustic Equilibrium (SAE) is a gated residual fusion. An MLP adapter projects a shallow encoder layer’s output into the deep feature space. A sigmoid gate, computed from the concatenated deep and shallow features, controls how much of that projection is added to the final-layer representation: H_combined = H_deep + g ⊙ MLP(H_shallow). The fused representation feeds the SAP decoder.

The framework of UniAudio-Token. (Left) The model is supervised by Semantic-Acoustic Primitives (SAP), which cover linguistic content, vocal attributes, and auditory scenes. (Center) Vector Quantization (VQ) converts hidden states into discrete audio tokens. (Right) Semantic-Acoustic Equilibrium (SAE) adaptively fuses shallow acoustic details with deep semantic features, mitigating the loss of fine-grained acoustic cues in deep layers.

Training has two stages. Stage 1 bypasses VQ and trains SAE and the decoder on the SAP prediction loss alone, turning the ASR decoder into an SAP decoder. Stage 2 inserts the VQ layer and mainly optimizes the codebook with the SAP loss plus standard VQ quantization and commitment losses (weights 10.0 and 2.5). Training data is about 182k hours: mostly public speech corpora (Emilia 96,750 h, Yodas, MLS, GigaSpeech, WenetSpeech and others) plus 4,922 h of AudioSet (Table 7).

To produce speech from the tokens, the authors follow the CosyVoice 2 and GLM-4-Voice recipe and train a flow-matching model that maps tokens back to speech. The paper gives no details of this decoder or the vocoder. For the downstream experiments, every tokenizer is paired with the same Qwen2.5 backbone under identical training: Qwen2.5-3B for audio understanding and Qwen2.5-0.5B for autoregressive TTS conditioned on a CAM++ speaker embedding.

Key Results

Speech reconstruction (Table 3). All tokens are decoded by a trained flow-matching model, and quality is scored with WER and MOSNet-predicted MOS on LibriSpeech test-clean/other and Seed-TTS eval en/zh. UniAudio-Token has the lowest WER on all four sets: 3.47 / 6.79 / 2.55 / 1.90, average 3.68. StableToken is next at 3.84 / 7.99 / 3.44 / 2.62, average 4.47. The CosyVoice 2 tokenizer (same 25Hz, 325 bps) averages 5.26, GLM-4-Voice-Tokenizer (12.5Hz, 175 bps) 5.04, and WavTokenizer (75Hz, 900 bps) 6.95. Predicted MOS averages 4.19 against 4.08 for GLM-4-Voice-Tokenizer and 4.03 for StableToken. UniAudio-Token is highest on LS-clean, LS-other and Seed-zh but trails GLM-4-Voice-Tokenizer on Seed-en (4.13 vs. 4.16). The bitrates differ: GLM-4-Voice-Tokenizer uses about half the bitrate, and WavTokenizer is an acoustic codec at nearly three times the bitrate. All MOS values come from a learned predictor, not listeners.

TTS with an LLM (Table 5, Seed-TTS eval, en | zh | avg). CosyVoice 2 is the only comparison, because the authors say the other tokenizers do not support speaker-embedding conditioning. With UniAudio-Token, WER is 1.78 / 1.29 / 1.54, against 2.71 / 1.39 / 2.05 for CosyVoice 2 tokens. Predicted MOS is 4.07 / 3.68 / 3.88 against 3.75 / 3.37 / 3.56. Speaker similarity (ERes2Net) is .792 / .742 / .767 against .758 / .762 / .760. UniAudio-Token is higher in English and lower in Chinese, with a small average edge. The Chinese WER gap (1.29 vs. 1.39) is much smaller than the English one.

Speech and audio understanding (Table 4, Qwen2.5-3B). UniAudio-Token has the highest overall accuracy on MMAU (61.10 vs. 55.20 for the next tokenizer, GLM-4-Voice-Tokenizer), MMAR (45.80 vs. 40.10) and MMSU (43.54 vs. 40.56, StableToken). On the speech-specific columns the gains are smaller. On MMAU-speech it ties StableToken at 45.05. On MMAR-speech it leads with 45.24 vs. 42.18. On the speech-only MMSU benchmark it scores 35.54 perception and 52.07 reasoning, against best baselines of 32.83 (WavTokenizer) and 49.71 (StableToken). Most of the MMAU and MMAR margin comes from the sound and music columns, for example MMAU-sound 70.27 vs. a best baseline of 61.56. Those columns are general-audio context and outside this wiki’s speech focus.

Non-speech discriminability (Table 2, context). On token-histogram clustering over ESC-10 and ESC-50, UniAudio-Token is the only tokenizer with positive silhouette scores (0.091 and 0.023). Its cluster purity is 0.73 and 0.39, against a best baseline of 0.468 and 0.216.

SAE fusion depth (Table 6). Without SAE, WER is 2.47 / 5.71 (LS-clean / LS-other) and the LLM-judged Non-Linguistic Score (NLS) on AudioSet is 2.93. Fusing layer 3 gives 2.43 / 5.58 and NLS 3.16. Layer 1 gives the best clean WER (2.41) but lower NLS (3.08), and layer 5 is close to the no-SAE baseline. The WER differences are small, at most 0.13 absolute. The paper does not say how Table 6’s WER is produced. It is not the token-to-speech pipeline of Table 3, whose WERs are higher.

Novelty Assessment

The two contributions are a change of supervision target and a light architectural addition. Training a Whisper-initialized quantized tokenizer to predict a structured caption (transcript plus voice attributes plus scene events) instead of a transcript is the main new idea. It extends the ASR-supervised semantic tokenizer line (CosyVoice 2, GLM-4-Voice, the same group’s StableToken) rather than replacing it. The authors note that their own earlier work used similar beyond-transcription supervision at the audio-LLM level, not at the tokenizer. SAE is a gated residual connection from a shallow layer, a familiar mechanism applied here to restore acoustic detail. The labels are built with large LLM and audio-captioner teachers (Qwen3 family), so part of the gain may come from distilling those teachers’ perception into the tokens.

Field Significance

Moderate. The paper shows that richer supervision targets can make a single-codebook semantic speech tokenizer more acoustically informative without lowering speech reconstruction or TTS intelligibility. Within its controlled comparisons, reconstruction and TTS WER actually go down, which goes against the assumption that a speech-focused tokenizer must trade acoustic coverage for linguistic fidelity. The evidence is objective-only, with one TTS baseline and small LLM backbones, so it is a promising data point rather than a settled result.

Claims

  • complicates: Purely ASR-supervised semantic speech tokens discard enough acoustic detail that adding voice-attribute and scene supervision can improve, rather than degrade, the intelligibility of speech resynthesized from them.

    Evidence: At equal 25Hz/325 bps, SAP-supervised UniAudio-Token tokens give an average reconstruction WER of 3.68 vs. 4.47 (StableToken) and 5.26 (CosyVoice 2 tokenizer) across LibriSpeech and Seed-TTS sets, with the same flow-matching token-to-speech recipe. The authors attribute this partly to better retention of accent and phonetic cues. (§5.1, Table 3)

  • supports: Single-codebook speech tokens that keep more paralinguistic and acoustic information can improve LLM-based TTS intelligibility and predicted quality without a loss in speaker similarity.

    Evidence: With the same Qwen2.5-0.5B backbone and CAM++ speaker-embedding conditioning on Seed-TTS eval, UniAudio-Token tokens give an average WER of 1.54 vs. 2.05 and predicted MOS of 3.88 vs. 3.56 compared with CosyVoice 2 tokens. Average SIM is .767 vs. .760, but Chinese SIM is lower (.742 vs. .762). (§5.2, Table 5)

  • refines: Fusing shallow encoder features into a deep semantic speech representation helps non-linguistic coverage most at intermediate depth. Very shallow layers favor clean-speech phonetic detail, and deeper layers add little.

    Evidence: Gated fusion from encoder layer 3 raises the LLM-judged non-linguistic score from 2.93 to 3.16 while LibriSpeech WER changes by at most 0.13 absolute. Layer 1 gives the best clean WER (2.41) with a smaller NLS gain (3.08), and layer 5 is close to the no-fusion baseline (2.95). (§5.3.1, Table 6)

  • complicates: Gains from a more acoustically informed speech tokenizer in speech LM understanding appear mainly on non-speech audio. Speech-specific understanding improves less or not at all.

    Evidence: With Qwen2.5-3B, UniAudio-Token ties StableToken on MMAU-speech (45.05) and leads by about 3 points on MMAR-speech and MMSU. On MMAU-sound and MMAU-music it leads the best baseline by 8.7 and 5.4 points. (§5.2, Table 4)

Limitations and Open Questions

Warning

All speech-quality evidence is objective. MOS is predicted by MOSNet, not rated by listeners, and the TTS comparison uses a single baseline (CosyVoice 2 tokens) with a 0.5B LLM. No human listening test supports the claim that acoustic coverage comes “without compromising” speech generation.

  • No ablation of SAP supervision. The only ablation varies SAE fusion depth (Table 6). There is no tokenizer trained with ASR-only targets under the same architecture and data, so the reconstruction and TTS gains cannot be attributed to SAP versus training data, the Whisper-large-v3 initialization, or SAE. The ~182k-hour training mix also differs from the baselines’ official checkpoints.
  • Where SAE sits relative to the tokens. Figure 2 places VQ between two stacks of encoder layers and feeds SAE into the SAP decoder. The text does not say exactly which features the shallow branch taps relative to the quantizer. It is therefore not fully clear how much of the SAE-restored detail is carried by the discrete tokens and how much is available only to the training-time decoder. The Table 6 ablation is measured with an unspecified WER and the decoder-generated NLS, not with downstream token-only use.
  • Opaque token-to-speech decoder. The flow-matching reconstruction model is mentioned in one sentence, with no architecture, data or vocoder details.
  • Teacher-generated labels. SAP targets come from Qwen3-family captioning and LLM models. A 500-sample human audit finds lower accuracy for emotion and event lists, and teacher errors or biases propagate into the tokenizer.
  • Language and fidelity scope. Evaluation covers English and Chinese only. The authors state that waveform reconstruction of complex non-speech audio falls short of high-bitrate acoustic codecs, and no non-speech reconstruction numbers are reported.
  • Small backbones. Downstream comparisons use Qwen2.5-3B and 0.5B. It is untested whether the tokenizer differences persist at larger LLM scales or in full spoken dialogue.

Wiki Connections

  • Neural Audio Codec — Proposes a single-codebook, 25Hz VQ semantic speech tokenizer whose supervision target extends ASR transcripts with voice-attribute and acoustic-scene fields.
  • Spoken Language Model — Trains Qwen2.5-3B on each tokenizer’s tokens and compares speech and audio understanding on MMAU, MMAR and MMSU under identical settings.
  • Autoregressive Codec TTS — Trains a Qwen2.5-0.5B autoregressive TTS model over its tokens and compares it with CosyVoice 2 tokens on Seed-TTS eval.
  • Flow Matching — Uses a trained flow-matching token-to-speech model (following the CosyVoice 2 recipe) for all reconstruction comparisons.
  • StableToken — Same group’s earlier noise-robust semantic tokenizer and the strongest speech baseline here. UniAudio-Token keeps the single-codebook ASR-initialized design but replaces transcript-only supervision with SAP targets and adds gated shallow-feature fusion, aiming at acoustic coverage rather than noise invariance.
  • CosyVoice 2 — Baseline tokenizer at the same 25Hz/325 bps rate, and the only comparison in the TTS experiment.
  • Scaling Speech-Text Pre-training with Synthetic Interleaved Data — Source of GLM-4-Voice-Tokenizer, a 12.5Hz baseline that edges UniAudio-Token on Seed-en predicted MOS.
  • WavTokenizer — Acoustic single-codebook baseline (75Hz), weakest on speech reconstruction WER in this comparison.
  • Whisper — Whisper-large-v3 initializes the tokenizer’s encoder and decoder.
  • Seed-TTS — Its evaluation sets are used for reconstruction and TTS benchmarks.
  • MMAU — Primary audio-understanding benchmark for the downstream LLM comparison.
  • WildSpeech-Bench — Same group’s speech LM benchmark, cited as related speech LM work.