Speech Generation Research Wiki

DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

9 min read

arXiv · 2026 · Preprint

Hanlin Zhang et al. (City University of Hong Kong) · → Paper · Demo: ✓ · Code: ?

Proposes DSA-Tokenizer, a dual-stream speech tokenizer with strictly separated ASR-supervised semantic tokens and reconstruction-supervised acoustic tokens fused via a hierarchical Flow-Matching decoder, and introduces cross-utterance semantic-acoustic recombination as a more rigorous evaluation task for tokenizer disentanglement than reconstruction quality alone.

Problem

Discrete speech tokenizers are the foundation of fully discrete Speech LLMs, but existing tokenizers fall short in one of three ways: semantic tokenizers (self-supervised or ASR-supervised) discard acoustic cues like timbre; semantic-acoustic mixed tokenizers achieve high-fidelity reconstruction but produce entangled representations that prevent independent attribute control; and shallowly disentangled tokenizers attempt to decouple semantic and acoustic information but suffer incomplete separation. The paper argues that reconstruction quality and ASR performance, the metrics typically used to evaluate tokenizers, do not directly test disentanglement: a more direct test is cross-utterance recombination, extracting pure linguistic content from one utterance and pure acoustic style (timbre, prosody) from a different utterance, then fusing them into new speech that retains the content of the former and the style of the latter. Existing tokenizers, the paper shows, perform poorly at this task, and a further structural limitation is that most tokenizers impose a rigid length constraint between semantic and acoustic token sequences, making recombination between different-length utterances impossible outright.

Method

DSA-Tokenizer uses two parallel discrete token streams with orthogonal, asymmetric optimization constraints. Semantic tokens are produced by a pretrained HuBERT encoder followed by a Finite Scalar Quantization (FSQ) layer (codebook size 1024, 25Hz frame rate), with the HuBERT-FSQ pipeline trained under a Connectionist Temporal Classification (CTC) loss to strictly filter out stylistic information and retain only linguistic content; the lightweight CTC decoder is discarded after training and the frozen HuBERT-FSQ module is used purely as a semantic token extractor. Acoustic tokens are produced by a SEANet-style encoder over mel-spectrograms followed by its own FSQ quantizer, trained end-to-end (gradients passed through the discrete bottleneck via a straight-through estimator) jointly with the decoder on a Flow Matching reconstruction objective, so acoustic tokens capture whatever spectral detail semantic tokens do not. Because acoustic and semantic token sequence lengths are independently determined, there is no rigid length constraint between them, which is what enables recombination between utterances of different lengths.

Overview of the proposed framework and training strategy. (a) DSA-Tokenizer framework: input audio X is encoded into discrete semantic and acoustic tokens, fed into a DiT decoder for audio generation. (b) Self-Reconstruction Mode: the model predicts the velocity field of the full mel-spectrogram from complete acoustic and semantic tokens. (c) Recombination (Contextual Inpainting) Mode: the model predicts the velocity field of a masked mel-spectrogram region from the acoustic tokens of the unmasked region and the full semantic token sequence.

The two token streams are fused in a DiT-based Flow Matching decoder via a hybrid injection strategy: semantic tokens are injected as a dense temporal condition through a ControlNet-style CNN adapter whose output is added directly to the noisy mel-spectrogram input (enforcing precise temporal/linguistic alignment as a structural backbone), while acoustic tokens are injected via cross-attention, letting the model flexibly draw on global or local acoustic style information without a length constraint. Training uses a joint reconstruction-recombination strategy: each batch is randomly assigned (50/50) to Self-Reconstruction Mode (predicting the full mel-spectrogram’s Flow Matching velocity field from complete semantic and acoustic tokens) or Recombination (Contextual Inpainting) Mode (masking the mel-spectrogram after a randomly sampled time-axis split point, and predicting the masked region’s velocity field from only the prefix acoustic tokens plus the full semantic token sequence), which forces the model to infer global acoustic style from partial context while strictly following semantic content, decoupling the two streams. Training combines this Flow Matching loss with a speaker-consistency loss that aligns pooled acoustic-token embeddings with a WavLM-derived reference speaker embedding via cosine similarity, and uses classifier-free guidance at inference (guidance scale 2).

Key Results

Against WavTokenizer (single-layer, non-decoupled), Mimi and EnCodec (multi-layer, non-decoupled), SpeechTokenizer (multi-layer, shallow decoupling), DualCodec and SAC (dual-branch, decoupled), DSA-Tokenizer achieves competitive reconstruction-task performance (UTMOS 3.38-3.46, WER 2.09-2.61%, SIM 0.72-0.82 across configurations, in English) at a lower bitrate (0.7-1.1 kbps) than most baselines. On the cross-utterance recombination task, the gap is dramatic: DSA-Tokenizer achieves 5.93-8.77% WER and UTMOS 3.54-3.67 in English, while baselines range from 12.98% WER (SpeechTokenizer, but with severely degraded 123.33% CER in Chinese) up to 90-107% WER for Mimi, EnCodec, and SAC, alongside far lower UTMOS and speaker similarity. Disentanglement probing (ASR and speaker-classification accuracy applied directly to each token stream) confirms this: DSA-Tokenizer’s semantic tokens achieve 6.28% WER with only 2.35% speaker-classification accuracy, indicating minimal cross-stream information leakage, while baselines like EnCodec show uniform ASR/SC performance across layers (indicating entanglement) and SpeechTokenizer and SAC show only partial separation. In LLM-based voice cloning (Qwen3-0.6B backbone predicting cloned-speech tokens from separate semantic-source and acoustic-source token sequences), DSA-Tokenizer achieves the best balance (UTMOS 3.9, WER 23.95%, SIM 0.41) versus SAC (UTMOS 3.3, WER 24.21%, SIM 0.37) and WavTokenizer, which despite strong standalone reconstruction fidelity performs worst on all three metrics and causes unstable LLM inference including endless generation. Ablation shows removing the speaker-consistency loss sharply reduces speaker similarity across both tasks, and removing the recombination training mode causes only mild reconstruction-task degradation but a severe recombination-task collapse (WER 5.93% to 107.68%, UTMOS 3.67 to 2.67).

Novelty Assessment

The paper’s most transferable contribution is arguably not the tokenizer architecture itself but the cross-utterance recombination task as an evaluation protocol: the authors explicitly frame it as a more rigorous, direct test of disentanglement than reconstruction quality or ASR performance, and the experimental results substantiate this by showing several baselines that look competitive on reconstruction collapse entirely under recombination, information reconstruction-only evaluation does not surface. On the architecture side, the combination of asymmetric per-stream training objectives (ASR-only for semantic, reconstruction-only for acoustic), length-unconstrained streams, and the specific hybrid ControlNet-plus-cross-attention fusion strategy is a coherent, well-motivated design rather than an incremental tweak, and each component’s necessity is demonstrated via ablation (recombination training mode, speaker loss) rather than merely asserted. The LLM-based voice cloning experiment is a useful downstream validation showing the disentanglement benefit transfers to actual generation-model compatibility, not just to the tokenizer’s own reconstruction metrics.

Field Significance

Tip

High, the cross-utterance recombination task is a genuinely useful new evaluation paradigm that other disentangled-tokenizer papers are likely to adopt, since it exposes a real failure mode (semantic-acoustic leakage) that reconstruction-only benchmarks miss. Combined with a substantial empirical gap over strong recent baselines (SAC, DualCodec, SpeechTokenizer) on that task and a demonstrated downstream benefit for LLM-based generation stability, this represents a meaningful advance for controllable, disentangled speech tokenization in fully discrete Speech LLMs.

Claims

  • supports: Explicit dual-stream tokenization, where semantic tokens are constrained purely by ASR supervision and acoustic tokens are constrained purely by spectral reconstruction, achieves substantially better semantic-acoustic disentanglement than shallow or partial disentanglement methods built on a shared or hierarchically-derived representation.

    Evidence: In cross-utterance recombination, DSA-Tokenizer achieves 5.93-8.77% WER in English versus 12.98-107.51% WER for shallowly disentangled or entangled baselines (SpeechTokenizer, Mimi, EnCodec, DualCodec, SAC), while also achieving the highest audio quality and speaker similarity among compared systems on this task. (§5.1, Table 1)

  • supports: Cross-utterance semantic-acoustic recombination is a more diagnostic evaluation task for tokenizer disentanglement than reconstruction quality or ASR performance alone, since models can achieve strong reconstruction metrics while failing catastrophically at recombination.

    Evidence: Baselines such as Mimi and SAC achieve competitive or strong reconstruction-task WER and UTMOS but collapse on the recombination task, with WER exceeding 90-107% and severely degraded UTMOS, revealing entanglement that reconstruction-only evaluation does not surface. (§5.1, Table 1)

  • complicates: High-fidelity speech reconstruction quality in a tokenizer does not guarantee stable, controllable generation when the tokens are used as input/output vocabulary for a speech language model.

    Evidence: WavTokenizer achieves strong standalone reconstruction fidelity (UTMOS 3.75) but performs worst among compared tokenizers on LLM-based voice cloning (WER 89.44%, SIM 0.28) and causes unstable LLM inference including endless generation, attributed to its lack of semantic-acoustic disentanglement. (§5.3, Table 2)

  • supports: Training a joint reconstruction-and-recombination objective, rather than reconstruction alone, is necessary for a disentangled tokenizer’s decoder to generalize to cross-utterance content-style recombination without requiring matched sequence lengths.

    Evidence: Removing the recombination training mode causes only slight speaker-similarity degradation on the reconstruction task but a severe collapse on the recombination task (WER rising from 5.93% to 107.68%, UTMOS dropping from 3.67 to 2.67), while removing the speaker-consistency loss causes a sharp similarity reduction across both tasks. (§5.4, Table 3)

Limitations and Open Questions

  • The authors state inference latency is a challenge for real-time applications, since the decoder’s 22-block DiT stack requires iterative Flow Matching sampling, computationally heavier than GAN-based counterparts; acceleration techniques are left to future work.
  • The current model is trained and evaluated exclusively on speech; generalization to other audio modalities (music, environmental sound) is explicitly untested and left to future extension.
  • Code and model release is promised only after acceptance, so results are not yet independently reproducible at the time of ingestion.
  • The recombination task’s realism as a proxy for real-world use cases (e.g. genuine voice conversion or style transfer applications) versus its value purely as a diagnostic disentanglement probe is not separately evaluated.

Wiki Connections

  • Neural Audio Codec — proposes a dual-stream discrete speech tokenizer with strict semantic-acoustic separation, positioned against several recent codec baselines on reconstruction, recombination, and disentanglement-probing tasks.
  • Disentanglement — enforces semantic-acoustic separation via asymmetric per-stream training objectives (ASR-only, reconstruction-only) and a joint reconstruction-recombination training strategy, validated with dedicated disentanglement-probing evaluation and ablations.
  • Flow Matching — fuses semantic and acoustic token streams via a DiT-based Flow Matching decoder with a hybrid ControlNet-style and cross-attention injection strategy.
  • Self-Supervised Speech — uses a pretrained HuBERT model, further constrained by CTC supervision, as the semantic token encoder.
  • Spoken Language Model — validates the tokenizer’s compatibility with a Qwen3-0.6B LLM backbone trained to generate cloned-speech tokens from separate semantic-source and acoustic-source token sequences.
  • SAC — the primary directly comparable prior dual-stream disentangled codec baseline throughout reconstruction, recombination, and disentanglement-probing experiments.
  • SpeechTokenizer — a shallow-disentanglement baseline (semantics distilled into layer 0 of an RVQ stack) compared throughout, and discussed as representative of incomplete disentanglement.
  • EnCodec — a multi-layer, non-decoupled codec baseline showing uniform (entangled) semantic/acoustic performance across layers in disentanglement probing.
  • WavTokenizer — a single-layer, non-disentangled baseline that achieves strong reconstruction fidelity but the worst LLM-based voice cloning performance among all compared tokenizers.
  • F5-TTS — used to synthesize the supervised target speech for the over 350,000 training triplets in the LLM-based voice cloning experiment.
  • Flow Matching for Generative Modeling — the paper’s Conditional Flow Matching decoder training objective follows this formulation directly.
  • Finite Scalar Quantization — the FSQ quantization scheme used to discretize both the semantic and acoustic token streams.
  • Emilia — the 100k-hour Chinese-English subset used to train the acoustic tokenizer and Flow Matching decoder, after a data-cleaning pass to remove multi-speaker samples.

Backlinks

Graph View