arXiv · 2025 · Preprint
Wang et al. · → Paper · Demo: ? · Code: ?
DualSpeechLM proposes a dual-token architecture that separates the speech token used as LLM input (a new semantics-oriented tokenizer) from the token used as output (an acoustic codec), resolving the fundamental tension between understanding and generation tasks in unified speech LLMs while enabling competitive performance with only 4.5K hours of training data.
Problem
Existing approaches to unified speech understanding and generation with LLMs share a single token type for both input and output. This creates an irreconcilable tension: acoustic tokens capture the fine-grained prosodic and speaker detail needed for high-fidelity generation but carry too little semantic content for understanding tasks such as ASR and speech question answering. Conversely, semantic tokens derived from SSL models (HuBERT) or ASR objectives improve understanding but degrade generation quality. Systems trained jointly on both task types show contradictory behaviour — improving one objective reliably degrades the other. A further practical obstacle is the large modality gap between speech and text, which has forced prior unified models (SpeechGPT, SpiritLM) to consume 70K–570K hours of paired training data to achieve alignment.
Method
DualSpeechLM addresses the shared-token bottleneck by explicitly decoupling the input and output token spaces within a single end-to-end framework. The system consists of two components trained in sequence: an Understanding-driven Speech Tokenizer (USTokenizer) and the DualSpeechLM LLM backbone.
The USTokenizer converts raw speech into USTokens by passing Whisper-medium encoder features through a 2× downsampling convolutional encoder, a single-layer vector quantizer with a 1024-entry codebook, and an upsampling decoder. What distinguishes this tokenizer is an understanding-driven loss: the quantized vectors are projected into a frozen LLaMA-3.2-1B’s embedding space, and the tokenizer is trained to minimise next-token prediction loss on understanding tasks (ASR, emotion recognition, speech question answering). This forces the VQ codebook to align with the semantic structure of the text LLM’s vocabulary, producing tokens that bridge the modality gap more directly than pure SSL quantisation. The resulting bitrate is 250 bps — substantially lower than competing tokenizers based on acoustic reconstruction.
DualSpeechLM then fine-tunes a text LLM (Phi-3.5-3B or Vicuna-7B) with LoRA (rank 16), using USTokens as input for all tasks. For generation tasks, an integrated AcousticGPT module — six-layer causal GPT-style transformer — is appended to the LLM and trained jointly. AcousticGPT autoregressively generates WavTokenizer acoustic tokens conditioned on LLM hidden states and a speaker embedding from a pretrained 3D-Speaker encoder. For understanding tasks (ASR, S2TT, SER, SQA), the model outputs text tokens directly. A semantic supervision loss trains the LLM to also predict USToken sequences for generation tasks, enforcing consistent semantic representations across pathways.
A Chain-of-Condition (CoC) strategy regularises AcousticGPT training by stochastically conditioning the module on one of three sources — prompt hidden states, predicted USTokens, or their concatenation — sampled with equal probability. At inference, the concatenated form is always used. This prevents overfitting to any single conditioning signal and mitigates errors introduced by imprecise USToken predictions.
Key Results
On understanding tasks with Phi-3.5-3B and 4.5K hours, DualSpeechLM-USToken achieves 4.22% WER on LibriSpeech test-clean and 9.71% on test-other, compared to 5.70%/14.32% for a HuBERT-input baseline trained on identical data. On speech-to-text translation (En→De CoVoST2), USToken yields 19.74 BLEU-4 against 11.13 for HuBERT input and 1.91 for the acoustic-token baseline. Speech emotion recognition accuracy rises to 60.92% from 51.91% (HuBERT) and 54.90% (acoustic).
On generation tasks, USToken input achieves 9.25% WER on LibriTTS-R test-clean TTS (versus 22.11% with acoustic tokens and 21.72% with HuBERT tokens), speaker similarity of 0.90, and DNSMOS 3.86. Subjective QMOS for TTS is 3.89 ± 0.31 and SMOS 3.74 ± 0.24, compared to 3.67/3.77 for the acoustic baseline and 2.26/2.76 for the semantic baseline — confirming that USToken’s modality alignment benefits generation as well as understanding.
The ablation study demonstrates that the semantic supervision loss is critical for generation: its removal causes WER to collapse to 167.56% on test-clean, indicating the text LLM cannot reliably produce usable USToken sequences without the explicit semantic objective. The CoC strategy provides a further ~0.7 percentage point WER improvement on TTS tasks.
Comparisons to models using much larger data budgets (SpeechGPT at 70K hours, SpiritLM at 570K hours) show DualSpeechLM-USToken achieves competitive or superior ASR performance despite a roughly 15–125× data reduction, attributable primarily to better modality alignment from the understanding-driven tokeniser.
Note
Comparisons to Qwen2.5-Omni (which uses 300B audio tokens and full fine-tuning) and Moshi (7M hours) are included for context but are not fair head-to-head comparisons given the enormous resource gap.
Novelty Assessment
The core architectural idea — decoupling input and output token types in a unified speech LLM — is genuinely novel in how it is instantiated. Prior work (SpeechTokenizer, SpiritLM) had explored multi-codebook schemes or interleaving semantic and acoustic tokens, but required either multi-stage pipelines or still relied on the same token for both LLM input and output. The direct optimisation of a speech tokenizer against an LLM’s text-generation objective (the understanding-driven loss) is a principled contribution that differs from standard SSL quantisation and ASR-objective approaches.
The AcousticGPT and CoC strategy are incremental engineering choices; both are competent but not surprising. The main contribution is the tokeniser design and the conceptual reframing of unified speech modelling as a dual-token problem rather than a single-token-with-multiple-objectives problem.
The evaluation is conducted on a relatively small, controlled data regime (4.5K hours), which limits claims about scalability. All comparisons to large-scale baselines (Qwen2.5-Omni, Moshi) are resource-asymmetric and should be read as positioning rather than as evidence of competitive parity at scale.
Field Significance
Moderate — DualSpeechLM provides a clean conceptual decomposition of the understanding-generation tension in unified speech LLMs and backs it with ablation evidence. The insight that speech tokeniser training objectives should explicitly reflect their downstream LLM role — rather than treating tokenisation and LM adaptation as separate stages — has implications for how the field approaches modality bridging. The contribution is primarily architectural and conceptual rather than empirical at scale, and the small training data regime limits direct comparison with frontier unified models.
Claims
- Separating the token used for LLM input from the token used for generation output can resolve the information-level conflict that makes joint optimisation of understanding and generation tasks difficult in a shared-token speech LLM. (§DualSpeechLM, §Results and Analyses)
- Training a speech tokenizer directly against a text LLM’s next-token prediction objective is more effective at reducing the speech-text modality gap than optimising against ASR or SSL reconstruction losses alone. (§USTokenizer, Table 1)
- Understanding-task supervision produces representations that transfer to generation quality improvements, but the converse — generation-task supervision improving understanding — is weaker and less consistent. (§Ablation Study, §H. Discussion)
- Stochastic conditioning during training (exposing a generation module to varied subsets of its conditioning signals) improves robustness to imperfect upstream predictions at inference time. (§Ablation Study, Table 6)
Limitations and Open Questions
Warning
The entire evaluation is conducted at 4.5K hours of training data with parameter-efficient LoRA fine-tuning. The claim that USTokens reduce data requirements is plausible but untested at the scale (70K–570K hours) where competing systems are evaluated. Whether the dual-token architecture and the understanding-driven tokeniser remain advantageous at scale is an open question.
No code or demo is linked in the paper, limiting reproducibility. The 4.5K-hour training regime excludes noisy, in-the-wild, and multilingual data, so generalisation to these conditions is untested despite the paper’s stated future direction of expanding to multilingual and cross-domain data. The USTokenizer’s understanding-driven loss requires a frozen LLM during tokeniser training, adding a significant computational overhead at the tokenisation stage (288% memory increase) even if this overhead disappears at DualSpeechLM inference. The model size of the full system (Phi-3.5-3B + AcousticGPT) is not explicitly stated in aggregate, and the AcousticGPT’s token generation speed relative to real-time is not reported.
Wiki Connections
The dual-token design directly addresses the longstanding challenge in autoregressive-codec-tts of finding a single token that serves both semantic and acoustic needs. The USTokenizer’s alignment objective connects to neural-codec literature, positioning USTokens as a hybrid between SSL-derived semantic tokens and codec acoustic tokens. The AcousticGPT module’s conditioning on speaker embeddings relates this work to spoken-language-model frameworks for zero-shot generation. The paper also evaluates voice conversion on VCTK, situating it within voice-conversion research on zero-shot speaker transfer.
The paper cites DiTAR as a related approach using combined autoregressive and diffusion objectives for speech generation, which represents an alternative strategy for handling acoustic detail in LLM-based generation.