arXiv · 2026 · Preprint
Yuxuan Lou et al. · → Paper · Demo: ? · Code: ?
Introduces DiffuSpeech, the first diffusion-based speech-text language model supporting both understanding and generation, which jointly generates internal text reasoning traces (“silent thought”) and spoken responses through iterative masked-diffusion denoising rather than autoregressive left-to-right decoding, paired with THINKINGTALK, the first speech QA dataset with paired reasoning traces.
Problem
Current speech language models generate spoken responses directly, without any explicit intermediate reasoning step, unlike text LLMs where chain-of-thought reasoning has proven highly effective. This is a particularly costly limitation for speech specifically, because speech generation is irreversible: once audio is produced, a factual or logical error embedded in it cannot be corrected mid-utterance the way text can be edited or backtracked. Existing speech-text language models are also exclusively autoregressive, generating tokens strictly left-to-right, which prevents parallel decoding and cannot leverage bidirectional context, a real constraint for speech, where prosody and content are inherently entangled and where jointly modeling reasoning and response could benefit from access to the full sequence rather than only preceding tokens.
Method
DiffuSpeech proposes “Silent Thought, Spoken Answer,” a paradigm where a speech LLM generates internal text reasoning traces jointly with its spoken response, with the reasoning informing the quality of the eventual speech. To realize this, the paper builds the first diffusion-based language model unifying speech and text under a single masked diffusion framework, extending the pretrained LLaDA-8B masked diffusion text LLM. Speech is tokenized via a frozen HuBERT encoder followed by a linear quantizer, producing discrete speech tokens that are added to the model’s vocabulary alongside text and special task-control tokens, so that speech and text share one unified token space rather than requiring separate modality-specific pathways. Sequences are structured with a task token (e.g., marking speech-to-speech-with-reasoning, ASR, TTS, or text-to-text) followed by conditioning input and target output; for the paper’s central task, a spoken user question is followed by a generated text reasoning trace and then generated speech tokens for the reply.
Training and generation both follow the masked diffusion recipe: the forward process independently masks each token in the target with a cosine-scheduled probability, and a transformer is trained to predict the original tokens at masked positions, with masking applied selectively only to the target portion of each sequence (the conditioning input stays intact) so the model learns proper conditional generation. Because different tasks require different components masked, a multi-task, two-stage curriculum is used: Stage 1 aligns speech and text via ASR, TTS, and text-LM objectives on LibriHeavy, VoxPopuli, and CommonVoice; Stage 2 fine-tunes on instruction-following tasks (speech-to-speech-with-reasoning, speech-to-text-with-reasoning, text-to-text-with-reasoning, plus continued ASR/TTS) using the paper’s own THINKINGTALK dataset. At inference, generation proceeds by iterative denoising: starting from an all-masked target, each step computes predictions and per-position confidence for all masked positions, then commits the most confident predictions (a linear unmasking schedule determines how many positions to reveal per step) while leaving less-confident positions masked for later refinement, allowing the model to lock in easy decisions early and resolve harder ones with more context available. Generated speech tokens are finally rendered to waveform by a frozen HiFi-GAN vocoder.
To train and evaluate the “Silent Thought” capability, the paper constructs THINKINGTALK, the first speech QA dataset with paired text reasoning traces (26,387 samples, 319 hours). It is built via a three-stage pipeline: existing text QA datasets (SmolTalk2 for conversational QA, SoundMind for logical reasoning) are rewritten by Qwen-32B into a spoken-suitable user question, a step-by-step text reasoning trace, and a natural spoken-style reply, avoiding content that cannot be verbalized (visual references, math notation, code); an LLM quality judge (Qwen3-32B) filters samples across seven dimensions (completeness, correctness, oral suitability, clarity, length, visual independence, conversational tone), retaining 81.7% of candidates; and audio is synthesized with MegaTTS3 (four diverse reference speakers, for varied user voices) for questions and Qwen3-Omni (one consistent voice, for assistant identity) for replies, with a words-per-second hallucination filter rejecting abnormally fast or slow synthesized audio.
Key Results
On speech-to-speech question-answering, DiffuSpeech achieves the best S→S accuracy across all evaluated benchmarks, outperforming the strongest autoregressive baseline (Moshi) by +4.3 points on LlamaQuestions (68.5%), +3.0 on TriviaQA (33.5%), and +9.0 on WebQuestions (49.7%), for a +5.5-point average S→S accuracy gain (50.6% vs. Moshi’s 45.1%); notably, while most baselines degrade sharply moving from text-output (S→T) to speech-output (S→S) evaluation (e.g., MinMo drops 12.7-15.1 points across benchmarks), DiffuSpeech’s S→T-to-S→S gap is comparatively small, indicating the joint reasoning-and-speech generation better preserves answer quality through to the spoken output. On speech processing, DiffuSpeech achieves competitive ASR WER (3.0% LS-Clean) and the best TTS WER among all speech-generative models tested (6.2% LS-Clean, 10.3% VoxPopuli, beating MinMo’s 6.7%/10.9% and Moshi’s 7.0%/10.6%). On text-only language understanding, DiffuSpeech scores higher than its own text-only diffusion base model (LLaDA) on all three benchmarks tested after multi-task speech-text training (MMLU 66.2% vs. 65.9%, TriviaQA 60.3% vs. 55.6%, GSM8K 72.8% vs. 70.3%), suggesting speech-text co-training does not impose the language-capability cost one might expect. Ablations directly validate both design choices: comparing matched-capability autoregressive (Llama-3.1-8B) and diffusion (LLaDA-8B) base models under identical speech-alignment training shows autoregressive adaptation converging faster initially, but diffusion overtaking it around 10K-15K steps and reaching substantially lower final WER (7.1% vs. 11.4% ASR, 10.5% vs. 14.7% TTS); and fine-tuning both an autoregressive model (SpiritLM) and DiffuSpeech with versus without paired thinking traces on the same data shows thinking traces improve both, but with a larger gain for the diffusion model (+13.4 average S→S points vs. +10.5 for SpiritLM). A sample-efficiency ablation further shows the model tolerates a 2x reduction in denoising steps (1024→512) with stable or even improved quality (TTS WER improves from 8.45% to 6.20%; SpeechQA accuracy holds at ~72%), and degrades only gracefully at a 4x reduction (256 steps: 70.52% accuracy, 7.60% TTS WER).
Novelty Assessment
This is a genuine architectural first, by the paper’s own account and consistent with the surveyed prior work: no previous diffusion-based language model supported both speech understanding and speech generation jointly (the closest prior system, DiFFA, supports diffusion-based speech understanding only, with no generation capability). Extending masked diffusion language modeling to a unified speech-text vocabulary, with modality-specific selective masking that keeps conditioning intact while diffusing only the target, is a substantive architectural adaptation, not merely applying an existing text-diffusion recipe unchanged. The “Silent Thought, Spoken Answer” paradigm itself, jointly generating a reasoning trace and a spoken response within one diffusion process rather than as separate stages, is empirically validated as architecture-relevant, not just a training-data trick: the paper’s own ablation shows the same reasoning-trace training data benefits an autoregressive baseline (SpiritLM) less than it benefits the diffusion model, suggesting the bidirectional, jointly-denoised generation process itself contributes to how effectively the model can exploit reasoning traces. The THINKINGTALK dataset fills a genuine gap (no prior speech QA dataset pairs spoken questions and answers with an explicit text reasoning trace) and is a reusable resource independent of the specific model architecture.
Field Significance
high — this paper is the first to demonstrate a diffusion-based language model capable of both speech understanding and speech generation, and directly validates, via controlled architecture-matched and data-matched ablations rather than assertion alone, that (1) diffusion-based speech adaptation ultimately surpasses autoregressive adaptation despite slower early training, and (2) jointly generating reasoning traces alongside spoken responses measurably improves answer accuracy, with the diffusion architecture specifically better positioned to exploit reasoning traces than an autoregressive counterpart trained on identical data.
Claims
- supports: Replacing autoregressive left-to-right generation with masked diffusion for a unified speech-text language model enables joint generation of internal text reasoning and speech tokens that measurably improves speech-to-speech question-answering accuracy over strong autoregressive baselines.
Evidence: The diffusion-based model achieves the best speech-to-speech QA accuracy across all benchmarks tested, outperforming the strongest autoregressive baseline (Moshi) by +4.3/+3.0/+9.0 points on LlamaQuestions/TriviaQA/WebQuestions and by +5.5 points on average S→S accuracy. (§5.2, Table 2)
- supports: Explicitly generating a text-based internal reasoning trace jointly with a spoken response substantially improves the accuracy of the resulting spoken answer, and this benefit is not exclusive to diffusion architectures, though diffusion appears to exploit it more effectively.
Evidence: Fine-tuning both an autoregressive model (SpiritLM) and the diffusion model on identical data with versus without paired thinking traces shows large accuracy gains from adding traces in both cases (+10.5 average points for SpiritLM, +13.4 for the diffusion model). (§5.3, Table 5)
- complicates: Autoregressive and diffusion-based approaches to adapting a pretrained language model for speech generation exhibit different training dynamics, with autoregressive adaptation converging faster early in training but diffusion adaptation ultimately surpassing it, so early-training comparisons between the two paradigms can be misleading about eventual relative quality.
Evidence: Tracking WER throughout speech-alignment training for matched-capability autoregressive (Llama-3.1-8B) and diffusion (LLaDA-8B) base models shows the autoregressive model improving faster in the first few thousand steps, but the diffusion model overtaking it around 10K-15K steps and reaching substantially lower final WER (7.1% vs. 11.4% ASR, 10.5% vs. 14.7% TTS). (§5.3, Figure 5)
- supports: A masked-diffusion-based multimodal language model can reduce the number of iterative denoising steps at inference substantially without a proportional loss in output quality, offering a favorable efficiency trade-off relative to autoregressive generation.
Evidence: Reducing inference denoising steps from 1024 to 512 (2x speedup) at fixed 1024-token target length maintains stable speech-QA accuracy (72.06% vs. 72.13%) and improves TTS WER (6.20% vs. 8.45%), while a further 4x speedup (256 steps) degrades performance only gracefully. (§5.3, Table 6)
- complicates: Training a speech-text language model jointly on speech and text tasks does not necessarily degrade the model’s underlying text-only language understanding capability relative to its text-only base model, and can even modestly improve it.
Evidence: After multi-task speech-text training, the model scores higher than its text-only diffusion base model (LLaDA) on all three text-only language-understanding benchmarks tested (+0.3 MMLU, +4.7 TriviaQA, +2.5 GSM8K). (§5.2, Table 4)
Limitations and Open Questions
The THINKINGTALK dataset that both trains and validates the "Silent Thought, Spoken Answer" capability is entirely synthetically constructed: source text QA is LLM-rewritten (Qwen-32B) into spoken style and reasoning traces, LLM-judged for quality (Qwen3-32B), and rendered to audio via TTS systems (MegaTTS3, Qwen3-Omni) rather than collected from naturally occurring human speech QA interactions with genuine human reasoning traces; the reported gains from thinking traces reflect this synthetic-data regime and have not been validated against a naturally-sourced reasoning-augmented speech QA benchmark.
The paper’s own conclusion identifies extending the paradigm to the visual modality and to Mixture-of-Experts architectures as future work, implying the current model handles only text and speech modalities and uses a dense (non-MoE) backbone. All experiments are conducted in English; generalization to other languages is untested.
Wiki Connections
- Spoken Language Model — extends a pretrained text diffusion LLM (LLaDA) to consume and generate external speech tokens (via a frozen HuBERT encoder) within a unified speech-text vocabulary and single masked-diffusion generation process.
- Speech-to-Speech — targets speech-to-speech question-answering as the primary evaluation task, jointly generating a text reasoning trace and a spoken reply from a spoken question.
- Diffusion TTS — generates speech tokens via iterative masked-diffusion denoising rather than autoregressive decoding, extending diffusion-based generation to the joint speech-text setting.
- Self-Supervised Speech — relies on a frozen, pretrained HuBERT self-supervised speech encoder as the core component of its speech tokenization pipeline.
- Moshi — used as the strongest autoregressive baseline in the main speech-to-speech QA and speech-processing comparisons, which the proposed model outperforms.
- LLaMA-Omni2 — used as a direct autoregressive baseline in the speech question-answering comparison table.
- Qwen2-Audio — used as a direct baseline for both speech question-answering and ASR evaluation, representative of autoregressive speech-text LLMs without speech-generation capability.
- SpeechGPT — used as a direct autoregressive baseline, and separately as the comparison architecture in the thinking-traces ablation isolating the reasoning-trace benefit from the diffusion-architecture benefit.
- MegaTTS 3 — used to synthesize diverse user-question audio (via four reference speakers from LibriSpeech) for constructing the THINKINGTALK dataset.
- HiFi-GAN — supplies the frozen vocoder used to convert generated speech tokens into the final output waveform.
- Whisper — used (Whisper-Large-v3) to transcribe generated speech for computing TTS Word Error Rate across all speech-processing evaluations.
- VoiceBench — supplies the evaluation framework used to assess speech reasoning performance on the MMSU benchmark.