Speech Generation Research Wiki

Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models

7 min read

arXiv · 2025 · Preprint

Xie et al. (NTU / NUS / Tencent) · → Paper · Demo: ? · Code: ✓

Mini-Omni-Reasoner introduces a “thinking-in-speaking” paradigm that interleaves silent reasoning tokens with spoken response tokens at the token level, enabling a large speech model to reason continuously during real-time speech generation rather than completing reasoning before speaking.

Problem

Spoken conversational agents that incorporate chain-of-thought reasoning face a fundamental latency problem: the standard “thinking-before-speaking” paradigm requires the model to finish its entire internal reasoning trace before producing any audio output. This either forces users to wait through a silent reasoning phase (high first-token delay) or requires verbalising the full reasoning chain, which generates verbose, slow-to-consume audio. Prior work on spoken dialogue models (Mini-Omni, LLaMA-Omni, Freeze-Omni, Qwen2.5-Omni) either lack structured reasoning entirely or suffer from excessive response length when reasoning is enabled. Text-domain solutions for reasoning efficiency (chain compression, adaptive reasoning, draft-token skipping) assume random-access output and do not transfer directly to speech, where output is consumed strictly in time-linear order.

Method

Mini-Omni-Reasoner is built on the Thinker-Talker architecture. The Thinker is a 3B-parameter language model (initialised from Qwen2.5-Omni-3B) that receives audio input via a pretrained audio encoder and a lightweight linear adapter. Instead of completing a reasoning trace before emitting response tokens, the Thinker alternates between generating response tokens and reasoning tokens in a fixed 2:8 ratio — two outward-facing response tokens followed by eight internal reasoning tokens per cycle. The Talker, a separately trained compact module, converts only the response tokens to speech via the SNAC codec; reasoning tokens remain silent throughout.

The interleaving ratio is derived from deployment constraints: a 3B model generates roughly 100 tokens per second on a GPU, while natural speech playback requires approximately 12.5 tokens per second; 20 response tokens per second suffice for smooth playback. Masked control tokens — inserted at reasoning/response boundaries but excluded from the loss — stabilise the alternation pattern and prevent the model from drifting out of the target ratio.

Overview of the Mini-Omni-Reasoner. Given a raw audio instruction, the audio encoder transforms it into a sequence of audio tokens embedded in language space. These tokens are used to pre-fill the Thinker LLM, initializing its context for autoregressive generation. The Thinker then generates an interleaved sequence of answer and reasoning tokens. Unlike conventional approaches that emit only answer tokens, our interleaved formulation enables the model to engage in internal reasoning while maintaining uninterrupted response generation. The answer tokens are streamed into the Talker module and decoded into speech in real time via a codec decoder, whereas the reasoning tokens remain silent and guide the generation process.

Training proceeds in five stages: (1) audio adapter alignment with the language backbone; (2) mixed mathematical pretraining in thinking-before-speaking format; (3) textual thinking-in-speaking training (language model parameters only); (4) acoustic transfer — audio encoder fine-tuning with the frozen LLM; (5) Talker training with the entire Thinker frozen. This staged approach isolates architectural adaptation from modality transfer and from acoustic synthesis.

To create training data for interleaved reasoning, the authors introduce SPOKEN-MATH-PROBLEMS-3M, a 3M-sample dataset constructed from public text-based maths QA corpora (GSM8K, Orca-Math, MetaMath, SimpleOP). Questions are converted to speech via CosyVoice2-0.5B TTS; reasoning and response streams are interleaved at the 2:8 ratio with a deliberate asynchronous alignment scheme — the response stream begins with soft contextual phrases before substantive content, ensuring reasoning temporally precedes its corresponding verbal output. A GPT-based verification stage filters examples where answer content appears before its corresponding reasoning.

Key Results

On the Spoken-MQA benchmark (arithmetic and contextual reasoning sub-tasks), Mini-Omni-Reasoner (3B) achieves:

  • Arithmetic: 77.25% average (short: 92.9%, long: 66.1%), compared to the 3B base model Qwen2.5-Omni-3B at 64.9% (+12.4%) and to the best 7B conversational model Qwen2.5-Omni-7B at 66.8% (+10.5%).
  • Contextual Reasoning: 68.1% average, up from the base model’s 64.0% (+4.1%), competitive with the 7B model (75.1%).
  • Response length: 42.9 words vs. 116.1 words for Qwen2.5-Omni-7B, representing a reduction of over 60% in audible output.
  • First-token latency: described as zero decoding latency relative to the thinking-before-speaking baseline at the same model scale.

The model outperforms all other 3B conversational models evaluated and matches or exceeds 7B-scale models on arithmetic tasks, despite the smaller parameter count. The cascade upper bound (Whisper + Qwen2.5-Math-7B-Instruct) reaches 77.3% on arithmetic, which the proposed model matches at 77.25%.

Note

Evaluation is limited to the Spoken-MQA benchmark, which focuses on mathematical and arithmetic reasoning. Naturalness and intelligibility of the spoken output are not reported with MOS or WER; the claim of zero latency refers to the absence of a silent pre-speech reasoning phase, not to absolute time-to-first-byte.

Novelty Assessment

The “thinking-in-speaking” interleaving principle is a genuine conceptual contribution — the observation that the token generation throughput of an LLM far exceeds the real-time audio playback rate creates a bandwidth surplus that can be allocated to silent reasoning, with no latency penalty to the user. This insight is clean and practically motivated. The execution builds on the existing Thinker-Talker architecture (Xu et al., 2025) rather than proposing a new model structure, and the interleaving mechanism is straightforward once the design principle is established. The SNAC codec and Qwen2.5-Omni-3B backbone are off-the-shelf. The dataset construction pipeline is systematic but not technically novel; converting text-domain maths datasets to speech via TTS is a standard procedure. The main contribution is thus the formulation and training recipe rather than an architectural advance.

Field Significance

Moderate — This paper addresses a specific but practically important bottleneck in spoken dialogue systems: how to incorporate chain-of-thought reasoning without introducing user-perceptible latency. The insight that inference throughput surplus can finance silent reasoning is transferable beyond this specific system and benchmark. The evaluation is limited to a single mathematical reasoning benchmark, which constrains how broadly the results generalise; performance on open-domain conversational tasks, naturalness, and robustness to non-maths queries remain uncharacterised.

Claims

  • In speech models, reasoning depth and response latency are not fundamentally in conflict when the model’s token generation rate substantially exceeds the real-time audio playback rate. (§1, §3.1)
  • Interleaving silent reasoning tokens with spoken response tokens at a fixed ratio can improve accuracy on structured reasoning tasks while reducing audible output length. (§2.3, §4.4, Table 2)
  • The “thinking-before-speaking” paradigm, when applied directly to speech, produces user-facing latency or verbosity that impairs conversational quality independently of reasoning correctness. (§1, §2.2)
  • Multi-stage training — separating modality alignment, reasoning transfer, and acoustic synthesis — is an effective strategy for progressively adapting an existing speech LLM to a new generation paradigm. (§3.3)
  • Synthetic speech-based mathematical reasoning datasets constructed from text corpora via TTS can provide sufficient training signal for spoken reasoning capabilities. (§3.2, §4.4)

Limitations and Open Questions

Warning

The entire evaluation uses a single benchmark (Spoken-MQA) focused on mathematics. There is no assessment of speech naturalness, intelligibility, or reasoning accuracy on open-domain conversational tasks. Reported latency claims refer to the absence of a pre-speech reasoning phase rather than to measured real-time performance metrics.

The fixed 2:8 interleaving ratio is derived from a throughput estimate for a specific GPU configuration; it is not adaptive and may be suboptimal for different deployment environments or model sizes. The training data is entirely synthetic — both the audio (produced by CosyVoice2-0.5B) and the reasoning traces (constructed algorithmically from text datasets). Whether the model generalises to naturalistic spoken queries beyond maths problems is untested. The GPT-based verification stage for dataset quality introduces a dependency on a proprietary closed model that is not reproducible.

Wiki Connections

Backlinks

Graph View