arXiv · 2025 · Preprint
Chiang et al. (National Taiwan University, Microsoft) · → Paper · Demo: ✓ · Code: ?
Introduces a general inference and training framework that lets spoken language models generate unspoken chain-of-thought reasoning while the user is still speaking, rather than only after the user’s turn ends.
Problem
Reasoning LLMs and spoken language models (speech LMs) currently think only after the full user input has arrived: the model waits for the turn to end, then generates a hidden chain-of-thought, then produces its response. This “think after listening” pattern is a poor fit for spoken dialogue, where every second of thinking after the user stops talking is added latency the user experiences directly. Humans do not work this way: psycholinguistic evidence shows listeners begin planning a response mid-utterance, well before the speaker’s turn ends. No prior spoken-dialogue framework had exploited this by giving a model an explicit, trainable mechanism to reason incrementally over partial input while the user is still speaking.
Method
SHANKS (Simultaneous Hearing and Thinking with chunked input Speech) is an inference and training recipe applied on top of an existing spoken language model rather than a new network architecture. At inference time, the user’s incoming speech is segmented into fixed-duration chunks (t_chunk, 4.0s in the main experiments). As soon as a chunk S_i arrives, the model is prompted (via a special [EOPA] “end of partial audio” token) to generate an unspoken thinking chunk R_i, conditioned on all previous speech chunks and all previous thinking chunks, while the user is still speaking the next chunk S_{i+1}. Thinking chunks are wrapped in <think>/</think> tokens and are capped at t_chunk × tokens-per-second so that reasoning for chunk i finishes before chunk i+1 arrives. When the user’s speech ends (marked with an [EOA] token), the model generates a final thinking chunk and then a spoken response O; only O is ever converted to audio output.
During training, a complete user utterance is segmented the same way, and target thinking chunks R_i and a final response O are constructed offline (using GPT-4o and GPT-4o-mini-TTS for speech synthesis) so that a standard next-token cross-entropy loss can be applied over the interleaved sequence {S_1, R_1, S_2, R_2, …, S_N, R_N, O}. Two downstream behaviors are trained into the thinking chunks: (1) interruption, where the model emits a special [INTERRUPT] token inside R_i and then speaks an interruption response, evaluated on a constructed dataset of GSM8K-based math questions with spoken step-by-step solutions (correct or deliberately erroneous, generated by Llama-2-7B/Llama-3.1-8B and voiced with GPT-4o-mini-TTS); and (2) tool calling, where R_i can contain API calls and their responses, evaluated on a speech-synthesized version of ComplexFuncBench, a multi-step API-calling benchmark for a travel-booking agent.
Two backbone instantiations are studied: SHANKS-E2E fine-tunes the thinker module of Qwen2.5-Omni (a thinker-talker end-to-end speech LM, where the thinker consumes speech-encoder features and the talker synthesizes the spoken output), passing only the final response tokens to the talker so intermediate reasoning is never spoken. SHANKS-Cascade instead pipes Whisper-large-v3 ASR transcripts into a separately fine-tuned Qwen2.5-7B-Instruct text LLM, isolating how much of SHANKS’s behavior depends on backbone reasoning strength versus the chunking mechanism itself.
Key Results
On the interruption task (Table 1), SHANKS-E2E interrupts 84.8% of instances in the wrong (erroneous-solution) subset with a 63.9% valid-interrupt ratio, versus 13.8% interrupt ratio and 26.8% valid-interrupt ratio for a “No-thinking” baseline trained on the identical data to predict [INTERRUPT]/[NO INTERRUPT] tokens without intermediate reasoning. SHANKS-Cascade, using the stronger Qwen2.5-7B-Instruct backbone with the same chunked training recipe, does even better (86.1% interrupt ratio, 78.3% valid-interrupt ratio on the wrong subset), suggesting the gains scale with backbone reasoning ability rather than being specific to the chunking mechanism. Varying the inference-time chunk duration (3s vs. 5s vs. the trained 4s) leaves interrupt/valid-interrupt ratios roughly unchanged but shifts interruption latency non-monotonically (1.56s at 3s chunks vs. 8.19s at 5s chunks).
On the tool-calling task (Table 2), both SHANKS variants successfully complete 56.9-63.9% of ground-truth API calls while the user is still speaking (the “early call accuracy”), but SHANKS alone reaches only 71.3% (E2E) or 69.3% (Cascade) total call accuracy and 34.9-35.2% task success rate, versus 86.5% total call accuracy and 63.2% success rate for a “Call-after-listen” baseline that waits for the complete user turn. The paper attributes this gap to SHANKS rarely retrying API calls that fail mid-stream. Combining SHANKS (for early calls) with a call-after-listen fallback pass after the user finishes recovers the best of both: 90.0% total call accuracy, 62.3% success rate, and only 117 tokens generated after the user’s turn ends versus 313 for call-after-listen alone (a 62.3% reduction in post-turn generation).
Both evaluation scenarios are constructed by the authors specifically for this paper (no prior benchmark exists for speech-in speech-out interruption or for spoken tool-calling with early-action credit), and all training and test audio is synthesized with GPT-4o-mini-TTS rather than recorded from real speakers.
Novelty Assessment
The core idea, generating unspoken reasoning chunks that lag the input stream by a fixed duration and training on interleaved speech/thinking/response sequences, is a genuinely new inference and training procedure for spoken language models; no architectural change to the backbone is required. The contribution is best described as a training-recipe and inference-protocol innovation layered on an existing thinker-talker architecture (Qwen2.5-Omni) and an existing cascade recipe (Whisper + Qwen2.5), rather than a new model family. The two evaluation scenarios (interruption, streaming tool calls) are also novel constructions built specifically to demonstrate the framework, which is a strength for illustrating the idea but means neither task has been validated by independent groups or on natural (non-synthesized) speech. The paper explicitly acknowledges a concurrent, closely related work (STITCH) that applies the complementary idea of thinking while the model itself is speaking, underscoring that “thinking during the interaction rather than only after it” is an idea multiple groups converged on independently in the same period rather than one this paper originated in isolation.
Field Significance
Moderate — SHANKS demonstrates, on two self-constructed tasks, that giving a spoken language model an explicit mechanism to reason incrementally over partial streaming input measurably improves both interruption accuracy and the fraction of downstream actions (tool calls) completed before the user’s turn ends. It is an early, proof-of-concept demonstration of “thinking while listening” for speech-in speech-out systems rather than a benchmark result on an established, widely used task, and its own results show real trade-offs (lower task success and response quality than a full-context baseline) that later work would need to resolve before the approach is deployable as-is.
Claims
- supports: Giving a spoken dialogue model an explicit incremental reasoning mechanism over partial input substantially improves its ability to correctly recognize errors in the user’s speech before the user finishes, compared to producing an interruption decision without an intermediate reasoning step.
Evidence: SHANKS-E2E reaches an 84.8% interrupt ratio and 63.9% valid interrupt ratio on the wrong (erroneous-solution) subset of the constructed math-interruption test set, versus 13.8%/26.8% for a “No-thinking” baseline trained on identical data to directly predict [INTERRUPT]/[NO INTERRUPT]. (§5.1, Table 1)
- complicates: Fixed-duration chunking of streaming input necessarily introduces a reasoning lag proportional to the chunk size, and changing that chunk size at inference time trades off latency against reasoning timing without a clear, monotonic effect on decision accuracy.
Evidence: Varying t_chunk from the trained value of 4.0s to 3.0s and 5.0s at inference (without retraining) leaves interrupt ratio and valid interrupt ratio on the wrong subset largely unchanged, while interruption latency varies non-monotonically (1.56s at t_chunk=3 vs. 8.19s at t_chunk=5). (§5.1, Table 1)
- complicates: Acting on partial streaming input to reduce response latency (e.g., issuing tool calls before a request is fully specified) can come at the cost of overall task success and response quality relative to waiting for the complete input, unless the early actions are paired with a full-context fallback pass.
Evidence: SHANKS-E2E alone reaches only 71.3% total API call accuracy and 35.2% task success rate versus 86.5%/63.2% for the call-after-listen baseline that waits for the full user turn; the paper attributes the gap to SHANKS rarely retrying API calls that fail mid-stream, and shows that combining SHANKS with a call-after-listen fallback restores success rate to 62.3% while still completing 57.3% of calls early. (§5.2, Table 2)
- supports: The quality of incremental “thinking while listening” reasoning in a streamed spoken-dialogue framework is governed primarily by the backbone language model’s reasoning strength rather than by the streaming/chunking mechanism itself.
Evidence: Swapping the same chunked training and inference recipe from an end-to-end thinker-talker backbone (Qwen2.5-Omni thinker) to a stronger text-only reasoning backbone (Qwen2.5-7B-Instruct, in the cascade variant) improves valid interrupt ratio on the wrong subset from 63.9% to 78.3% and improves total tool-call accuracy, despite the streaming mechanism being identical across both variants. (§5.1-§5.2, Tables 1-2)
Limitations and Open Questions
Warning
All training and evaluation speech in both scenarios is synthesized with GPT-4o-mini-TTS rather than recorded from real speakers, and the interruption and tool-call tasks are entirely author-constructed with no prior benchmark or independent replication; the reported gains have not been tested on natural, spontaneous speech with disfluencies, overlapping speech, or background noise.
Beyond the synthetic-speech and self-constructed-benchmark limitation, the paper’s own conclusion identifies three further constraints: the approach requires the user’s speech to have enough duration and sequential structure for incremental reasoning to be meaningful (very short utterances give SHANKS no opportunity to think while listening); the fixed chunk size means the model’s reasoning always lags the true input by at least one chunk duration, and the paper explicitly leaves more sophisticated (non-fixed) chunking strategies to future work; and because the user’s intent may still be ambiguous mid-utterance, some generated thinking chunks may be redundant or unhelpful, while the scheme also meaningfully increases inference-time compute cost relative to think-after-listening. The tool-calling results additionally show that SHANKS’s lack of a retry mechanism for failed mid-stream API calls is an unresolved practical gap relative to a full-context baseline.
Wiki Connections
- Spoken Language Model — extends an existing end-to-end thinker-talker speech LM (and a cascaded ASR+LLM pipeline) with a chunked incremental-reasoning training and inference recipe driven by genuine external streaming user speech.
- Speech-to-Speech — both SHANKS instantiations are evaluated as full speech-in, speech-out systems where only the final response (not the intermediate reasoning) is synthesized to audio.
- Evaluation Metrics — introduces new task-specific metrics (interrupt ratio, valid interrupt ratio, interruption latency, early/late call accuracy) to measure reasoning-while-listening behavior, since no prior metrics existed for this setting.
- Qwen2.5-Omni — SHANKS-E2E is built by fine-tuning this thinker-talker end-to-end speech LM’s thinker module to emit unspoken chunked reasoning before its final response.
- Moshi — cited as a full-duplex spoken dialogue model that, despite supporting duplex interaction by design, is found unable to interrupt the user at all, motivating SHANKS’s explicit interruption mechanism.
- Full-Duplex-Bench — referenced as related work on evaluating the turn-taking and full-duplex capabilities that SHANKS’s user-interruption scenario draws on, though SHANKS studies the model interrupting the user rather than the reverse.
- Stream RAG — a concurrent work studying tool/retrieval calls during the user’s turn; SHANKS positions itself as studying a broader thinking-while-listening paradigm with tool-calling as one application, versus Stream RAG’s narrower focus on when to issue retrieval queries without an explicit silent reasoning process.
- Qwen3-Omni — cited as another example of a speech-in speech-out spoken language model for which think-after-listening reasoning is impractical due to real-time latency requirements.