arXiv · 2026 · Preprint
Zhiyuan Song et al. (University of Pennsylvania) · → Paper · Demo: ? · Code: ✓
Recasts streaming spoken reasoning in a large audio-language model as a learned wait-think-answer control policy, trained with SFT and then DAPO on a six-term trajectory reward, so that the model emits short reasoning updates while the user is still speaking and leaves less deliberation for after the speech endpoint.
Problem
Spoken reasoning evaluations for audio-language models usually follow a complete-audio protocol: the model hears the whole question, then reasons, then answers. In live interaction this pushes all deliberation into the user-visible gap after the speaker stops, while answering early risks committing before the decisive information arrives. Prior “think while listening” work (Shih et al., ICLR 2026, not in this corpus) uses an inference-time question-completeness trigger followed by a single post-utterance chain of thought, and SHANKS and related systems interleave reasoning with streams. None trains the model itself to decide, under partial audio, when to stay silent and when to externalize a reusable intermediate reasoning state.
Method
The controller is Qwen2.5-Omni-7B (Thinker only; output is text) with a serialized three-action vocabulary: <wait/>, <think>...</think>, and <answer>...</answer>. Audio arrives on a fixed 0.5 s decision grid. At each decision step the controller observes the full audio prefix heard so far plus all previously committed visible thoughts, and chooses WAIT (advance without changing memory) or THINK (append a short semantic state update). At the speech endpoint it must emit one final THINK and then an ANSWER, so answer timing is endpoint-gated rather than learned. Because Qwen2.5-Omni exposes no controller-style KV-cache interface, the reported system uses full-prefix replay (each call re-encodes the whole prefix); a cache-native prototype is described only as a wiring-level smoke test (Appendix B.2).

Training data: GPT-4o (GPT-4o System Card) generates 80,000 semantic controller traces (spoken surface form, TTS style instruction, lexical anchors, wait-think-answer trace, answer), which after filtering yield 75,723 records (38,213 verifiable). Each input is rendered as one full utterance with Qwen3-TTS, CTC forced alignment maps words to time, and action anchors are snapped to the 0.5 s grid.
Training proceeds in two stages. SFT uses LoRA (rank 8) on all linear layers of the language model for one epoch, with the audio encoder and aligner frozen. DAPO (a GRPO-family method with asymmetric clipping 0.20/0.28, KL 0.01, 8 rollouts per prompt, 1000 steps) is then initialized from the SFT adapter and run on the 37,180 verifiable records. The reward combines four rule-based terms (answer correctness R_a, protocol validity R_f, post-endpoint final-think length R_s, update timing R_u) and two terms scored by a local Qwen3.6-35B-A3B judge (thought quality R_t, chain consistency R_c, the latter only credited when the answer is correct). A protocol gate returns only the format penalty for any malformed or early-answer rollout. Ablations add R_t and R_c progressively, giving 4-, 5- and 6-reward variants.
Key Results
All latency claims use post-endpoint final-think length in tokens, a controller-level proxy for residual deliberation, not wall-clock response delay.
On synthetic spoken SRQA (8,959 TTS-rendered items, Table 2), within the deployment-mode Qwen controller block the 6-reward DAPO controller has the highest row-weighted accuracy (70.3% vs. 67.6% for the base controller, 66.1% for SFT, 68.5% and 69.2% for the 4- and 5-reward variants) and the shortest final-think length (8.99 vs. 10.44 tokens for the base, a reduction of about 14%). It is highest or tied-highest in that block on every task except LLaMA-QS, where the SFT controller leads (71.7 vs. 71.0). The SFT stage alone lowers synthetic accuracy below the base controller (66.1 vs. 67.6), and the 4- and 5-reward DAPO variants raise accuracy but lengthen the final think (10.87 and 10.94 tokens), so the latency gain depends on adding the chain-consistency term.
Against the complete-audio (offline) Qwen2.5-Omni-7B, the 6-reward streaming controller is slightly lower on average (70.3 vs. 70.8) and lower on PIQA (69.2 vs. 71.3) and GSM8K (25.9 vs. 26.7), but higher on ARC-C (81.7 vs. 81.1) and SIQA (71.0 vs. 70.9), tied on ARC-E (89.6) and LLaMA-QS (71.0). It also exceeds the offline Audio Flamingo 3 (47.4), Audio Flamingo 3 + AF-Think (30.5) and GLM-4-Voice-9B (40.2) averages, and the literature-reported Moshi rows from Shih et al. (best average 56.6). Those comparisons cross evaluation protocols and, for Moshi, are copied from another paper with a different latency unit, so the authors treat them as context only.
On Real Audio Bench (186 human recordings, five speakers, Table 3), the picture is less favorable to the 6-reward controller. SFT has the highest accuracy (68.8%, +4.8 over the 64.0% base), the 5-reward variant is next (67.7%), and the 6-reward variant gains only +1.1 points (65.1%). The 6-reward variant is the only learned controller whose final-think length falls below the base (6.33 vs. 6.52 tokens, a 0.19-token reduction); the others lengthen it (+0.12 to +1.22). The 95% bootstrap intervals overlap across all lanes, including the base (Table 8), so no controller ranking on this set is statistically supported.
The replay-harness runtime audit (Table 5) shows real-time factors above 1 for every lane because of repeated prefix prefill; the 6-reward controller (mean RTF 1.134) roughly matches the base (1.137) and is faster than SFT (1.353).
Novelty Assessment
The contribution is a training formulation rather than a new model: the base architecture is unchanged, and SFT with LoRA plus DAPO are established recipes. What is new is the action space (stateful visible thoughts that accumulate across streaming windows instead of one post-audio rationale or a completeness trigger), the aligned data pipeline that grounds thought anchors in TTS-rendered audio time, and the six-term trajectory reward with a protocol gate that scores timing and consistency of intermediate reasoning, not only the final answer. The empirical gains are modest (+2.7 points synthetic accuracy, 1.45 fewer final-think tokens) and come from a single base model; the most directly comparable method (question-completeness control from Shih et al.) is not re-implemented in the same harness, and answer timing itself is not learned. The Real Audio Bench is a small new transfer set but not a primary contribution.
Field Significance
Moderate. The paper shows that when a speech LM reasons during listening, and not only after it, can be made a trainable policy with an RL reward over the whole streaming trajectory, giving a concrete recipe for shifting the accuracy versus residual-deliberation operating point of spoken conversational agents without changing the backbone. Its evidence is limited to token-count latency proxies, endpoint-gated answers and one model family, so it is a methodological step rather than a demonstrated deployment gain.
Claims
- supports: Reinforcement learning over full streaming trajectories can move a speech LM toward reasoning during the audio stream, improving accuracy while shortening post-endpoint deliberation.
Evidence: Under the same deployment harness, the 6-reward DAPO controller raises synthetic SRQA row-weighted accuracy from 67.6% (base) to 70.3% and lowers mean final-think length from 10.44 to 8.99 tokens. (§5.1, Table 2)
- complicates: Supervised imitation of interleaved reasoning traces teaches the streaming action format but can reduce task accuracy relative to the untuned model.
Evidence: The SFT controller falls to 66.1% synthetic SRQA accuracy against 67.6% for the base controller; task-level reward from DAPO is needed to recover and exceed it. (§5.1, Table 2)
- refines: Rewarding answer correctness, format and timing alone does not shorten post-endpoint reasoning in a streaming speech LM; content-level rewards on intermediate thoughts are needed for the latency gain.
Evidence: The 4- and 5-reward DAPO variants improve accuracy but lengthen final-think length to 10.87 and 10.94 tokens; only adding the chain-consistency term brings it to 8.99. (Appendix D, Table 7)
- complicates: Streaming-reasoning gains measured on TTS-rendered spoken questions may not transfer reliably to human-recorded speech.
Evidence: On 186 human recordings, the controller that is best on synthetic data gains only +1.1 accuracy over the base while SFT gains +4.8, and the 95% bootstrap intervals overlap across all lanes. (§5.2, Table 3; Appendix D.1, Table 8)
- complicates: Token-count proxies for response latency in streaming speech LMs can diverge from wall-clock cost when the serving stack lacks incremental caching.
Evidence: Full-prefix replay gives real-time factors above 1 for every controller (mean 1.134 to 1.353), so final-think length reductions are reported separately from runtime, and a cache-native implementation exists only as an unscored prototype. (Appendix B.1, Table 5; Appendix B.2)
Limitations and Open Questions
Warning
Latency is measured as post-endpoint final-think token count inside the authors’ replay harness, answer timing is fixed to the speech endpoint, and the actual replay system runs slower than real time; no wall-clock response-delay improvement is demonstrated.
- Training and the main benchmark both use TTS-rendered speech (Qwen3-TTS for training inputs); the only human-speech evaluation is 186 recordings from five speakers with overlapping confidence intervals.
- A single base model (Qwen2.5-Omni-7B) is studied; external baselines run only in offline mode, and Moshi numbers are copied from Shih et al. with an incomparable latency unit.
- The closest prior method (question-completeness control) is not re-implemented in the same harness, which the authors list as future work.
- Two reward terms depend on a local LLM judge, and reward weights were tuned on the synthetic validation set.
- Output is text only; the model’s speech-generation path is not exercised or evaluated.
Wiki Connections
- Spoken Language Model — Trains a large audio-language model consuming streaming user speech to decide when to emit intermediate reasoning during listening.
- RLHF Speech — Applies DAPO with a six-term trajectory reward to a speech-input model, with ablations over reward composition.
- Qwen2.5-Omni — Base model for every controller variant and the offline complete-audio reference.
- Qwen3-TTS — Renders all synthetic training inputs as full utterances before forced alignment.
- SHANKS — Related simultaneous hearing-and-thinking approach for spoken language models that the paper positions against.
- Moshi — Literature-reported streaming baseline rows (from Shih et al.) shown for context on synthetic SRQA.
- Audio Flamingo 3 — Offline baseline, with and without AF-Think reasoning, on synthetic SRQA.
- GLM-4-Voice — Offline end-to-end spoken chatbot baseline on synthetic SRQA.
- Audio-CoT — Prior work on chain-of-thought in audio LLMs that motivates explicit intermediate reasoning.
- GPT-4o System Card — GPT-4o generates the synthetic controller traces and Real Audio Bench candidate items.