ACL · 2026 · Conference

Tianle Liang et al. (Zhejiang University / China University of Petroleum-Beijing at Karamay / Xiamen University) · → Paper · → Preprint · Demo: ? · Code: ✓

Fine-tunes an end-to-end spoken dialogue model to reason in text before acting, call tools from spoken requests, and offload tool retrieval to a parallel auxiliary LLM, using a 470-hour synthesized spoken agent corpus (AgentChat).

Problem

End-to-end spoken dialogue models are mostly trained for reactive conversation. They lag text LLMs at reading tool descriptions and producing well-formed tool calls with correct arguments, and there is little speech data annotated with reasoning traces or tool interactions. Speech input also costs many more tokens than text, so putting a large tool library in the context raises inference latency. The paper targets all three gaps: a definition of an end-to-end spoken agent, agentic training data in speech, and tool use that scales to large tool libraries.

Method

The paper first defines an end-to-end spoken agent along four dimensions: a profile (static persona and timbre, plus context-dependent style), a memory with semantic and acoustic channels, planning through an intermediate reasoning step (input to reasoning to output rather than input to output), and action execution through tool selection and invocation. Only planning and action execution are trained and evaluated in the experiments; profile and memory are part of the definition only.

VoxMind fine-tunes Step-Audio 2 as the main agent. For each spoken user turn, the model first generates an explicit chain-of-thought trajectory (“Think-before-Speak”), then samples an action conditioned on that trajectory, the history and a small local tool set. The action is either a spoken response or a tool call. A special retrieval action (searchTools()) signals that the local tools are insufficient.

Tool scaling uses a second, auxiliary language model (Multi-Agent Dynamic Tool Management). As soon as the reasoning trajectory is produced, the auxiliary model reads it and proposes candidate tools from the global pool, in parallel with the main agent’s action sampling. If the main agent emits the retrieval action, the candidates are merged into the local tool set for the next step; otherwise the local set is unchanged. The paper does not name the auxiliary model or report its size.

AgentChat (about 470 hours) combines a tool-interaction corpus (about 109 hours, 14,805 samples from ToolACE, APIGen-MT and self-built subsets for tool selection, parameter filling, parallel calls, tool search and observation handling) with a general-dialogue corpus (about 361 hours, 38,681 samples from ARC, GSM8K, SciQ, textbook content and general conversation). Text is cleaned with rules and an LLM, then synthesized with CosyVoice 2 using over 600 prompt timbres from Seed-TTS. Reasoning chains are generated backwards from the gold answer, scored 0 to 10 by an LLM, kept at a score of 7 or higher (up to three regeneration attempts), then compressed. Two training mixtures are compared: 1:1 and 1:0.5 (agent-interaction hours to general-dialogue hours), plus small No-Tool, safety and text-only supplementary sets. Training is supervised fine-tuning on 2 H20 GPUs with learning rate 1e-5, batch size 1 with 8 gradient-accumulation steps, and DeepSpeed ZeRO-3. Model size and the speech output path are inherited from the base model and not described.

Key Results

All agent scores come from Gemini-2.5-Flash grading tool calls against ground truth (averaged over three runs) on a held-out AgentChat split, which is synthesized with the same TTS pipeline as the training data. Baselines run zero-shot with prompting.

  • Overall vs. baselines (Table 2). VoxMind reaches 74.57 Overall, the highest in the table, against 71.51 for Gemini-2.5-pro, 68.40 Gemini-2.5-flash, 64.00 for a Qwen3-8B + Whisper cascade (Qwen3, Whisper), 54.94 Kimi-Audio, 54.77 GPT-4o-audio, 39.85 Qwen2.5-Omni and 34.88 for the Step-Audio 2 base. The abstract’s “task completion rate from 34.88% to 74.57%” refers to this Overall score. The paper does not give its formula; the reported values match the mean of the ten columns with FC rescaled from its 5-point scale to 100 (our arithmetic, checked for VoxMind, Gemini-2.5-pro and Step-Audio 2).
  • Cell-by-cell against Gemini-2.5-pro (Table 2). VoxMind is higher on Single-Task TS (98.50 vs 90.98), Task Decomposition TS (95.24 vs 82.54), Parallel TS (89.52 vs 88.57), Contextual Planning PF (62.33 vs 61.64) and Proactive Seeking TU (68.66 vs 26.87). It is lower on Single-Task PF (72.18 vs 75.19), Task Decomposition PF (38.10 vs 52.38), Parallel PF (61.59 vs 69.52), Contextual Planning TS (80.82 vs 84.25) and Result Feedback FC (3.94 vs 4.16). The Overall lead depends on the Proactive Seeking column, which tests the searchTools() convention that only VoxMind was trained on. Under the averaging above, the 41.79-point TU gap contributes about 4.2 points, more than the 3.06-point Overall margin (our arithmetic).
  • Column leaders other than VoxMind (Table 2). Gemini-2.5-flash leads Single-Task PF (77.44) and both Contextual Planning cells (86.99 / 65.75); Gemini-2.5-pro leads Task Decomposition PF and Parallel PF; GPT-4o-audio leads FC (4.22). VoxMind leads Single-Task TS, Task Decomposition TS, Parallel TS, TU and Overall.
  • Base model. Step-Audio 2 scores 4.34 TS and 1.60 PF on Contextual Planning, far below every other system, so part of the 113.79% relative gain reflects a near-failure of the base model on that category.
  • Reasoning ablation (Table 3). Adding the reasoning step raises Overall from 68.83 to 71.97 (1:1) and from 70.97 to 74.57 (1:0.5), with the largest change on Proactive Seeking TU (31.34 to 59.70; 37.31 to 68.66). Reasoning lowers Contextual Planning: the selected 1:0.5 model with reasoning scores 80.82 TS / 62.33 PF there, against 86.30 / 75.34 without reasoning.
  • General conversation (Table 4, VoiceBench). The selected model scores 64.21 Overall vs 64.15 for the base model, effectively a tie. Without reasoning, the score drops to 59.72 (1:1) and 54.80 (1:0.5). Per subset, the selected model gains on CommonEval (3.94 vs 3.12), WildVoice (3.69 vs 3.36), MMSU (53.04 vs 50.82), OBQA (71.87 vs 68.13) and AdvBench (100.00 vs 92.88), but loses on AlpacaEval (3.98 vs 4.19), SD-QA Panda (49.73 vs 55.15), SD-QA GPT (44.85 vs 52.80), BBH (54.69 vs 58.53) and IFEval (18.83 vs 39.64, roughly half).
  • Tool scaling (Figure 4, Table 10). Without the auxiliary model, normalized inference time grows from about 1 to about 32 as the tool pool grows from 1 to 100, and FS/PF accuracy collapses beyond 25 tools (FS below 20 at 75 to 100 tools). With it, inference time stays near 2 and FS stays near 93 to 96 across sizes; at a single tool the auxiliary setup is slightly slower. Auxiliary retrieval takes 1.31 s at 10 tools and 2.64 s at 100 tools, but the main agent’s waiting overhead stays under 0.015 s on average because retrieval overlaps with reasoning. These are tool-orchestration latencies; speech-output latency is not measured.
  • Real speech (Table 9). On 150 ToolMind queries recorded by humans (including stutter, hesitation and noise subsets), FS is 86.00% and PF 60.67%, against 93.33% and 67.33% for the same queries synthesized with CosyVoice.
  • Reasoning overhead (Table 11). Reasoning adds about 88 tokens on average against 701.2 answer tokens in speech-output mode (12.6%), and 84.4 against 52.6 in text-output mode (160.5%).

Novelty Assessment

The contribution is mainly an engineering integration plus a dataset. Chain-of-thought before tool calls, tool retrieval from a large pool, and multi-agent delegation are established in text agents; the paper carries them into an end-to-end speech model and supplies synthesized speech training data for them. The parallel auxiliary retrieval that hides retrieval latency behind the main model’s reasoning is the most distinctive design element. The “formal definition” of a spoken agent is a conceptual framing; two of its four dimensions (profile, memory) are not implemented or tested.

The evaluation favours the proposed system. The main test set is an in-domain split of the authors’ synthesized data, baselines are not fine-tuned on it, and the Proactive Seeking category rewards a retrieval token that only VoxMind is trained to produce. Speech output quality is never evaluated, and there is no comparison with other open spoken tool-use systems such as Stream RAG or AURA, or on an external spoken agent benchmark such as VoiceAgentBench.

Field Significance

Moderate — The paper gives a concrete recipe and an open, 470-hour synthesized corpus for adding tool use to an end-to-end speech LM, and shows that decoupling tool retrieval from the main model keeps tool-call accuracy and latency stable as the tool pool grows. Its ablations also suggest that an explicit reasoning step can limit the loss of general conversational ability during agentic fine-tuning, though instruction following on IFEval still drops by about half.

Claims

  • supports: Training a speech LM to produce an explicit reasoning trace before acting improves tool-use accuracy, most strongly for deciding when available tools are insufficient.

    Evidence: With the same data mixture, adding the reasoning step raises Overall from 70.97 to 74.57 and Proactive Seeking TU from 37.31 to 68.66 (1:0.5 mixture); at 1:1, Overall rises from 68.83 to 71.97. (§4.2, Table 3)

  • complicates: Agentic fine-tuning of an end-to-end spoken dialogue model trades general conversational skills against tool-use skills, and the aggregate score can hide large per-skill losses.

    Evidence: Without reasoning, VoiceBench Overall falls from 64.15 (base) to 54.80 at the 1:0.5 mixture. With reasoning it matches the base (64.21), yet IFEval drops from 39.64 to 18.83, SD-QA GPT from 52.80 to 44.85 and AlpacaEval from 4.19 to 3.98. (§4.2, Table 4)

  • supports: Delegating tool retrieval to a parallel auxiliary model keeps a speech agent’s latency and tool-call accuracy roughly independent of tool-library size, while placing the full library in context does not scale.

    Evidence: Without the auxiliary model, normalized inference time rises to about 32 at 100 tools and FS falls to about 41 at 50 tools and below 20 at 75 and 100 tools; with it, inference time stays near 2 and FS near 93 to 96. Main-agent waiting overhead stays below 0.015 s while auxiliary retrieval grows from 1.31 s to 2.64 s. (§4.2, Figure 4; §Appendix I, Table 10)

  • complicates: Agent evaluations built on TTS-synthesized queries overestimate performance on real recorded speech.

    Evidence: On 150 matched queries, tool selection accuracy is 93.33% on CosyVoice-synthesized speech and 86.00% on human recordings with stutter, hesitation and noise subsets; parameter filling is 67.33% vs 60.67%. (§Appendix H, Table 9)

  • complicates: Explicit reasoning does not improve every agentic skill; context-dependent planning across multi-turn history can degrade.

    Evidence: Contextual Planning TS/PF is 86.99/73.29 and 86.30/75.34 without reasoning, but 84.93/65.75 and 80.82/62.33 with reasoning; the final model also trails Gemini-2.5-flash (86.99/65.75) there. (§4.2, Tables 2 and 3)

Limitations and Open Questions

Warning

All agent results use an LLM judge (Gemini-2.5-Flash) on tool-call correctness over a held-out split of the authors’ own TTS-synthesized data; the generated speech is never evaluated, and the main real-speech check covers only 150 queries.

  • The training and main test data are synthesized from written text corpora, which the authors note lack the disfluency and spontaneity of real speech (§Limitations). The real-speech study shows a 6 to 7 point drop.
  • The reasoning step adds latency before the spoken response (§Limitations). Table 11 counts tokens, but end-to-end response latency and time to first audio are not reported.
  • Base model size, the auxiliary LLM’s identity and size, and the speech decoder are not specified.
  • Profile and memory, half of the proposed spoken-agent definition, are not trained or evaluated.
  • The 1:0.5 mixture improves agent scores but its VoiceBench effect is mixed by subset, and IFEval falls sharply in every fine-tuned variant (17.40 to 23.61 vs 39.64 base), leaving instruction following as an open problem.
  • Baselines are prompted zero-shot while only VoxMind was trained on the searchTools() retrieval convention, so the Proactive Seeking column is not a like-for-like comparison.

Wiki Connections

  • Spoken Language Model — Adapts a speech LM to consume spoken requests and produce reasoning traces and structured tool calls.
  • Speech-to-Speech — Extends an end-to-end spoken dialogue model with tool use while keeping spoken input and spoken responses.
  • Step-Audio 2 — Base end-to-end model that VoxMind fine-tunes, and the weakest baseline in the agent evaluation.
  • CosyVoice 2 — TTS system used to synthesize all AgentChat speech.
  • Seed-TTS — Source of the 600-plus prompt timbres used for speaker diversity during data synthesis.
  • Kimi-Audio — Strongest open end-to-end baseline in Table 2, which VoxMind exceeds on Overall.
  • Qwen2.5-Omni — Open end-to-end baseline that scores 39.85 Overall and 0.00 on Proactive Seeking TU.
  • Qwen3 — Text LLM in the Qwen3-8B + Whisper cascade baseline, which beats all open end-to-end baselines on Overall.
  • VoiceBench — Benchmark used to test whether agentic fine-tuning preserves general conversational ability.
  • Stream RAG — Prior spoken tool-use work cited as limited to isolated retrieval functions; not compared experimentally.