arXiv · 2025 · Preprint
Yayue Deng et al. (StepFun) · → Paper · Demo: ? · Code: ✓
Introduces Multi-Bench, the first benchmark that evaluates spoken dialogue models in genuinely interactive multi-turn conversation, scoring both the textual content and the acoustic emotional alignment of the model’s own generated speech.
Problem
Existing spoken dialogue model (SDM) benchmarks concentrate on single-turn exchanges or on “multi-turn” settings that are in practice concatenations of independent single-turn queries rather than a genuine back-and-forth conversation. Several also restrict scoring to textual correctness (e.g., transcript accuracy, recall of an earlier statement) and ignore the acoustic and prosodic content of the SDM’s own generated speech. Narrow multi-turn benchmarks such as C3-Bench and ContextDialog test only whether a model can recall or repeat an earlier utterance, which the authors argue is a limited proxy for sustained emotionally intelligent dialogue. No prior benchmark combines interactive multi-turn structure with a systematic, hierarchical assessment of emotional intelligence (EI) across both text and speech modalities.
Method
Multi-Bench evaluates SDMs across two hierarchical dimensions: Emotion Understanding and Recognition (Emotion Recognition, Paralinguistic Recognition, Style Inference) and Emotion Reasoning and Application (Emotion Inference, Interactive Dialogue), totaling about 3,212 samples drawn from UnderEmotion, NVSpeech, StyleTalk, PsyQA, PsyDTCorpus, and MultiDialog. The first four tasks are single-turn understanding/reasoning probes; Interactive Dialogue is the paper’s central contribution.
The Interactive Dialogue task runs a closed-loop, multi-turn conversation between a simulated user and the SDM under test. A user profile (scenario, goal, character) is extracted from real English and Chinese dialogues using GPT-4o and DeepSeek-R1 respectively, sampled for topic diversity across daily-life and emotional scenarios. The initial user utterance is converted into emotional speech with Step-Audio-TTS to seed the dialogue. In subsequent turns, the simulated user’s text response is generated by a chat LLM (DeepSeek-V3.1), and an emotion-conditioning mechanism selects the most contextually appropriate emotion (via GPT-4o) from 38 curated emotional audio prompts spanning categories such as sadness, fear, happiness, relaxation, excitement, humor, hesitation, and empathy; Step-Audio-TTS then conditions on the retrieved prompt to synthesize the next emotional user turn. A separate LLM decides when to terminate the conversation (explicit user termination, sufficient emotional relief, or repeated stagnation), with a hard cap of ten turns. The SDM under test receives each user audio turn and must produce both a spoken and a textual response.
Evaluation is dual-modality. Gemini 2.5 Pro acts as an audio-aware LLM (ALLM) judge that scores the emotional alignment of the SDM’s generated speech output, using prompted timestamp-referenced textual and acoustic analysis of the dialogue history, the latest user audio, and the SDM’s response, plus a final sanity-check step. DeepSeek-R1 separately scores dialogue-level EI from the text transcript. The five understanding/reasoning tasks use accuracy or an LLM-judge score depending on question format (multi-choice, semi-open, or open-domain).
Key Results
On the single-turn understanding and reasoning tasks (Table 3), GPT-4o leads overall except on emotion recognition, where Step Audio 2 scores highest (70.80% vs. GPT-4o’s 65.65%) and also leads on style inference (56.14%) and paralinguistic emotion inference (40.00%) among non-GPT-4o systems. The hard paralinguistic-recognition variant (requiring full correctness) remains difficult for every system, topping out at 17.84% (GPT-4o).
The Interactive Dialogue evaluation covers 1,500 dialogues across four datasets, yielding 157,262 total turns at an average of 8 turns per dialogue. GPT-4o achieves the best overall acoustic-dimension (Gemini judge) and textual-dimension (DeepSeek judge) scores, but the gap to Step Audio 2 is small on Chinese-heavy emotional dialogues (3.861 vs. 3.866 on the combined Psy subset) while GPT-4o pulls far ahead on English dialogues, indicating current SDMs handle multi-turn English emotional dialogue less well than Chinese. GLM-4-Voice scores lowest overall, which the authors attribute to its lack of dialogue-history support. Kimi Audio performs weakly on English dialogues, frequently mixing Chinese and English. To validate the ALLM judge, the authors compare its rankings against ten human evaluators under an identical scoring pipeline and report a 0.885 rank correlation.
Novelty Assessment
The primary contribution is evaluation methodology and a new dataset, not a new model or SDM architecture. The genuinely novel elements are: (1) a dynamic, terminatable multi-turn dialogue loop (rather than concatenated single-turn queries) as the evaluation vehicle, and (2) scoring the SDM’s own generated speech for emotional alignment via an audio-aware LLM judge, validated against human raters, alongside a separate text-dimension judge. The emotion-conditioning mechanism that drives the simulated user’s TTS output is a retrieval-plus-classification pipeline over a curated prompt bank rather than a trained or architecturally novel component. Overall this is an incremental but useful engineering and evaluation-methodology contribution built on existing TTS, LLM, and ALLM-judge tools.
Field Significance
moderate — Multi-Bench extends the spoken dialogue model benchmarking literature by adding genuine multi-turn interactivity and by scoring the acoustic/emotional content of a model’s own generated speech rather than only its transcript. It provides a reusable evaluation framework and a validated ALLM-judge methodology for future SDM emotional-intelligence assessment, though its own contribution is evaluation and data curation rather than a new model or architecture.
Claims
- supports: Scoring a spoken dialogue model’s own generated speech for emotional alignment, rather than only its text transcript, surfaces model differences that text-only evaluation would miss.
Evidence: Multi-Bench combines an acoustic-dimension score (Gemini 2.5 Pro judging the SDM’s spoken response for emotional alignment) with a separate text-dimension score (DeepSeek-R1); model rankings and gaps (e.g., GLM-4-Voice trailing across both dimensions due to missing dialogue-history support) are reported from both scores jointly. (§2.2, Table 4)
- complicates: Benchmarks that concatenate independent single-turn exchanges to simulate “multi-turn” evaluation understate the difficulty of sustaining genuinely interactive spoken dialogue.
Evidence: The paper contrasts prior benchmarks such as URO-Bench, whose multi-turn subtasks are concatenations of independent single-turn exchanges rather than a true conversational loop, with Multi-Bench’s adaptively terminated, up-to-ten-turn interactive dialogue loop. (§1, Table 1)
- complicates: Current spoken dialogue models show substantially larger performance gaps on emotionally-charged dialogue than on casual daily conversation, making emotional intelligence a harder and more differentiating capability than general dialogue competence.
Evidence: On the daily-topic NVSpeech subset most models perform similarly except GLM-4-Voice, but on the emotion-centric Psy subset (PsyQA/PsyDTCorpus) the spread widens substantially, with GPT-4o and Step Audio 2 clearly ahead of Qwen 2.5 Omni. (§3.2, Table 4)
- supports: An audio-aware large language model can serve as a scalable proxy judge for human ratings of emotional alignment in multi-turn spoken dialogue evaluation.
Evidence: The paper reports a 0.885 rank correlation between Gemini-2.5-Pro ALLM judge scores and scores from ten human evaluators using the same scoring pipeline and instructions. (§3.2)
Limitations and Open Questions
The benchmark's emotional user turns are entirely TTS-synthesized (Step-Audio-TTS conditioned on retrieved emotional prompts) rather than natural human emotional speech, and the primary acoustic-dimension judge (Gemini 2.5 Pro) and textual judge (DeepSeek-R1) are both closed, proprietary models, which limits reproducibility and independent auditing of the scoring pipeline.
Human validation of the ALLM judge is reported as a single aggregate rank correlation (0.885) across ten evaluators rather than per-task or per-language correlations, so it is unclear whether judge reliability holds uniformly across the five tasks or across English and Chinese. The evaluation covers only six SDMs, several of them large closed or near-closed systems (GPT-4o, Step Audio 2, Qwen 2.5 Omni), leaving open how smaller or fully open-source SDMs would perform. Coverage is limited to English and Chinese.
Wiki Connections
-
Spoken Language Model — evaluates six representative SDMs (GPT-4o, Qwen 2.5 Omni, GLM-4-Voice, Step-Audio-AQAA, Step Audio 2, Kimi Audio) on multi-turn emotional dialogue.
-
Speech-to-Speech — the benchmark’s dialogue loop is a genuinely interactive audio-in, audio-out exchange between a simulated user and the SDM under test.
-
Evaluation Metrics — introduces a dual acoustic/textual scoring framework combining an ALLM judge, a text LLM judge, and UTMOS.
-
Subjective Evaluation — validates the automated ALLM judge against ten human evaluators using an identical scoring pipeline.
-
Emotion Synthesis — scores the emotional alignment of the SDM’s own generated speech and constructs its dialogue inputs through an emotion-conditioned TTS module drawing on 38 curated emotional prompts.
-
Kimi-Audio — evaluated as one of six SDMs under test; shown to mix Chinese and English and underperform on English dialogues.
-
GLM-4-Voice — evaluated as one of six SDMs under test; scores lowest overall, attributed to its lack of dialogue-history support.
-
Step-Audio 2 — evaluated as one of six SDMs under test; the closest competitor to GPT-4o, leading on emotion recognition accuracy.
-
VoiceBench — cited as a prior single-turn SDM benchmark that Multi-Bench positions itself against by adding genuine multi-turn interactivity.
-
AIR-Bench — cited as a prior chat-dimension benchmark using free-form LLM-judged scoring that Multi-Bench extends with multi-turn interactivity and acoustic-dimension evaluation.