arXiv · 2026 · Preprint
Zhixian Zhao et al. (Northwestern Polytechnical University (ASLP@NPU)) · → Paper · Demo: ? · Code: ?
Summarizes the first Human-like Spoken Dialogue Systems Challenge (HumDial) at ICASSP 2026, a two-track benchmark using authentically recorded (not synthetically mixed or concatenated) human conversations to evaluate emotional intelligence and full-duplex turn-taking, drawing over 100 registered teams and 15 valid submissions.
Problem
Spoken dialogue systems built on Audio-LLMs and omni-modal architectures (GPT-4o Realtime, Qwen2.5-Omni, GLM-4-Voice, and others) have rapidly improved interaction quality, raising the question of how close they actually are to human-level conversational naturalness. Measuring this requires standardized evaluation of two capabilities: emotional intelligence (perceiving and responding to users’ emotional states across a conversation) and real-time interaction mechanisms (navigating interruptions and turn-taking under genuine listening-while-speaking conditions). Existing benchmarks fall short on both fronts: emotion-focused datasets and challenges largely use “static classification” or “pseudo-multi-turn” designs (concatenated single-turn dialogues or synthetic speech) that disrupt natural emotional flow, while full-duplex benchmarks rely on “synthetic mixing,” artificially overlaying separately recorded audio tracks, which fails to capture natural cognitive synchronization (hesitations, cooperative barge-ins) and generally overlooks a system’s ability to correctly stay silent amid background noise.
Method
HumDial is structured as a two-track challenge built on authentic, professionally performed dialogue recordings rather than synthetic audio mixing.
Track I (Emotional Intelligence) evaluates three tasks: Emotional Trajectory Detection (identifying and summarizing emotional changes across a multi-turn conversation), Emotional Reasoning (inferring the underlying causes of a user’s emotional state), and Empathy Assessment (generating empathetic responses in both text and audio). Its dataset is built via a hybrid pipeline: Gemini 2.5 Pro generates coherent 3-5-turn user thought-flow scripts (Task 1 scripts embed disruptive interference events into an initial emotional trajectory; Task 2 scripts embed implicit causal chains driving emotional transitions), professional actors record these scripts across six balanced emotion categories, and Task 3 samples critical emotional segments directly from the resulting multi-turn dialogues. The final Track I score combines automated LLM-judge scores (Qwen3-Omni-30B judging trajectory, reasoning, and textual empathy) with human ratings (20 evaluators, split evenly between Chinese and English groups, all with bachelor’s degrees and 6+ months of annotation experience) for emotional appropriateness and audio naturalness, weighted 0.2/0.2/0.1/0.25/0.25 respectively.
Track II (Full-Duplex Interaction) evaluates real-time decision-making under concurrent listening and speaking, covering two scenario families: Interruption (follow-up questions, negation/dissatisfaction, repetition requests, topic switching, silence/termination) and Rejection (correctly ignoring user backchannels, pauses, third-party speech, and speech directed at others). Its dataset uses DeepSeek to generate naturalistic dialogue scripts embedding specific interaction cues (e.g. barge-ins, side-talk), which professional actors then perform, ensuring interruptions land at cognitively meaningful semantic junctures (such as mid-hesitation) rather than random timestamps, preserving natural overlap timing and prosody. All submitted systems are evaluated in standardized Docker environments on NVIDIA RTX A6000 GPUs, scored on Interruption success/latency, Rejection success (including Early Interrupt Rate), and overall first-response delay, combined into a final score weighted 0.4/0.4/0.2.
Key Results
The challenge drew over 100 registered teams and 15 valid submissions across both tracks. In Track I, the top three teams (TeleAI, NJU-TencentHY, BJTU Unisound) achieved near-ceiling scores on Task 1 (Emotional Trajectory Detection, ~4.8-5.0/5) and Task 2 (Emotional Reasoning, ~4.9-5.0/5), but scored notably lower on Task 3 (Empathy Assessment, generating actual empathetic responses, ~3.7-4.1/5), a pattern that held across nearly all participating teams. In Track II, Badcat achieved the highest raw Interruption success rate (89.7) but a comparatively low Rejection score (57.8), while Cookie asr won overall by balancing a strong Interruption score (79.3) with the best trade-off against latency and Rejection (72.2), for a top final score of 76.6. Across nearly every submitted system, Rejection scores trailed Interruption scores, indicating that distinguishing genuine user turns from background noise or non-instructional speech remains harder than responding correctly to clear interruptions. The organizers state a follow-up publication will use the HumDial datasets to benchmark leading commercial and open-source models directly, beyond the challenge submissions reported here.
Novelty Assessment
The paper’s central methodological claim, that authentic professional-actor recordings of LLM-generated scripts produce more naturalistic evaluation data than concatenated single-turn dialogues or synthetically mixed audio, is a direct and well-reasoned response to specific, named limitations in prior benchmarks (ContextDialog, Multi-Bench for the “pseudo-multi-turn” problem; Full-Duplex-Bench, MTalk-Bench for the “synthetic mixing” problem). Rather than proposing a new modeling technique, the contribution is entirely in dataset construction and evaluation-protocol design: a hybrid LLM-script-plus-human-performance pipeline for two distinct interaction dimensions (emotional intelligence and full-duplex timing) that prior work addressed only separately or with weaker data fidelity. The challenge’s substantial community uptake (100+ registered teams, 15 valid submissions) is itself evidence the benchmark filled a real gap rather than being a purely academic exercise. As a competition summary paper, it reports what happened rather than deeply analyzing why particular systems succeeded or failed, and a promised follow-up paper is where deeper comparative analysis against leading commercial/open-source models is expected.
Field Significance
Tip
High, as the first challenge to jointly benchmark emotional intelligence and full-duplex interaction using authentically recorded (rather than synthetically constructed) human dialogue, HumDial addresses genuine, previously identified gaps in prior emotion and turn-taking benchmarks and demonstrates real community engagement at a major venue. The resulting dataset and evaluation protocol are likely to be reused as a standard reference point for evaluating human-like spoken dialogue systems beyond the challenge itself.
Claims
- supports: Recording professional actors performing LLM-generated dialogue scripts, rather than concatenating single-turn utterances or synthetically mixing separately recorded audio tracks, produces more naturalistic evaluation data for both long-term multi-turn emotional consistency and full-duplex interaction timing.
Evidence: The dataset construction pipeline (LLM-scripted dialogues performed by professional actors) is explicitly designed to address the “pseudo-multi-turn” limitation of prior context-aware emotion benchmarks and the “synthetic mixing” limitation of prior full-duplex benchmarks, which fail to place interruptions at cognitively meaningful semantic junctures like hesitations. (§2, §3.A, §4.A)
- complicates: Current spoken dialogue systems can perform emotional trajectory tracking and causal emotional reasoning at near-ceiling accuracy while still struggling substantially to generate empathetic vocal and textual responses.
Evidence: In the Emotional Intelligence track, top teams scored near-ceiling on Task 1 (Emotional Trajectory Detection, ~4.8-5.0/5) and Task 2 (Emotional Reasoning, ~4.9-5.0/5) but markedly lower on Task 3 (Empathy Assessment, ~3.7-4.1/5 for even the top three teams). (§5, Table 2)
- complicates: In full-duplex spoken dialogue systems, correctly rejecting non-instructional speech without falsely interrupting is a harder capability to achieve than correctly responding to genuine user interruptions.
Evidence: Rejection scores were consistently lower than Interruption scores across nearly every submitted system, including the overall winner (Cookie asr: 79.3 Interruption vs. 72.2 Rejection) and the highest raw Interruption scorer (Badcat: 89.7 Interruption vs. only 57.8 Rejection). (§5, Table 3)
- supports: A composite evaluation score combining automated LLM-judge scoring for semantic and reasoning tasks with human ratings for emotional appropriateness and audio naturalness can differentiate competing spoken dialogue systems across both textual and acoustic dimensions of empathetic response generation.
Evidence: The Track I scoring formula combines Qwen3-Omni-30B-judged trajectory, reasoning, and textual empathy scores with separately collected human ratings (20 evaluators with bachelor’s degrees and 6+ months of annotation experience) for emotional appropriateness and naturalness, and this composite scoring produced a clear final ranking across the 8 submitted teams. (§3.B, Table 2)
Limitations and Open Questions
- As a challenge summary paper, deep comparative analysis of why particular systems succeeded or failed is deferred to a stated follow-up publication rather than presented here.
- The Emotional Intelligence track’s dataset is generated via a hybrid LLM-script-plus-actor pipeline (Gemini 2.5 Pro for scripts, professional actors for performance) rather than sourced from real spontaneous human conversations, so some degree of scripted artificiality remains despite the improvement over synthetic mixing.
- Both tracks’ scoring depends partly on automated LLM judges (Qwen3-Omni-30B for Track I trajectory/reasoning/textual-empathy scores); the reliability and potential biases of this automated judging component relative to full human evaluation are not independently validated within the paper.
- Human evaluation for Track I involves 20 evaluators split into Chinese and English groups; cross-lingual or cross-cultural consistency of emotional-appropriateness judgments is not analyzed.
Wiki Connections
- Speech-to-Speech — benchmarks spoken dialogue systems’ real-time turn-taking and full-duplex interaction capability, directly targeting the dialogue sub-paradigm of speech-to-speech generation.
- Evaluation Metrics — establishes a new two-track benchmark and composite scoring methodology (automated LLM-judge plus human ratings) specifically for human-like spoken dialogue evaluation.
- Subjective Evaluation — uses 20 trained human evaluators to rate emotional appropriateness and audio naturalness of generated empathetic responses.
- Emotion Synthesis — directly benchmarks systems’ ability to perceive, reason about, and generate empathetic emotional responses across multi-turn dialogue.
- Multi-Bench — critiqued as exhibiting a “pseudo-multi-turn” nature (concatenated single-turn dialogues or synthetic speech) that HumDial’s authentic-recording pipeline is designed to avoid.
- Full-Duplex-Bench — critiqued for relying on synthetic audio-track mixing rather than genuine cognitively-timed interruptions, which HumDial’s professional-actor recording pipeline addresses.
- MTalk-Bench — critiqued alongside Full-Duplex-Bench for relying on synthetic mixing that fails to capture natural cognitive synchronization such as hesitations or cooperative barge-ins.
- Qwen3-Omni — used as the automated LLM-judge model for scoring the Emotional Trajectory, Reasoning, and Textual Empathy tasks in Track I.
- GPT-4o — cited as the proprietary model (via its Realtime variant) that set the benchmark for authentic, low-latency speech-to-speech interaction motivating the challenge.
- Qwen2.5-Omni — cited as one of the leading open-source omni-modal dialogue systems the challenge’s evaluation targets are motivated by.
- GLM-4-Voice — cited as one of the leading open-source end-to-end spoken chatbot systems motivating the challenge.