arXiv · 2026 · Preprint

Chengyou Wang et al. (Northwestern Polytechnical University (ASLP@NPU) / Nanjing University / AISHELL) · → Paper · Demo: ? · Code: ?

Describes the Full-Duplex Interaction Track of the ICASSP 2026 HumDial Challenge. It releases a bilingual, dual-channel, actor-recorded dataset of interruption and rejection scenarios, defines the HumDial-FDBench protocol (behaviour accuracy plus three latency measures combined into one weighted score), and reports a leaderboard covering challenge submissions, open-source speech LMs, and Gemini 2.5.

Problem

Most spoken dialogue systems still use rigid turn-taking, triggering a response only after voice activity detection or endpointing decides the user has finished. Full-duplex behaviour (responding to barge-in, ignoring backchannels, waiting through hesitation pauses, rejecting third-party or side-directed speech) is hard to develop for two reasons. Public dialogue corpora are mostly single-channel or scripted task dialogues that simplify overlap and interruption. And there is no shared benchmark that scores interactional skills such as interruption handling, response timing, and resuming after disruption, as opposed to recognition accuracy or task completion. Earlier full-duplex benchmarks (Full-Duplex-Bench, FD-Bench) introduced overlap-centred evaluation, but the authors argue that interaction data realistic enough for both training and testing is still missing.

Method

The paper’s contribution is a dataset, an evaluation protocol, and a challenge leaderboard. It proposes no new model.

Dataset. Data is built in two stages. DeepSeek generates dialogue scripts with targeted interaction cues (interjections, asides, utterances addressed to another person), then professional actors perform and record them on two channels. The authors chose this over synthetic track mixing so that natural overlap timing, prosody, and interaction rhythm are preserved. The corpus has more than 100 hours of Chinese and English speech in train/dev/test splits (Table 1). It covers two categories. Interruption has five scenarios: follow-up question, negation or dissatisfaction, repetition request, topic switch, and silence or stop. Rejection has four: user real-time backchannels, pause handling, third-party speech, and speech directed to others. Each test scenario has 600 instances, except speech directed to others with 200. The training split is very uneven: 1,211 to 1,507 instances for most scenarios, but only 120 for third-party speech and none for speech directed to others.

HumDial-FDBench. The protocol extends Full-Duplex-Bench v1.5. Model output is transcribed with time alignment (Paraformer for Chinese, Parakeet-TDT for English). DeepSeek-V3 then sorts each response after overlap into Respond, Resume, Uncertain, or Unknown. In interruption scenarios only Respond counts as correct. For backchannels, third-party speech after the query, and side-directed speech, only Resume counts as correct. For pause handling and third-party speech before the query, a timestamp rule marks the rejection as failed if the model speaks before the user’s semantic content is complete. Latency boundaries come from Silero-VAD. Interruption scenarios measure stop latency, response latency, and a newly added first response latency (end of the first user question to start of the first model response). Rejection scenarios measure only first response latency, and only on successful rejections. The mean latency L is converted to a delay score by log normalisation: 100 at the fastest system’s latency and 60 at the baseline’s latency. The final score is 0.4 × interruption + 0.4 × rejection + 0.2 × delay (§4.3).

Baseline and evaluated systems. The official baseline combines the Easy-Turn turn-taking model with OSUM-EChat. The leaderboard also includes the open-source end-to-end models Freeze-Omni and Moshi, the closed-source Gemini 2.5, and nine ranked team submissions plus one late submission (§5.1).

Key Results

The top three submissions score well above the baseline and all reference models (Table 2):

SystemInt.Rej.Delay (s)Delay scoreFinal
Cookie asr (rank 1, cascaded)79.372.21.26079.976.6
Badcat (rank 2, semi-cascaded)89.757.81.63272.673.5
SenseDialog (rank 3)76.460.91.23780.571.0
Gemini-2.5 (reference)79.836.51.30179.062.3
Baseline (Easy-Turn + OSUM-EChat)75.935.22.53160.056.4
HelloWorld (rank 7, end-to-end)51.336.30.624100.055.0
Freeze-Omni (reference)29.650.22.57859.543.8
Moshi (reference)35.422.82.87656.334.5

Main findings:

  • Rejection is the weaker skill. Every system except Freeze-Omni scores lower on rejection than on interruption. Rejection scores range from 22.8 (Moshi) to 72.2, and only the top three submissions exceed 57. Gemini 2.5 handles interruptions well (79.8) but rejects poorly (36.5). The authors attribute its low latency to a VAD-based interruption strategy, which helps explain the gap.
  • Latency and behaviour come apart. HelloWorld, an end-to-end system, has the lowest latency (0.624 s, delay score 100) but only 51.3 on interruption, ranking 7th. The top-ranked system is not the fastest.
  • Architecture paradigm alone does not decide the ranking. Two of the top three use cascaded or semi-cascaded pipelines with a dedicated turn-decision component (§6.1). But the team named “Cascade”, also cascaded, ranks 9th (37.7), below several end-to-end systems.

Challenge participants who used the training split trained on it, while the reference models were evaluated off the shelf. The leaderboard therefore mixes adapted systems with zero-shot systems. Results are not broken down by language or sub-scenario, and the paper reports no inter-rater or human validation of the DeepSeek-V3 behaviour classifier.

Novelty Assessment

The contribution is resources and evaluation, not modelling. The scenario taxonomy and the Respond/Resume classification protocol come from Full-Duplex-Bench v1.5. What is new:

  1. A bilingual, dual-channel, actor-performed corpus with both training and test splits. Prior full-duplex benchmarks mostly offered test data only, or used synthetic overlap.
  2. Added rejection scenarios: pause handling, third-party speech before and after the query, and speech directed to others.
  3. A first-response-latency metric and a single weighted composite score.
  4. A shared leaderboard that puts challenge systems next to Moshi, Freeze-Omni, and Gemini 2.5.

The analysis of team designs in §6 is qualitative. It sorts submissions by architecture (cascaded, semi-cascaded, end-to-end), turn-taking strategy (heuristic rules, specialised classifiers, LLM-as-judge), and training strategy (extensive fine-tuning vs. frozen pipelines), but runs no controlled ablation to attribute score differences to those choices. The composite score’s weights (0.4/0.4/0.2) and its normalisation to the baseline are design choices the paper does not justify empirically.

Field Significance

Moderate. The paper provides a public, bilingual full-duplex corpus whose training split covers interruption and rejection behaviours, and a leaderboard where cascaded systems with explicit turn-decision modules beat open-source end-to-end speech LMs and Gemini 2.5. It is mainly a benchmark and dataset resource. The architectural lessons it draws come from cross-team comparison rather than controlled experiments.

Claims

  • supports: Full-duplex spoken dialogue systems with an explicit, separately designed turn-taking decision component currently handle interruptions and non-actionable speech better than end-to-end speech LMs that rely on implicit turn control.

    Evidence: The top-ranked cascaded and semi-cascaded submissions reach final scores of 76.6 and 73.5, against 59.2 and 55.0 for the best end-to-end submissions, and 43.8 (Freeze-Omni) and 34.5 (Moshi) for open-source end-to-end reference models. However, the cascaded “Cascade” team scores only 37.7, so pipeline design alone does not guarantee good turn-taking. (§5.2, Table 2; §6.1)

  • complicates: Responsiveness to user barge-in and robustness against non-actionable speech (backchannels, pauses, third-party speech) pull in opposite directions in current full-duplex systems.

    Evidence: Gemini 2.5 scores 79.8 on interruption but 36.5 on rejection, and Badcat scores 89.7 vs. 57.8. Freeze-Omni shows the reverse pattern (29.6 interruption, 50.2 rejection). Only the top-ranked system exceeds 70 on both. (§5.2, Table 2)

  • complicates: Low turn-taking latency does not imply correct turn-taking behaviour, so latency-only or latency-weighted evaluations of full-duplex speech LMs can hide behavioural failures.

    Evidence: HelloWorld has the lowest mean latency (0.624 s, delay score 100) but scores only 51.3 on interruption and 36.3 on rejection, ranking 7th of 9. The ranked winner has a mean latency of 1.260 s. (§4.3; §5.2, Table 2)

  • refines: Delegating turn decisions to an LLM improves handling of semantically ambiguous pauses but leaves systems weaker on purely acoustic rejection cues such as third-party speech without lexical evidence.

    Evidence: The authors’ cross-team analysis contrasts LLM-as-judge systems (Unity Squad with Qwen2.5 completeness labels, Badcat with Qwen3-Omni Switch/Continue tokens) against dedicated acoustic classifiers (Whisper-tiny four-class, TinyBERT plus rules). They also note degraded performance across systems with multi-speaker overlap and background noise. The argument is qualitative, with no per-scenario breakdown. (§6.2; §5.2)

Limitations and Open Questions

  • Behaviour labels come from an unvalidated LLM judge. Every Respond/Resume decision is made by DeepSeek-V3 from ASR transcripts, and the paper reports no agreement with human annotators. Because the classifier only sees transcripts, prosodic and acoustic aspects of the response are not assessed.
  • Coarse reporting. Only aggregate interruption, rejection, and delay scores are reported. Per-scenario, per-language (Chinese vs. English), and per-latency-type (stop, response, first response) results are absent, so the paper’s statements about multi-speaker and noise failures cannot be checked against numbers.
  • Scripted data. Dialogues are LLM-scripted and then acted. This preserves natural overlap timing better than synthetic mixing, but the interactions are not spontaneous.
  • Uneven training split. Third-party speech has 120 training instances and speech directed to others has none, while the test set has 600 and 200 respectively. Rejection results for these scenarios therefore partly measure generalisation without matching training data.
  • Mixed comparison conditions. Challenge submissions could use the training split, while Moshi, Freeze-Omni, and Gemini 2.5 were evaluated without adaptation.
  • Composite score design. The 0.4/0.4/0.2 weights and the log normalisation anchored to the fastest system and the baseline are not justified empirically. The delay score is also relative to the systems submitted, so it is not comparable across leaderboard snapshots if the fastest system changes.
  • Textual inconsistencies. §3.1 says the dataset has “a total of eight scenarios” and that Rejection “includes four sub-scenarios”, but Table 1 and the scenario descriptions list nine (five interruption plus four rejection). §4.3 says L_base “is set to 60”. However, the formula and Table 2 show L_base is the baseline’s latency (2.531 s), with 60 being the resulting delay score. The baseline description cites reference [17] for Easy-Turn, but [17] is the Unit-Based Agent paper that is also cited for the Badcat submission.

Wiki Connections

  • Spoken Language Model — Evaluates end-to-end speech LMs (Moshi, Freeze-Omni, Gemini 2.5, and end-to-end team submissions) on full-duplex interruption and rejection behaviour, where they trail cascaded pipelines with explicit turn modules.
  • Speech-to-Speech — Benchmarks spoken dialogue systems across cascaded, semi-cascaded, and end-to-end paradigms under real-time overlap and barge-in.
  • Evaluation Metrics — Introduces first response latency and a weighted composite of interruption accuracy, rejection accuracy, and log-normalised delay for full-duplex dialogue.
  • The ICASSP 2026 HumDial Challenge — The parent challenge overview. This paper details its Full-Duplex Interaction Track.
  • Full-Duplex-Bench v1.5 — HumDial-FDBench builds directly on its overlap-handling protocol and Respond/Resume/Uncertain/Unknown response classification.
  • Full-Duplex-Bench — The first overlap-centred full-duplex benchmark, positioned as the starting point that this track extends with real recorded dual-channel data.
  • FD-Bench — A prior full-duplex benchmarking pipeline covering longer multi-turn interaction, cited as related evaluation work.
  • OSUM-EChat — Response generator in the official baseline, paired with the Easy-Turn turn-taking model.
  • Unit-Based Agent for Semi-Cascaded Full-Duplex Dialogue — The Badcat submission (rank 2), a semi-cascaded system that emits Switch/Continue decision tokens from Qwen3-Omni.
  • Moshi — Open-source end-to-end reference model, with the lowest final score on the leaderboard (34.5).
  • Freeze-Omni — Open-source reference model that rejects better than it handles interruptions (50.2 vs. 29.6), the reverse of most systems.