arXiv · 2025 · Preprint
He Zhang et al. · → Paper · Demo: ? · Code: ✓
Introduces a benchmark that segments continuous full-duplex spoken dialogues into discrete, evaluable turns and scores full-duplex speech language models across four dimensions (conversational features, dialogue quality, instruction following, safety) over up to ten conversational rounds.
Problem
Existing evaluations of full-duplex speech language models (FD-SLMs) focus almost entirely on single-round interactions: a benchmark presents one user turn (with a pause, interruption, or overlap) and scores the model’s handling of that isolated event. Real deployments, however, involve extended multi-round exchanges, and prior single-round benchmarks cannot reveal whether a model’s conversational competence holds up as a dialogue continues. Extending evaluation to multiple rounds is non-trivial for two reasons specific to full-duplex communication: turn boundaries are blurred, since full-duplex audio has no explicit turn-taking structure the way half-duplex transcripts do, so it is unclear where one round ends and the next begins; and context inconsistency arises because a model’s own generated response in an earlier round can diverge from the ground-truth response that the rest of the dialogue was scripted against, causing later rounds to evaluate the model in scenarios it would never encounter naturally. Prior multi-round attempts (e.g., FD-Bench, Talking Turns) either sidestep turn-level scoring in favor of aggregate quality, or restrict themselves to a single evaluation scenario, and none of the existing full-duplex benchmarks jointly cover conversational dynamics, dialogue quality, instruction following, and safety.
Method
MTR-DuplexBench contributes two things: a turn segmentation methodology for converting continuous full-duplex audio into discrete, evaluable rounds, and an evaluation suite spanning four dimensions applied turn-by-turn across up to ten rounds.
The turn segmentation algorithm proceeds in four steps. First, dual-channel audio (user and assistant) is transcribed with timestamps using Whisper-timestamped plus Silero VAD and medium-sized Whisper, producing fragmented per-channel segments. Second, the sorted, merged timeline of segments is passed to GPT-4o, which proposes turn boundaries; this is repeated six times to account for GPT-4o’s segmentation variability. Third, a majority-voting step clusters the six candidate segmentations by requiring at least 30% temporal overlap to merge into a candidate turn, taking the median start/end time of merged turns and discarding candidates with only a single vote. Fourth, any residual overlapping candidate turns are merged. Once user turns are fixed, the assistant’s response window for a given turn is defined from the start of the current user turn to the end of the next user turn, with all earlier assistant turns populated by ground-truth speech (teacher forcing) so that each evaluated round only depends on the model’s response to the current turn, not on its own possibly-diverging prior outputs. This directly targets the two identified challenges: turn boundaries are made explicit, and context inconsistency is bounded by keeping every turn except the one under evaluation grounded in ground truth.
Four evaluation dimensions are then applied on top of the segmented turns, each with its own data source and metric. Conversational features (smooth turn-taking, interruption, pause handling, background speech, backchanneling) reuse feature definitions from Full-Duplex-Bench and Full-Duplex-Bench v1.5 but extend them to ten-round synthetic dialogues generated by GPT-4o and synthesized into speech with CosyVoice 2, scored with a binary per-round success criterion and, for smooth turn-taking, an improved “takeover” metric that also penalizes speaking over the user. Dialogue quality is scored on natural (non-synthetic) 120-second Candor conversations, segmented with the turn methodology above, transcribed with Whisper-large-v3 and stable-ts, and rated 0-5 by a GPT-4o judge prompt (rescaled because GPT-4o’s own scoring of ground-truth dialogue is conservative, averaging 2.97/5). Instruction following and safety reuse existing spoken-query datasets (Llama Question from OpenAudioBench for instructions; AdvBench from VoiceBench for harmful requests), reorganized into ten-round sequences under either smooth turn-taking or interruption interaction styles, with GPT-4o making a binary success/refusal judgment per round. The benchmark evaluates four baseline systems: Moshi (end-to-end FD-SLM), Freeze-Omni (cascaded FD-SLM built on a frozen LLM), VocalNet (a half-duplex speech LLM, included as an upper-bound reference), and a cascaded pipeline built from SenseVoice ASR, GPT-4o, and ChatTTS.
Key Results
Across all four conversational features, every evaluated model’s per-round success rate declines monotonically as the number of rounds increases (e.g., Moshi’s smooth turn-taking success falls from 73.0% at round 1 to 57.4% averaged over rounds 1-10; Freeze-Omni’s background-speech handling collapses from already-poor single-digit values toward near zero). Latency also grows with round count for every model on every feature, including the half-duplex VocalNet baseline, and the cascaded pipeline exhibits the largest and most erratic latency (roughly 9-12 seconds versus under 2 seconds for the end-to-end and cascaded-lightweight systems). When conversational features are combined (smooth turn-taking plus interruption plus pause handling plus background speech), success rates degrade further than in any single-feature condition, and even the previously-perfect VocalNet and cascaded baselines drop to roughly 50% once background speech is added to the combination.
On dialogue quality (rescaled GPT-score, 0-5), the half-duplex VocalNet scores highest (3.96), the cascaded FD-SLM Freeze-Omni is second (3.48), and the end-to-end FD-SLM Moshi scores lowest (3.13); the fully cascaded ASR+LLM+TTS pipeline scores lowest overall (2.19), which the authors attribute to its high latency preventing full responses within the allotted window (its average response length is roughly half that of the other systems). On instruction following and safety (Tables 6-7), all baselines maintain refusal rates at or above roughly 90% across all ten rounds under both smooth turn-taking and interruption conditions, but Moshi’s instruction-following success rate degrades substantially with round count (68% at round 1 down to 41.9% averaged over rounds 1-10 under smooth turn-taking), while the other three baselines remain comparatively stable.
Novelty Assessment
The contribution is evaluation methodology and empirical characterization, not a new model or architecture. The turn segmentation algorithm (GPT-4o-based boundary proposal with repeated-sampling majority voting) is a concrete, reusable mechanism for a problem (blurred turn boundaries in full-duplex audio) that prior benchmarks either avoided by using only single-round scenarios or handled less rigorously. The four-dimension evaluation suite is an aggregation of existing evaluation ideas (conversational features from Full-Duplex-Bench and its v1.5 extension, dialogue-quality scoring in the style of prior GPT-judged approaches, and instruction-following/safety datasets borrowed from OpenAudioBench and VoiceBench) rather than a conceptually new evaluation axis, but the paper’s own comparison table shows no prior full-duplex benchmark combines multi-round dialogue, turn-by-turn evaluation, conversational features, dialogue quality, instruction following, and safety in one suite. The empirical finding that performance degrades with round count, and specifically that this degradation is invisible to single-round benchmarks, is the paper’s most transferable result.
Field Significance
moderate — this paper contributes a methodological building block (turn segmentation for full-duplex audio) and a multi-dimensional evaluation protocol that extends prior single-round full-duplex benchmarks into the multi-round regime, and it produces evidence that current FD-SLMs degrade over conversational rounds in ways single-round benchmarks cannot detect. Its significance is bounded by reliance on an LLM judge for most of its scoring dimensions and by evaluating only two genuinely full-duplex open-source systems.
Claims
- supports: Full-duplex spoken dialogue systems that handle single-round conversational events well can still degrade substantially in success rate and response latency as the number of conversational rounds increases, a failure mode that single-round evaluation protocols cannot detect.
Evidence: Across all four conversational features (smooth turn-taking, interruption, pause handling, background speech), every baseline’s per-round success rate and latency worsen monotonically from round 1 to the round 1-10 average, and combining multiple features compounds the degradation further. (§4.2.1, Tables 3-4)
- supports: Full-duplex conversational capability and dialogue semantic coherence trade off against each other among current systems: greater duplex sophistication (end-to-end full-duplex generation) is associated with lower dialogue meaningfulness than cascaded or half-duplex alternatives.
Evidence: On the rescaled GPT-score dialogue-quality metric, the end-to-end FD-SLM (Moshi, 3.13) scores below the cascaded FD-SLM (Freeze-Omni, 3.48), which scores below the half-duplex baseline (VocalNet, 3.96). (§4.2.2, Table 5)
- supports: Under multi-round and interruption stress, safety refusal behavior in speech language models generalizes more robustly than instruction-following behavior for at least some current architectures.
Evidence: Refusal rates on the multi-round safety evaluation stay at or above roughly 90% across all ten rounds and both interaction styles for every baseline, while Moshi’s instruction-following success rate drops from 68% at round 1 to 41.9% averaged over rounds 1-10 under smooth turn-taking, a decline not mirrored by the other baselines. (§4.2.3, Tables 6-7)
- complicates: Benchmark scores for dialogue quality and task success in full-duplex speech evaluation that are produced by an LLM judge rather than real human listeners require calibration and should not be read as equivalent to human-rated naturalness or quality judgments.
Evidence: The GPT-4o judge used for the dialogue-quality metric assigns a conservative average score of 2.97 out of 5 even to ground-truth human dialogues, forcing the authors to rescale all GPT-scores from a 0-2.97 range to 0-5 before they are interpretable as absolute quality; conversational-feature success, instruction-following success, and safety refusal are likewise all determined by a binary GPT-4o decision rather than by human raters. (§4.2.2, §3.2.1, §3.2.3)
Limitations and Open Questions
All of the benchmark's headline metrics (conversational-feature success, dialogue-quality GPT-score, instruction-following success, safety refusal rate) are computed by an automated GPT-4o judge, not by human listeners or raters. The paper's own Appendix B human evaluation validates only the turn-segmentation boundaries (with modest inter-annotator agreement on omitted turns), not the quality judgments themselves; no listening test or human rating of model responses is reported.
The authors themselves note several additional limitations. The benchmark mixes natural (Candor) and synthetic (GPT-4o-scripted, CosyVoice-2-synthesized) data, which may not capture the full diversity of real-world conversation. Coverage is English-only, limiting applicability to multilingual or cross-lingual FD-SLMs. The conversational-feature taxonomy is inherited from prior single-round benchmarks (Full-Duplex-Bench) to preserve comparability, so some interaction dynamics, such as user backchannels, are not evaluated as a model-response feature. Evaluation design is inconsistent across dimensions: dialogue-quality assessment uses teacher-forced inference (grounding all rounds except the one under test in ground truth), while conversational-feature, instruction-following, and safety evaluation instead let the model build autoregressively on its own outputs, which the authors acknowledge introduces a gap between the teacher-forced and free-running evaluation setups. Only two genuinely full-duplex, end-to-end or cascaded open-source systems (Moshi and Freeze-Omni) were available for evaluation at the time of writing; the half-duplex and fully cascaded baselines are included for contrast rather than as full-duplex competitors.
Wiki Connections
- Spoken Language Model — evaluates full-duplex speech language models that continuously consume a live external user audio stream and generate speech responses in real time, the defining setting this concept covers.
- Speech-to-Speech — targets the dialogue sub-paradigm of speech-to-speech interaction, specifically the full-duplex variant where listening and speaking overlap.
- Evaluation Metrics — introduces a turn segmentation methodology and a four-dimension evaluation protocol (conversational features, dialogue quality, instruction following, safety) purpose-built for multi-round full-duplex assessment.
- Moshi — evaluated as the primary end-to-end FD-SLM baseline across all four benchmark dimensions.
- Freeze-Omni — evaluated as the cascaded FD-SLM baseline, contrasted against Moshi’s end-to-end design.
- VocalNet — used as a half-duplex speech LLM reference point to contrast full-duplex and non-full-duplex behavior.
- Full-Duplex-Bench — supplies the conversational-feature taxonomy (smooth turn-taking, interruption, pause handling, background speech, backchanneling) that MTR-DuplexBench extends to multi-round settings.
- Full-Duplex-Bench v1.5 — prior single-round benchmark whose overlap/background-speech scenario design this paper builds on and extends into multiple rounds.
- FD-Bench — prior multi-round full-duplex benchmark that this paper directly contrasts itself against for prioritizing aggregate quality over turn-level, round-by-round evaluation.
- Full-Duplex-Bench-v2 — concurrent multi-round full-duplex benchmark, contrasted for relying on an automated Speech LLM examiner with less consistent per-task evaluation data than this paper’s fixed evaluation set.
- TurnGuide — supplies the GPT-score dialogue-quality evaluation methodology that this paper’s dialogue-quality dimension follows.