ACL · 2026 · Conference

Yifu Chen et al. (Zhejiang University / Alibaba Group / Beijing University of Technology) · → Paper · → Preprint · Demo: ? · Code: ?

Trains an audio LLM to judge full-duplex spoken dialogue from dual-track audio, producing separate reasoning on semantic coherence and on turn-taking timing before a single binary reward, and shows it matches human labels more closely than zero-shot audio LLMs and commercial models on most test sets.

Problem

Reinforcement learning for full-duplex spoken dialogue models needs a reward that judges both what the system says and when it says it. Existing automatic proxies cover only one side: oracle-aligned timing metrics are largely blind to content, behavioural statistics (overlap counts, gap lengths, as used with Moshi) count events without judging whether they were appropriate, and metric suites such as Full-Duplex-Bench give fragmented, high-latency signals that are hard to fuse into one online reward. LLM-as-judge evaluators, including the earlier WavReward, were built for turn-based, single-channel dialogue and cannot perceive overlaps, gaps, and interruptions. Human evaluation is reliable but too slow and costly for an RL loop.

Method

The paper first defines a taxonomy of dyadic interaction (§3.1). Each speaker’s track is segmented into speech and silence; from these it derives pauses (intra-speaker silence), gaps (inter-speaker silence), and overlaps, and four functional events: smooth turn transition, successful interruption, backchannel, and failed interruption. Failures are organised on two axes: semantic errors (contextual incoherence, including “interruption amnesia”, where the system loses the discourse state after being interrupted) and timing errors, split into over-reactive (inappropriate barge-in, overly deferential ceding to a backchannel) and under-reactive (delayed turn transition, ignored interruption). The taxonomy is grounded in public conversation corpora and about 10 hours of in-house human-machine dialogue annotation.

The reward model maps a dual-track dialogue to two parallel chain-of-thought analyses, one on semantic coherence and one on turn management, followed by a binary score that is 1 only if both axes are acceptable (§3.2). The authors argue a binary score matches user judgments and yields higher annotator agreement than Likert scales.

Training data is mostly synthetic: text dialogues from SODA, DialogSum, MultiWOZ 2.2 and UltraChat are rewritten by LLMs (Qwen3, DeepSeek-R1) to script specific events and errors, then rendered into dual-track audio with GPT-4o-mini-tts and Gemini TTS, giving programmatic timestamps for every overlap and silence (§3.2, Appendix A-B). The synthetic set has 6,361 samples (about 146 h), roughly 60% successful and 40% failure cases. Real data adds 100 human-human segments from Seamless Interaction, filtered with pyannote diarization and speaker-embedding checks and then manually annotated, and 289 human-machine dialogues collected by seven trained testers who deliberately provoked good and bad behaviour from GPT-4o voice mode, Gemini Live, and Doubao (Appendix C).

The backbone is Qwen2.5-Omni-7B, trained in three stages (§3.3, Table 5):

  1. SFT-1, interaction grounding (4,904 samples, full model): multi-task supervised fine-tuning on event detection with timestamps, speaker diarization, and timestamped diarized transcription, so the model learns to perceive dual-track structure.
  2. SFT-2, reasoning distillation (2,670 samples, full model): Gemini-2.5-Pro is given the text metadata (timestamps, transcripts, speakers) and generates structured CoT analyses and scores, which the model learns to produce from audio.
  3. GRPO (3,513 synthetic scenarios plus the real-world data; LLM only, audio encoder frozen; K=4, about 400 steps): the reward is an equal-weighted sum of a format indicator (required think and score blocks present) and an accuracy indicator (score matches the ground-truth label).

Key Results

All models predict the binary interaction-quality label zero-shot; metrics are accuracy and macro-F1 (§4.1, Table 1).

Test setProposed (Acc / F1)Best baseline (Acc / F1)Base Qwen2.5-Omni (Acc / F1)
In-Distribution0.9853 / 0.9852Gemini-2.5-Pro 0.7473 / 0.74230.0733 / 0.0811
Fine-Grained0.8500 / 0.8476Gemini-2.5-Pro 0.6750 / 0.67910.2500 / 0.2500
Out-of-Distribution0.9614 / 0.9610Gemini-2.5-Pro 0.6781 / 0.67150.0858 / 0.0837
RW-HH0.8679 / 0.6931GPT-4o 0.9245 / 0.64670.4906 / 0.4851
RW-HM0.7727 / 0.7647GPT-4o and Gemini-2.5-Flash 0.6818 / 0.68180.5227 / 0.5227

The proposed model has the highest accuracy and F1 on the three synthetic sets and on RW-HM, and the highest macro-F1 on RW-HH. On RW-HH, however, GPT-4o has higher accuracy (0.9245 vs 0.8679). The paper notes that the human-human set is heavily imbalanced towards fluent conversations (Appendix C.2), so accuracy there rewards a judge that rates most samples as fine; macro-F1 is the more informative column, and the proposed model’s lead on it is modest (0.6931 vs 0.6467). The confusion matrices in Figure 4 show GPT-4o labelling most errors as correct, while the proposed model separates semantic, over-reactive, and under-reactive errors. The real-world test sets are small (53 and 44 samples), so the gaps there are sensitive to a handful of items.

In a blind human-expert review of CoT outputs on 40 snippets (Table 2), the proposed model has the highest consistency with a reference rationale on a 5-point scale (4.125 vs 3.95 for Gemini-2.5-Pro and 2.5 for GPT-4o) and is picked as the best reasoning in 55% of cases (Gemini-2.5-Pro 40%, GPT-4o 5%). Only two baselines were included in this comparison.

The ablation (Table 3) shows each stage matters, and in a non-monotonic way:

  • SFT-2 alone gives 64.96% ID, 45.28% RW-HH, 63.64% RW-HM.
  • Adding SFT-1 before it raises RW-HH to 84.91% and RW-HM to 72.73%, but ID is only 76.64%.
  • Adding GRPO on synthetic data only raises ID to 97.45% but lowers RW-HH to 83.02% and RW-HM to 56.82%, which the authors attribute to overfitting to synthetic patterns.
  • Skipping SFT-1 (SFT-2 + GRPO) collapses RW-HM to 38.64%.
  • Adding the small real-world set to GRPO gives the full model’s 98.54% ID, 86.79% RW-HH, and 77.27% RW-HM.

Novelty Assessment

The model and RL algorithm are existing components (Qwen2.5-Omni, SFT, CoT distillation from Gemini, GRPO with a rule-based accuracy reward). What is new is the problem framing and data: a taxonomy that pairs full-duplex timing events with content errors, a synthetic pipeline that injects each failure type with exact timestamps, and a judge whose reasoning is split into a semantic axis and a timing axis. Its predecessor WavReward judged turn-based dialogue; this paper’s contribution is extending generative reward modelling to dual-track, overlapping speech. Despite the framing as a reward model for RL, the paper never uses it to train a dialogue model, so its value as a reward signal is untested. The comparison is also lopsided: the proposed model is trained on the same taxonomy and label definition it is scored on, while all baselines are zero-shot with a prompt.

Field Significance

Moderate. The paper gives a concrete taxonomy and labelled dual-track test sets for judging full-duplex interaction quality, and shows that general audio LLMs, including commercial ones, perceive turn-taking violations poorly without targeted training. It is an evaluation and data contribution; its central motivation (an online RL reward for full-duplex models) is still unvalidated.

Claims

  • supports: Zero-shot audio LLM judges fail to perceive turn-taking violations in full-duplex dialogue and tend to rate flawed interactions as acceptable.

    Evidence: Zero-shot open-source audio LLMs, including the untrained base model, score 0.07-0.44 accuracy on the synthetic in-distribution set; GPT-4o misclassifies most error types as correct in the fine-grained confusion matrix, and in a case study GPT-4o and Gemini-2.5-Pro both describe an inappropriate barge-in as smooth or natural. (§4.2, Table 1, Figure 4, Figure 5)

  • supports: Explicit training on dual-track interaction structure (event detection, diarization, timestamped transcription) is necessary for an audio LLM to judge real conversational timing.

    Evidence: Removing the SFT-1 grounding stage drops RW-HM accuracy from 77.27% to 38.64% and RW-HH from 86.79% to 77.36% (SFT-2 + GRPO), and SFT-2 alone reaches only 45.28% on RW-HH. (§4.2, Table 3)

  • complicates: Interaction-quality judges trained mainly on TTS-synthesised dialogue overfit to synthetic patterns and need some real recorded interactions to generalise.

    Evidence: GRPO on synthetic data alone raises in-distribution accuracy to 97.45% but cuts RW-HM accuracy to 56.82%, below the 72.73% of the SFT-only model; adding 389 real samples to GRPO restores RW-HM to 77.27%. (§4.2, Table 3)

  • complicates: Accuracy on naturally imbalanced human-human conversation data can favour weak judges that rate most interactions as fluent, so balanced metrics are needed to compare dialogue evaluators.

    Evidence: On RW-HH, GPT-4o reaches 0.9245 accuracy but only 0.6467 macro-F1, while the trained model scores 0.8679 accuracy and 0.6931 macro-F1; the paper notes the human-human data is highly imbalanced towards fluent conversations. (§4.2, Table 1; §Appendix C.2)

  • complicates: Interaction-quality reward models for full-duplex speech LMs are often validated only as offline classifiers, leaving their benefit as an RL training signal undemonstrated.

    Evidence: The authors state they could not integrate the model into an online RL loop, so no dialogue model is optimised with the proposed reward. (§Limitations)

Limitations and Open Questions

Warning

The model is presented as a reward for online RL of full-duplex dialogue models, but it is never used to train one (§Limitations). All evidence is offline agreement with labels, so whether it improves a policy, or is exploitable by one, is unknown.

  • The real-world test sets are small (53 human-human and 44 human-machine samples), so differences of a few points there correspond to a few samples.
  • Training and in-distribution test data are generated with the same LLM-rewrite and commercial-TTS pipeline, and the CoT targets are distilled from Gemini-2.5-Pro working from text metadata, so judgments may be tied to these synthesis artefacts and to the teacher’s reasoning.
  • The binary score is coarse; the authors suggest separate semantic and timing scores or graded scores as future work, but neither is evaluated.
  • The CoT quality study compares only against GPT-4o and Gemini-2.5-Pro and does not report inter-rater agreement or the number of experts.
  • The real-world human-machine data covers three commercial systems and English speakers only.

Wiki Connections

  • Spoken Language Model — The reward model is an audio LLM (Qwen2.5-Omni) adapted to consume dual-track dialogue audio, and the baselines are existing audio LLMs evaluated on the same recordings.
  • Evaluation Metrics — Proposes and validates an automatic interaction-quality judge for full-duplex dialogue against human-labelled test sets and human ratings of its reasoning.
  • RLHF Speech — Builds a reward model meant to supply the reward signal for online RL of full-duplex dialogue models, and trains the judge itself with GRPO, though it does not yet use the reward to train a dialogue policy (§Limitations).
  • WavReward — Earlier generative reward model for turn-based spoken dialogue from the same group; this paper extends the approach to full-duplex timing and overlaps.
  • SDiaReward — Related spoken dialogue reward modelling and benchmark work from the same group, cited as prior audio LLM judging work.
  • Qwen2.5-Omni — The backbone model fine-tuned in all three stages, and the zero-shot base baseline.
  • Full-Duplex-Bench — A metric suite for full-duplex turn-taking that the paper contrasts as fragmented and too slow for an online reward.
  • Moshi — A full-duplex model whose behavioural-statistics evaluation is cited as a proxy that counts events without judging them.
  • Qwen2-Audio — Zero-shot open-source baseline judge.
  • Kimi-Audio — Zero-shot open-source baseline judge.
  • Audio Flamingo 3 — Zero-shot open-source baseline judge.