ACL · 2026 · Conference
Dongwook Lee et al. (Seoul National University / Yonsei University / University of Seoul) · → Paper · → Preprint · Demo: ✓ · Code: ✓
Shows that current speech LMs merge a bystander’s interjection into the primary user’s request, and builds a training set and benchmark (TPI-Train, TPI-Bench) whose key ingredient is same-transcript single-speaker counterexamples that force the model to use the speaker change in the audio rather than the wording.
Problem
Voice assistants built on speech LMs are trained and evaluated almost entirely on dyadic exchanges: one user, one assistant. When a third party interjects (“Should we order the new pasta?” followed by another person saying “No, let’s just get the usual”), the models tend to read the whole audio as one speaker’s self-repaired request and act on the interrupter’s words. The authors argue the root cause is perceptual, not dialogue reasoning: the models are insensitive to the acoustic cue of a speaker change. No prior dataset or benchmark targets this triadic third-party interruption (TPI) setting, and there is no single correct way to respond to an interruption, since the right behaviour depends on user preference.
Method
The task is speech-input, text-output: given the primary utterance followed by a possibly overlapping third-party utterance, the model produces a text response (§2.1). The paper defines a configurable two-stage response framework: each interruption is first labelled Actionable (it carries information that serves the primary user’s goal) or Ignorable (off-topic or disruptive), then handled by a matching policy. The reference strategy answers only the primary user for Ignorable cases and, for Actionable cases, folds the interruption into a proposal and asks the primary user to confirm. Four Actionable categories are defined: correction and disambiguation, feasibility constraint, goal-oriented suggestion, and cooperative addition (§2.2, §3.1, Appendix D).
TPI-Train. Primary utterances come from VoiceAssistant-400K. Seven interruption categories from dyadic conversation-analysis taxonomies (Murata, 1994; Goldberg, 1990) are recast for a three-party setting and expanded into 26 scenarios (e.g., recall assistance, veto with justification, priority alert). For each primary utterance, Qwen3-235B-A22B-Instruct writes a scenario-conditioned interruption and the reference response; both utterances are voiced with Chatterbox TTS using distinct reference speakers drawn from about 5,000 English Multilingual LibriSpeech speakers. Interruptions occur either mid-sentence or after completion, with a time overlap sampled from N(-0.5, 0.1) seconds following OmniFlatten. A reasoning model filters unrealistic samples (§3.1, Appendix F). The result is 80K two-speaker samples split evenly between Actionable and Ignorable (Table 7).
Speaker-aware hard negatives. An LLM selects transcripts whose concatenation could plausibly be one person’s monologue (self-repair, stream of consciousness). These are re-synthesized entirely in the primary speaker’s voice, with the transcript unchanged, so the speaker change in the audio is the only signal separating them from genuine interruptions. TPI-Train contains about 8K such samples (§3.1, Table 7).
TPI-Bench. TPI-Test has 2K held-out two-speaker samples balanced across scenarios. Janus-Test has 2K semantically ambiguous samples re-synthesized as single-speaker audio, so a model that treats them as interruptions is reacting to wording, not acoustics. A 100-clip real-world set is curated from the AMI Meeting Corpus, Friends-MMC, and the authors’ own re-enacted recordings in reverberant, hallway, and outdoor conditions, after LLM pre-filtering (over 99% rejected) and author verification (§4.1, Appendix G). All benchmark samples are filtered to zero WER under Whisper large-v3 (§4.1).
Metrics. Response Strategy Following (RSF) is a binary LLM-judge score for adherence to the reference strategy; on Janus-Test a score of 1 marks an error (a false interruption), so lower is better there. Overall Helpfulness (OH) is a 1 to 5 LLM-judge rating with access to ground-truth speaker information. BLEU and ROUGE-L between a model’s responses to the paired TPI-Test and Janus-Test items measure whether the response changes at all when only the acoustics change (§4.2). The judge is Qwen3-235B-A22B-Instruct; its scores correlate with GPT-5-mini at 0.83 to 0.86 and with human raters at 0.87 (RSF) and 0.69 (OH) on TPI-Test (Pearson, Table 8). Human-LLM correlation on Janus-Test is not reported.
Model. Qwen2.5-Omni-7B is fine-tuned with LoRA (rank 16, alpha 32) on all trainable layers, next-token cross-entropy, 2 epochs, batch size 16, learning rate 1e-4 (Appendix H). Three ablations: TPI-Base (corpus only), TPI-VA (corpus plus VoiceAssistant-400K dyadic data), and TPI-Full (adds the hard negatives) (§5.1).
Key Results
Off-the-shelf speech LMs fail at TPI (Table 2). ChatGPT-4o-audio, Kimi-Audio-Instruct-7B, VITA-Audio-Instruct-7B, Qwen2.5-Omni-7B, and Qwen3-Omni-30B-A3B-Instruct reach TPI-Test RSF of only 0.19 to 0.24, with OH 3.22 to 4.06. Their low Janus-Test RSF (0.01 to 0.13) is consistent with treating almost every input as single-speaker, which Janus-Test rewards by design; their BLEU/ROUGE-L between paired responses ranges from 0.20/0.39 (Qwen3-Omni) to 0.94/0.99 (Kimi-Audio), meaning several produce near-identical responses whether or not a second voice is present. Note that RSF measures adherence to the authors’ reference strategy, which these models were not told about.
Hard negatives remove the semantic shortcut (Table 2). All three fine-tuned variants reach similar TPI-Test RSF (0.82, 0.82, 0.83), but they differ sharply on Janus-Test:
| Variant | TPI-Test RSF ↑ | TPI-Test OH ↑ | Janus RSF ↓ | Janus OH ↑ | BLEU ↓ | ROUGE-L ↓ |
|---|---|---|---|---|---|---|
| Qwen2.5-Omni-7B (base) | 0.24 | 3.22 | 0.12 | 4.44 | 0.31 | 0.53 |
| TPI-Base | 0.82 | 4.32 | 0.86 | 3.54 | 0.46 | 0.63 |
| TPI-VA | 0.82 | 4.29 | 0.67 | 3.75 | 0.39 | 0.58 |
| TPI-Full | 0.83 | 4.16 | 0.16 | 4.80 | 0.12 | 0.34 |
Without hard negatives the fine-tuned models flag most single-speaker ambiguous inputs as interruptions; with them, Janus RSF falls to 0.16. TPI-Full has the lowest BLEU and ROUGE-L of all eight systems in Table 2. The trade-off: TPI-Full’s TPI-Test OH (4.16) is below TPI-Base (4.32) and TPI-VA (4.29), and on Janus-Test it does not match the untuned models that never detect interruptions (Qwen3-Omni RSF 0.01; ChatGPT-4o-audio OH 4.95). t-SNE of LLM representations shows separate clusters for two-speaker, single-speaker, and hard-negative inputs only for TPI-Full, with hard negatives sitting between the other two (§5.3.1, Figure 4).
General ability (Table 3, VoiceBench). Training on the corpus alone hurts general skills: TPI-Base drops to 66.81 on OpenBookQA (base 80.44), 50.80 on MMSU (base 61.22), and 2.83 on WildVoice (base 3.53). Mixing in dyadic data recovers most of this. TPI-Full exceeds the base on six of eight subsets (AlpacaEval 4.12 vs 3.78, CommonEval 3.93 vs 3.67, SD-QA 36.08 vs 28.39, IFEval 0.46 vs 0.42, AdvBench 1.00 vs 0.98, WildVoice 3.64 vs 3.53) and is below it on two (MMSU 59.14 vs 61.22, OpenBookQA 80.00 vs 80.44). TPI-VA is slightly higher than TPI-Full on CommonEval (3.97). Appendix K reports significant gains on 5 of 8 subsets and parity on the rest. VoiceBench is scored with Qwen3-235B as judge instead of the official judge (§5.1).
Real-world transfer (Table 4). On the 100 real clips, TPI-Full raises RSF from 0.17 to 0.60 (+0.43) and OH from 3.21 to 4.25 (+1.04), against +0.59 RSF and +0.94 OH on TPI-Test. The RSF gain is smaller on real data; the OH gain is slightly larger.
Prompting is not enough (Table 6). Giving the base model an instruction chosen according to the ground-truth scenario (“reflect the second speaker’s interruption” or “ignore the second speaker’s interruption”) lifts TPI-Test RSF only from 0.24 to 0.37 while OH slightly drops (3.22 to 3.17); TPI-Full scores 0.83 / 4.16. The authors attribute the RSF gain to the model ignoring the second utterance indiscriminately.
Human judgements (Table 5, Table 1). In an MTurk study rating text responses, TPI-Full’s responses were “preferred” in 66.05% of 400 ratings versus 64.75% for the corpus reference responses, and in a 200-rating A/B test TPI-Full was preferred over the base model 60.50% to 28.00% (11.50% ties). Benchmark realism was rated 2.87 (text scenario) and 2.63 (with synthesized audio) on a 3-point scale.
Novelty Assessment
The contribution is a task definition, data, and evaluation design, not a model. The model side is standard LoRA supervised fine-tuning of an existing speech LM. The most transferable idea is the counterfactual control: holding the transcript fixed and changing only whether one or two voices speak it, used both as a test (Janus-Test) and as training hard negatives. This cleanly isolates acoustic speaker sensitivity from lexical cues, and the TPI-VA vs TPI-Full ablation shows it matters. The interruption taxonomy adapts existing conversation-analysis work to a three-party setting. The comparisons against off-the-shelf models are not like-for-like, since RSF rewards a specific strategy those models never saw; the ablations within Qwen2.5-Omni and the oracle-prompting control (Table 6) are the fairer evidence.
Field Significance
Moderate — The paper defines third-party interruption handling as a measurable task for speech LMs and supplies an 88K-sample training set, a two-part benchmark, and LLM-judge metrics checked against human ratings. Its main finding, that interruption fine-tuning teaches lexical shortcuts unless same-transcript single-speaker counterexamples are included, is a concrete data-design lesson for any speech LM behaviour that should depend on who is speaking rather than what is said.
Claims
- supports: Speech LMs trained on dyadic data treat speech from two different speakers in one input as a single user’s request, and several give nearly the same response whether or not a second voice is present.
Evidence: Five speech LMs, including ChatGPT-4o-audio and Qwen3-Omni, reach TPI-Test RSF of 0.19 to 0.24; BLEU/ROUGE-L between responses to identical transcripts spoken by one versus two voices reaches 0.89/0.93 for ChatGPT-4o-audio and 0.94/0.99 for Kimi-Audio. (§5.2, Table 2; Appendix A)
- supports: When fine-tuning speech LMs for behaviour that should depend on a speaker change, models learn lexical shortcuts unless training includes same-transcript counterexamples in which only the acoustics differ.
Evidence: Fine-tuning Qwen2.5-Omni-7B on two-speaker interruption data gives TPI-Test RSF 0.82 but Janus-Test false-interruption RSF 0.86 (0.67 with dyadic data added); adding 8K single-voice re-syntheses of ambiguous transcripts drops it to 0.16 while TPI-Test RSF stays at 0.83, and t-SNE shows separate clusters only in that case. (§3.1, §5.2, §5.3.1, Table 2, Figure 4)
- complicates: Narrow task-specific fine-tuning of a speech LM degrades its general spoken-dialogue abilities unless general conversational data is mixed into training.
Evidence: Training on the interruption corpus alone lowers VoiceBench OpenBookQA from 80.44 to 66.81, MMSU from 61.22 to 50.80, and WildVoice from 3.53 to 2.83; adding VoiceAssistant-400K data recovers OpenBookQA to 80.22 (and 80.00 with hard negatives), though MMSU stays below base (58.88 and 59.14). (§5.2, Table 3)
- supports: Explicit speaker-aware instructions in the prompt do not give a speech LM the acoustic speaker attribution it lacks.
Evidence: Prompting Qwen2.5-Omni-7B with the instruction matching each sample’s ground-truth Actionable or Ignorable label raises TPI-Test RSF only from 0.24 to 0.37 and lowers OH from 3.22 to 3.17, versus 0.83 and 4.16 after fine-tuning. (§5.3.3, Table 6)
- supports: Speaker-change handling learned from TTS-synthesized multi-speaker dialogue transfers to real multi-party recordings.
Evidence: On 100 real clips from AMI, Friends-MMC, and re-enacted recordings in noisy rooms, the fine-tuned model improves RSF from 0.17 to 0.60 and OH from 3.21 to 4.25, compared with gains of +0.59 RSF and +0.94 OH on the synthetic TPI-Test; the real-world set is small. (§4.1, §5.2, Table 4, Appendix G)
Limitations and Open Questions
Warning
Training data and both main benchmarks are synthesized by one TTS system (Chatterbox) from LLM-written interruptions, and every TPI-Bench score comes from an LLM judge. Real-world evidence rests on 100 clips, 59 of them re-enactments of TPI-Test scripts, and the judge’s agreement with humans is reported only for TPI-Test (r = 0.69 for OH).
- Janus-Test rewards treating input as single-speaker, so a model that never detects interruptions scores near-perfectly on it (Qwen3-Omni RSF 0.01). It is only informative read together with TPI-Test.
- RSF measures adherence to one reference strategy. The authors stress that interruption handling is subjective and user-dependent, and all remaining non-Actionable cases are grouped as Ignorable for practicality (§3.1, §Limitations).
- The setting is turn-based and text-output. Deciding whether and when to speak in a full-duplex multi-party conversation is left for future work (§Limitations).
- Human studies rate text responses and the realism of the benchmark audio, not the model’s output speech; no speech output is evaluated.
- Gains are shown for a single backbone (Qwen2.5-Omni-7B); transfer of the data recipe to other speech LMs is untested.
- All training and synthetic benchmark data are English.
Wiki Connections
- Spoken Language Model — Evaluates five speech LMs on multi-speaker audio and fine-tunes one to discriminate a third-party speaker from the primary user using acoustic cues.
- Evaluation Metrics — Proposes RSF and OH LLM-judge metrics plus a paired-response BLEU/ROUGE-L sensitivity probe, and validates the judge against GPT-5-mini and human raters.
- Qwen2.5-Omni — Fine-tuning backbone and main baseline; the ablations and the prompting control are all run on it.
- Qwen3-Omni — Evaluated baseline; lowest Janus-Test RSF (0.01) but TPI-Test RSF only 0.19.
- Kimi-Audio — Evaluated baseline with the most response-invariant behaviour to a second speaker (BLEU 0.94, ROUGE-L 0.99).
- VITA-Audio — Evaluated baseline that fails on TPI-Test (RSF 0.21).
- VoiceBench — Used to check that interruption training preserves general voice-assistant abilities across its eight subsets.
- Mini-Omni — Cited source of the VoiceAssistant-400K data that supplies the primary utterances and the dyadic training mix.
- OmniFlatten — Source of the overlap-duration sampling used when mixing primary and third-party utterances.
- Multilingual LibriSpeech — Supplies the English reference voices for synthesizing primary and third-party speakers.