EMNLP · 2025 · Conference
Yifu Chen et al. · → Paper · Demo: ✓ · Code: ?
InteractSpeech is a 150-hour English spoken dialogue dataset with over 10 types of interaction annotations (interruptions, backchannels, gaps, pauses) and precise speaker timestamps, created via multi-stage LLM prompt programming and advanced TTS synthesis, that enables models to learn fine-grained turn-taking and interactive conversational behavior.
Problem
Spoken Dialogue Models (SDMs) such as GPT-4o voice mode and Moshi have demonstrated natural speech generation but struggle with fine-grained interactional dynamics: handling interruptions from users, providing timely backchannels, managing conversational flow, and determining when to yield the speaking floor. A bottleneck is the lack of high-quality, publicly available training data annotated for these nuanced behaviors. Existing resources are limited: Fisher and Switchboard lack fine-grained timestamps and contemporary content; Command FDM and Voice FDM cover only single-turn command-cessation interactions; Moshi uses the outdated Fisher corpus. No open-source dataset combines multi-turn dialogue, diverse interruption types, precise speaker timestamps, and modern conversational topics.
Method
InteractSpeech is a mixed dataset combining 112 hours of synthetic and 38 hours of real-world speech, totaling 150 hours and 90 k text utterances.
Formal Framework for Interaction Dynamics. The paper formalizes dyadic interaction as alternating speech utterances U_i^X and silence periods S_j^X with precise temporal boundaries. Key interaction events are defined: Turn Initiation (TI), Interruption (INT — speaker H begins while M is speaking), Backchannel (BC — short utterance during M’s turn without claiming the floor), Pause (P — silence within a turn), and Gap (G — joint silence at turn boundary).
Synthetic Data Generation (112 hours). A multi-stage GPT-4o prompt programming pipeline enriches four text dialogue corpora (SODA, Dialogsum, PLACES, MultiWOZ 2.2) with interaction markers. Stage 1 identifies semantically appropriate locations for interruptions, backchannels, and other events; Stage 2 generates the interruptive content constrained by naturalness and dialogue coherence. Expert prompt-testing achieved >95% satisfaction before large-scale deployment. CosyVoice-300M-SFT synthesizes dual-track audio with configurable overlap timing (e.g., interruption begins after n words of ongoing speech). Multi-stage prompting substantially outperforms single-stage (98% vs. 21% human overall quality rating).
Real-World Data Filtering (38 hours). The CANDOR corpus (large-scale naturalistic English conversations) is filtered using pyannote.audio for speaker diarization to detect overlapping segments, and a vocal consistency model checks timbre similarity before/after overlap to classify whether a turn change occurred (interruption) or not (backchannel). Temporal constraints ensure segment quality.
OOD Interactive Test Set (1 hour, excluded from 150h total). Human participants conduct free-form conversations with GPT-4o voice mode, Doubao, and Gemini Live under assigned interaction modes (interruption, pause, normal, backchannel). Professional annotators label event timestamps.
Downstream Evaluation.
- Speech understanding model: Qwen-2.5-Omni fine-tuned with GRPO reinforcement learning on binary classification of BC/INT/G/P events from audio. Trained for 1,500 steps (AdamW, lr 1e-6, batch 2).
- Text interaction model: LLaMA 3-8B fine-tuned with LoRA on InteractSpeech text scenarios to predict dialogue continuations following interruptions.
Key Results
Speech interaction event classification (binary accuracy %):
| Model | IS Overall | RWS Overall |
|---|---|---|
| Random Guess | 50.0 | 50.0 |
| GPT-4o (zero-shot) | 54.2 | 53.0 |
| Qwen-Omni (zero-shot) | 52.5 | 50.8 |
| Qwen-Omni-FT (ours) | 73.4 | 63.3 |
Qwen-Omni-FT achieves 73.4% on the InteractSpeech test set — a 19 pp improvement over GPT-4o — and 63.3% on real-world scenarios. The model transfers well out-of-domain, particularly for Backchannel (70.7% on RWS).
Ablation: removing synthetic data drops IS by 10.8 pp; removing real data drops RWS by 7.0 pp. Both data types are complementary — synthetic data drives in-domain feature learning, real data improves generalization.
Text interaction quality: LLaMA 3-8B fine-tuned on InteractSpeech text achieves 86% overall human agreement (naturalness 86%, consistency 85%, specificity 88%) vs. 21% for single-stage prompt-generated text.
Audio quality: MOS 4.35 ± 0.12, higher than Fisher (4.21) and Switchboard (4.09), despite being predominantly synthetic.
Novelty Assessment
The primary contribution is the dataset resource itself. InteractSpeech is the first open-source corpus specifically designed for fine-grained spoken dialogue interaction with over 10 annotated interruption types, precise timestamps, and dual-track audio. The multi-stage prompt programming methodology for generating diverse and natural interactive scenarios is a practical methodological contribution. The formal interaction dynamics framework provides a principled vocabulary for the field. The GRPO-based fine-tuning for event classification is a sensible application of recent RL-from-feedback approaches to audio understanding. The work is a Findings of EMNLP 2025 paper, consistent with its dataset-contribution focus.
Field Significance
Moderate — InteractSpeech addresses a concrete data gap for spoken conversational agents: no prior open-source dataset provided multi-turn dialogue with over 10 annotated interaction event types and precise dual-track timestamps. The formal framework for interaction dynamics (Turn Initiation, Interruption, Backchannel, Pause, Gap) provides a reusable vocabulary for training and benchmarking turn-taking models. The dataset and event classification benchmark can serve as a reference resource for subsequent work on full-duplex spoken dialogue systems.
Claims
- Large-scale spoken dialogue datasets with fine-grained interaction annotations (interruptions, backchannels, gaps, pauses) and precise speaker timestamps are necessary for training models to classify conversational events accurately from audio. (§4.1, Table 2)
- Fine-tuning on interaction-specific data enables speech models to classify conversational event types at accuracy substantially above zero-shot performance from large proprietary models. (§4.1, Table 2)
- Synthetic and real-world data provide complementary benefits for interaction event modeling: synthetic data improves in-domain performance while real-world data is critical for out-of-domain generalization. (§4.1)
- Multi-stage prompt programming for generating interactive dialogue scenarios yields substantially higher quality than single-stage prompting on naturalness, consistency, and specificity dimensions. (§4.4, Table 4)
- Advanced TTS synthesis can produce spoken dialogue datasets with audio quality comparable to or exceeding legacy real-world corpora such as Fisher and Switchboard. (§4.2, Table 3)
Limitations and Open Questions
- The current validation is limited to sub-tasks (event classification, text generation quality). End-to-end duplex SDM fine-tuning on InteractSpeech is not demonstrated due to lack of open-source full-duplex systems at evaluation time.
- The dataset is English-only; interactive phenomena may be language-specific.
- The OOD test set covers only 1 hour; broader evaluation across speakers and dialogue domains would be needed.
- Qwen-Omni-FT’s 63.3% accuracy on real-world scenarios leaves substantial room for improvement; the gap between in-domain (73.4%) and OOD (63.3%) suggests distribution shift challenges.
- Standardized metrics for holistic turn-taking evaluation remain an open problem in the field.
Wiki Connections
- spoken-language-model — core dataset contribution addressing data scarcity for fine-grained interaction modeling in full-duplex spoken conversational agents
- evaluation-metrics — introduces a benchmark task (interaction event classification) and methodology for evaluating spoken dialogue model interactivity
- rlhf-speech — the speech understanding model uses GRPO reinforcement learning for event classification from audio
- autoregressive-codec-tts — CosyVoice-300M-SFT is used to generate the synthetic audio component of the dataset
- zero-shot-tts — speaker-consistent synthesis across dual-track dialogues relies on zero-shot voice cloning capabilities of CosyVoice