arXiv · 2025 · Preprint
Pan, Banerjee, Hebbar et al. (Zhejiang University and UC Berkeley) · → Paper · Demo: ✓ · Code: ?
A behavior-perception-and-reasoning framework for full-duplex spoken dialogue that reframes duplex interaction from black-box next-token prediction to an explicit, interpretable pipeline: detecting hierarchical speech acts from audio, then reasoning over their causal and temporal dependencies via a Graph-of-Thoughts model to predict the next behavior and generate a natural-language rationale for it.
Scope note: This paper does not train or evaluate a novel speech generation model. Its own system takes audio and text as input and produces discrete behavior labels and text rationales as output; CosyVoice2 is used only as an off-the-shelf tool to synthesize its training corpus, and Moshi/dGSLM are used only as benchmarked existing full-duplex speech-generation systems, not extended or improved. It is ingested as a scope exception, following the precedent that full-duplex/spoken-dialogue understanding research with clear relevance to SCA system design can be accepted even without training a generative model, extending the reasoning already applied to spoken dialog state tracking (2510.09424) and prosody/text information-theoretic analysis (2512.16832).
Problem
Full-duplex spoken dialogue systems, which listen and speak simultaneously, have mostly been framed as sequence-prediction problems: either predicting an agent’s complete response turn, or generating simultaneous dual token streams for both speakers to handle overlap. Both framings treat conversation as pattern matching over token sequences, bypassing the cognitive layer that actually governs human interaction, in which a listener first perceives a speaker’s behavior (e.g., recognizing a constative statement), reasons about it (deciding whether to interrupt, backchannel, or wait), and only then produces a response. This gap between black-box prediction and interpretable causal reasoning limits both the naturalness and the auditability of full-duplex dialogue systems, and existing conversational-behavior research in spoken dialogue systems has focused narrowly on low-level mechanics (turn-taking, backchannel, interruption) while overlooking the higher-level communicative intent that drives those mechanics.
Method
The framework has two stages. The first, a Conversational Behavior Detection system, processes two-channel speech in streaming 1-second chunks, extracting HuBERT acoustic features and Whisper-transcript semantic features, fusing them via gating, and passing the result through a causal Transformer to jointly classify each chunk at two hierarchical levels: high-level speech acts (constative, directive, commissive, acknowledgment, drawn from speech-act theory) capturing communicative intent, and low-level speech acts (turn-taking, backchannel, interruption, continuation) capturing interaction mechanics. The second stage, a Graph-of-Thoughts (GoT) Behavior-Reasoning system, builds a streaming, per-second directed graph whose nodes are either text entities (subject-relation-object triples extracted via OpenIE from the incremental ASR transcript) or the predicted high/low-level speech-act labels, with edges encoding subject-to-object co-occurrence within a strictly causal, past-only sliding window. This graph, together with the raw audio and transcript, is encoded by separate frozen encoders (HuBERT for audio, T5 for text, a graph attention network for the graph structure) and fused via gating into a contextual representation that conditions a fine-tuned T5 decoder, which autoregressively generates a natural-language rationale explaining the predicted next behavior for that chunk, trained with teacher forcing against human-authored ground-truth rationales. To train and evaluate this pipeline, the authors construct a hybrid corpus: a large synthetic portion generated by prompting GPT-4o to convert ExploreToM narratives into two-speaker dialogues with explicit interruption and backchannel markers, then synthesizing audio with CosyVoice2 using a novel overlap-based dialogue-stitching mechanism (each utterance is synthesized independently, then inserted into the other speaker’s channel at a computed timestamp to produce genuine acoustic overlap for interruptions and backchannels), plus a real-world portion drawn from the CANDOR conversational corpus with added human rationale annotations.
Key Results
The behavior detection model achieves strong ranking performance on synthetic data (AUC 0.78-0.95 across classes) but more variable classification accuracy (F1 0.47-0.88), with constatives and continuation the easiest classes to detect and directives, commissives, interruption, and backchannel the hardest; on real CANDOR speech, AUC remains consistently high (0.70-0.80) across turn-taking, backchannel, and continuation classes. A data-quality analysis shows the synthetic simulation corpus reproduces key structural properties of genuine human conversation (comparable gap and overlap rates to human reference and to dGSLM/Moshi baselines, though with denser micro-segmentation), and its speaking-style statistics (words-per-minute, filler-word rate) pattern consistently with continuous grounding behavior rather than overt turn-grabbing. The GoT reasoning pipeline produces rationales with BLEU-1/ROUGE-1/ROUGE-L in the 0.42-0.58 range and semantic similarity of 0.52 on synthetic data, improving to 0.66 on CANDOR (a 17-27% relative gain across metrics), which the authors attribute to real dialogues having more natural discourse cues than simulated ones. Streaming ablations show a small look-ahead (5-10s) and medium context windows (20-30s) work best, and naively adding text input can hurt performance unless the GoT graph structure with conservative gating is used to stabilize it. In a human evaluation of rationale quality on genuinely full-duplex audio (comparing the simulation corpus, GPT-4, and Moshi on a 1-10 scale for correctness, plausibility, and coherence), the paper reports two different rankings in two places (Section 6.3 text states simulation 8.93 > GPT-4 7.06 > Moshi 4.25, while Table 6 reports GPT-4 7.07 > Moshi 6.85 > simulation 6.30), but both versions support the qualitative conclusion that GoT rationales trained on simulated dialogues transfer effectively to reasoning about real full-duplex speech-generation model output.
Novelty Assessment
The individual components, HuBERT/Whisper feature fusion for behavior classification, Graph-of-Thoughts reasoning (adapted from prior LLM-reasoning work), and OpenIE-based graph construction, are each drawn from existing techniques; the paper’s contribution is applying and adapting Graph-of-Thoughts specifically to real-time causal reasoning over conversational speech acts, which the authors state is a first for this domain. The hierarchical speech-act labeling scheme itself, explicitly separating high-level communicative intent from low-level interaction mechanics, is a genuine conceptual contribution addressing a gap the authors identify in prior full-duplex behavior research, which has focused on low-level mechanics alone. The overlap-based TTS dialogue-stitching mechanism used to construct the synthetic training corpus is a practical, reusable data-generation technique independent of the reasoning framework itself.
Field Significance
moderate — This paper reframes full-duplex spoken dialogue modeling around an explicit, interpretable perceive-reason-generate pipeline rather than opaque next-token prediction, and its synthetic full-duplex conversational speech corpus (with a genuinely novel overlap-stitching TTS construction method) and human-annotated CANDOR rationales could serve as reusable resources for future full-duplex SCA research. Because the system itself performs understanding and reasoning rather than speech generation, and its own reported human-evaluation results contain an internal inconsistency between text and table, its direct field impact should be read as a conceptual and methodological contribution to full-duplex SCA research rather than an advance in speech generation quality.
Claims
- supports: Framing full-duplex conversational dynamics as a two-level hierarchy, distinguishing high-level communicative intent (speech acts like constatives and directives) from low-level interaction mechanics (turn-taking, backchannel, interruption), is a learnable structure that a streaming classifier can detect from real conversational audio with consistently strong ranking performance.
Evidence: On real CANDOR speech, the behavior detection system achieves AUC scores of 0.796 (turn-taking), 0.701 (backchannel), and 0.720 (continuation), and on synthetic data achieves per-class AUC in the 0.78-0.95 range across all eight high- and low-level classes. (§6.1, Table 3, Table 4)
- supports: A synthetic full-duplex conversational speech corpus, generated by inserting independently-synthesized TTS utterances into overlapping timestamps computed from explicit backchannel/interruption markers, can reproduce structural turn-taking properties (gap and overlap rates) comparable to genuine human conversation and to existing full-duplex speech-generation systems.
Evidence: The simulation corpus’s cumulative overlap duration (4.2%) and gap rate (7.3 events/minute) fall within the same range as the human reference (6.7%, 5.5/minute), dGSLM (9.7%, 7.2/minute), and Moshi (3.1%, 6.7/minute) baselines, despite denser micro-segmentation (higher IPU and pause counts) than the human reference. (§5.1, Table 1)
- supports: Reasoning explicitly over a causal graph of perceived speech acts and extracted discourse entities, rather than generating a rationale directly from raw context, produces interpretable justifications whose quality improves measurably when applied to more naturalistic (real, rather than simulated) conversational discourse.
Evidence: Rationale generation quality improves from BLEU-1/ROUGE-1/ROUGE-L/Similarity of 0.48/0.47/0.42/0.52 on synthetic dialogue data to 0.58/0.56/0.49/0.66 on the real CANDOR corpus, a 17-27% relative gain across all four metrics. (§6.2, Table 5)
- complicates: Certain categories of communicative intent and interaction mechanics remain substantially harder to classify from conversational speech than others, indicating that behavior-level annotation schemes may need class-specific handling rather than being treated as uniformly learnable.
Evidence: On the synthetic dataset, directives (F1 0.471), commissives (F1 0.474), interruption (F1 0.515), and backchannel (F1 0.536) all score substantially lower than constatives (F1 0.705) and continuation (F1 0.878), a gap the authors attribute in part to imbalanced data generation rather than to inherent unlearnability. (§6.1, Table 4)
Limitations and Open Questions
The paper's own human-evaluation results for rationale quality on full-duplex audio are internally inconsistent: the main text (Section 6.3) reports simulation-trained rationales scoring highest (8.93 > GPT-4's 7.06 > Moshi's 4.25), while Table 6 in the same section reports a different ranking and different absolute values (GPT-4 7.07 > Moshi 6.85 > simulation 6.30). Both versions are reproduced as-authored in the source paper; this page does not attempt to resolve the discrepancy, and readers should treat the specific human-evaluation numbers as unreliable pending clarification from the authors.
The authors’ own limitations section identifies discrete speech-act labels as a simplification of inherently ambiguous, continuous conversational cues, and notes system performance is sensitive to upstream ASR errors, with robustness to noisy conditions left as future work. The synthetic training corpus, while shown to approximate several structural properties of human conversation, may not capture the full diversity of speaking styles, accents, or cultural conversational norms present in real dialogue. Because the system’s own function is understanding and reasoning about existing conversational audio rather than generating speech, its relevance to this wiki’s core scope is as background/tooling research for full-duplex SCA design rather than a speech-generation contribution in itself.
Wiki Connections
- Spoken Language Models — analyzes and reasons over full-duplex conversational speech from external audio input, and benchmarks existing full-duplex spoken dialogue systems (Moshi, dGSLM) on genuine turn-taking corpus statistics, though the reasoning system itself does not generate speech.
- CosyVoice 2 — used as the TTS engine for the paper’s novel overlap-based dialogue-stitching mechanism, synthesizing the full synthetic training corpus’s two-channel conversational audio.
- Moshi — benchmarked as an existing full-duplex speech-generation system, both for corpus-level turn-taking statistics comparison and in the human evaluation of full-duplex reasoning rationales.