NAACL · 2025 · Conference
Sehun Lee et al. (Seoul National University) · → Paper · Demo: ✓ · Code: ✓
Introduces a large-scale synthetic dataset and a behavior-conditioned dialogue language model that generates full-duplex spoken conversations with explicit, controllable turn-taking, interruptions, filler words, and backchannels.
Problem
Spoken conversational agents and spoken language models typically generate dialogue with strict, sequential turn-taking, but real human conversation is full-duplex: speakers overlap, interrupt, insert backchannels (“yeah,” “uh-huh”), and use filler words at rates that vary by individual and social context. Existing spoken dialogue datasets either record real conversations without speaker-level behavior labels (Fisher, Switchboard, CANDOR) or focus on sequential turn-taking without modeling overlapping speech and interruption (DailyTalk, StyleTalk). As a result, prior dialogue generation systems such as dGSLM, which use dual-tower transformers to predict two channels simultaneously, lack any mechanism to condition the overall behavioral character of a conversation, and cascaded LLM+TTS pipelines cannot maintain consistent conversational dynamics (turn-taking, backchanneling) across an entire dialogue.
Method
The paper makes two contributions: a data generation pipeline producing the Behavior-SD dataset, and BeDLM, a behavior-conditioned dialogue language model trained on that dataset.
Data generation. Each dialogue is built from a narrative sampled from the SODA dataset and four speaker-level behavior traits, each on a 3-level scale (0/1/2): verbosity, filler words, backchannels, and interruptions. GPT-4o generates an 8-12 turn text dialogue conditioned on the narrative and per-speaker behaviors, with interruption scenarios injected from a bank of seven interruption types (agreement, disagreement, floor-taking, tangentialization, clarification, assistance, topic change). GPT-4o-mini detects backchannel opportunity points (BOPs) at natural pause points, and GPT-4o fills in contextually appropriate backchannel text at a subsampled fraction of BOPs (set by the backchannel behavior level). Each sentence is captioned with pitch, speaking rate, and emotion via GPT-4o-mini. The resulting rich-text dialogues are synthesized into two-channel audio with CosyVoice-Instruct/CosyVoice-Base, using a speaker bank of 26 male and 26 female voices selected via Max-Min sampling for diversity. To preserve speaker identity and smooth transitions across independently synthesized utterances, each utterance is conditioned on both the speaker’s first utterance and their immediately preceding utterance as speech prompts. Backchannel and interrupt utterances are time-aligned to the conversation using Gaussian-distributed gap/overlap statistics drawn from the CANDOR corpus.
BeDLM. Rather than training a transformer from scratch, BeDLM fine-tunes a pretrained Llama3.2-1B. Speech is discretized with HuBERT into 500 unit tokens, added to the LLM vocabulary. To represent full-duplex two-channel dialogue without doubling sequence length (the inefficiency of dGSLM’s alternating two-channel scheme), the paper introduces a streamlined single-stream representation: control tokens <A>/<B> mark each speaker’s turn start, <GAP>/<OVERLAP> (each covering 40ms) represent silence or overlapping speech, and <BC_S>/<BC_E> bracket backchannel spans. BeDLM is trained in two stages: first a text-dialogue pretraining stage (supervised fine-tuning to generate text dialogues from narrative+behavior prompts), then fine-tuning to generate the streamlined unit sequence directly from the same conditions. At inference, streamlined units are split back into two channels and each channel is vocoded independently via a HuBERT2Mel module (transformer+convolutional layers with GLUs, predicting mel spectrograms from HuBERT tokens plus a speaker embedding) followed by a fine-tuned HiFi-GAN, producing stereo audio.
BeDLM is trained on 94K dialogues (1.87K hours) for 18k iterations on 8 A40 GPUs with a batch size of 32×4096 tokens. HuBERT (k-means component only, dense model from TWIST), HuBERT2Mel, and HiFi-GAN are all fine-tuned on the Behavior-SD corpus.
Key Results
Human evaluation (5-point Likert, three raters per sample) on naturalness, meaningfulness, and sound quality shows BeDLM (streamlined units, with pretraining) scoring 4.09/4.04/4.05, closely approaching ground-truth audio (4.14/4.02/4.12) and resynthesized audio (4.09/4.02/4.04), and outperforming a cascaded GPT-4o+CosyVoice-SFT baseline (4.12/3.97/4.09) on meaningfulness despite the cascaded system’s stronger text generation, attributed to the cascaded model’s speaker confusion and misplaced backchannels. The dGSLM baseline scores lowest across all three axes (3.96/3.88/3.92).
On behavioral adherence (Wasserstein distance between generated and ground-truth behavior-feature distributions, lower is better, except narrative adherence which is rated 1-5 by GPT-4o), BeDLM achieves the best scores among generative baselines on backchannel (0.87) and interruption (0.58) adherence, beating both dGSLM (3.25/2.62) and the cascaded LLM+TTS baselines (GPT-4o cascade: 1.73/0.91; Llama3-70B cascade: 1.18/0.68), though its narrative adherence score (3.11) trails the cascaded models (4.58 for GPT-4o, 4.09 for Llama3-70B) since those use much larger text LLMs for the dialogue content itself.
Ablations isolate two design choices: removing the text-dialogue pretraining stage drops narrative adherence from 3.11 to 2.80 and naturalness MOS from 4.09 to 4.00, and replacing the streamlined representation with a dGSLM-style alternating two-channel format drops naturalness to 3.90 and interruption adherence from 0.58 to 1.21, despite using the same Llama3.2-1B backbone, isolating the streamlined representation itself (not just model scale) as the source of the gain. A speaker-consistency ablation using WavLM-Base+ cosine similarity shows that prompting with the speaker’s first utterance is essential (raising similarity from 0.680 to 0.889 for global identity), while additionally including the immediately preceding utterance mainly improves local transition smoothness (0.850 to 0.872).
The Behavior-SD dataset itself is compared to five prior spoken dialogue datasets (Fisher, Switchboard, CANDOR, MELD, DailyTalk, StyleTalk) via the same three-axis human evaluation; Behavior-SD scores highest on all three (3.94/3.78/3.87) while being roughly 2-4x larger in audio hours than the next-largest comparator and the only dataset in the comparison with speaker-level behavior labels.
Novelty Assessment
The dataset contribution is the stronger half of the paper: an LLM-driven generation pipeline that produces speaker-level annotated behavior labels (verbosity, fillers, backchannels, interruptions) at a scale (108K dialogues, 2164 hours) not matched by prior full-duplex datasets, none of which carry behavior labels at the speaker-turn level. The pipeline itself (narrative sampling, LLM dialogue generation, LLM-detected backchannel insertion, LLM-generated style captions, TTS synthesis with speaker-consistency prompting) is an engineering integration of existing components (GPT-4o, CosyVoice, Max-Min speaker sampling) rather than a new generation technique.
The model contribution, BeDLM, combines a known backbone (a pretrained text LLM fine-tuned with HuBERT discrete units, following the SpeechGPT/TWIST line of work) with one genuinely new architectural element: the streamlined single-stream duplex representation that encodes two-channel turn-taking, overlap, and backchannels as control tokens in one sequence instead of dGSLM’s doubled alternating-channel scheme. The ablation isolating this representation against an alternating-unit variant on the same backbone and dataset is a fair, controlled comparison and supports the claim that the representation itself (not just data scale or backbone size) drives the gain in naturalness and behavioral adherence.
Field Significance
Moderate — this paper provides a dataset and modeling approach for full-duplex behaviorally annotated dialogue, an underexplored intersection of spoken language modeling and conversational pragmatics (turn-taking, interruption typology, backchanneling). Its contribution is primarily a controlled empirical demonstration that an efficient single-stream duplex representation outperforms a doubled-channel alternative on the same backbone, plus a synthetic dataset large enough to support such training. The evaluation is entirely on the paper’s own synthetic test set and does not establish how the approach transfers to real conversational recordings.
Claims
- supports: A single-stream control-token representation for full-duplex dialogue can match or exceed a doubled-channel alternating representation in both naturalness and behavioral adherence, while using fewer tokens per unit of audio duration.
Evidence: On the same Llama3.2-1B backbone and Behavior-SD training data, the streamlined-unit variant scores 4.09 naturalness MOS and 0.58 interruption adherence vs. 3.90 MOS and 1.21 interruption adherence for the alternating-unit variant. (§7.1, §7.2, Table 4, Table 5)
- supports: Conditioning a speech-prompted TTS synthesizer on a speaker’s very first utterance, in addition to their most recent utterance, materially improves long-range speaker identity consistency in independently synthesized multi-turn dialogue.
Evidence: WavLM-Base+ cosine similarity between the prompt and synthesized speech rises from 0.680 (no prompt) to 0.889 when conditioning on the first utterance alone, and combining first-utterance and previous-utterance prompts yields the best transition smoothness (0.872) without degrading global identity (0.885). (§7.3, Table 6)
- complicates: Pretraining a dialogue generation model on text-only dialogue before fine-tuning on speech-unit sequences improves adherence to narrative content but is not necessary for behavioral adherence.
Evidence: Removing the text-dialogue pretraining stage drops narrative adherence (GPT-4o-rated) from 3.11 to 2.80, while behavioral adherence scores for filler words, backchannels, and interruptions remain comparable (0.10/0.87/0.64 vs. 0.15/0.87/0.58 with pretraining). (§7.2, Table 5)
- complicates: Cascaded LLM-then-TTS pipelines for spoken dialogue generation can achieve strong narrative coherence and sound quality but struggle to maintain conversational meaningfulness because they lack any mechanism to track behavioral context (speaker turn identity, backchannel placement) across the full dialogue.
Evidence: GPT-4o and Llama3-70B cascaded baselines achieve the highest narrative adherence scores (4.58 and 4.09) but their meaningfulness MOS (3.97 for both) is lower than the proposed model (4.04), attributed by the authors to speaker confusion and misplaced backchannels in cascaded outputs. (§7.1, §6.2, Table 3)
Limitations and Open Questions
Warning
All quantitative evaluation (human MOS, behavioral adherence, speaker consistency) is conducted entirely on Behavior-SD’s own synthetic test split, which is itself generated by the same LLM and TTS pipeline used for training data. No evaluation is reported on real, human-recorded spoken dialogue, so it remains untested whether behavioral adherence and naturalness gains transfer outside the paper’s own synthetic data distribution.
The authors note occasional mispronunciation, limited control over complex emotions beyond the five discrete categories used for style captioning, and limited non-lexical vocalization beyond laughter. The word error rate on the Behavior-SD test split (3.55%, measured via Whisper-Large V3) is higher than the underlying CosyVoice TTS system’s WER on LibriSpeech (2.89%), attributed to filler-word insertion and proper-name misrecognition rather than a fundamental synthesis quality gap. The dataset and model are limited to two-speaker dialogues; the authors flag multi-speaker extension as future work. Behavior-SD’s narratives derive from SODA, which itself derives from a smaller set of social-commonsense scenarios, so diversity of conversational topics is bounded by that source. As with any LLM/TTS-synthesized dataset, the authors flag inherited biases from training data and the risk of misuse for voice impersonation or deepfake audio.
Wiki Connections
- Spoken Language Model — extends the discrete-unit LLM paradigm (HuBERT tokens fine-tuned into a pretrained text LLM) to full-duplex two-speaker dialogue rather than single-speaker generation.
- Speech-to-Speech — generates direct audio-to-audio dialogue dynamics (overlap, interruption, backchannel timing) without an intermediate cascaded text-to-speech step at inference.
- Self-Supervised Speech — relies on HuBERT discrete units, fine-tuned on the proposed dataset, as the core speech representation fed into the dialogue language model.
- Prosody Control — conditions TTS synthesis on per-sentence pitch, speaking rate, and emotion captions generated by an LLM, providing an explicit style control channel during dataset construction.
- Evaluation Metrics — introduces a behavioral adherence metric based on Wasserstein distance between generated and ground-truth behavior-feature distributions, alongside standard MOS-style human evaluation.