EMNLP · 2025 · Conference
Dongjie Fu et al. (Zhejiang University / Meituan) · → Paper · Demo: ✓ · Code: ?
Introduces the first spoken dialogue dataset and system designed to recognize speaker identity in multi-party conversations and generate persona-consistent responses grounded in each speaker’s background profile.
Problem
LLM-based spoken dialogue systems have made rapid progress on semantic content, and some more recent systems mine style or emotional cues, but almost none use speaker identity to shape who a response is tailored for. The paper distinguishes three levels of role understanding a voice assistant might have: speaker-aware (correctly addressing the current speaker), speaker-understanding (mapping different timbres to different roles within one conversation), and persona-aware (linking a recognized voice to that person’s background information to generate a response tailored to them). Existing spoken dialogue models mostly stop at simple speaker-recognition Q&A and have not been evaluated on genuinely personalized response generation in multi-party settings. Progress has been blocked less by modeling difficulty than by data: speech data with rich, labeled persona information is expensive and privacy-sensitive to collect, so no prior spoken dialogue corpus combines multi-party structure with persona annotations at scale.
Method
The paper’s first contribution is a data construction pipeline for Persona-Dialogue, a large-scale synthetic multi-party spoken dialogue dataset with speaker profiles. The pipeline proceeds in stages: (1) a large language model generates diverse user profiles (five-sentence background descriptions covering identity, interests, and scenario-relevant traits) across 21 curated multi-party assistant scenarios (family life, company meetings, classrooms, etc.); (2) textual dialogue scripts are generated between randomly paired users and an assistant, using two complementary strategies (direct single-pass dialogue generation, and sequential generation where user-to-user dialogue is produced first and then a selected user interacts with the assistant), with and without profile information visible to the generator to increase diversity; (3) an LLM-as-judge verifies faithfulness (consistency with assigned profiles), relevance (profile-dialogue alignment), and toxicity of generated scripts; (4) text is converted to speech using the zero-shot TTS system CosyVoice2, conditioned on reference timbres drawn from VoxCeleb2, so each synthetic user maintains one consistent voice across turns; (5) a speaker verification model (PyAnnote) confirms timbre consistency per user and a Whisper-based ASR pass filters out samples with high word error rate. The resulting corpus contains 21,760 dialogues, 159,933 turns, and 217 hours of audio.
The second contribution is PAChat, a spoken dialogue architecture built on this dataset. The model formalizes the task as: given a set of registered user profiles and a dialogue history spanning multiple speakers (any of whom may be the assistant), generate a personalized text response for the user who just addressed the assistant, then synthesize that response as speech. The backbone is a frozen Llama 3.1-Instruct 8B model fine-tuned only through a LoRA adapter. Incoming user speech is processed by two parallel, frozen encoders: a Whisper encoder extracts frame-level semantic features, which are then compressed by a window-level Q-Former (one trainable query per ~0.33-second window) into a sequence aligned with the LLM’s input space; separately, a PyAnnote speaker encoder extracts a single sequence-level speaker embedding per utterance, projected through a linear layer. The speaker embedding is concatenated with the frame-level semantic features and combined with the textual user-profile and dialogue-history context as the LLM’s input sequence. Only the LoRA adapter and the alignment modules (Q-Former, speaker projection) are trained, via a standard autoregressive cross-entropy objective over the assistant’s response tokens conditioned on the audio features and dialogue history. At inference, the LLM’s text response is converted to speech using the same CosyVoice2 TTS model used for dataset construction.
Key Results
On the Persona-Dialogue test split (1,000 held-out dialogues with unseen scenarios), PAChat outperforms all baselines (FunAudioLLM, Salmonn, Qwen-Audio, Qwen2-Audio, adapted with character/speech prompts since they were not designed for multi-party input) across every metric: BLEU 5.2 vs. next-best 4.3 (FunAudioLLM given speaker identity as an explicit text prompt), GPT-eval 3.8 vs. 3.1 for the best audio-input baseline (Qwen2-Audio, +0.7), and human evaluation 3.6 vs. 3.0 for the best audio-input baseline (+0.6). On the out-of-domain Persona-Chat dataset (a text persona dialogue benchmark reformatted into spoken form for this evaluation), PAChat’s BLEU margin over baselines narrows, but its subjective scores remain clearly ahead (GPT-eval 4.04, human eval 3.45 vs. the next-best baseline’s 3.15/3.21). An ablation removing user-profile or dialogue-history context from the input (Table 3) shows GPT-eval dropping to as low as 2.3, confirming both signals are needed for persona-consistent generation. A separate data-scale ablation shows a steep-then-flat pattern: models trained on 1k-2k dialogues perform poorly, gains continue through roughly 20k dialogues, and further scaling brings diminishing returns given the fixed set of scenarios. In a supplementary end-to-end speech evaluation (Appendix C.2), PAChat’s synthesized speech scores MOS 3.87, essentially tied with a Qwen2-Audio + CosyVoice2 baseline (MOS 3.88) since both use the identical TTS backend, while PAChat’s real-world dialogue-understanding human score is higher (3.03 vs. 2.81).
Novelty Assessment
The genuine contribution here is task framing and data, not architecture. Persona-Dialogue is presented as the first synthetic spoken dialogue corpus combining multi-party structure with persona (user profile) annotations, filling a real gap: prior spoken multi-party corpora (MELD, DailyTalk, SpokenWOZ, StyleTalk) lack persona labels, and prior persona corpora (Synthetic-Persona-Chat, LiveChat) are text-only. PAChat’s architecture, by contrast, is a fairly direct application of an established pattern in the speech-LLM literature: a frozen audio encoder plus windowed Q-Former alignment feeding a LoRA-tuned frozen LLM, a design lineage shared with prior audio-language models such as SALMONN and Qwen-Audio, with a speaker-embedding channel added alongside the semantic channel and an off-the-shelf zero-shot TTS module bolted on for output. No new training objective, alignment mechanism, or generative component is proposed. The paper’s own framing supports this reading: it positions itself as the first work to combine speaker identity understanding with personalized generation in a multi-party spoken setting, not as introducing a new modeling technique.
Field Significance
moderate - PAChat demonstrates, with a purpose-built dataset, that adding an explicit dedicated speaker-identity channel to a speech-LLM pipeline measurably improves persona-consistent response generation over baselines that only have access to transcribed content or a text-prompted identity label. Its primary value to the field is the new task framing and dataset (Persona-Dialogue) rather than a structural advance in speech-LLM architecture, and the improvements over baselines are consistent but modest in scale, especially on the out-of-domain Persona-Chat evaluation.
Claims
- supports: Providing a dedicated speaker-identity representation alongside content features to a spoken dialogue LLM yields more persona-consistent responses than relying on transcribed content alone or on speaker identity injected only as a text label.
Evidence: PAChat, with a separate speaker encoder and alignment channel, outperforms FunAudioLLM given identity as a text prompt and outperforms all audio-input-only baselines on GPT-eval and human-eval across both Persona-Dialogue and Persona-Chat. (§5.3.1, §5.3.2, Table 2)
- supports: Large-scale LLM-generated synthetic dialogue with zero-shot TTS-synthesized speech can train spoken dialogue systems that generalize to out-of-domain, differently-sourced conversational data.
Evidence: A model trained only on the synthetic Persona-Dialogue corpus still achieves the best subjective scores (GPT-eval 4.04, human-eval 3.45) when zero-shot evaluated on Persona-Chat, a separate text-dialogue dataset reformatted into spoken form. (§5.3.2, Table 2)
- complicates: Persona-aware response generation in multi-party spoken dialogue depends jointly on dialogue history and static user-profile context; neither signal alone is sufficient, which constrains what minimal-context deployments can achieve.
Evidence: Ablating either user-profile or dialogue-history input individually drops GPT-eval from 3.8 to between 2.3 and 3.2, though the authors also note that scenarios requiring both signals jointly are not universal in real dialogue. (§5.5, Table 3)
- complicates: Evaluating “spoken” dialogue systems primarily through text-based metrics on the LLM’s output can decouple the measured gains from the quality of the actual synthesized speech, since the two stages are typically produced by independently trained modules.
Evidence: PAChat and a Qwen2-Audio + CosyVoice2 baseline that share the identical TTS backend obtain nearly identical MOS (3.87 vs. 3.88) despite PAChat’s clearly better dialogue-understanding human score (3.03 vs. 2.81), showing the persona-aware gains are entirely in text content, not the shared downstream TTS stage. (§Appendix C.2, Table 4)
Limitations and Open Questions
PAChat is explicitly restricted to single-user interaction per turn and does not model conversational turn-taking. The authors state this limits its readiness for real deployment in genuinely multi-party live conversations, where knowing when to speak or interrupt is essential.
Beyond turn-taking, the authors note two further open points. First, the data-scale ablation shows continued (if diminishing) gains up to the largest tested scale, so the point at which additional synthetic data stops helping is not established. Second, PAChat currently generates responses only as text content matched to a persona; it does not yet adapt vocal tone or prosody to the addressee (e.g., a gentler tone for a child, a more careful tone for an elderly user) since speech synthesis is delegated to a fixed, off-the-shelf TTS module rather than being unified into the model itself.
Wiki Connections
- Spoken Language Model — PAChat is a speech-LLM that consumes external user speech through a frozen Whisper encoder and windowed Q-Former, aligned into a LoRA-tuned Llama 3.1 backbone, in a genuine multi-party spoken-dialogue setting.
- Speech-to-Speech — the system takes speaker-attributed spoken input and produces spoken output end-to-end (via an LLM response stage followed by TTS synthesis), functioning as a spoken dialogue assistant rather than a text-only chat model.
- Zero-Shot TTS — both the synthetic training corpus and PAChat’s own output speech are generated by zero-shot voice cloning (CosyVoice2) conditioned on reference timbres, rather than any speaker-specific TTS training.
- Subjective Evaluation — the paper uses human raters (GPT-eval alongside a five-volunteer human-eval panel, and a separate MOS listening test) as its primary measure of persona-consistent response quality.
- FunAudioLLM — used as a primary baseline, both in its default ASR-transcription mode and with speaker identity injected as an explicit text prompt.
- SALMONN — used as a baseline audio-input dialogue model and architecturally the closest prior design (frozen audio encoder plus Q-Former alignment into a frozen LLM).
- Qwen-Audio — used as a baseline general-purpose audio-language model, evaluated zero-shot on the multi-party persona task it was not designed for.
- Qwen2-Audio — the strongest audio-input baseline across both evaluation datasets, and reused (paired with CosyVoice2) as the comparison system in the end-to-end speech MOS evaluation.
- CosyVoice2 — the zero-shot TTS system used both to synthesize the Persona-Dialogue training corpus and to generate PAChat’s final spoken responses at inference.
- GPT-4 — used throughout the data pipeline as the generator for user profiles and dialogue scripts, and as an automated judge for dialogue verification and subjective evaluation (GPT-eval).