arXiv · 2025 · Preprint
Hanke Xie et al. (Northwestern Polytechnical University) · → Paper · Demo: ✓ · Code: ✗
Introduces a dual-track LLM and chunked flow-matching architecture for zero-shot multi-turn dialogue speech synthesis, paired with a data pipeline for constructing dual-speaker conversational training data with overlap annotation.
Problem
Standard TTS systems synthesize speech for a single speaker from a single stream of text, which fails to capture the interactional dynamics of real conversation, turn-taking, overlapping speech, and speaker consistency across many turns. Prior multi-speaker dialogue systems have partial solutions: dual-channel approaches trained on small, lower-quality corpora (Fisher) lack scale and audio quality, while LLM-scripted podcast generators that rely on a single interleaved token stream cannot represent genuinely overlapping speech between the two channels. There is also a shortage of large, well-annotated dual-track (per-speaker, per-channel) conversational speech data with overlap labels needed to train such models at scale.
Method
DialoSpeech decomposes dialogue synthesis into two stages, text-to-token (T2T) and token-to-waveform (T2W). In the T2T stage, dialogue text and two speaker prompts are consumed by DiaLM, a 0.5B-parameter LLaMA-based transformer that autoregressively predicts two parallel streams of semantic tokens (one per speaker), extracted from real speech using the S3Tokenizer from CosyVoice2. A causal cross-attention mechanism operates between the two token streams so that each speaker’s representation can attend to the other’s content and prosodic intent, letting the model learn timing of turn transitions, backchanneling, and overlaps without explicit timing signals; when a speaker is not talking, the model emits a special <SIL> token for that channel. Speaker identity for zero-shot cloning is supplied via ECAPA-TDNN embeddings from short reference clips per speaker, and [spkchange] tags mark turn boundaries in the input text. Training uses a dual-channel cross-entropy loss summed over both speaker streams.
In the T2W stage, each speaker’s predicted semantic tokens (excluding <SIL>) are converted to mel-spectrograms by a conditional flow matching (CFM) model built on an F5-TTS-style DiT backbone (22 layers, hidden dimension 768, ~150M parameters), conditioned on the semantic tokens concatenated with the speaker embedding. To keep memory and latency bounded on long dialogues, the authors introduce block-wise guided attention: input tokens are partitioned into fixed-size blocks and a configurable attention mask (causal, history, or future) controls which neighboring blocks each block’s DiT computation can see, enabling chunked decoding with a fixed memory footprint rather than full-sequence attention. Mel-spectrograms are converted to 24kHz waveforms with a pretrained BigVGANv2 vocoder.
To supply training data at scale, the paper also describes a modular dual-track data pipeline: long-form podcast and vlog audio is segmented, transcribed with a Paraformer-based ASR system, diarized with Pyannote, and passed through punctuation restoration and a Conformer-based overlapped speech detection (OSD) module. A multi-stage filter (SNR, speaker-cluster coherence, intra-speaker embedding similarity, DNSMOS) removes low-quality segments, and SpatialNet separates overlapping regions into per-speaker channels. This pipeline produced roughly 10,000 hours of training data (3,000h studio Chinese dialogue, 5,000h spontaneous Chinese podcast, 2,000h English Fisher telephone dialogue).
Key Results
On Chinese dialogue generation, DialoSpeech achieves the lowest CER (2.27) among the compared systems, versus 2.81 for CosyVoice2 (run as an independent single-speaker system per turn, then concatenated) and 3.61 for MoonCast, while matching baseline speaker similarity (SIM-O 0.69 vs. 0.75 for CosyVoice2 and 0.74 for MoonCast) and leading on subjective Spontaneity (3.96 vs. 3.44 and 3.87 respectively) (§IV.D, Table 2). On English dialogue, trained only on the Fisher corpus (2,000h), DialoSpeech substantially outperforms a re-implemented CoVoMix baseline trained on the same Fisher data (UTMOS 2.836 vs. 1.735; WER 8.62 vs. 9.71) and is competitive with CosyVoice2 on subjective Spontaneity despite CosyVoice2 having access to much larger English training data (§IV.D, Table 1). The authors highlight that DialoSpeech reaches these results with a 0.5B-parameter LM and ~10K hours of training data, compared to over 1M hours reportedly used by CosyVoice2 and MoonCast (§IV.D).
The comparisons mix training-data regimes (CoVoMix retrained on the same Fisher subset as DialoSpeech; MoonCast and CosyVoice2 used as topline references trained on orders of magnitude more data), so the “efficiency” framing is only partially controlled: it is fair evidence against CoVoMix on equal data, but the comparisons against CosyVoice2/MoonCast conflate architecture and data-scale differences.
Novelty Assessment
The core architectural contribution is the combination of a causal cross-attention mechanism between two parallel LLM-predicted semantic token streams (extending the dual-tower idea from dGSLM and the dual-channel modeling in CoVoMix with LLM-based contextual conditioning) and a block-wise guided attention scheme that lets flow matching operate in fixed-memory chunks over long dialogue sequences. Both are genuine, if incremental, architectural extensions rather than a wholesale new paradigm: dual-channel dialogue generation (CoVoMix), LLM-scripted podcast synthesis (MoonCast), and F5-TTS-style conditional flow matching are each pre-existing components; this paper’s contribution is integrating them with a new inter-speaker attention mechanism and a streaming-oriented attention masking scheme for the acoustic model. The data pipeline in Section II is a substantial engineering effort (ASR, diarization, OSD, multi-stage filtering, and source separation chained together) but assembles established tools rather than introducing a new method; it is not released as a public dataset. The efficiency claim (competitive quality at 10K hours vs. baselines’ 1M+ hours) is a meaningful practical finding but is not isolated in a controlled ablation with matched architectures.
Field Significance
Moderate — DialoSpeech provides an integrated architecture and data pipeline for zero-shot dual-speaker dialogue synthesis that models overlapping and turn-taking speech more directly than single-stream LLM dialogue scripting, and demonstrates competitive quality using markedly less training data than the two systems it compares against. Its significance is bounded by only preprint-stage evaluation, comparisons that are not fully matched across training data scale, and reliance on internal (non-public) datasets for both training and evaluation.
Claims
- supports: Modeling dual-speaker dialogue as two parallel token streams with explicit cross-stream attention allows a synthesis system to represent turn-taking and overlapping speech without external timing annotations at inference time.
Evidence: DiaLM inserts a
<SIL>token per inactive-speaker channel and relies solely on causal cross-attention between the two LLM-predicted token streams to learn when each speaker should speak, pause, or overlap, without explicit timing control signals. (§III.B) - supports: Chunked, block-wise attention masking can bound the memory cost of flow-matching-based acoustic reconstruction for long, multi-turn audio without requiring a separate streaming-specific model.
Evidence: The block-wise guided attention mechanism partitions tokens into fixed-size blocks and applies causal/history/future mask patterns to the DiT backbone, enabling chunked decoding with a fixed memory footprint at inference. (§III.C)
- complicates: Training-data efficiency claims for LLM-based dialogue TTS systems are difficult to isolate from architectural differences when baselines are trained on far larger, non-matched corpora.
Evidence: DialoSpeech is trained on ~10,000 hours and reports competitive or superior CER/subjective scores against CosyVoice2 and MoonCast, which the paper states are trained on over 1,000,000 hours, but the systems also differ architecturally, so the comparison does not isolate a pure data-efficiency effect. (§IV.D)
- complicates: Cross-lingual generalization for a dual-speaker LLM dialogue model trained on a single, telephone-quality language corpus remains uneven across intelligibility and perceptual quality metrics.
Evidence: Trained on English only via the 2,000-hour Fisher corpus, DialoSpeech achieves higher UTMOS and speaker similarity than a Fisher-trained CoVoMix baseline but is outperformed by CosyVoice2 on WER and UTMOS, which the authors attribute to CosyVoice2’s access to much larger English training data. (§IV.D)
Limitations and Open Questions
Warning
The evaluation relies on internal, non-public datasets for both training and testing (Biaobei Corp. recordings, internally crawled podcast/vlog audio, and internal Chinese/English test sets), and no held-out public benchmark comparison is reported, limiting independent reproducibility of the reported numbers.
The paper’s own conclusion notes that jointly training the language model and flow matching module becomes difficult at minute-scale context lengths due to memory limitations, and identifies continuous latent representations (rather than discrete tokens) as a promising direction to address this. The English evaluation is also limited to telephone-band Fisher-style speech, so generalization to higher-fidelity or more diverse English conversational styles is untested.
Wiki Connections
- Flow Matching — DialoSpeech’s token-to-waveform stage is a conditional flow matching model built on an F5-TTS-style DiT backbone, extended with block-wise guided attention for chunked, memory-bounded decoding.
- Autoregressive Codec TTS — DiaLM autoregressively predicts dual-speaker semantic token streams from dialogue text, following the VALL-E-style next-token-prediction paradigm but extended to two simultaneous channels.
- Zero-Shot TTS — the system clones both dialogue speakers’ voices from short reference prompts via ECAPA-TDNN speaker embeddings, without speaker-specific fine-tuning.
- Multilingual TTS — DialoSpeech is trained on Chinese and English dialogue data and is evaluated for cross-lingual generalization using Chinese speaker prompts to generate English dialogue.
- Neural Audio Codec — the model’s semantic token representation is produced by the S3Tokenizer from CosyVoice2, reused here as the per-speaker discretization target for the dual-track LM.
- VALL-E — DialoSpeech’s overall next-token-prediction framing for the T2T stage builds on the AR discrete-token TTS paradigm this paper introduced.
- CosyVoice — cited as part of the lineage of supervised semantic token TTS systems that DialoSpeech’s S3Tokenizer-based token representation builds on.
- CosyVoice2 — used as a strong single-speaker zero-shot TTS baseline (run per-turn and concatenated) and as the source of the S3Tokenizer used for DialoSpeech’s semantic tokens.
- MoonCast — used as a topline zero-shot podcast generation baseline for Chinese dialogue comparison, trained on orders of magnitude more data than DialoSpeech.
- F5-TTS — DialoSpeech’s chunked conditional flow matching module is built directly on the F5-TTS DiT architecture.