arXiv · 2026 · Preprint
Han Zhu et al. (Xiaomi Corp.) · → Paper · Demo: ? · Code: ✓
ZipVoice-Dialog is a 123M-parameter non-autoregressive flow matching model for zero-shot spoken dialogue generation that achieves 15× faster inference than autoregressive baselines by combining curriculum learning (monologue pretraining then dialogue fine-tuning) with learnable speaker-turn embeddings to solve the speech-text alignment and turn-taking problems.
Problem
Generating spoken dialogue from text is harder than monologue TTS because: (1) multiple distinct speaker timbres must be maintained across turns while learning speech-text alignment; (2) precise speaker turn-taking (who speaks when) is non-trivial; and (3) existing autoregressive (AR) systems (MoonCast, Dia/Parakeet) suffer from high latency and exposure bias leading to word repetition or skipping. Preliminary experiments confirm that vanilla flow matching fails completely on dialogue data (WER ~116% without curriculum learning), demonstrating that the challenges are not merely incremental. Furthermore, the field lacks large-scale open-source dialogue training data.
Method
ZipVoice-Dialog extends ZipVoice, a state-of-the-art flow matching monologue TTS system built on Zipformer encoders and a conditional flow matching (CFM) objective with speech infilling.
Curriculum learning. Two-stage training: Stage 1 initializes from ZipVoice pretrained on 100k hours of monologue (Emilia); Stage 2 fine-tunes on 7.6k hours of dialogue data for 60k updates (batch 4k seconds). Stage 1 establishes robust speech-text alignment; Stage 2 adapts to multi-speaker conversational dynamics. Without Stage 1 initialization, WER collapses to ~84–116% (Table 1).
Speaker-turn embeddings. Two randomly initialized learnable embeddings (one per speaker, denoted [S1] and [S2]) are added to the text encoder output features after encoding but before average upsampling. These are distinct from speaker verification embeddings — they are binary role embeddings optimized end-to-end. This simple mechanism reduces cpWER from 31.34% (with two distinct speaker tokens but no embeddings) to 5.82% (Table 2), demonstrating that explicit embedding-based disambiguation is far more effective than textual speaker tokens alone.
Input format. Multi-turn utterances are sorted chronologically and interleaved into a single text sequence. Adjacent same-speaker utterances are merged. The model implicitly learns token and turn durations from the CFM objective — no external duration predictor or timestamp annotation is needed.
Stereo generation extension. ZipVoice-Dialog-Stereo renders each speaker to a separate audio channel using weight initialization from the mono model, single-channel regularization, and a speaker-exclusive loss.
OpenDialog dataset. A 6.8k-hour open-source spoken dialogue corpus (5.1k hrs English, 1.8k hrs Chinese) mined from in-the-wild audio using VAD → diarization → WhisperD speaker-attributed transcription → LLM-based dialogue classification → DNSMOS filtering (DNSMOS OVRL ≥ 2.8). This is the first large-scale open-source spoken dialogue dataset.
Evaluation benchmark. Two out-of-domain test sets (test-zh: 357 dialogues, test-en: 280 dialogues) from real-world spontaneous speech. Metrics: WER, cpWER (concatenated minimum-permutation WER for turn-taking accuracy), cpSIM (speaker cosine similarity with permutation matching), UTMOS, RTF.
Inference. 16-step Euler ODE solver with time-dependent classifier-free guidance.
Key Results
ZipVoice-Dialog trained on all data (7.6k hours) vs. baselines (Table 4):
| Model | Params | RTF | test-en WER | test-en cpWER | test-en cpSIM | test-en UTMOS |
|---|---|---|---|---|---|---|
| Dia (AR) | 1.61B | 1.663 | 11.8% | 12.59 | 0.333 | 1.87 |
| MoonCast (hybrid AR/NAR) | 2.67B | 0.953 | 23.62% | 16.53 | 0.356 | 2.37 |
| ZipVoice-Dialog | 123M | 0.063 | 3.25% | 3.27 | 0.437 | 3.07 |
ZipVoice-Dialog is 15× faster than MoonCast and 26× faster than Dia (RTF 0.063 vs. 0.953/1.663), while dramatically outperforming both on intelligibility and speaker turn-taking accuracy. The WER gap (3.25% vs. 11.8%/23.62%) reflects the instability of AR models under exposure bias.
Subjective CMOS/SMOS evaluation on test-zh (Table 5): ZipVoice-Dialog scores CMOS 0.00 (reference) and SMOS 3.86±0.11, while MoonCast scores CMOS -1.17±0.12 and SMOS 2.35±0.14. Despite MoonCast’s larger parameter count, its AR instability substantially degrades perceptual quality.
OpenDialog alone (6.8k hours) vs. in-house data (0.8k hours, Table 3): OpenDialog yields better WER but slightly lower cpSIM and UTMOS, attributed to higher quality and annotation accuracy of the manually transcribed in-house data. Combining both achieves the best balance.
Novelty Assessment
The core technical contributions are modest but effective: curriculum learning (an established technique applied to a new domain) and speaker-turn embeddings (a conceptually simple but empirically impactful addition). The key insight is that dialogue generation is not simply multi-speaker TTS but requires these specific adaptations to flow matching. The dataset (OpenDialog) is a significant community contribution. The evaluation benchmark (cpWER for turn-taking accuracy) is a useful methodological contribution. The work is primarily a systems paper — the value is in making NAR dialogue generation work, not in novel algorithmic development.
Field Significance
Tip
High — ZipVoice-Dialog demonstrates that non-autoregressive flow matching can be adapted to multi-speaker spoken dialogue generation with relatively simple modifications, upending the assumption that AR models are necessary for this task. The simultaneous release of OpenDialog (the first large-scale open dialogue corpus) and a reproducible evaluation benchmark with cpSIM and cpWER metrics provides the field with infrastructure for future dialogue synthesis research.
Claims
- Non-autoregressive flow matching can achieve spoken dialogue generation only when combined with curriculum learning; direct fine-tuning on dialogue data without monologue pretraining causes near-complete intelligibility collapse. (§4.1, Table 1)
- Learnable role embeddings added to text encoder features are substantially more effective for multi-speaker turn-taking disambiguation than text-level speaker tokens alone. (§4.2, Table 2)
- Non-autoregressive dialogue systems can outperform larger autoregressive baselines in intelligibility, turn-taking accuracy, and speaker similarity at a fraction of the inference cost. (§8.4, Table 4)
- Automatically mined in-the-wild spoken dialogue data is sufficient to train competitive dialogue TTS models, with performance approaching manually annotated in-house data on intelligibility metrics. (§8.3, Table 3)
- Subjective quality scores for autoregressive spoken dialogue models are strongly penalised by instability artefacts even when model size and expressiveness are larger than the non-autoregressive baseline. (§8.4, Table 5)
Limitations and Open Questions
Model expressiveness is limited by the 123M parameter budget; the authors acknowledge that larger models may yield more expressive results. Subjective evaluation was conducted only in Chinese by native speakers. The model is limited to two-speaker dialogues (though the method generalizes in principle). The lack of prosody-conditioned or emotion-conditioned generation limits expressiveness. Future work should explore larger model scales and multilingual dialogue beyond English/Chinese.
Wiki Connections
- flow-matching — ZipVoice-Dialog extends the flow matching TTS paradigm to multi-speaker dialogue, showing that curriculum learning and role embeddings are necessary adaptations
- zero-shot-tts — the model uses speech infilling-based zero-shot speaker conditioning inherited from ZipVoice
- spoken-language-model — contrasts with textless spoken LM approaches (dGSLM) and hybrid AR/NAR systems (MoonCast) for dialogue generation
- evaluation-metrics — introduces cpSIM (concatenated maximum permutation speaker similarity) and a reproducible dialogue benchmark built on out-of-domain real-world speech