arXiv · 2026 · Preprint

Ruiqi Li et al. (ByteDance / Zhejiang University) · → Paper · Demo: ✓ · Code: ?

A single non-autoregressive flow-matching DiT generates whole 1-4-speaker conversations and monologues from a speaker-turn-tagged script, trained through a monologue-to-dialogue curriculum on a pause-corrected in-the-wild corpus and then post-trained with DiffusionNFT on pronunciation and speaker-similarity rewards.

Problem

Multi-party content such as podcasts and short dramas is usually produced by synthesizing each turn separately with a monologue TTS model and concatenating the waveforms. Each turn may sound fine on its own, but adjacent turns disagree in room acoustics, ambience, loudness, and pause timing, so the result sounds assembled rather than recorded as one scene. Existing dialogue TTS systems mostly use autoregressive language-model decoders, which bring sequential latency and skipping/repetition failures in long outputs, and they are typically fine-tuned from a monologue model on dialogue data, which can erode monologue quality. The authors also argue that the bottleneck is partly data: dialogue training needs reliable speaker segmentation, transcripts whose punctuation reflects real pauses, and enough non-neutral speech to learn affect.

Method

Data (SwanData-Speech). From a raw pool of about 2.59M hours (2.24M Chinese, 0.35M English, mainly internal), the pipeline runs vocal separation, 3D-Speaker diarization (FSMN VAD, CAM++ embeddings, spectral clustering), and splits segments into a monologue pool (same-speaker merges up to 60 s) and a dialogue pool (2-4-speaker merges up to 120 s, no silence over 2 s, sliding-window greedy merging). SenseVoice-Small transcribes with inverse text normalization disabled; dialogue turns are wrapped in <S{id}>…</S{id}> tags. A forced aligner then rewrites punctuation to match measured pauses: gaps of 0.08-0.18 s become a <|sp|> token, 0.18-0.45 s a comma, and longer gaps sentence-final punctuation, while punctuation with no observed pause is deleted. Samples are filtered with DNSMOS and torchaudio-SQUIM PESQ/STOI, and emotion2vec+ selects high-confidence non-neutral samples as a high-expressiveness subset. A synthetic subset, RobustMegaTTS3, renders LLM-generated sentences for every GCIDE word and every Level-1/2 standard Chinese character, plus 40K polyphone/homograph hard cases and 100K Chinese-English code-switching texts, with MegaTTS 3 as a phoneme-based renderer.

Swan Forced Aligner (Appendix A-B). The aligner used for punctuation correction is itself a new model of about 400M parameters: a WavLM acoustic encoder plus Transformer text and audio encoders, a <|wbd|> anchor token that pools each word’s subword tokens, an interleaved word/blank state lattice with neighbour-conditioned blank prototypes, stay/advance/skip transitions, per-frame score canonicalization with separate learned gains for unary and transition terms, and a combined cross-entropy, CRF, duration, and monotonicity objective. It is trained on an 80K-hour subset pseudo-labelled with Montreal Forced Aligner and decoded with Viterbi or posterior decoding.

Model. A VAE downsamples 24 kHz audio to 25 latent frames per second; its encoder follows WavTokenizer’s design and its decoder is HiFi-GAN-based, trained with reconstruction, light KL, and LSGAN losses against MPD/MSD/MRD discriminators. Text enters as raw BPE tokens from the CosyVoice tokenizer (one token per Chinese character) with no G2P frontend, extended with the <|sp|> pause token and 1,549 pinyin syllables; random pinyin substitution during training lets users force a pronunciation at inference. A per-token speaker-turn label sequence is derived from the turn tags. The 2B-parameter flow-matching DiT pads text and turn embeddings to latent length, passes them through a small Transformer before merging with the speech latent (following F5-TTS’s observation that this beats naive early concatenation), and uses RMSNorm with AdaLN global adapters. Training masks a contiguous target span after a reference span; for dialogue the reference must contain speech from every speaker.

Curriculum and post-training. Stage 1 trains from scratch on about 2M hours of monologue speech plus the synthetic hard cases (64 A100s, 500k steps); stage 2 mixes monologue with concatenated 2-4-speaker pseudo-dialogues, oversampling conversations so speaker switches are frequent (32 A100s, 600k steps); stage 3 replaces the concatenations with real conversations from film, TV, and podcasts (32 A100s, 300k steps). Post-training applies DiffusionNFT, a value-free online RL method that optimizes the forward flow-matching objective, on 3K real conversations with pause-corrected transcripts. The reward averages a phone-tone recognition reward, exp(-phone WER), and prompt speaker cosine similarity, with a regularizer toward the reference policy.

Inference. The reference speech is transcribed with SenseVoice-Small, target duration comes from a per-speaker speaking-rate heuristic, sampling uses F5-TTS’s sway sampling, and a “staircase” classifier-free guidance combines null, text-only, and full conditions with separate text and reference scales so content guidance and timbre/style guidance can be tuned independently.

Key Results

All TTS evaluation uses SwanBench-Speech (Pan et al., 2026), a benchmark from an overlapping author group, with fully automatic metrics: segment-level speaker similarity, SRMR variability for reverb consistency, SQUIM-PESQ, CER/WER from FunASR-Nano, SpeechJudge prosody scores, and Gemini-3-Pro judgements of expressive richness and hierarchy. There is no human listening test.

Monologue (Table 1, Expressive Challenge subset, 10 open-source baselines). SwanVoice has the highest richness (3.81; next VibeVoice 3.42, IndexTTS-2 3.32) and hierarchy (3.62; next VibeVoice 3.06, MegaTTS-3 3.01) of all 11 systems. On every other axis it trails at least one baseline. Its content error of 0.172 is ninth of 11, worse than the baseline average of 0.12 and than every baseline except ZipVoice (0.213) and SparkTTS (0.314); FishSpeech has the lowest (0.066). Prosodic coherence (3.56) is fourth, behind FishSpeech (3.77), VibeVoice (3.75), and IndexTTS-2 (3.63); sound fidelity (3.60) is fourth behind FishSpeech (4.09), GLM-TTS (3.90), and CosyVoice-3 (3.80); reverb consistency (2.06, lower is better) is fifth behind GLM-TTS (1.64), IndexTTS-2 (1.77), FishSpeech (2.00), and SparkTTS (2.04); timbre (0.93) is 0.01 below GLM-TTS (0.94) and level with six other baselines.

Dialogue (Table 2, six open-source dialogue/podcast baselines). SwanVoice again has the highest richness (3.62 vs. VibeVoice 3.09) and hierarchy (3.71 vs. SoulX-Podcast 3.15) of all seven systems, and ties SoulX-Podcast for highest timbre consistency (0.92). SoulX-Podcast leads on sound fidelity (3.98 vs. 3.77), prosody (3.89 vs. 3.70), and content error (0.101 vs. 0.145, with ZipVoice-Dialog at 0.116 also ahead); VibeVoice leads on reverb consistency (2.09 vs. 3.02). SwanVoice’s dialogue content error is below the baseline average (0.180) but third among seven.

Aligner (Table 3, AAS in ms, lower is better). Swan Forced Aligner scores 45.19 on GTSinger-Speech-ZH and 27.67 on LibriSpeech-Clean, lower than every publicly released aligner (next Qwen3 Forced Aligner, 47.31 and 27.84); the unreleased LattifAI Aligner is lower on both (31.60, 25.70). On LibriSpeech-Others, Qwen3 Forced Aligner is lowest (29.74), with Swan at 29.92 and LattifAI at 36.00. The paper’s prose describes Swan as about 10 ms behind LattifAI on LibriSpeech-Others, but the table image shows Swan 6 ms ahead of LattifAI there.

The expressiveness lead is consistent in size across both settings (about 0.4-0.56 points on 1-5 judge scales), but it rests on a single LLM judge, and the paper reports no ablation showing which ingredient (emotion-filtered data, real dialogue SFT, RL, guidance) produces it.

Novelty Assessment

The contribution is primarily engineering integration and data construction. The components are established: a HiFi-GAN-decoded latent VAE, a flow-matching DiT with F5-TTS-style in-context infilling and sway sampling, per-token speaker-turn labels (as in earlier dialogue TTS work), and DiffusionNFT as an off-the-shelf flow-model RL method with ASR and speaker-similarity rewards. The less common choices are the explicit pause-calibrated punctuation (mapping measured silences to <|sp|>, comma, or period), pinyin substitution as an opt-in pronunciation override on a raw-text model, the staircase CFG that separates text and reference guidance, and a phone-tone rather than character-level ASR reward. The Swan Forced Aligner, with its neighbour-conditioned blank states and calibrated lattice decoding, is the most architecturally distinctive piece, but it is a data-pipeline tool reported in an appendix. None of these design choices is ablated, so their individual value is asserted rather than demonstrated. The data scale (about 2M hours) and corpus are internal and unreleased.

Field Significance

Moderate. The paper adds evidence that a non-autoregressive flow-matching model can generate multi-speaker conversations as one sequence while keeping competitive monologue quality, using a curriculum from monologue to synthetic to real dialogue data. Its clearest empirical signal, higher LLM-judged expressiveness across both monologue and dialogue, comes with weaker content accuracy than several open-source baselines, which makes it a useful data point on the expressiveness versus intelligibility trade-off in long-form synthesis.

Claims

  • supports: Non-autoregressive flow-matching models can generate multi-speaker dialogue as a single sequence when speaker identity is supplied as a per-token turn label aligned with the text.

    Evidence: A 2B flow-matching DiT with token-level speaker-turn embeddings synthesizes 2-4-speaker conversations and has the highest judged richness and hierarchy among seven dialogue systems while tying for best timbre consistency (0.92). (§3.2, §3.3, §4.5, Table 2)

  • refines: Training a dialogue-capable speech generator benefits from staging data from monologue to synthetic concatenated conversations before real conversations, rather than training on conversational data directly.

    Evidence: The authors report that training directly on conversational data from scratch produces unintelligible speech, and use a three-stage curriculum with monologue examples retained at every stage; the resulting model stays at or above the open-source average on timbre, fidelity, and prosody in monologue. No ablation of the curriculum is reported. (§3.4, §4.4, Table 1)

  • complicates: Gains in judged expressiveness for long-form TTS can coexist with worse content accuracy than simpler baselines.

    Evidence: In monologue, SwanVoice has the highest richness (3.81) and hierarchy (3.62) but a content error of 0.172, worse than the 10-baseline average (0.12) and ninth of 11 systems, despite RL post-training on a phone-level WER reward. (§3.5, §4.4, Table 1)

  • supports: Rewriting transcript punctuation from measured acoustic pauses gives TTS training text whose pause symbols correspond to actual silences, which ASR punctuation chosen for readability does not guarantee.

    Evidence: The pipeline maps aligner-measured gaps to <|sp|>, comma, or sentence-final punctuation by duration thresholds (0.08/0.18/0.45 s), deletes punctuation where no pause occurs, and uses a dedicated forced aligner whose AAS is lower than released baselines on Chinese and LibriSpeech-Clean. The effect on synthesized pause behaviour is not measured. (§2.4.2, Appendix A.1, Table 3)

  • complicates: Expressiveness comparisons for long-form speech that rely only on an audio-LLM judge are hard to interpret without human listening tests.

    Evidence: Richness and hierarchy, the only axes where SwanVoice leads outright in monologue, are both scored by Gemini-3-Pro on 1-5 rubrics, with no human rating reported to validate the judge. (§4.2, Tables 1-2)

Limitations and Open Questions

Warning

Every TTS result is an automatic score (expressiveness from a single Gemini-3-Pro judge, prosody from SpeechJudge) with no human listening test, and no ablation isolates the curriculum, DiffusionNFT post-training, staircase CFG, pinyin substitution, or pause-corrected data, so the attribution of the expressiveness gains to these components is untested.

  • Content accuracy. The authors name content accuracy as the main limitation: monologue content error is worse than most baselines, and speaker switching still fails for acoustically similar voices or short prompts (§5).
  • Benchmark provenance. SwanBench-Speech comes from an author group that overlaps with this paper’s, and the dialogue evaluation does not name its subset. Results are not broken out by language, so Chinese and English performance cannot be separated.
  • Reproducibility. Training data is mostly internal (about 2.59M raw hours), the 3K post-training set and the 80K-hour aligner set are internal, and no code or checkpoint release is stated; only a demo page is provided.
  • 3-4-speaker dialogue. The model supports up to four speakers, but quantitative evaluation is not separated by speaker count; 3-4-speaker cases appear only on the demo page (§4.5).
  • RL side effects. Improvements in recording-environment consistency and expressiveness after post-training are described only from qualitative inspection (§3.5).
  • Reporting inconsistencies. The aligner prose places Swan about 10 ms behind LattifAI on LibriSpeech-Others, whereas Table 3 shows Swan ahead (29.92 vs. 36.00 ms). The reference attached to the MOSS-TTSD baseline is the MOSS-Speech report, so the exact MOSS-TTSD version evaluated is unclear.
  • Synthetic data dependence. Pronunciation-hard and code-switching coverage comes from MegaTTS 3 renderings, and MegaTTS 3 is also a baseline; the effect of this synthetic subset on content error is not measured.

Wiki Connections

  • Flow Matching — The generator is a 2B flow-matching DiT trained with a velocity objective, sampled with sway sampling, and post-trained with a forward-process flow RL method.
  • Zero-Shot TTS — Both monologue and dialogue settings are evaluated as prompt-based zero-shot cloning against open-source zero-shot baselines.
  • Neural Audio Codec — The paper trains its own 25 Hz continuous VAE latent with a HiFi-GAN-style adversarially trained decoder as the generation space.
  • RLHF Speech — DiffusionNFT online RL with a phone-tone recognition reward and a speaker-similarity reward is applied after supervised training.
  • Multilingual TTS — The model is trained on Chinese and English with code-switching synthetic data and evaluated with CER on Chinese and WER on English.
  • MegaTTS 3 — Renders the synthetic RobustMegaTTS3 pronunciation-hard subset used in pretraining and also serves as a monologue baseline.
  • F5-TTS — Source of the in-context text-conditioning design and sway sampling the DiT adopts; also a monologue baseline.
  • CosyVoice 3 — Supplies the raw-text BPE tokenizer SwanVoice extends with pause and pinyin tokens; also a monologue baseline.
  • VibeVoice — The strongest baseline on monologue richness and hierarchy and the leader on dialogue reverb consistency.
  • SoulX-Podcast — The dialogue baseline that beats SwanVoice on content error, sound fidelity, and prosody while trailing on expressiveness.
  • ZipVoice-Dialog — A non-autoregressive flow-matching dialogue baseline with lower content error than SwanVoice.
  • FireRedTTS-2 — An autoregressive dialogue baseline cited as an example of turn-controlled conversational TTS.
  • SpeechJudge — Provides the prosodic coherence scorer used in all TTS comparisons.