arXiv · 2025 · Preprint
Ziqi Chen et al. (Tsinghua University, Giant Network AI Lab) · → Paper · Demo: ✗ · Code: ✓
Introduces an IPA-standardized, open-data pipeline for Chinese dialect TTS that adds a dialect-aware Mixture-of-Experts module to F5-TTS and adapts to new dialects with only a few hours of data via LoRA and Conditioning Adapters.
Problem
Dialect TTS lags far behind standard-language TTS because public dialect resources are largely ASR-oriented rather than TTS-ready, orthographies are inconsistent across dialects, and grapheme-to-phoneme mapping is ambiguous when the same character can carry different pronunciations and prosody across dialect varieties. The paper argues no unified, open-data pipeline previously existed for end-to-end zero-shot dialect TTS, forcing dialect systems to rely on scarce, proprietary, or dialect-specific resources that do not scale to the long tail of low-resource dialects.
Method
DiaMoE-TTS is built on top of F5-TTS, a non-autoregressive flow-matching TTS model that conditions a Diffusion Transformer (DiT) backbone on masked Mel-spectrograms and a reference audio prompt via Optimal Transport Conditional Flow Matching. The authors first construct a unified International Phonetic Alphabet (IPA) pronunciation lexicon over existing open-source ASR corpora (Common Voice Cantonese, Emilia Mandarin, KeSpeech, Southern Min) plus commercially acquired Shanghai and Tianjin data, producing aligned text-IPA-speech triples that replace pinyin or character input with explicit phoneme sequences shared across dialects.
Training proceeds in four stages. Stage 0 initializes from a pretrained F5-TTS checkpoint. Stages 1 and 2 jointly train on the unified IPA multidialect dataset, aligning pinyin/character inputs to the shared IPA phoneme space; Stage 2 additionally inserts a residual Mixture-of-Experts (MoE) module after the text embedding layer, where each expert specializes in the phonological characteristics of a subset of dialects and a learnable gate routes each input sequence to the relevant experts. An auxiliary dialect-classification cross-entropy loss on the gate’s mean-pooled logits (weighted by λ=0.1 in Stage 2, λ=0 elsewhere) is added to the flow-matching objective to counteract “style averaging,” the homogenization of dialect-specific prosody that emerges from naive joint multidialect training. Stage 3 freezes the MoE routing and the rest of the backbone, then adapts to new low-resource dialects (as little as ~3 hours of data) using only LoRA (rank 16) on the query/value projections of the attention layers plus lightweight Conditioning Adapters in the text embedding module, combined with pitch/time-scale data augmentation (factors 0.85-1.15) to expand the small fine-tuning set.
Key Results
The system is trained on roughly 0.7k hours of Mandarin plus 0.4k hours of dialect data (Table 1), several orders of magnitude less than CosyVoice2 (~150k hours) or Qwen-TTS (~3,000k hours). On the main comparison (Table 2), DiaMoE-TTS trails commercial systems substantially on WER for most dialects it evaluates against a baseline average (e.g., 76.59% vs. an average of 31.98% on Cantonese, 37.12% vs. 7.43% on Chengdu), though it is competitive or ahead on UTMOSv2 in several dialects (e.g., 3.33 vs. 3.11 average on Chengdu) and closes much of the MOS gap on some dialects (3.15 vs. 4.01 average on Xi’an) while lagging badly on others (1.80 vs. 2.96 on Shanghai, 1.66 vs. 2.64 on Tianjin). Crucially, the paper reports results on several dialects and specialized styles (Southern Min, Shijiazhuang, Nanjing, Peking Opera Jingbai/Yunbai) for which no commercial baseline numbers exist at all, since those systems do not support them.
The ablation (Table 3) isolates the two proposed components on four dialects: removing the MoE module while keeping IPA input degrades MOS (e.g., 2.46 vs. 3.15 on Xi’an) and WER (45.08% vs. 33.00% on Xi’an); replacing IPA with pinyin input while keeping MoE causes a much larger collapse (MOS drops to ~1.0-1.2, WER rises to ~90-93%), indicating the unified IPA frontend is the more load-bearing design choice of the two.
Novelty Assessment
The core novelty is narrow but genuine: a residual MoE module gated by an auxiliary dialect-classification loss, inserted specifically into the text embedding layer to counteract style-averaging in joint multidialect training, combined with a staged LoRA/Conditioning-Adapter recipe for extending to extremely low-resource dialects (a few hours) without catastrophic forgetting of the base multidialect model. The IPA-based frontend itself is not new (the paper cites prior IPA-based Tacotron and Tacotron2 systems), and the base architecture (F5-TTS’s flow-matching DiT) is adopted wholesale rather than modified. The LoRA and adapter-based PEFT recipe is also an established technique applied here to a new setting (cross-dialect low-resource TTS adaptation) rather than a novel training method in itself. The paper is candid that its aim is feasibility with open, modest-scale data rather than matching proprietary systems’ performance, and the results bear this out: DiaMoE-TTS is not competitive with commercial baselines on WER in most dialects tested, though it is the only system evaluated on several dialects and styles that commercial systems do not cover at all.
Field Significance
Moderate — the paper packages an unusual combination of ingredients (unified cross-dialect IPA representation, a dialect-gated MoE with an auxiliary classification loss, and PEFT-based extreme low-resource adaptation) into a fully open-data, open-code pipeline for a task (Chinese dialect and specialized-style TTS, including Peking Opera recitation) that had little prior open tooling. Its empirical results substantially lag commercial systems on core intelligibility (WER), so its contribution is best read as a reproducible open-source baseline and dataset-construction methodology for low-resource dialect TTS rather than a state-of-the-art result.
Claims
- supports: Joint training across multiple dialects induces “style averaging,” where dialect-specific prosodic identity is diluted unless the model has a mechanism to route dialect-specific information separately.
Evidence: Ablating the dialect-aware MoE module (removing it while keeping the IPA frontend) degrades both MOS (e.g., 2.46 vs. 3.15 on Xi’an) and WER (45.08% vs. 33.00% on Xi’an) relative to the full model, isolating the MoE’s contribution to preserving per-dialect quality in a jointly trained model. (§4.4.2, Table 3)
- supports: A standardized phonetic representation (IPA) resolves grapheme-to-phoneme ambiguity better than orthography-based inputs (e.g., pinyin) when a single model must serve multiple dialects with divergent character-to-sound mappings.
Evidence: Replacing the IPA frontend with pinyin input (keeping the MoE module) collapses MOS to roughly 1.0-1.2 and raises WER to roughly 90-93% across four dialects, a far larger degradation than removing the MoE alone. (§4.4.2, Table 3)
- supports: Parameter-efficient fine-tuning (LoRA plus lightweight adapters) on a frozen multidialect backbone can extend zero/few-shot speech synthesis to a new, extremely low-resource dialect using only a few hours of data.
Evidence: Stage 3 adapts the frozen Stage-2 model to Peking Opera (Jingbai/Yunbai) and Nanjing dialect using only ~3 hours of data per target style, training only LoRA (rank 16) and Conditioning Adapters while the backbone stays frozen, producing usable synthesis (e.g., Jingbai MOS 3.04, UTMOSv2 1.75) for styles with no commercial TTS baseline available for comparison. (§3.3, §4.1, Table 2)
- complicates: Open, modest-scale multidialect training data can substantially underperform proprietary, large-scale commercial systems on standard intelligibility metrics even when subjective naturalness scores are comparable.
Evidence: Trained on ~1.1k total hours versus CosyVoice2’s ~150k hours and Qwen-TTS’s ~3,000k hours, DiaMoE-TTS’s WER trails the baseline average by wide margins in several dialects (e.g., 44.61 points worse on Cantonese, 29.69 points worse on Chengdu), even where its UTMOSv2 score is on par with or better than the commercial average. (§4.4.1, Table 1, Table 2)
Limitations and Open Questions
Warning
Word error rate is substantially worse than commercial baselines across nearly every dialect evaluated (e.g., 76.59% vs. a baseline average of 31.98% on Cantonese), and the paper does not report a controlled comparison isolating whether this gap stems from training data scale, IPA/G2P errors, or the MoE routing itself.
The evaluation compares against commercial black-box systems (Edge TTS, CosyVoice2, Qwen-TTS) whose training data, architectures, and exact evaluation conditions are not fully known, making some of the reported gaps difficult to attribute precisely. Coverage is also uneven: several dialects and styles (Southern Min, Shijiazhuang, Nanjing, Peking Opera) have no baseline comparison at all, so the paper’s claim of being “almost comparable” to commercial systems rests on a subset of dialects where baselines exist. The MoE’s number of experts and their assignment to specific dialects are not detailed in the text, and no comparison is given for how the dialect-classification auxiliary loss weight (λ) was chosen or how sensitive results are to it.
Wiki Connections
- Flow Matching — inherits the OT-CFM/DiT flow-matching backbone from F5-TTS without modification, applying it unchanged to a new dialect-adaptation setting.
- Zero-Shot TTS — extends zero-shot voice cloning (via reference audio prompting, inherited from F5-TTS) to previously unseen Chinese dialects and specialized vocal styles.
- Multilingual TTS — constructs a unified IPA-based text-speech alignment dataset spanning Mandarin and multiple Chinese dialects to enable shared phonetic modeling across them.
- Speaker Adaptation — uses LoRA and Conditioning Adapters on a frozen backbone to rapidly adapt to new, extremely low-resource dialects with only a few hours of data.
- F5-TTS — used as the pretrained flow-matching backbone and initialization checkpoint that DiaMoE-TTS extends with a dialect-aware MoE and PEFT adaptation.
- CosyVoice 2 — used as a large-scale commercial baseline in the main comparison table across several Chinese dialects.
- Spark-TTS — cited as a related zero-shot LLM-based TTS approach in the introduction’s survey of recent zero-shot TTS progress.
- MoE-TTS — cited as prior work applying Mixture-of-Experts to TTS, motivating the dialect-aware MoE design used in this paper’s text embedding module.