arXiv · 2025 · Preprint
Xue et al. (Kunlun Inc.) · → Paper · Demo: ✓ · Code: ?
MoE-TTS addresses the robustness gap between in-domain and out-of-domain text descriptions in instruction-conditioned TTS by embedding speech-modality experts into a frozen LLM via a modality-based Mixture-of-Experts approach, thereby preserving the LLM’s pre-trained text understanding capabilities while learning speech generation.
Problem
Description-based TTS systems (also called instruction-conditioned or prompt-based TTS) are trained on curated datasets whose natural language descriptions ultimately derive from finite tag sets, regardless of how much LLM-generated paraphrase diversity they contain. At inference, real users produce descriptions that routinely fall outside this tag space — using metaphors, analogies, implied attributes, or figurative language that the model was never exposed to. Prior work either trained a text encoder jointly with the TTS model (limiting generalisation to in-domain language), relied on pre-trained encoders like T5 with limited capacity for complex linguistic phenomena, or fine-tuned the entire LLM backbone within the TTS framework — causing catastrophic forgetting that erodes the very text understanding capabilities the LLM was chosen for.
Method
MoE-TTS is built on a pre-trained textual LLM (Qwen3-4B) extended to 8B parameters through the addition of speech-modality experts. The basic task formulation follows a standard autoregressive codec language model: text descriptions and transcriptions are provided as user messages in the LLM’s chat template, and speech tokens are generated autoregressively as assistant responses in a next-token prediction framework. The CosyVoice2 speech tokenizer at 25 Hz converts waveforms into 6,561 discrete speech tokens, which are appended to the LLM vocabulary.
The key architectural contribution is the modality-based Mixture-of-Experts (MoE) integration. Rather than fine-tuning the entire LLM — which leads to catastrophic forgetting — MoE-TTS inserts parallel speech-expert parameter sets alongside the frozen original LLM parameters within each Transformer block. The critical components converted to MoE layers are multi-head attention (Q, K, V, O projections), feed-forward networks, and layer normalization. A deterministic modality-based routing strategy assigns text tokens exclusively to the original (frozen) text experts and speech tokens exclusively to the newly trained speech experts. Speech experts are initialised from the text expert parameters. During training, only the speech-modality expert weights are updated; the LLM backbone remains entirely frozen throughout.
Acoustic modelling uses a two-stage pipeline downstream of the LLM. A Diffusion Transformer (DiT) following the EDM framework (0.7B parameters, from Stable Audio) converts predicted discrete speech tokens into Gaussian latent representations. A VAEGAN (also from Stable Audio, operating at 50 Hz with 32-dimensional features) then decodes these latents to 48 kHz waveforms. Training proceeds in two phases: a pre-training phase on pure TTS data (VoxBox, Emilia-YODAS) to establish speech generation capability, followed by fine-tuning on description-based TTS datasets (TextrolSpeech, SpeechCraft, Parler-TTS, LibriTTS-P, ParaspeechCaps) for style-description alignment. All LLM training uses 16 kHz audio; the VAEGAN is trained separately at 48 kHz.
Key Results
Evaluation uses a custom subjective listening test with 21 professional evaluators rating samples on a 5-point scale across six dimensions: speech quality (SQ), word and sentence segmentation accuracy (WSSA), pronunciation accuracy (PA), stylistic expressiveness alignment (SEA), overall alignment (OA), and overall score (OS). The test sets are small: 20 in-domain and 40 out-of-domain samples. Baselines are ElevenLabs and MiniMax commercial APIs.
On in-domain descriptions, MoE-TTS is competitive with the commercial systems on basic speech quality dimensions but does not lead; MiniMax achieves the highest OS (3.83 vs 3.82 for MoE-TTS). On the alignment dimensions — which directly measure description-following — MoE-TTS outperforms both commercial systems (OA: 3.61 vs 3.26 ElevenLabs, 3.46 MiniMax on in-domain).
On out-of-domain descriptions, the alignment advantage is more pronounced. MoE-TTS achieves the highest OS (3.79 vs 3.73 ElevenLabs, 3.70 MiniMax) and the highest OA (3.75 vs 3.39 ElevenLabs, 3.30 MiniMax). Attribute-level accuracy (gender, accent, speed) also favours MoE-TTS in most categories on both test sets, with the exception of age estimation on out-of-domain descriptions where commercial systems perform better.
Note
The test sets are very small (20 and 40 samples respectively) and were constructed by the authors specifically to challenge out-of-domain understanding. The commercial systems were accessed via public API; internal model versions, training data, and conditioning mechanisms are unknown, making the comparison informative but not controlled.
Novelty Assessment
The MoE formulation for modality decoupling is transferred from the multimodal LLM literature (Mono-InternVL, EVEv2) and applied here to TTS for the first time. The core insight — that deterministic modality routing preserves LLM text understanding by keeping backbone weights frozen while learning speech-specific parameters — is novel in the TTS context, though not conceptually new in the broader multimodal learning context. The downstream acoustic modelling pipeline (discrete LLM tokens → diffusion → VAEGAN) recombines components from CosyVoice2 and Stable Audio without architectural modifications.
The framing around out-of-domain description robustness is a genuine and underexplored problem in the instruction-conditioned TTS literature. The construction of a carefully designed out-of-domain test set using metaphors, analogies, and figurative language is a useful methodological contribution, though the 40-sample scale limits statistical power. The comparison against commercial APIs rather than open-source baselines makes it harder to isolate the contribution of the MoE approach from training data and scale effects.
Field Significance
Moderate — MoE-TTS introduces a practically motivated adaptation of modality-based MoE from vision-language models to instruction-conditioned TTS, with a plausible solution to the persistent problem of catastrophic forgetting when adapting LLMs for speech generation. The out-of-domain generalisation framing is a useful lens that the field has not explicitly examined at this level of detail, and the custom evaluation methodology provides a template for future work. However, the small test set scale and opaque commercial baselines limit the strength of the empirical conclusions.
Claims
- Freezing the backbone LLM and routing modality-specific tokens to separate expert sets can preserve pre-trained text understanding capabilities during speech generation fine-tuning. (§3.2)
- Instruction-conditioned TTS systems trained on tag-derived description datasets exhibit significant performance degradation when faced with figurative or metaphorical natural language at inference time. (§1, §4.1)
- Commercial speech synthesis products are not immune to the out-of-domain description challenge, suggesting that this generalisation gap is not solved by scale alone. (§4.2, Table 2)
- Modality separation techniques from multimodal vision-language research transfer meaningfully to the speech domain, reducing catastrophic forgetting without requiring multi-modal data mixing during pre-training. (§2.2, §3.2)
Limitations and Open Questions
Warning
The evaluation is based on 20 in-domain and 40 out-of-domain test samples, annotated by 21 evaluators. These are very small test sets for drawing strong comparative conclusions. Both the in-domain and out-of-domain test sets were constructed by the MoE-TTS authors, introducing potential design bias toward cases where the proposed approach excels.
The system currently supports only English text descriptions, due to the limited scope of available open-source description-based TTS datasets. The LLM architecture sensitivity is unexplored — all experiments use Qwen3-4B, and the impact of model scale (smaller or larger LLM backbones) on the MoE approach is left for future work. The diffusion and VAEGAN components are adapted from Stable Audio without fine-tuning on description-based data, and their contribution to description alignment is not ablated.
Wiki Connections
The modality-based routing approach builds directly on MoE techniques developed in multimodal LLM research, applied here to the autoregressive-codec-tts paradigm. The core motivation — preventing catastrophic forgetting when adapting a textual LLM backbone for speech — connects to instruction-conditioned-tts systems such as Parler-TTS and CosyVoice2, both of which use pre-trained text encoders or LLMs for description following. The acoustic modelling pipeline relies on neural-codec representations via the CosyVoice2 tokenizer, and the downstream diffusion reconstructor is a standard component shared with many recent systems.
In-corpus references: ParaspeechCaps (scaling study for style-prompted TTS datasets), EmoVoice (LLM-based instruction-conditioned TTS), Emilia (multilingual speech generation dataset).