arXiv · 2025 · Preprint
Cheng Gong et al. (Tianjin University) · → Paper · Demo: ✓ · Code: ?
Proposes EMM-TTS, a two-stage cross-lingual emotional TTS system that disentangles speaker timbre from emotion by predicting perturbed self-supervised representations, enabling independent control of emotion and timbre across languages.
Problem
Cross-lingual emotional TTS requires a system to transfer the emotion of a reference speaker, potentially recorded in a different language, onto a target speaker’s voice while keeping that target speaker’s own timbre and producing intelligible speech in the target language. Prior emotion-transfer approaches typically rely on adversarial learning or classifier constraints to separate speaker and emotion information, which can destabilize training and limit the diversity of the synthesized emotion. Recent large-scale voice-cloning systems built on codec language models or flow matching prioritize zero-shot timbre cloning from a single reference clip but determine emotion and timbre jointly from that reference, offering little independent control over the two. Self-supervised representation-based multilingual TTS reduces the need for per-language pronunciation dictionaries, but the effect of modern SSL-based speaker anonymization methods (as opposed to classic signal-processing perturbations such as formant shifting) on emotion-timbre disentanglement in this setting had not been systematically studied.
Method
EMM-TTS extends the authors’ earlier ZMM-TTS system into a two-stage pipeline: a txt2vec stage that predicts emotion-bearing self-supervised (SSL) representations from text, and a vec2wav stage that restores the target speaker’s timbre from perturbed versions of those representations. The txt2vec module combines an XPhoneBERT phoneme encoder, a Mel-style encoder that learns a sentence-level implicit style embedding capturing speaker and emotion cues, and a variance adaptor with three predictors (pitch, energy, duration extracted from paired text-speech data via a learnable aligner) that models emotion explicitly alongside the implicit embedding.
In the vec2wav stage, the model performs a speaker perturbation on the training waveform (formant shifting via Praat, or an SSL-based language-independent speaker anonymization method) before re-extracting multilingual discrete/continuous XLSR representations and an emotion representation from the perturbed audio, forcing the model to reconstruct target timbre from speaker-independent input rather than leaking timbre from the predicted representation. These perturbed features pass through a multi-stage encoder and frame decoder; speaker identity is reintroduced via speaker ID (rather than a pretrained embedding, which the authors found leaks emotion information) and a global emotion representation extracted from a pretrained SSL-based emotion recognition model (emotion2vec). To combine these two conditioning signals, the paper introduces Speaker-Emotion Adaptive Layer Normalization (SEALN), which predicts the scale and shift of the frame decoder’s layer-normalized output from the speaker and emotion representations jointly, generalizing single-condition mechanisms such as SALN and CLN to simultaneous emotion-and-timbre control. The SEALN output is passed to a HiFi-GAN generator to produce the final waveform. A Speaker Consistency Loss (SCL), which maximizes cosine similarity between speaker embeddings (from a pretrained speaker encoder) of generated and ground-truth audio, is added to the vec2wav reconstruction objective to further pull the recovered timbre toward the target speaker. The model has 183M parameters; no neural audio codec is used, since the pipeline operates on XLSR self-supervised representations rather than codec tokens.
Key Results
In monolingual zero-shot voice cloning on LibriSpeech test-clean, EMM-TTS (183M params) reaches WER 2.28%, UTMOS 4.11, and SECS 0.661, improving speaker similarity over its direct predecessor ZMM-TTS (SECS 0.644) and over HierSpeech++ (SECS 0.591) at a fraction of HierSpeech++‘s inference cost (RTF 0.027 vs. 0.217), though ZMM-TTS remains faster (RTF 0.003) (§4.1, Table 2).
In cross-lingual emotional synthesis against DiCLET-TTS and M3, EMM-TTS obtains the best subjective naturalness, speaker similarity, and emotion similarity in both Chinese and English test conditions, e.g. Chinese-speaker MOS/DMOS/EMOS of 4.12/3.95/3.97 vs. DiCLET-TTS’s 4.04/3.88/3.85, and shows the smallest degradation when the target speaker’s language differs from the reference emotion speaker’s language (§4.2.1, Tables 3-4). Objectively, EMM-TTS achieves the best SECS and lowest CER across all four language/speaker combinations tested, while DiCLET-TTS (which uses adversarial speaker/emotion classifiers) has the worst intelligibility (§4.2.1, Table 5).
The perturbation ablation shows that adding speaker perturbation of any kind improves SECS relative to no perturbation (e.g., 0.662 vs. 0.627 for Chinese speech/Chinese speaker), but SSL-based speaker anonymization degrades intelligibility far more than formant shifting (CER 12.24% vs. 7.13% in the same condition) (§4.2.2, Table 7). Isolated on perturbed audio itself (not full synthesis), anonymization causes a larger SECS drop (0.354) than formant shift (0.514) but with much worse CER (20.57% vs. 9.07%), indicating anonymization disturbs speaker identity more aggressively at the cost of content fidelity (§4.2.2, Table 6). Component ablations show that removing SCL, SEALN, or the pretrained emotion representation each reduces the corresponding target metric (speaker or emotion similarity), and that emotion similarity and speaker similarity move in opposite directions in these ablations (§4.2.3, Table 8).
Novelty Assessment
The contribution is primarily architectural but incremental: EMM-TTS is an explicit extension of the authors’ own ZMM-TTS system, adding an emotion-modeling stage, the SEALN multi-condition normalization module (a generalization of existing single-condition mechanisms like SALN and CLN), and a speaker-consistency loss borrowed from prior zero-shot TTS work. None of the individual components (perturbation-based disentanglement, adaptive layer normalization conditioning, cosine-similarity speaker losses) is new in isolation; the paper’s genuine contribution is combining them into a working two-stage pipeline and running a controlled comparison of formant-shift versus SSL-based speaker-anonymization perturbation for emotion-timbre disentanglement, a comparison the authors note was previously underexplored for SSL-representation-based TTS. Baseline comparisons use specialized cross-lingual emotion-transfer models (DiCLET-TTS, M3) rather than the large-scale codec-LM systems (Seed-TTS, CosyVoice2, F5-TTS) cited as motivation in the introduction, so the paper does not establish how EMM-TTS compares to that class of system on emotion-timbre control.
Field Significance
Moderate — this paper provides a concrete architectural mechanism (SEALN) for jointly conditioning a TTS decoder on independently controlled speaker and emotion representations, and gives a direct empirical comparison of two speaker-perturbation strategies for disentangling them in an SSL-representation-based pipeline. Its contribution builds incrementally on the authors’ own prior system rather than introducing a new synthesis paradigm.
Claims
- supports: Perturbing speaker-identifying information in self-supervised speech representations during training improves disentanglement of speaker timbre from other speech attributes, measurable as higher speaker-similarity scores at inference when perturbation is used versus omitted.
Evidence: Adding either formant-shift or SSL-based speaker-anonymization perturbation during vec2wav training raises SECS relative to the no-perturbation ablation across all four language/speaker test combinations (e.g., 0.662 vs. 0.627 for Chinese speech, Chinese speaker). (§4.2.2, Table 7)
- complicates: Stronger speaker-identity perturbation for disentanglement purposes trades off against synthesized speech intelligibility.
Evidence: SSL-based speaker anonymization produces a larger drop in speaker-embedding similarity of the perturbed audio itself (SECS 0.354 vs. 0.514 for formant shift) but a far worse character error rate on that same perturbed audio (20.57% vs. 9.07% CER), and this intelligibility gap persists in the full synthesized output (12.24% vs. 7.13% CER in one test condition). (§4.2.2, Tables 6-7)
- complicates: In cross-lingual emotional speech synthesis, emotion similarity and speaker similarity behave as competing objectives rather than independent ones.
Evidence: Component ablations (removing SCL, SEALN, or the pretrained emotion representation) show that gains in speaker-similarity metrics (DMOS) are accompanied by losses in emotion-similarity metrics (EMOS) and vice versa across the tested configurations. (§4.2.3, Table 8)
- supports: Combining explicit acoustic features (pitch, energy, duration) with pretrained latent SSL representations improves voice-cloning quality beyond using latent representations alone.
Evidence: The authors report that pretrained high-dimensional latent features from speaker or emotion encoders cannot fully substitute for explicit acoustic feature modeling, and that incorporating explicit pitch/energy/duration prediction alongside the SSL-based txt2vec pipeline improves voice-cloning performance relative to their ZMM-TTS predecessor, which lacks this explicit modeling. (§5, §4.1)
Limitations and Open Questions
Warning
The cross-lingual evaluation covers only two languages (Chinese and English) using a single emotional corpus (ESD, 350 base sentences per language upsampled 5x for training), so the paper’s cross-lingual and “multilingual” framing is not tested beyond this bilingual pair, and results may not generalize to lower-resource language pairs or larger emotion vocabularies.
The paper does not compare EMM-TTS against the large-scale codec-language-model or flow-matching voice-cloning systems it cites as motivating context (Seed-TTS, CosyVoice2, F5-TTS), so it is unclear how the proposed disentanglement mechanism would fare against that class of system on emotion-timbre control. The negative correlation observed between speaker similarity and emotion similarity in the ablation study is acknowledged but not resolved; the authors defer finer-grained, adjustable control over this trade-off to future work. The specific XLSR checkpoint/variant used to produce the SSL representations predicted by txt2vec is not stated in the running text, limiting exact reproducibility of that component. No code repository is mentioned; only a demo page with audio samples is referenced.
Wiki Connections
- Emotion Synthesis — introduces the SEALN normalization module and Speaker Consistency Loss to enable independently controllable emotion transfer alongside timbre preservation in a cross-lingual setting.
- Multilingual TTS — builds on an XPhoneBERT-based multilingual phoneme representation to synthesize emotional speech across Chinese and English with a single shared model.
- Disentanglement — uses training-time speaker perturbation of self-supervised representations, combined with SEALN conditioning, to explicitly separate speaker timbre from emotional content.
- Self-Supervised Speech — predicts and perturbs self-supervised speech representations as the core generation target, and uses a pretrained SSL-based emotion recognition model to extract emotion embeddings.
- Zero-Shot TTS — evaluates monolingual voice cloning with a pretrained speaker embedding standing in for unseen target speakers, following the same zero-shot protocol as its ZMM-TTS predecessor.
- emotion2vec — used as the pretrained SSL-based emotion recognition model that supplies the global emotion representation conditioning the vec2wav stage.
- LibriTTS — used as training data for the monolingual voice-cloning experiments and as one of the corpora underlying the cross-lingual training mixture.
- NaturalSpeech 3 — its evaluation protocol (randomly sampling 25 utterances per speaker from LibriSpeech test-clean) is adopted for the monolingual voice-cloning benchmark.
- Seed-TTS — cited as an example of large-scale voice-cloning systems whose focus on zero-shot timbre cloning comes at the expense of independent emotion control, motivating this paper’s disentanglement approach.
- CosyVoice 2 — cited alongside Seed-TTS and F5-TTS as a large-scale codec-language-model voice-cloning system that determines emotion and timbre jointly from a single reference, contrasted with this paper’s independent-control design.
- F5-TTS — cited as an example of recent large-scale flow-matching-based voice cloning that prioritizes zero-shot timbre reproduction over flexible emotion control.
- XTTS — cited as prior work in multilingual zero-shot TTS that this paper’s cross-lingual emotional synthesis setting builds upon conceptually.