arXiv · 2026 · Preprint
Lucas H. Ueda et al. (UNICAMP, CPQD) · → Paper · Demo: ✓ · Code: ✓
Introduces SelfTTS, a VITS-based TTS model that transfers emotion to neutral (non-expressive) speakers using an explicit, label-free embedding disentanglement mechanism and a voice-conversion-based self-augmentation stage, without relying on external pre-trained speaker or emotion encoders.
Problem
Cross-speaker style transfer lets a TTS system apply the prosodic and emotional character of an expressive reference speaker to a target speaker who only provided neutral recordings, which matters because collecting expressive data for every target speaker is often infeasible. Existing approaches typically rely on style/emotion encoders trained with cross-entropy classification heads and, in more recent designs, Gradient Reversal Layers (GRL) applied against external speaker or emotion labels to suppress unwanted information. The paper argues this label-based adversarial approach has a structural weakness: when a dataset’s expressive speech is concentrated in only a subset of speakers, emotion and speaker labels become correlated, so a GRL trained against labels alone cannot fully separate the two attributes. Prior mitigations, such as filtering the input mel-spectrogram to low bins or applying formant/timbre perturbation to the reference audio, are heuristic and lossy. Separately, when expressive data is scarce, synthetic-data pipelines that fine-tune on voice-converted audio are bounded by the quality of the conversion itself. SelfTTS targets both problems with a single self-contained architecture.
Method
SelfTTS extends the VITS conditional VAE-normalizing-flow-adversarial framework by conditioning the Posterior Encoder, the Residual Coupling (normalizing flow) blocks, the Stochastic Duration Predictor, and the Waveform Decoder on dedicated speaker (g) and emotion (e) embeddings, both produced by a shared Reference Encoder design (a 6-layer CNN followed by a GRU) applied separately to speaker- and emotion-labelled mel-spectrogram slices.
Two mechanisms shape these embeddings during training. First, Multi Positive Contrastive Learning (MPCL) is applied to both the speaker and emotion encoder outputs, pulling together embeddings that share a label and pushing apart embeddings that do not, producing label-oriented clusters without requiring the carefully constructed batch sampling that other contrastive objectives (e.g. N-Pair) need. Second, an explicit embedding disentanglement scheme applies a Gradient Reversal Layer directly between the speaker and emotion embeddings themselves (via a small linear “processor” network on each side) using a cosine-similarity loss, rather than reversing gradients against external classification labels. The same cosine-based GRL mechanism is applied a second time at the output of the normalizing flow (z_p, the style-neutral, speaker-agnostic latent), using convolutional processors, to prevent the Posterior Encoder’s linear-spectrogram input from leaking speaker or emotion identity into the shared linguistic latent space.
Because the normalizing flow is invertible, the same trained model performs voice conversion: a source utterance is mapped to the shared latent z_p via the forward flow, then the inverse flow reconstructs it conditioned on a different target speaker/emotion pair. SelfTTS exploits this to implement a self-augmentation refinement stage, applied only after the base model has already learned cross-speaker style transfer: speaker references are randomly permuted while holding the source emotion fixed, generating synthetic same-emotion, different-voice samples that are mixed into a fraction of each training batch and fed to the emotion encoder. Emotion (rather than speaker) references are held fixed during permutation because the model’s conversion does not explicitly control duration, and altering the emotional reference for emotions like “Sad” was found to distort speaking rate. The full model is initialized from a VITS checkpoint pre-trained on VCTK for 800k steps (compatible layers only), trained for 200k further steps on the target data, then refined for 50k steps of self-augmentation at a reduced learning rate.
Key Results
On the ESD dataset (English speakers, five emotion categories, two held-out neutral-only target speakers), SelfTTS achieves the best perceptual emotion-similarity score (eMOS 2.853) among all compared systems, and the best emotion-embedding cosine similarity (EECS 0.8423) and speaker-embedding cosine similarity (SECS 0.8163) among externally-comparable published cross-speaker baselines (E3-VITS and VECL), while ranking second on perceptual naturalness (nMOS 2.746) and speaker similarity (sMOS 3.112) and on UTMOS (3.6104). The E3-VITS baseline achieves the highest naturalness (nMOS, UTMOS) but the worst emotion similarity (eMOS 2.237, EECS 0.5367), which the authors attribute to it collapsing toward the training set’s average (near-neutral) emotion. VECL attains the best WER, attributed to its dedicated intelligibility consistency loss.
An ablation over encoder loss and GRL configuration (Table 2) shows the proposed cosine-based embedding GRL achieves the lowest CKA between speaker and emotion representations (0.0139, indicating the least entanglement) while maintaining competitive SECS and the best EECS, whereas plain MPCL without any GRL leaves representations highly entangled (CKA 0.3148). Self-augmentation raises UTMOS from 3.4899 (without it) to 3.6104 with only a modest EECS trade-off (0.8793 to 0.8423); a 0.25 mixing proportion, using synthetic samples only as additional emotion-encoder references (not as literal training targets), gave the best naturalness/adherence balance across the tested configurations (Tables 5-6). In a cross-corpus setting (training on LJSpeech and VCTK speakers p226/p231, evaluating against ESD-derived emotion centroids), SECS remained high (0.86-0.91) but WER rose sharply to roughly 1.08-1.15, and EECS dropped to 0.60-0.61, reflecting acoustic mismatch between the ESD-trained emotion references and the new recording conditions.
Novelty Assessment
The contribution is a training-recipe and mechanism-level refinement rather than a new generative architecture: SelfTTS is built directly on the existing VITS framework, and its ingredients (GRL-based adversarial disentanglement, contrastive clustering of style embeddings, and voice-conversion-based data augmentation) all have precedents cited by the paper itself. What is new is the specific combination: applying the adversarial disentanglement directly between the speaker and emotion embeddings via a cosine-similarity loss (rather than against external classification labels), which the paper argues removes a structural failure mode of label-based GRL when label correlations exist in the training data; adapting MPCL (originally proposed for other representation-learning settings) to jointly cluster TTS speaker and emotion embeddings without specialized batch construction; and using the model’s own voice-conversion output as a curriculum-style self-augmentation stage applied after the base model converges, rather than as an offline data-generation step. The ablation studies (Tables 2 and 3) are a genuine strength, systematically isolating the contribution of each design choice against reasonable alternatives (CE vs. MPCL, GRL variants, alternative emotion encoders and input perturbations).
Field Significance
moderate — This paper offers a concrete recipe improvement for a well-established problem (cross-speaker emotion transfer) and demonstrates, with controlled ablations, that adversarial disentanglement applied at the embedding level rather than the label level can reduce speaker-emotion entanglement without external pre-trained encoders. Its scope is limited to a single VITS-style architecture, a single emotional speech corpus (ESD) with only two evaluated target speakers, and a small-scale subjective test; the cross-corpus experiment shows the approach’s fidelity to speaker identity generalizes better than its intelligibility does under acoustic mismatch.
Claims
- supports: Applying an adversarial disentanglement loss directly between two learned embeddings (rather than between each embedding and its external classification label) can more effectively suppress cross-attribute leakage when label correlations exist in the training data.
Evidence: The proposed cosine-similarity GRL applied directly between speaker and emotion embeddings achieves the lowest CKA (0.0139, least entangled) of all tested encoder-loss/GRL configurations, while retaining the highest EECS (0.8793) and competitive SECS (0.8103), outperforming both no-GRL and label-based cross-entropy GRL variants. (§3.1.1, Table 2)
- supports: Using a generative model’s own voice-conversion output as a post-hoc self-augmentation stage, applied only after the base model has already learned the target task, can improve output naturalness without substantially degrading target-attribute fidelity.
Evidence: Adding a 50k-step self-augmentation refinement stage raised UTMOS from 3.4899 to 3.6104 while EECS decreased only from 0.8793 to 0.8423 and SECS improved slightly; using the synthetic samples as literal training targets (rather than as additional style references) instead degraded naturalness, attributed to VC-introduced artifacts. (§3.1.3, Tables 5-6)
- complicates: Cross-speaker style transfer systems trained and evaluated within a single recording corpus do not necessarily preserve intelligibility when applied across corpora with different acoustic conditions, even when speaker identity is preserved.
Evidence: In a cross-corpus configuration (training on LJSpeech and VCTK speakers, evaluating against ESD-derived emotion centroids), WER rose to roughly 1.08-1.15 compared to about 0.23 in the matched-corpus setting, while SECS remained high (0.86-0.91), which the authors attribute to acoustic mismatch between the ESD emotional references and the target recording conditions. (§3.2, Table 7)
- refines: For emotion-embedding leakage suppression, the compatibility between an encoder’s architecture and the disentanglement mechanism matters more than heuristic input perturbations applied to that encoder.
Evidence: Adding 20-mel-bin filtering or formant-based timbre perturbation to the Reference Encoder, or substituting alternative emotion encoders (StyleSpeech, StyleTTS), monotonically reduced EECS (down to 0.48-0.59 in the most perturbed configurations) relative to the unperturbed Reference Encoder (EECS 0.8793), and no substitute encoder outperformed it. (§3.1.2, Table 3)
Limitations and Open Questions
Cross-corpus generalization is not yet reliable: word error rate rose above 1.0 in the cross-corpus evaluation (§3.2), an order of magnitude worse than the in-domain result, indicating the transcribed output diverges substantially from the intended text when recording conditions differ from training, even though speaker-similarity metrics stay high.
Beyond the cross-corpus intelligibility gap, the evaluation is limited to two held-out target speakers within a single corpus (ESD, English subset only) and a modest 30-participant listening test. The method does not yet support zero-shot style transfer to unseen speakers; the authors state this, along with cross-lingual scalability of MPCL and self-augmentation, as future work rather than a demonstrated capability. The paper also does not report model parameter counts or inference latency, and speaker/emotion conditioning at test time uses centroid prototypes from the training style space rather than arbitrary reference audio, so behavior with freely chosen reference clips is untested.
Wiki Connections
- Emotion Synthesis — proposes an explicit disentanglement and self-augmentation recipe specifically for transferring emotional expressivity to speakers with only neutral training data.
- Disentanglement — introduces a cosine-similarity-based adversarial loss applied directly between speaker and emotion embeddings (and again at the shared flow latent), validated with CKA/LK-CKA ablations against label-based GRL alternatives.
- Voice Conversion — exploits the invertibility of the model’s normalizing flow to perform voice conversion, which is then repurposed as a self-augmentation data source for TTS training.
- Subjective Evaluation — reports a 30-participant MOS listening test covering naturalness, emotion similarity, and speaker similarity across all compared systems.