arXiv · 2025 · Preprint
Joonyong Park et al. (The University of Tokyo; Preferred Networks Inc.) · → Paper · Demo: ✓ · Code: ?
EmoSSLSphere combines spherical Arousal-Valence-Dominance (AVD) emotion vectors with HuBERT-derived k-means-discretized prosody tokens inside a FastSpeech2-style decoder, enabling fine-grained multilingual emotional TTS with improved intelligibility, acoustic quality, and naturalness over purely label-based or spherical-vector-only baselines.
Problem
Existing multilingual emotional TTS systems struggle to simultaneously maintain speaker identity, produce consistent emotional expression across languages, and model fine-grained prosodic nuances. Prior work either relies on coarse explicit emotion labels (which lose continuous expressiveness) or adopts spherical AVD representations (EmoSphere-TTS) without language-independent prosodic grounding, leading to unnatural intonation, speaker drift, and inconsistent emotional delivery when moving across languages. Self-supervised learning (SSL) approaches exist for zero-shot multilingual synthesis but offer limited emotional control and consistency.
Method
EmoSSLSphere is a four-module framework built on top of the NATSpeech/FastSpeech2 architecture and extending EmoSphere-TTS:
Emotional Style and Intensity Encoder. A pre-trained encoder extracts AVD vectors from reference Mel-spectrograms and converts them from Cartesian (x, y, z) to spherical (r, θ, φ) coordinates to enable smooth interpolation across emotional states. One-hot Emotion ID embeddings (categorical: Angry, Sad, Happy, Surprised, etc.) are concatenated with the spherical AVD vector, providing both continuous nuance and discrete stylistic priors.
SSL-based Feature Encoder. HuBERT-base (pre-trained on LibriSpeech) is used to extract hidden states from the 9th transformer layer, chosen as a middle-ground between fine phonetic detail and high-level semantics. These continuous features are vector-quantized via k-means clustering (K=200 codebook, trained separately per language on the English ESD and Japanese JVNV datasets) to produce discrete prosody tokens capturing rhythm, pitch contour, and intensity in a speaker-agnostic fashion.
Text Encoder. A parallel two-component design: (1) a phoneme encoder using Montreal Forced Aligner for English and pyopenjtalk/ESPNet for Japanese produces phoneme sequences passed directly to the decoder; (2) a frozen DeBERTaV3 semantic encoder (language-adapted tokenizer only) produces multilingual semantic embeddings that condition the emotional and prosodic modules via cross-attention but are not directly decoded.
Mel-spectrogram Decoder. Based on FastSpeech2. Training is two-stage: individual encoders are pre-trained and then frozen while the decoder is fine-tuned. The four embeddings (phoneme, spherical emotion+ID, discrete SSL tokens, speaker embedding) are projected to a shared latent space, concatenated, linearly projected to decoder input dimension, and decoded to a Mel-spectrogram.
During inference, a reference waveform provides emotional and prosodic cues; a fixed speaker embedding maintains speaker identity independently of the reference. The reference need not match the target text content. Separate encoder instances are instantiated per language to avoid cross-lingual interference. Model size is not reported.
Key Results
Results are drawn from Tables 1–3 in the paper, evaluated on 80 English utterances (ESD) and 60 Japanese utterances (JVNV) from single female speakers in each language. Baselines are NATSpeech with Emotion Labels and EmoSphere-TTS.
Naturalness (nMOS, Table 2): EmoSSLSphere scores 4.13 ± 0.08 (English) and 3.94 ± 0.15 (Japanese), compared to EmoSphere-TTS at 4.05 ± 0.12 (English) and 3.63 ± 0.16 (Japanese). Ground truth is 4.30 ± 0.09 (English) and 4.40 ± 0.12 (Japanese).
Intelligibility (Table 1): English WER drops from 20.96% (EmoSphere-TTS) to 19.58%; Japanese CER drops from 19.26% to 18.33%. SpeechBLEU improves from 0.209 to 0.225 (English) and 0.188 to 0.218 (Japanese).
Acoustic quality (Table 2): MCD improves from 7.979 to 7.282 (English) and 9.131 to 8.719 (Japanese). LogF0RMSE improves from 0.359 to 0.341 (English) and 0.428 to 0.409 (Japanese). UTMOSv2 improves from 2.403 to 2.427 (English) and 1.875 to 1.912 (Japanese).
Emotional expressiveness (Table 3): Average AVD RMSE falls from 0.0798 to 0.0773 (English) and 0.0867 to 0.0783 (Japanese).
Ablation studies confirm each component contributes: removing reference waveform conditioning, the semantic text encoder, or k-means discretization all degrade performance. Speaker-mismatched reference audio has minimal adverse impact, suggesting robust speaker-independent prosody modeling.
Comparisons are not entirely fair at scale — evaluations use small, single-speaker datasets (80 English / 60 Japanese samples) and limited listener panels (9 English, 21 Japanese native speakers), limiting generalizability claims.
Novelty Assessment
The combination of spherical AVD emotion vectors with SSL-derived discrete prosody tokens is genuinely novel at the integration level. Each component individually builds on established prior work (EmoSphere-TTS for spherical representation, HuBERT k-means discretization from the discrete speech token literature, FastSpeech2 for the backbone), but their fusion with a dual-component text encoder and per-language encoder instantiation is a new configuration. The contribution is primarily architectural/integration — there is no new training paradigm, no new codec, and no large-scale data contribution. The gains over EmoSphere-TTS are real but modest on the reported metrics. The paper is accepted as a workshop paper (SSW 13), which fits the scope of the contribution.
Field Significance
Moderate — EmoSSLSphere demonstrates that integrating SSL-derived discrete prosody tokens with continuous spherical emotion representations improves emotional expressiveness and intelligibility in multilingual TTS, providing incremental evidence for the utility of combining SSL features with explicit affective representations. The evaluation scope is narrow (two languages, single-speaker corpora, small listener panels), so the result is confirmatory rather than definitive, but the fusion approach can serve as a reusable recipe for future multilingual emotional TTS work.
Claims
- Discretizing self-supervised speech features via k-means produces more stable prosodic control signals than retaining continuous SSL representations in encoder-decoder TTS. (§5.6, Table 1)
- Combining continuous spherical emotion vectors with discrete SSL prosody tokens improves both naturalness and intelligibility compared to spherical emotion vectors alone. (§5.1, §5.3, Tables 1–2)
- Speaker-independent prosody conditioning via SSL tokens can generalize across reference speakers with minimal degradation, enabling robust cross-speaker emotion transfer. (§5.6, Table 1)
- Semantic text encoders contribute to emotional and prosodic consistency in multilingual TTS even when they do not serve as direct decoder inputs. (§5.6, Table 1)
Limitations and Open Questions
- Evaluations are limited to single-speaker datasets in two languages, making it unclear whether EmoSSLSphere generalises to multi-speaker, low-resource, or unseen-language scenarios.
- Subjective listener panels are small, and emotional authenticity is assessed primarily via AVD RMSE as a proxy rather than direct perceptual emotion ratings.
- Speaker similarity (SPK-SIM) is not evaluated, making it hard to quantify speaker fidelity claims.
- Cross-speaker emotion transfer is described but not formally evaluated; inference always uses same-speaker reference audio.
- Separate per-language encoder instances do not scale to many-language settings without significant parameter overhead.
- Integration with semi-supervised training (EmoSphere++) and extension to zero-shot speaker scenarios are listed as future work.
Wiki Connections
This paper advances emotion-synthesis by extending the spherical AVD representation from EmoSphere-TTS with SSL-based discrete prosody tokens, improving emotional controllability and prosodic consistency. The use of HuBERT features discretized via k-means is a direct application of self-supervised-speech representations for prosodic grounding. The FastSpeech2 backbone and dual phoneme/semantic text encoding are characteristic of transformer-enc-dec-tts. The language-specific SSL token codebooks and multilingual DeBERTaV3 conditioning address multilingual-tts challenges. The spherical AVD and discrete token fusion contributes to prosody-control through explicit mechanisms for independent prosody and emotional style control.