ASRU · 2025 · Workshop
Yoon et al. (POSTECH) · → Paper · Demo: ✓ · Code: ?
Maestro-EVC introduces a three-way reference-guided emotional voice conversion framework that independently controls content, speaker identity, and emotional style, with novel modules for temporal emotion alignment and explicit prosody transfer under prosody-mismatched conditions.
Problem
Most emotional voice conversion systems couple speaker identity and emotional style within a single latent space, making independent control difficult. Reconstruction-based frameworks in particular constrain the content and emotion reference to share the same prosody during training, which breaks down at inference when the references come from different speakers and utterances. Prior work on prosody modeling either predicted prosodic patterns implicitly from latent representations or handled only pitch without energy, leaving temporal dynamics of emotional expression poorly captured in cross-speaker scenarios.
Method
Maestro-EVC uses three separate reference utterances at inference — one for linguistic content, one for speaker identity, and one for the target emotion — each encoded by a dedicated module. Content is extracted using a pre-trained HuBERT base model (trained on LibriSpeech 960h) discretized through K-means clustering into a 500-token vocabulary with 256-dimensional embeddings, yielding a content representation that is largely phonetic.
The temporal content-aware emotion modeling (TCEM) module applies a cross-attention mechanism in which the content representation queries the frame-level emotion embeddings produced by a pre-trained speech emotion diarization model. This generates temporally aligned, content-aware emotion embeddings. A gradient reversal layer (GRL) with a content classifier then suppresses residual phonetic information leaking into the emotion representation, enforcing disentanglement.
Explicit emotional prosody transfer (EEPT) conditions F0 and energy prediction directly on smoothed prosodic features extracted from the emotion reference using a Savitzky-Golay filter. To bridge the training-inference gap introduced by prosody mismatch, a prosody augmentation strategy randomly applies either temporal shifting or piecewise time warping to the smoothed features during training, teaching the FE predictor to generalize across misaligned prosody sequences. Duration is predicted jointly from the unique content unit sequence, smoothed reference durations, and the disentangled emotion embedding.
The emotion-invariant speaker encoder (EISE) fine-tunes additional layers on top of a frozen pre-trained ECAPA-TDNN using two adversarial objectives: a GRL suppressing emotional classification from speaker embeddings, and a triplet loss enforcing cross-emotion speaker consistency with margin 0.3. The vocoder is HiFi-GAN. Total training uses adversarial, feature matching, reconstruction, prosody, speaker, and content-disentanglement losses.
Key Results
On the seen ESD test set (700 test triples), Maestro-EVC outperforms both StyleVC and ZEST on all six objective metrics. WER improves from 17.18% (ZEST) to 11.78%, and CER from 9.85% to 6.54%. Emotion embedding cosine similarity (EECS) rises from 0.779 to 0.819, and F0 Pearson correlation from 0.432 to 0.551. In subjective MOS tests with 25 participants, Maestro-EVC scores 4.06 naturalness, 4.11 emotion similarity, 4.02 speaker similarity, and 4.15 prosody similarity — the largest margin is in prosody similarity, where ZEST scores 2.86.
Zero-shot generalization to unseen speakers (18 VCTK speakers) and unseen emotion categories (fear/disgust from CREMA-D, frustration/excitement from IEMOCAP) consistently shows improvements over baselines on all reported metrics. On unseen speakers, CER drops from 9.69% (ZEST) to 6.10% and F0-PCC rises from 0.486 to 0.577.
Ablation results confirm that each proposed component contributes: removing content GRL causes the largest degradation (WER rises to 20.98%), removing temporal emotion representation reduces SCA by over 16 percentage points, and removing prosody augmentation raises WER from 11.78% to 17.56%.
Novelty Assessment
The three distinct reference inputs with dedicated encoders is an established design pattern in controllable voice conversion, and the use of HuBERT discrete units for content and ECAPA-TDNN for speaker is standard practice. The genuine contributions are narrower: (1) using a frame-level speech emotion diarization model rather than utterance-level embeddings for temporal emotion alignment, (2) the cross-attention content-query scheme with GRL for disentangling residual phonetics from emotion, and (3) the prosody augmentation strategy (shift + piecewise time warp) as a training-time data augmentation for prosody mismatch robustness. Each component is independently validated in ablation. The combination of energy alongside F0 in explicit prosody transfer is noted as a differentiation from prior EVC work, which focused on F0 only.
Field Significance
Moderate — Maestro-EVC advances the controllability and temporal expressiveness of reference-guided EVC by pairing frame-level emotion representations with a practical training augmentation for prosody mismatch. The result is meaningful improvement on both intelligibility and prosody transfer metrics over published baselines on ESD, though the small dataset (350 utterances, 10 speakers, 5 emotions) and proprietary-by-construction test conditions limit how broadly the gains generalise. The approach is a useful template for explicit prosody conditioning in emotion conversion tasks.
Claims
- Frame-level emotion representations improve speaker emotion classification accuracy and prosody transfer fidelity over utterance-level representations in voice conversion. (§IV.C, Table I)
- Adversarial disentanglement via gradient reversal layers reduces phonetic leakage into emotion embeddings, improving intelligibility under cross-linguistic-content conversion. (§IV.C, Table I)
- Explicit conditioning on extracted prosodic features (F0 and energy) from an emotion reference transfers temporal dynamics more faithfully than implicit prediction from latent codes. (§IV.A, Table I)
- Training-time prosody augmentation through temporal shifting and warping improves robustness of prosody transfer under mismatched reference conditions without sacrificing naturalness. (§IV.C, Table I)
Limitations and Open Questions
Warning
Training and primary evaluation use only the English ESD corpus — 350 parallel utterances across 10 speakers and 5 emotion categories. This is a narrow domain; generalisation to spontaneous, noisy, or multilingual emotional speech is entirely untested.
The small, parallel ESD corpus makes it difficult to assess whether the disentanglement holds under more naturalistic or non-parallel conditions. The ablation study evaluates the seen scenario only; it is not clear whether the ablated variants degrade similarly on unseen speakers and emotions. Speaker classification accuracy (SCA) is reported as a zero-shot metric for the seen-speaker scenario but becomes undefined for unseen speakers, so that dimension of the zero-shot evaluation lacks a corresponding metric. Model size and inference speed are not reported, which matters for the real-time dubbing applications the paper motivates.
Wiki Connections
This paper contributes to voice-conversion, emotion-synthesis, prosody-control, and disentanglement. It builds on self-supervised-speech representations (HuBERT discrete units for content, ECAPA-TDNN for speaker) and evaluates using both objective and subjective criteria covered by evaluation-metrics and subjective-evaluation.