arXiv · 2025 · Preprint
Fengping Tian et al. (Alibaba International Digital Commerce) · → Paper · Demo: ? · Code: ✓
Marco-Voice combines voice cloning and emotion-controllable TTS in a single CosyVoice1-based framework via rotational emotion embeddings, cross-orthogonal speaker-emotion disentanglement, and in-batch contrastive learning, achieving substantially higher speaker similarity (0.8275) and emotional expressiveness than CosyVoice2 baselines.
Problem
Existing TTS systems conflate speaker identity and emotional expression, making it difficult to independently control the two. Most emotion-conditioned TTS systems treat emotions as discrete categorical labels, failing to capture the continuous and multidimensional nature of human affect. When voice cloning is combined with emotion control, systems tend to degrade speaker similarity or emotional naturalness. There is also a scarcity of high-quality multilingual emotional speech data, particularly for Mandarin.
Method
Marco-Voice extends CosyVoice1 with four key modifications to its LM + flow-matching architecture:
Rotational Emotion Embedding (REE). For a target emotion, the system encodes paired samples of emotional and neutral speech from the same speaker using a pre-trained emotion encoder. The difference vector (emotional minus neutral, normalized) is aggregated over N=10 pairs to produce a speaker-independent emotion direction vector e. This difference-in-means approach extracts emotion content while removing speaker-specific variation. The resulting embedding is injected as a conditioning signal into both the language model and the flow-matching module.
Cross-orthogonal constraint for speaker-emotion disentanglement. Speaker embeddings s and emotion embeddings e are forced to be perpendicular via an orthogonality loss on their cross-dot-product matrix. This ensures that manipulating one conditioning axis does not bleed into the other, enabling independent voice identity and emotional style control.
In-batch contrastive learning. For each minibatch, the system penalizes high cosine similarity between distinct emotion embeddings (across different emotional categories). This pushes emotion representations apart in the embedding space, improving the separability of emotion categories.
Cross-attention emotion-text integration. In the flow-matching module, the emotion embedding serves as the query attending over the language model’s output token sequence as keys/values. This allows the emotion embedding to dynamically modulate linguistic content throughout acoustic generation, producing emotionally coherent outputs.
Training uses a combined loss: main TTS loss + weighted cross-orthogonality regularization (λ_orth=0.1) + contrastive penalty (λ_contrast=0.5). Training was conducted on 8 A100 GPUs on ESD (29h, 5 emotions, 20 speakers) and the authors’ proprietary CSEMOTIONS dataset (10h, 7 emotions, 10 professional Mandarin speakers).
Ablation versions. The paper compares v1 (REE conditioning only) through v4 (all components: REE + cross-orthogonal + contrastive + cross-attention). v4 is the full system.
Key Results
Human evaluation results (5-point Likert scale) show Marco-Voice v4 substantially outperforms CosyVoice1 and CosyVoice2 across all dimensions. Voice cloning: Marco-Voice achieves speaker similarity 0.8275 vs CosyVoice2 0.605 and CosyVoice1 0.700. Overall satisfaction: 4.430 vs CosyVoice2 3.330. Naturalness: 4.205 vs CosyVoice2 3.150. Emotional expression: 4.225 vs CosyVoice2 3.240.
A/B preference tests show Marco-Voice is preferred over CosyVoice1 in 60% of comparisons and over CosyVoice2 in 65%.
Objective evaluation on LibriTTS (Table 4): Marco-Voice v4 achieves WER 11.4 (lowest among Marco-Voice variants, comparable to CosyVoice1 at 12.1), SS-SpeechBrain 63.2, SS-ERes2Net 74.3, DNSMOS 3.86. On AISHELL-3 (Table 5): WER 17.6 for v4 (higher than CosyVoice1 at 3.0 due to emotional prompt complexity and vocalized fillers that are not in the reference text).
Emotion recognition accuracy analysis shows Marco-Voice v4 achieves 0.77–0.78 overall accuracy across English and Chinese, with strong cross-lingual generalization for neutral and angry categories.
Novelty Assessment
The core novelty is the combination of rotational emotion embeddings (using difference-in-means from EmoKnob) with a cross-orthogonal constraint and in-batch contrastive learning in a unified voice cloning + emotion TTS system. Each component individually has precedent, but their integration within a CosyVoice1 framework to enable simultaneous zero-shot speaker cloning and fine-grained emotion control is new. The CSEMOTIONS dataset (7 emotions, 10 Mandarin professional speakers) is also a new resource. The cross-attention mechanism for emotion-text integration in the flow-matching stage is a targeted architectural improvement over simple additive conditioning. The contribution is primarily engineering and recipe rather than fundamentally new methodology.
Field Significance
Moderate — Marco-Voice demonstrates that speaker identity and emotional expression can be disentangled effectively within a unified flow-matching framework, using an explicit orthogonality constraint rather than learned separation alone. The cross-orthogonal training objective and CSEMOTIONS dataset both provide reusable resources for expressive multilingual TTS research. The system’s gains over CosyVoice baselines are substantial on subjective metrics, though the evaluation scope is limited to 7 emotion categories and proprietary data.
Claims
- Difference-in-means emotion embeddings, computed from paired emotional and neutral speech, enable speaker-independent emotion representations that generalise across speakers. (§2.2)
- Enforcing orthogonality between speaker and emotion embedding spaces via a cross-dot-product loss enables independent control of voice identity and emotional style without degrading either. (§2.3, Table 1)
- Unified joint modeling of voice cloning and emotion control produces higher speaker similarity and emotional naturalness than systems that address the two capabilities separately. (§4.1, §4.2, Tables 1–2)
- Expressive emotional TTS systems that include naturalistic vocalized fillers exhibit higher word error rates on clean-transcript benchmarks, even when perceived speech quality is high. (§4.4, Table 4–5)
- Gender imbalance in emotional speech training data produces systematic differences in emotion recognition accuracy, with male speakers underperforming female speakers across most emotion categories. (§4.4, Figure 6)
Limitations and Open Questions
- Requires paired emotional/neutral speech data from the same speaker for emotion embedding extraction, which limits scalability and language coverage.
- Higher WER on Mandarin AISHELL-3 relative to the CosyVoice1 baseline: the system inserts vocalized fillers (ah, um) that are natural in expressive speech but inflate WER against clean transcripts.
- Model size not reported; training efficiency uncharacterized.
- Evaluation is limited to 7 emotions — more naturalistic, multi-emotion or low-intensity affects are not tested.
- No streaming or latency evaluation.
Wiki Connections
Marco-Voice builds on CosyVoice and extends the same pipeline toward emotion control, comparing against CosyVoice 2. The difference-in-means emotion embedding technique relates to the emotion-synthesis and disentanglement concepts. The flow-matching backbone connects to flow-matching. Speaker similarity improvements situate the work within zero-shot-tts and voice-conversion literature. The cross-orthogonal constraint is a new disentanglement technique not previously documented in the disentanglement concept page.