arXiv · 2026 · Preprint
Suvendu Sekhar Mohanty · → Paper · Demo: ? · Code: ?
Reformulates emotion-conditioned FastSpeech2 training around a structural causal model, adding two counterfactual loss terms that push emotion’s effect on speech to route entirely through explicit prosody features (duration, pitch, energy).
Problem
Non-autoregressive TTS models such as FastSpeech2 handle the one-to-many text-to-speech mapping problem with explicit duration, pitch, and energy predictors, but the base architecture has no emotion awareness: expressiveness only emerges implicitly. Prior emotion-conditioned extensions typically inject an emotion embedding or reference-derived style code (e.g. Global Style Tokens) into the network with no constraint on how that signal is allowed to affect the output. Nothing in a standard label-conditioned model guarantees that emotion is expressed only through prosodic changes, so the added conditioning can bleed into speaker timbre or linguistic content, degrading intelligibility or speaker consistency when transferring an emotion across voices. The paper targets this entanglement: it argues that a principled way to keep emotion effects confined to prosody requires a causal, not just correlational, training constraint.
Method
The paper introduces Causal Prosody Mediation (CPM), which pairs an emotion-augmented FastSpeech2 backbone with two counterfactual training losses derived from a structural causal model (SCM) of the TTS generation process.
The SCM defines five variables: text content X, emotion E, speaker identity S, prosody features M (duration, pitch, energy), and synthesized speech Y. It posits that emotion should influence speech only indirectly, through the path E → M → Y, with the direct path E → Y treated as an unwanted shortcut to be eliminated (formally, the target property is the conditional independence Y ⊥ E | X, M, S).
Architecturally, the backbone extends standard FastSpeech2 (4 FFT blocks each for encoder and decoder, hidden dimension 256, 2 attention heads, 1024 FFN inner dimension) with learned speaker (128-dim) and emotion (64-dim) embeddings added at the phoneme encoder. The duration, pitch, and energy predictors are each conditioned on the emotion embedding, while the decoder receives only the speaker embedding, not emotion directly, to keep the direct emotion-to-decoder pathway minimal by construction. A pretrained HiFi-GAN vocoder (fine-tuned on the target data) converts predicted mel-spectrograms to waveform; all training operates on mel-spectrogram prediction.
Two loss terms operationalize the causal constraint. The Indirect Path Constraint (IPC) runs a counterfactual decoder pass with the emotion swapped to a randomly sampled alternative E’ while holding the encoder’s expanded hidden states and prosody fixed, then penalizes any difference from the original output with an L1 term — pressuring the decoder to ignore any residual direct emotion signal. The Counterfactual Prosody Constraint (CPC) runs a full counterfactual generation with the swapped emotion E’ (re-encoding, re-predicting prosody, and re-decoding), and combines a content-consistency term (measured as 1 − WER between the original and counterfactual transcripts) with an auxiliary emotion-classifier loss that rewards the counterfactual output for being recognizable as E’. The combined objective sums the base FastSpeech2 losses (mel reconstruction plus duration/pitch/energy prediction) with weighted IPC and CPC terms (β_IPC = 1.0, β_CPC = 0.5 during fine-tuning). Training proceeds in two stages: 200k steps of neutral-only pretraining on the combined LibriTTS + VCTK corpora (all samples labeled neutral), followed by 50k steps of fine-tuning on EmoV-DB with emotion labels and the causal losses active. The counterfactual passes roughly double per-sample compute and increase total training time by about 1.5x relative to baseline FastSpeech2.
Key Results
On held-out test utterances synthesized in five emotions across two EmoV-DB voices, CPM reports the lowest WER among compared systems (3.1% vs. 3.5% for vanilla FastSpeech2 and 4.0% for FastSpeech2 + emotion embedding without causal losses), speaker-similarity cosine of 0.88 (comparable to the 0.87 of the naive emotion baseline and well above 0.79 for a post-hoc activation-editing baseline, CAE), a content-consistency score above 0.95, and 94% emotion-classification accuracy versus 80% for the naive emotion baseline and 88% for CAE (Table 1). Subjectively, CPM achieves MOS 4.45 ± 0.05 versus 4.21 ± 0.06 (naive emotion baseline), 4.10 ± 0.07 (vanilla FastSpeech2), and 4.00 ± 0.06 (CAE), and a DMOS emotion-similarity rating of 4.3 versus 3.8 and 4.0 for the two baselines. An ABX listening test found listeners preferred CPM’s speaker-identity match over CAE’s in 85% of trials. Ablations (Table 2) isolate each loss’s contribution: removing IPC drops emotion-classification accuracy to about 88% and slightly increases MOS variance from timbre inconsistency, while removing CPC leaves content well preserved but drops emotion accuracy to about 75% and produces perceptibly under-expressed, more monotonic emotional renderings.
Novelty Assessment
The novelty here is in the training recipe and the causal framing, not the backbone: the underlying model is standard FastSpeech2 with emotion and speaker embeddings, an architecture already used by several prior emotion-conditioned TTS systems. What is new is casting emotion-to-prosody-to-speech as a mediation problem in a structural causal model and deriving two loss terms (IPC, CPC) directly from that formalization, evaluated against a naive emotion-embedding baseline and a post-hoc counterfactual-editing baseline (CAE) on the same architecture and data. The comparison to CAE is a useful contribution in itself, since it isolates the effect of baking the causal constraint into training versus applying it post hoc at inference. The paper’s own honest limitation, that its prosody mediator (duration, pitch, energy) cannot represent voice-quality cues like breathiness or creakiness, is a fair caveat on how far the disentanglement claim can be pushed.
Field Significance
moderate — the paper offers a concrete, ablated demonstration that counterfactual training losses derived from an explicit causal model can reduce emotion-content-speaker entanglement in a non-autoregressive TTS backbone, compared to both a naive conditioning baseline and a post-hoc editing baseline on the same architecture. The evaluation is confined to a single backbone (FastSpeech2), a single small emotional corpus (EmoV-DB, four actors, five emotion categories), and two evaluation voices, which limits how far the specific numbers generalize, but the causal-mediation framing and the IPC/CPC loss design are a genuinely new mechanism for this problem, applicable in principle beyond this one architecture.
Claims
- supports: Explicit counterfactual training constraints that penalize a direct conditioning-to-output pathway can reduce cross-attribute leakage (e.g. emotion bleeding into speaker timbre or content) more effectively than conditioning on the attribute embedding alone.
Evidence: Removing the Indirect Path Constraint (IPC) loss while keeping the emotion-augmented backbone drops emotion-classification accuracy from 94% to about 88% and slightly lowers content-consistency, indicating the suppressed direct pathway reappears once the counterfactual constraint is removed. (§7.3, Table 2)
- complicates: A prosody mediator limited to duration, pitch, and energy cannot represent all emotion-relevant acoustic variation, so causal or disentanglement objectives built solely on these features leave some emotional cues unmodeled.
Evidence: The “disgusted” emotion category is repeatedly confused with anger by both the model and an external emotion classifier, which the authors attribute to voice-quality cues (e.g. creaky voice, spectral tilt) that are not captured by duration, pitch, or energy alone. (§7.5 Error Analysis; §8 Discussion and Limitations)
- supports: Training-time mechanisms that structurally constrain how a style or attribute signal reaches the output can preserve target-speaker identity better than post-hoc editing of a pretrained model’s internal activations.
Evidence: A post-hoc counterfactual-activation-editing baseline (CAE) applied to a pretrained emotion-conditioned FastSpeech2 model shows lower speaker-similarity cosine (0.79) than the causal-mediation model (0.88), and listeners preferred the causal model’s speaker match over CAE’s in 85% of ABX trials. (§7.1, Table 1)
- complicates: Enforcing counterfactual consistency constraints during training carries a nontrivial computational overhead relative to standard conditioned training.
Evidence: The extra IPC and CPC forward passes roughly double per-sample computation, and total training time increases by about 1.5x compared to baseline FastSpeech2 training. (§4.4 Combined Training Objective; §8 Discussion and Limitations)
Limitations and Open Questions
The central disentanglement claim rests on the assumption that all emotion-relevant acoustic effects are captured by duration, pitch, and energy alone. The authors acknowledge this is not fully true (voice quality and spectral-tilt cues are excluded by construction), so the causal constraint may suppress emotional information the mediator cannot represent rather than route it correctly.
The evaluation is limited to a single backbone architecture, English speech, and a small emotional corpus (EmoV-DB: five emotions, four actors, about eight hours), with subjective ratings from 20 listeners and only two synthesis voices tested for cross-speaker emotion transfer. Emotion labels are coarse categorical classes rather than continuous arousal/valence descriptors, which the authors note limits the granularity of controllable expression. The counterfactual training procedure adds roughly 1.5x training time and doubles per-sample compute, a cost the paper does not evaluate at larger model or data scale. No code or demo availability is stated in the paper.
Wiki Connections
- Emotion Synthesis — introduces an emotion-augmented FastSpeech2 backbone with counterfactual losses purpose-built to control how emotion is rendered in prosody.
- Prosody Control — the Indirect Path Constraint and Counterfactual Prosody Constraint give an explicit training mechanism for routing an attribute’s effect through duration, pitch, and energy specifically, rather than leaving prosody control implicit.
- Disentanglement — derives its loss terms directly from a structural causal model and validates the resulting separation of emotion, content, and speaker identity with ablations that remove each loss in turn.
- Transformer Encoder-Decoder TTS — builds on the FastSpeech2 feed-forward-transformer encoder-decoder backbone, adding emotion and speaker conditioning without changing the base architecture.
- Subjective Evaluation — reports MOS and DMOS listening tests with human raters (n=20) alongside ABX preference tests comparing speaker-identity retention across systems.