arXiv · 2025 · Preprint
Rui Liu et al. (Inner Mongolia University) · → Paper · Demo: ✓ · Code: ✓
Simulates the director-actor rehearsal process in movie dubbing by retrieving emotionally similar reference footage and progressively fusing it into a graph-based speech generator, improving the emotional expressiveness of script-to-speech dubbing conditioned on silent video and a timbre prompt.
Problem
Automatic movie dubbing (also called Visual Voice Cloning, V2C) generates speech from a script that replicates a target speaker’s timbre from a brief timbre prompt while staying lip-synced to a silent video. Prior dubbing systems condition only on the target utterance’s own text, video, and timbre prompt, modeling emotion through direct cross-modal fusion of these three signals. The paper argues this reproduces a simplified dubbing workflow in which an actor performs without preparation, and that it overlooks a defining feature of real dubbing sessions: directors show actors reference footage beforehand so they can internalize emotional context before performing. Without this preparatory step, prior models exhibit limited emotional expressiveness, even when pronunciation quality and audio-visual synchronization are otherwise strong.
Method
Authentic-Dubber reframes dubbing as a three-stage, retrieval-augmented process that mirrors a director-actor rehearsal.
Multimodal Reference Footage Construction. From the training portion of the V2C-Animation dataset, the authors build a Multimodal Reference Footage Library (MRFL). For every clip, four modality-specific emotion extractors produce emotion vectors: a Scene Emotion Extractor and a Face Emotion Extractor that each first prompt VideoLLaMA2 (a video-understanding LLM) to produce an emotion caption, then pass that caption through a RoBERTa-based text emotion recognition model (RTER); a Text Emotion Extractor that combines an RTER-derived self-emotion vector from the script with a COMET-generated commonsense-reaction caption also passed through RTER; and an Audio Emotion Extractor that uses Emotion2Vec, a self-supervised speech emotion representation model, to embed the clip’s ground-truth audio directly.
Emotion-Similarity-based Retrieval-Augmentation (ESRG). At inference, the target utterance’s own scene, face, and text emotion vectors are computed and used as three separate queries against the MRFL via cosine similarity, since animated dubbing has limited speaker-specific reference material and the retrieval is deliberately speaker-agnostic. Each query retrieves its Top-K most similar indirect emotion items (scene, face, or text) together with the direct emotional audio matched to those items via index lookup.
Progressive Graph-based Speech Generation (PGSG). The retrieved emotional knowledge is fused through a construct-and-encode paradigm over three successive graphs, each encoded with a Graph Attention Encoder and each initialized from the previous stage’s output: a Basic Emotion Graph connecting the target utterance’s own scene, face, and text emotion vectors; an Indirect Emotion Extended Graph that adds the retrieved indirect (scene/face/text) emotion nodes, connected to same-modality basic emotion nodes; and a Direct Emotion Extended Graph that adds the retrieved direct emotional audio nodes. An Emotion Knowledge-based Speech Synthesizer then aggregates the three graphs’ learned representations with the output of a Cross-Modal Aligner (adopted from StyleDubber, which handles audio-visual synchronization from the script and video frames and injects the timbre prompt) via successive Conv1D and cross-attention layers, producing a representation that is decoded to a mel spectrogram and converted to waveform by a vocoder.
Key Results
On the V2C-Animation benchmark (the only public movie-dubbing dataset with emotion annotations), Authentic-Dubber achieves the best score among six compared systems on every emotion-related metric reported in Table 1: EMO-ACC 47.21% (vs. 45.73% for StyleDubber and 44.55% for Speaker2Dubber), MOS-DE 3.792 ± 0.055, and MOS-SE 3.889 ± 0.053 (both the highest of any system, though still below the ground-truth speech’s 4.416 and 4.497). Speaker similarity (SECS) is also highest at 84.4, close to but not exceeding StyleDubber’s 83.46. However, its WER (25.95) is worse than both StyleDubber (24.70) and Speaker2Dubber (18.27), the two strongest prior baselines on intelligibility, indicating the emotional-expressiveness gains come with a pronunciation-accuracy trade-off relative to the best prior system.
Ablations (Table 2) show each mechanism contributes: removing the LLM-derived scene and face captions in favor of conventional feature extractors (I3D, EmoFAN) drops EMO-ACC from 47.21 to 46.02; removing all three retrieval channels drops EMO-ACC to 45.23 (the largest single-component ablation drop); and removing the graph-based modeling, the progressive construct-and-encode structure, or the hierarchical aggregation each degrades EMO-ACC and MOS scores by smaller but consistent margins. A supplementary analysis (Fig. 4) shows EMO-ACC peaks at Top-K = 3 under speaker-agnostic retrieval and degrades with larger K, and cosine similarity outperforms dot product and Euclidean distance as the retrieval metric (Fig. 6).
Novelty Assessment
The contribution is architectural: the paper is the first, per its own claim, to explicitly model the director-actor rehearsal process in movie dubbing as a retrieval-augmented generation problem, with a purpose-built progressive graph fusion mechanism (Basic → Indirect → Direct Emotion Extended Graphs) rather than the single-stage cross-modal fusion used by prior dubbing models. The individual components are largely established techniques recombined for this task: retrieval-augmented generation, graph attention encoders, and LLM-based captioning are all prior art, and the Cross-Modal Aligner is adopted unmodified from StyleDubber. The novelty lies in the specific staged construct-and-encode design and the emotion-similarity-based, speaker-agnostic retrieval strategy tailored to animated dubbing’s limited per-speaker data, rather than in any single new primitive. The gains over the strongest prior baselines are real but modest on the primary emotion metric (EMO-ACC +1.48 over StyleDubber) and come with a WER regression relative to Speaker2Dubber, so the overall improvement is not uniformly positive across all evaluated axes.
Field Significance
moderate — this paper introduces a genuinely new mechanism, retrieval-augmented director-actor interaction simulation via progressive graph fusion, for the narrow but real sub-problem of emotionally expressive movie dubbing, validated with both objective and subjective evaluation and a thorough ablation study on the only public benchmark for the task. Its scope is limited to a single dataset (V2C-Animation), a single language, and animated dubbing specifically, and the improvement is not clean across all metrics (intelligibility regresses relative to the strongest prior baseline), so it reads as a solid incremental advance within the dubbing sub-field rather than a broader shift in speech synthesis methodology.
Claims
- supports: Retrieving multimodal emotional context from a reference library and fusing it into the generation process can improve emotional expressiveness in speech synthesis beyond what direct cross-modal conditioning on the target utterance alone achieves.
Evidence: Removing all three retrieval channels (scene, face, text) from the full model drops EMO-ACC from 47.21% to 45.23% and MOS-SE from 3.889 to 3.527, the largest single-component degradation observed in the ablation study, while the full retrieval-augmented model outperforms five prior dubbing baselines on every emotion-related metric. (§Ablation Studies, Table 1, Table 2)
- supports: Deriving emotion captions with a video-understanding LLM before encoding them into an emotion vector can capture richer emotional signal than encoding low-level visual features with conventional emotion recognition models.
Evidence: Replacing the LLM-generated scene and face emotion captions with embeddings from the conventional feature extractors I3D and EmoFAN reduces EMO-ACC from 47.21% to 46.02% and both MOS-DE and MOS-SE, with the largest drop occurring when both captions are replaced simultaneously. (§Ablation Studies, Table 2)
- complicates: Retrieval-augmented conditioning for expressive speech generation is sensitive to retrieval scale and the choice of similarity metric, and excessive retrieval can degrade rather than improve output quality.
Evidence: Under speaker-agnostic retrieval, EMO-ACC peaks at Top-K = 3 and declines as K increases further; across similarity functions, cosine similarity consistently yields the best performance while dot product shows larger fluctuations and Euclidean distance a lower performance ceiling. (§Speaker-Agnostic vs. Speaker-Specific Retrieval, §Analysis of Similarity Metrics, Figures 4 and 6)
- complicates: Improving emotional expressiveness in conditioned speech generation does not automatically preserve intelligibility relative to the strongest prior non-emotion-focused system.
Evidence: Authentic-Dubber achieves the best EMO-ACC (47.21%) and MOS scores among all compared systems, but its WER (25.95) is higher than both StyleDubber (24.70) and Speaker2Dubber (18.27), the two strongest baselines on pronunciation accuracy. (§Results and Discussion, Table 1)
Limitations and Open Questions
Evaluation is restricted to a single benchmark, V2C-Animation, described by the authors as the only public movie-dubbing dataset with emotion annotations; all reported gains are relative to baselines re-run on this one animated-movie corpus, so generalization to live-action dubbing or other languages is untested.
The retrieval strategy is explicitly speaker-agnostic, motivated by the limited speaker-specific reference footage available for virtual animated characters; the paper does not establish whether the same design would hold for live-action dubbing, where speaker-specific footage is more available and speaker-specific retrieval might instead be preferred. The subjective evaluation uses only 20 raters scoring 12 generated samples, a small panel by TTS evaluation norms. The emotional-expressiveness gains come with a WER regression relative to the strongest prior intelligibility baseline (Speaker2Dubber), a trade-off the paper reports but does not attempt to close. The authors themselves note that future work should extend the framework to controllable timbre, speaking rate, and prosody, which the current model does not explicitly control.
Wiki Connections
- Emotion Synthesis — proposes a retrieval-augmented, progressive graph-based mechanism specifically to improve the emotional expressiveness of generated dubbing speech, evaluated with both an emotion-accuracy classifier and emotion-focused MOS tests.
- Zero-Shot TTS — clones a target character’s timbre from a brief timbre prompt via a cross-modal aligner adapted from StyleDubber, following the standard prompt-based voice cloning pattern used in zero-shot TTS.
- Self-Supervised Speech — uses Emotion2Vec, a self-supervised speech emotion representation model, as the Audio Emotion Extractor that embeds direct emotional audio for the retrieval library and the fused graph.
- Subjective Evaluation — validates emotional expressiveness with a 20-rater Mean Opinion Score test (MOS-DE, MOS-SE) alongside objective EMO-ACC, WER, SECS, and MCD-DTW-SL metrics.
- FastSpeech 2 — included as one of five prior dubbing/TTS baselines compared against on the V2C-Animation benchmark, where it scores lowest among all baselines on EMO-ACC and MOS-DE/MOS-SE.
- Emotion2Vec — adopted directly and without modification as the Audio Emotion Extractor that produces direct emotional audio embeddings for the Multimodal Reference Footage Library.