arXiv · 2025 · Preprint
Cho et al. (KAIST) · → Paper · Demo: ? · Code: ✓
MAVFlow introduces a conditional flow matching renderer with dual audio-visual guidance (x-vector speaker embeddings and facial emotion embeddings) to preserve paralinguistic speaker characteristics in zero-shot audio-visual-to-audio-visual (AV2AV) multilingual translation across five languages.
Problem
Audio-visual-to-audio-visual (AV2AV) translation systems that bypass intermediate text representations (textless, direct systems) have made progress on linguistic accuracy but fail to preserve speaker-specific paralinguistics such as voice identity, timbre, and emotional expression during cross-lingual translation. Prior direct systems such as AV2AV rely on a simple speaker d-vector injected at the vocoder level, which provides only weak speaker conditioning and ignores the complementary information available in the speaker’s facial expressions. The result is synthesized speech that lacks the speaker consistency required for natural dubbing or video conferencing across languages.
Method
MAVFlow is a four-stage pipeline built on top of an existing audio-visual speech unit translation framework. The unit extractor (m-AVHuBERT, pretrained on 7,000 hours of multilingual audio-visual data via self-supervised masked cluster prediction) and the unit-to-unit translation module are borrowed unchanged from prior work; MAVFlow’s contribution is the renderer that converts translated discrete units into high-quality mel-spectrograms.
A Duration Length Regulator first expands the deduplicated output units and interpolates the result to match the duration of the original source audio, an important constraint for real-world dubbing where video length must remain fixed before and after translation.
The core component is an OT-CFM (Optimal Transport Conditional Flow Matching) Transformer decoder that generates mel-spectrograms from the discrete units under dual multimodal guidance. Speaker voice embeddings are x-vectors extracted from audio: during training, multiple utterances from the same speaker are averaged into a global speaker embedding concatenated at every frame; during inference, a single utterance-specific embedding is used to capture fine-grained variation. Facial emotion embeddings come from EmoFAN, applied frame-by-frame to capture time-varying emotional states (anger, happiness, neutral, and so on) independently of phonetic content, on the grounds that emotion is shared across languages even when phonetics are not.
The two conditioning signals are complementary in scope: the speaker embedding provides stable global identity (timbre, speaking style) while the emotion embedding provides frame-level dynamic expression. Both are projected to 80 dimensions and injected into the CFM’s Transformer decoder alongside the masked mel-spectrogram and unit sequence. A HiFi-GAN vocoder fine-tuned on LRS3 converts mels to audio, and Wav2Lip handles face generation. The CFM module is warm-started from a checkpoint pretrained on LibriTTS and fine-tuned on MuAViC (1,200 hours, English, Spanish, French, Italian, Portuguese) on 8 RTX A6000 GPUs.
Key Results
On MuAViC zero-shot speaker similarity (Table 1), MAVFlow achieves SS scores of 0.49 (Es-En), 0.51 (Fr-En), 0.53 (It-En), and 0.48 (Pt-En) against 0.35, 0.31, 0.37, and 0.30 for the AV2AV baseline, and against 0.42, 0.34–0.35, 0.41, and 0.35–0.36 for 3- and 4-stage cascaded systems. This constitutes an average 36% improvement in speaker similarity over AV2AV. MCD-DTW and MCD-DTW-SL metrics also improve, confirming better duration and spectral alignment.
Translation quality (ASR-BLEU, Table 3) is maintained or slightly improved over AV2AV (26.97 vs 26.57 for Es-En), and is competitive with cascaded systems, demonstrating that the paralinguistic improvements do not degrade linguistic accuracy.
Subjective MOS evaluation by 21 participants (Table 4) shows MAVFlow at 3.49 similarity and 4.01 naturalness vs AV2AV at 3.33 and 3.58 and the best cascaded system at 3.33 and 3.58.
On visual quality (LRS3, Table 5), MAVFlow achieves LSE-C 8.30 and FID 5.69 vs AV2AV’s 7.43 and 6.30, demonstrating that higher-quality mel-spectrograms improve downstream face generation even though the Wav2Lip face decoder is unchanged. On the CREMA-D emotion dataset, MAVFlow achieves 36.46% audio emotion accuracy and 72.68% visual emotion accuracy (Table 2, Table 6) vs AV2AV at 33.66% and 67.2%.
Ablation (Table 8) shows that audio guidance is the dominant contributor to speaker similarity, while visual guidance provides modest incremental gains; visual guidance alone is insufficient and can mildly reduce emotion accuracy relative to no guidance.
Novelty Assessment
The contribution is primarily engineering integration: OT-CFM, x-vectors, EmoFAN, m-AVHuBERT, HiFi-GAN, and Wav2Lip are all established components applied here together to the AV2AV translation problem. The key design choice is dual-modal conditioning in the CFM generator rather than relying on a single speaker embedding at the vocoder stage, which is a meaningful but incremental change over the AV2AV baseline. The Duration Length Regulator’s duration-matching strategy for dubbing alignment is a practical refinement not present in prior AV2AV work. The paper does not introduce a new architecture, training objective, or loss function; the OT-CFM formulation is taken directly from Matcha-TTS and related work. Evaluation is on publicly available datasets (MuAViC, CREMA-D, LRS3), and the code is released, supporting reproducibility.
Field Significance
Moderate — MAVFlow makes a concrete case that integrating flow matching with dual audio-visual conditioning can substantially improve speaker paralinguistic preservation in AV2AV translation relative to simpler vocoder-level conditioning, and that this benefit carries through to downstream face generation without additional training of the face synthesis module. The paper’s contribution is confined to the AV2AV translation niche, and the core findings, that richer conditioning in the mel generator improves both audio and visual output quality, are consistent with broader trends in conditional speech synthesis.
Claims
-
supports: Conditioning a flow matching mel-spectrogram generator on rich multimodal speaker representations produces more consistent speaker identity in zero-shot cross-lingual speech synthesis than injecting a single speaker embedding at the vocoder stage.
Evidence: MAVFlow achieves an average 36% improvement in speaker similarity (SS) over AV2AV across four language pairs on MuAViC, using OT-CFM conditioned on x-vector speaker embeddings plus facial emotion embeddings, while AV2AV uses d-vector conditioning in the vocoder only. (§5.4, Table 1)
-
supports: Higher-quality intermediate mel-spectrogram synthesis propagates benefits to downstream talking-face generation even when the face decoder itself is unchanged.
Evidence: MAVFlow improves LSE-C by +0.87, LSE-D by -0.49, and FID by -0.61 relative to AV2AV on LRS3 visual evaluation, despite using the same Wav2Lip face decoder, suggesting that the mel quality bottleneck affects face sync accuracy. (§5.5, Table 5)
-
complicates: Visual emotion conditioning is insufficient on its own to improve emotion expression in synthesized speech and requires concurrent audio speaker conditioning to be effective.
Evidence: Adding only visual guidance to the CFM model marginally maintains speaker similarity (SS 0.056 vs 0.057 without guidance) but reduces emotion accuracy from 28.66% to 26.83% on CREMA-D; the combination of audio and visual guidance is needed to reach 36.46%. (§5.6, Table 8)
-
supports: Paralinguistic and linguistic generation objectives are compatible in zero-shot cross-lingual speech synthesis; improving speaker fidelity does not require sacrificing translation accuracy.
Evidence: MAVFlow maintains competitive ASR-BLEU scores (26.97 Es-En vs 26.57 for AV2AV and 28.66–30.55 for cascaded systems) while substantially improving speaker similarity, using the same unit translation module as AV2AV. (§5.4, Tables 1 and 3)
-
complicates: Emotion recognition accuracy in synthesized cross-lingual speech remains far below ground-truth levels even with multimodal conditioning, indicating that paralinguistic preservation is an unsolved challenge.
Evidence: MAVFlow achieves 36.46% audio emotion accuracy vs a GT upper bound of 81.95% on CREMA-D, even with dual audio-visual guidance; additional training on an emotion-rich dataset (CREMA-D) raises this to 51.46% but still far below GT. (§5.4, Table 2; §5.6, Table 7)
Limitations and Open Questions
The system relies on emotional cues from facial video alone; audio-side paralinguistics such as prosody and timbre variation are not used as emotion conditioning signals, which may limit emotion transfer when visual input is unavailable or low quality. The unit extractor and unit-to-unit translation modules are adopted unchanged from AV2AV, so improvements in semantic translation quality require addressing those upstream components separately. The Duration Length Regulator interpolates to match source duration, which may introduce length-related artifacts when source and translated speech have substantially different natural lengths. Evaluation is limited to five European languages with English as the target; generalization to typologically distant language pairs (e.g., tonal languages, right-to-left scripts) is untested.
Wiki Connections
- Flow Matching — MAVFlow’s core generative module is an OT-CFM Transformer decoder for mel-spectrogram synthesis, demonstrating flow matching’s applicability to multimodal-conditioned cross-lingual speech rendering.
- Zero-Shot TTS — the renderer operates without target-speaker training data, using a reference utterance x-vector at inference to synthesize translated speech in the original speaker’s voice.
- Multilingual TTS — the system covers five languages (English, Spanish, French, Italian, Portuguese) on MuAViC and addresses the specific challenge of preserving speaker identity across language boundaries.
- Emotion Synthesis — frame-level EmoFAN facial emotion embeddings guide mel-spectrogram generation per frame, enabling emotion transfer from visual input across languages without explicit audio emotion labels.
- Speaker Adaptation — x-vectors averaged over multiple training utterances allow the model to learn generalizable speaker representations that transfer to unseen speakers at inference.
- Self-Supervised Speech — the pipeline uses m-AVHuBERT (a self-supervised masked multimodal prediction model) as its discrete unit extractor, which is the first stage of the MAVFlow system.
- CosyVoice (CosyVoice) — cited as a contemporary multilingual zero-shot TTS system using supervised semantic tokens, providing context for the multilingual speaker-preserving synthesis approach.
- VALL-E (VALL-E) — cited as a representative zero-shot neural codec language model, representing the class of speaker-conditioned synthesis systems that MAVFlow’s rendering approach is positioned alongside.