arXiv · 2026 · Preprint
Ngoc-Son Nguyen et al. (FPT Software AI Center) · → Paper · Demo: ✓ · Code: ?
Introduces the first video dubbing framework built on a discrete flow matching backbone, pre-training a zero-shot TTS model on factorized speech tokens and then adapting it to video-conditioned dubbing through a cross-modal alignment module that replaces text-only duration prediction.
Problem
Video dubbing (visual voice cloning) must simultaneously preserve the target speaker’s voice, match video-driven prosody and emotion, and stay precisely lip-synchronized, but existing systems struggle across all four requirements at once. The paper identifies two specific gaps in prior work: Speaker2Dubber pre-trains a phoneme encoder on large TTS corpora but does not transfer the prosody and acoustic modeling advantages of large-scale TTS pre-training into the dubbing stage, and ProDubber improves prosody-aware acoustic quality but still relies on a duration predictor to upsample text features to the target length. Because that duration predictor is not constrained by the actual video length, it produces speech whose timing drifts from the video, degrading lip-sync error scores. A further practical bottleneck is the scarcity of large-scale, legally shareable video dubbing data, which motivates transferring capability from TTS corpora rather than training dubbing systems from scratch.
Method
DiFlowDubber decomposes speech into three factorized discrete token streams (prosody, content, acoustic) using a pre-trained FACodec, plus a continuous speaker embedding, at 80 tokens/s. The system is trained in two stages.
In the first stage, a zero-shot TTS model is pre-trained on 470 hours of LibriTTS. Content is modeled deterministically: a phoneme encoder feeds a duration predictor, length regulator, and Feed-Forward Transformer (FFT) blocks to predict content tokens. Prosody and acoustic tokens are instead modeled generatively by the Discrete Flow-based Prosody-Acoustic (DFPA) module, a Diffusion Transformer (DiT, 8 layers, hidden size 768) that learns a discrete flow matching denoiser over the joint prosody-acoustic token distribution, conditioned on reference-speech tokens, a speaker embedding, and the latent content representation. Training uses a cross-entropy-based discrete flow matching loss over masked token positions.
In the second stage, the pre-trained TTS model is adapted to video-conditioned dubbing. The Face-to-Prosody Mapper (FaPro) extracts a global prosody prior directly from face-cropped video frames using ConvNeXt V2 blocks and an FFT-based Prosody Predictor. The Content-Consistent Temporal Adaptation (CCTA) module reuses the content-modeling architecture initialized from the pre-trained TTS model, but replaces its duration predictor with the Synchronizer: a dual cross-attention alignment mechanism that separately aligns lip-cropped video features with phonemes (video-text alignment) and phoneme embeddings with speech token positions (speech-text alignment), each supervised with a contrastive loss against Montreal Forced Aligner-derived alignment matrices, plus an auxiliary CTC loss. A distillation loss enforces cosine-similarity consistency between the CCTA student’s pre-content-predictor representation and the teacher TTS model’s corresponding representation. The FaPro prosody prior and the Synchronizer’s output are fused and fed to the Content Predictor, and the resulting content-prosody correlation representation replaces the reference-content conditioning of DFPA (whose denoiser weights are initialized from stage 1 and fine-tuned), so DFPA generates prosody and acoustic tokens consistent with both the video and the target transcript. A Codec Decoder synthesizes the final waveform from the predicted tokens and speaker embedding.
Key Results
On the Chem benchmark (Table 1), DiFlowDubber achieves the best LSE-C/LSE-D synchronization scores among all compared systems in both Dub 1.0 and Dub 2.0 settings, and the best UTMOS naturalness score. WER is second-best in Setting 1.0 and best in Setting 2.0; ProDubber attains competitive WER but its weak LSE scores indicate it behaves more like a text-driven TTS system than a synchronized dubbing model. SECS (speaker cosine similarity) does not favor DiFlowDubber, which the authors attribute to baselines being evaluated with the same speaker-embedding extractor used to train them; a 20-listener MOS-Similarity study instead gives DiFlowDubber the highest score (4.18 ± 0.18) among the four most recent baselines. Results on GRID (Table 2) follow a similar pattern, with DiFlowDubber leading on LSE-C, LSE-D, WER, and UTMOS, and again showing a SECS gap alongside the highest MOS-S. An NFE ablation (Table 4) shows DiFlowDubber matches EmoDubber’s inference-time RTF (0.05) at only 8 NFEs despite a larger 250M parameter count (vs. EmoDubber’s 116M), and that most metrics besides UTMOS are stable from 8 to 128 NFEs. A supplementary evaluation with pitch and emotion-consistency metrics (Table 5) shows DiFlowDubber achieving the lowest F0 Frame Error (0.395 vs. EmoDubber’s 0.426) and the highest emotion similarity (0.983) among compared baselines.
Novelty Assessment
The core novelty is architectural: this is presented as the first video dubbing framework built on a discrete flow matching backbone, and the paper’s ablations (Table 3) show each proposed component (zero-shot TTS pre-training, the Synchronizer’s dual-alignment losses, the distillation and CTC losses, and FaPro) contributes measurably to the final performance, rather than being decorative additions. The Synchronizer’s replacement of a duration predictor with a dual video-text/speech-text cross-attention alignment mechanism is a targeted fix for a specific, previously identified failure mode (ProDubber’s unconstrained duration-based upsampling causing synchronization drift). The zero-shot TTS pre-training and video-domain adaptation two-stage recipe itself extends a pattern already used by Speaker2Dubber and ProDubber, so the training-recipe framing is incremental relative to that line of work; the distinguishing contribution is the discrete-flow generative backbone plus the cross-modal synchronization mechanism layered on top of it.
Field Significance
moderate — This paper contributes a targeted architectural advance within the established two-stage TTS-pretrain-then-adapt paradigm for video dubbing, demonstrating that discrete flow matching can serve as an efficient generative backbone for prosody-acoustic modeling in this setting and that explicit cross-modal alignment losses can materially improve lip synchronization over duration-predictor-based approaches. Its scope is specific to the video dubbing subtask rather than TTS more broadly, and its own ablations and limitations section are transparent about where the approach falls short (speaker-similarity metrics, voice-cloning fidelity, and dependence on a third-party codec).
Claims
- supports: A discrete flow matching generative backbone can model joint prosody-acoustic token distributions with fewer denoising steps than continuous flow or diffusion counterparts of similar generative capacity, at comparable inference-time cost.
Evidence: At only 8-10 NFEs, the proposed 250M-parameter discrete flow matching model matches the real-time factor (RTF ≈ 0.05-0.06) of a smaller 116M-parameter continuous flow-based baseline (EmoDubber), while most quality metrics besides UTMOS remain stable from 8 to 128 NFEs. (§4.4, Table 4)
- supports: Explicit cross-modal alignment supervision between visual, textual, and speech streams improves audio-visual synchronization more than relying on text-only duration prediction in video-conditioned speech generation.
Evidence: Removing both alignment losses (video-text and speech-text) from the Synchronizer drops the AVSync score from 0.721 to 0.652 and raises WER from 9.65 to 17.15 on the Chem test set; adding the speech-text loss back is the primary driver of the AVSync improvement. (§4.4, Table 3)
- supports: Transferring representations from a large-scale zero-shot TTS pre-training stage into a downstream, domain-specific adaptation stage improves output naturalness and pronunciation accuracy relative to training the adaptation stage without such a pre-trained initialization.
Evidence: Removing zero-shot TTS pre-training lowers UTMOS from 4.02 to 3.53 and raises WER from 9.65 to 12.04 on the Chem ablation set. (§4.4, Table 3)
- complicates: Speaker-similarity scores computed with a cosine-similarity speaker encoder can be biased in favor of systems evaluated with the same embedding extractor used elsewhere in their pipeline, so automatic speaker-similarity metrics alone can misstate true perceptual speaker fidelity.
Evidence: DiFlowDubber shows no clear SECS advantage over baselines on either benchmark, which the authors attribute to shared speaker-embedding-extractor bias favoring those baselines, while a 20-listener MOS-Similarity study rates DiFlowDubber highest among the four most recent baselines in the same setting. (§4.2, Table 1)
- complicates: Building a video dubbing pipeline on top of a fixed, off-the-shelf factorized speech codec inherits that codec’s representational limitations, particularly around speaker-timbre fidelity, regardless of downstream architectural improvements.
Evidence: The paper’s own Limitations section states the framework depends on the third-party FACodec and inherits its constraints, and that the current design “does not fully meet expectations in voice cloning,” identifying speaker-timbre mimicry in real-world dubbing as an open problem. (§Limitations)
Limitations and Open Questions
The system depends on a fixed, off-the-shelf FACodec for its discrete token representations and speaker embedding, and the authors state this constrains voice-cloning fidelity; they plan to explore alternative codecs in future work rather than address this within the current paper.
Beyond the codec dependency, the paper reports that automatic speaker-similarity (SECS) results do not favor the proposed method on either benchmark, and the authors’ explanation (shared embedding-extractor bias) is not independently verified with an extractor-agnostic metric. Evaluation is limited to two benchmarks (Chem, a single-speaker lecture corpus, and GRID, a controlled studio-recording corpus); neither reflects the more variable acoustic and visual conditions of real-world film or television dubbing.
Wiki Connections
- Flow Matching — applies discrete flow matching (rather than continuous flow matching) as the generative backbone for jointly modeling prosody and acoustic token distributions.
- Zero-Shot TTS — pre-trains a zero-shot TTS system on LibriTTS as the first stage of its pipeline, then transfers its representations into the video dubbing domain.
- Prosody Control — introduces the Face-to-Prosody Mapper, an explicit mechanism that predicts a global prosody prior from facial expressions, decoupled from the content and speaker-identity pathways.
- Subjective Evaluation — runs a 20-listener MOS-Similarity study to corroborate speaker-similarity results where the automatic SECS metric is suspected of being biased.
- Neural Audio Codec — builds its entire token representation and training pipeline on top of the pre-trained FACodec’s factorized prosody/content/acoustic RVQ tokens.
- NaturalSpeech 3 — reuses NaturalSpeech 3’s pre-trained FACodec as the discrete tokenizer, detokenizer, and speaker embedding extractor underlying the entire framework.
- DiFlow-TTS — shares the same author group and a discrete flow matching approach to zero-shot TTS, applied here as one stage of a larger video dubbing system.
- OZSpeech — prior work from the same group on learned-prior-conditioned flow matching for zero-shot TTS, part of the lineage this paper’s TTS pre-training stage builds on.
- F5-TTS — cited as a representative flow-matching zero-shot TTS approach in related work, contrasted with this paper’s discrete-token formulation.
- E2 TTS — cited as a non-autoregressive flow-based zero-shot TTS baseline informing the design of the TTS pre-training stage.
- VoiceCraft — cited among prior zero-shot TTS systems that motivate large-scale TTS pre-training as a source of transferable representations.
- MaskGCT — cited as a masked generative codec transformer approach to zero-shot TTS in the related work discussion of discrete representation learning.
- VALL-E 2 — cited among neural codec language model approaches to zero-shot TTS that the paper positions its discrete flow matching approach against.