arXiv · 2026 · Preprint
Wenhao Guan et al. (Xiamen University / Shanghai Innovation Institute / Shanghai Jiao Tong University / Joy Future Academy) · → Paper · Demo: ✓ · Code: ?
Extends a patch-based autoregressive diffusion-transformer text-to-audio model with patch-level audio-visual cross-attention so that a single model generates lip-synchronized speech together with ambient sound and music for a silent video, steered by a structured text description and an optional reference voice.
Problem
Video dubbing systems (lip-synchronized TTS from a script and a silent video) generate speech only. Real video soundtracks also contain ambient sound, sound effects, and music, which practitioners add afterwards with separate text-to-audio tools and manual mixing. Prior video-to-soundtrack systems that do produce speech and sound together are framed by the authors as using decoupled dual-head designs and as relying on the video as the main condition, which limits explicit text control over what non-speech audio appears. The authors target one generative process that produces the whole acoustic scene from text, video, and optionally a reference voice.
Method
HoliDubber takes a structured text description, a silent video at 25 FPS, and optionally a reference utterance, and outputs a single audio stream containing speech and non-speech sound. It is built in three stages.
Audio-VAE. A DAC-style encoder/decoder (residual units, Snake activations, multi-scale dilations) with the RVQ bottleneck replaced by a VAE bottleneck, giving continuous 64-dimensional latents at 25 Hz from 32 kHz audio. It is trained on about 30,000 hours split 1:1:1 across speech, general audio/music, and naturally mixed speech-plus-background recordings, with reconstruction, KL (β = 0.1), and a multi-scale adversarial loss following DAC (§4.4, App. D).
Text-to-audio (TTA) backbone. A DiTAR-style patch-based autoregressive diffusion model (DiTAR): latents are grouped into patches of size 5; an Aggregation Encoder (4 Transformer blocks) compresses each patch into one embedding via a learnable token; a Causal Autoregressive Transformer initialized from Qwen2.5-1.5B models the sequence of patch embeddings conditioned on text; and a Local Diffusion Transformer (LocDiT, 12 blocks) generates the continuous latents of the next patch, conditioned on the causal transformer’s output and the previous patch, trained with a conditional flow-matching (velocity-prediction) loss and sampled with 10 ODE steps. The backbone is pre-trained on about 130,000 hours: 80,000 h of Emilia speech with plain transcripts, 20,000 h of mixed speech-audio, 20,000 h of speech annotated with speaker and style descriptions, and 10,000 h of non-speech audio and music (§4.4).
Structured captions. Qwen3-Omni-30B-A3B-Instruct annotates each clip with three natural-language fields, <speaker profile> (gender, age, accent, timbre), <speech instruct> (speaking rate, emotional tone, vocal qualities), and <audio caption> (background music, ambient sound, spatial dynamics). The <text> field comes from source separation followed by Whisper-large-v3 ASR. A stratified sample is human-checked and failing samples are discarded (§4.2, App. B).
Video conditioning. AV-HuBERT, a self-supervised audio-visual speech model, extracts visual features at 25 Hz. A new Patch-based AV-Fusion module (randomly initialized) applies cross-attention with audio latents as queries and video patch features as keys/values; each audio patch attends to the video of its upcoming temporal segment so the autoregressive generator can anticipate articulation. Cross-attention is chosen over concatenation to avoid changing the input distribution of the pre-trained backbone (§3.2). The full model is fine-tuned on VoxCeleb2 plus CelebV-Dub training clips with the single flow-matching loss.

Two inference modes. During fine-tuning, the three descriptive fields are dropped together with probability 0.5, leaving only <text>. At inference this supports (1) “zero-shot dubbing” with a transcript plus reference speech as an audio prompt, and (2) text-prompt-guided dubbing, where the full structured description (including the <audio caption>) replaces the reference voice (§3.1, §4.4).
Key Results
All speech metrics are computed on the model’s full output, which in HoliDubber’s case is a jointly generated speech-plus-background stream; the paper does not say that speech is separated before scoring. Baselines are speech-only dubbing systems. The authors attribute HoliDubber’s higher WER partly to the co-generated background interfering with Whisper ASR (§5.1).
Zero-shot dubbing mode (Table 1) against AlignDiT, VoiceCraft-Dub, and FunCineForge:
| Metric | VoxCeleb2: HoliDubber | VoxCeleb2: best other system | CelebV-Dub: HoliDubber | CelebV-Dub: best other system |
|---|---|---|---|---|
| LSE-C ↑ | 6.83 | 6.92 (AlignDiT) | 6.65 | 6.63 (AlignDiT) |
| LSE-D ↓ | 7.87 | 7.90 (AlignDiT) | 7.92 | 7.66 (AlignDiT) |
| SPK-SIM ↑ | 0.68 | 0.55 (AlignDiT) | 0.46 | 0.41 (AlignDiT) |
| EMO-SIM ↑ | 0.981 | 0.98 (AlignDiT) | 0.951 | 0.955 (VoiceCraft-Dub) |
| WER ↓ | 24.12 | 20.92 (AlignDiT) | 17.95 | 12.31 (AlignDiT) |
| UTMOS ↑ | 2.79 | 3.7 (FunCineForge) | 3.24 | 3.65 (FunCineForge) |
| MOS ↑ | 3.83 | 3.91 (FunCineForge) | 3.96 | 4.06 (FunCineForge) |
HoliDubber has the highest SPK-SIM on both sets. Lip sync is roughly tied with AlignDiT (each wins one of the two LSE measures per set), and both are far ahead of FunCineForge (LSE-C 1.56 and 2.63). AlignDiT has the lowest WER on both sets; HoliDubber ranks third of four on VoxCeleb2 and second on CelebV-Dub. FunCineForge has the highest UTMOS and MOS on both sets; HoliDubber is second on MOS. Ground-truth WER is high (19.91 on VoxCeleb2, 9.19 on CelebV-Dub) and ground-truth UTMOS low (2.51, 2.87), reflecting in-the-wild recordings. The authors argue UTMOS should be close to GT rather than maximal and state that HoliDubber’s UTMOS is closest to GT; in Table 1, AlignDiT’s UTMOS is marginally closer to GT on both sets (2.77 vs. 2.79 against GT 2.51; 3.16 vs. 3.24 against GT 2.87).
Text-prompt-guided mode (Table 2), compared only with FunCineForge (which still needs reference speech in its prompt mode): HoliDubber is better on LSE-C/D, EMO-SIM, WER (20.03 vs. 21.1 on VoxCeleb2; 19.42 vs. 22.17 on CelebV-Dub), MOS (3.92 vs. 3.86; 3.99 vs. 3.96), and all four audio-generation metrics (e.g. FAD 3.82 vs. 5.18 and 3.16 vs. 4.11). FunCineForge has higher UTMOS on both sets (3.55 vs. 3.04; 3.65 vs. 3.12), so the paper’s summary that HoliDubber is superior “across all evaluated dimensions” does not hold for UTMOS.
HoliDub-Bench (Table 3), text-prompt mode, against HoliDubber’s own TTA backbone without video: LSE-C 6.44 vs. 1.21, WER 12.81 vs. 15.28 (GT 14.51), UTMOS 3.02 vs. 2.97, MOS 3.96 vs. 3.81, FAD 3.08 vs. 10.51, KLD 1.86 vs. 1.89. The TTA backbone is better on FD (9.52 vs. 10.95), IS (1.54 vs. 1.51), and EMO-SIM (0.993 vs. 0.992). No external system is evaluated on HoliDub-Bench.
Ablations (Table 4, VoxCeleb2, zero-shot mode). Removing video at inference drops LSE-C from 6.83 to 4.29 while WER (21.43) and UTMOS (2.91) improve slightly. Replacing cross-attention fusion with concatenation collapses the model (SPK-SIM 0.15, WER 62.19, UTMOS 1.76). Training without prompt-field dropout lowers LSE-C to 4.61 and SPK-SIM to 0.56.
Decoupled pipeline (Table 5, App. C). HoliDubber speech-only output mixed at equal loudness with AudioLDM background audio scores worse than joint generation on HoliDub-Bench on every reported metric except IS (1.65 vs. 1.51), with the largest gap on UTMOS (2.03 vs. 3.02).
Novelty Assessment
The generative backbone is DiTAR’s patch-based AR-plus-local-diffusion design applied to a continuous audio VAE rather than a speech-only one, with an LLM initialization; this part is adaptation. The new pieces are (1) the patch-level audio-visual cross-attention fusion with lookahead to the upcoming video segment, inserted as a side channel into a pre-trained text-to-audio backbone, which the ablation shows matters a great deal relative to concatenation; (2) the formulation of dubbing as joint speech-plus-ambient generation from one structured text prompt, using a captioning pipeline to supply speaker, style, and scene descriptions; and (3) HoliDub-Bench, a 1,000-clip, 2.4-hour subset of VoxCeleb2 and CelebV-Dub selected for background sound or emotional variation and annotated with the same pipeline. The “first holistic framework” claim is made against a backdrop of video-to-soundtrack systems (VSSFlow, DualDub, DeepAudio) that also generate speech and sound together; the paper distinguishes itself by text-driven, single-stream generation but does not compare against any of them experimentally.
Field Significance
Moderate. The paper demonstrates that a speech-plus-ambient text-to-audio backbone can be turned into a lip-synchronized dubbing model by adding a lightweight cross-attention video branch, and it provides an evaluation set for dubbing in acoustically complex scenes. Its evidence for speech quality is harder to interpret than in speech-only dubbing work, because speech metrics are computed on mixed output against speech-only baselines, and the comparisons favour HoliDubber on speaker similarity and lip sync but not on WER or perceived quality.
Claims
- supports: Injecting visual lip features into a pre-trained text-to-speech or text-to-audio generator through cross-attention preserves the generator’s capabilities better than concatenating visual features into its input.
Evidence: Replacing the patch-level cross-attention fusion with feature concatenation drops SPK-SIM from 0.68 to 0.15, raises WER from 24.12 to 62.19, and lowers UTMOS from 2.79 to 1.76 on VoxCeleb2. (§5.4, Table 4)
- complicates: In video dubbing, conditioning on video for lip synchronization trades off against speech intelligibility and automatically predicted quality.
Evidence: Removing video at inference lowers LSE-C from 6.83 to 4.29 but improves WER (24.12 to 21.43) and UTMOS (2.79 to 2.91); across systems, FunCineForge has the highest UTMOS and MOS but by far the worst lip sync. (§5.1, §5.4, Tables 1 and 4)
- supports: Generating speech and background sound jointly in one model yields better speech quality and intelligibility than generating them separately and mixing afterwards.
Evidence: Against a speech-only HoliDubber output mixed at equal loudness with AudioLDM background audio, joint generation scores UTMOS 3.02 vs. 2.03, WER 12.81 vs. 16.67, and FAD 3.08 vs. 5.53 on HoliDub-Bench; the pipeline is better only on IS. (§C, Table 5)
- complicates: Standard speech metrics such as ASR-based WER and predicted MOS become hard to compare across systems when one system’s output contains co-generated background sound and the others’ do not.
Evidence: HoliDubber’s zero-shot WER (24.12 on VoxCeleb2, 17.95 on CelebV-Dub) exceeds AlignDiT’s (20.92, 12.31), which the authors attribute to co-generated ambient sound interfering with Whisper; the authors also argue that UTMOS above the ground-truth level indicates over-clean speech rather than better quality. (§5.1, Table 1)
- supports: Randomly dropping descriptive prompt fields during fine-tuning lets one model serve both reference-audio voice cloning and description-only generation.
Evidence: Training without the 50% field dropout lowers zero-shot-mode LSE-C from 6.83 to 4.61 and SPK-SIM from 0.68 to 0.56, which the authors attribute to train-test mismatch when only the transcript is given. (§5.4, Table 4)
Limitations and Open Questions
Warning
Speech metrics (WER, UTMOS, SPK-SIM, EMO-SIM, MOS) are computed on HoliDubber’s mixed speech-plus-background output and compared with speech-only baselines; the paper does not report separating speech before scoring, so differences reflect both speech quality and the presence of ambient sound. The test sets are random clip-level holdouts from VoxCeleb2 and CelebV-Dub whose remaining clips form the training set, so evaluated speakers very likely appear in training and the “zero-shot” label does not establish cloning of unseen voices. The MOS protocol (number of raters, scale, instructions, whether raters judged speech or the whole scene) is not reported.
- FD, FAD, KLD, and IS are described as measuring non-speech audio, but they are computed against ground-truth audio that contains speech too; the paper does not describe isolating the non-speech component.
- In zero-shot mode, the reference utterance used for prompting and for SPK-SIM is not specified (for example, whether it is a different clip of the same speaker).
- HoliDub-Bench is compared only against HoliDubber’s own backbone and a pipeline built from HoliDubber plus AudioLDM; no external holistic or video-to-soundtrack system is evaluated on it.
- The FunCineForge comparison is cross-domain: it was trained mainly on Chinese and English television data, while HoliDubber is fine-tuned on the training splits of the two test corpora. The text-prompt-mode comparison also gives FunCineForge a reference voice that HoliDubber does not use.
- Captions come from an LLM with only a stratified human check; caption error rates are not reported.
- Multilingual dubbing and multi-speaker dialogue scenes are left for future work (§6). Code release is not stated.
Wiki Connections
- Flow Matching — The LocDiT patch decoder is trained with a conditional flow-matching velocity loss and sampled with a 10-step ODE solver.
- Instruction-Conditioned TTS — In text-prompt-guided mode the system generates speech from natural-language speaker-profile and speaking-style descriptions without any reference audio, alongside a scene caption for the background.
- Subjective Evaluation — Human MOS is reported in every comparison, but on mixed speech-plus-background output with no reported protocol details.
- DiTAR — HoliDubber adopts DiTAR’s aggregation encoder, causal AR transformer, and local diffusion transformer design and adds video conditioning to it.
- VSSFlow — A video-conditioned joint sound-and-speech generator the authors position against as video-driven rather than text-driven; it is not evaluated as a baseline.
- F5-TTS — SPK-SIM follows its protocol and the concatenation ablation follows its input-fusion style; AlignDiT, the strongest baseline, extends this model with visual cues.
- AudioLDM — Supplies the audio-generation metric protocol (FD, FAD, KLD, IS) and the background generator in the decoupled-pipeline comparison.
- Emilia — Provides 80,000 hours of the text-to-audio backbone’s pre-training speech.
- Qwen3-Omni — Used as the captioning model that produces the speaker, style, and scene descriptions for training data and HoliDub-Bench.
- UTMOS — Used as the automatic quality metric; the authors argue scores near the ground-truth level are preferable to higher scores for in-the-wild dubbing.
- emotion2vec — Provides the emotion embeddings behind the EMO-SIM metric.