arXiv · 2025 · Preprint
Xin Cheng et al. (Renmin University of China, Apple) · → Paper · Demo: ✓ · Code: ✗
Unifies video-to-sound (V2S) and visual text-to-speech (VisualTTS) generation into a single flow-matching Diffusion Transformer, routing each conditioning signal to cross-attention or self-attention based on its temporal density.
Problem
Video-conditioned audio generation splits into two historically separate lines of work: V2S, which produces ambient/environmental sound synchronized to a silent video, and VisualTTS, which produces intelligible speech synchronized to a talking video and a transcript. V2S models cannot produce intelligible speech, and VisualTTS models cannot produce non-speech environmental sound, so neither can serve applications that require both (e.g., dubbing a clip that has both dialogue and background sound). Prior attempts to unify the two tasks either rely on complex multi-stage curriculum learning to acquire V2S and VisualTTS capabilities separately, or cannot generate sound and speech jointly at all because of the scarcity of paired video-sound-speech training data.
Method
VSSFlow is a single flow-matching model built on a Diffusion Transformer (DiT) backbone (DiT blocks follow the Stable Audio Open implementation) that is trained end-to-end on V2S and VisualTTS data simultaneously, without task-specific stages. Audio is represented as a latent from the AudioLDM2 VAE (25 Hz, 64-dim) over a mel-spectrogram, decoded back to waveform via a frozen HiFi-GAN vocoder; a continuous-time ODE with a learned velocity field transports Gaussian noise to this audio latent under the standard conditional flow-matching objective.
The paper’s central architectural contribution is a disentangled condition aggregation mechanism that routes each of four conditioning signals to a different attention pathway based on its temporal structure, rather than using a single conditioning mechanism uniformly. Video semantic features (CLIP embeddings, sparse and global in nature) are injected via cross-attention. Three temporally-dense, frame-aligned signals — sound synchronization features (from Synchformer), phoneme-level speech transcript features, and lip-movement speech-synchronization features (from AV-HuBERT) — are instead concatenated with the audio latent along the channel dimension and processed through self-attention, so the model can exploit fine-grained temporal correspondence directly. 1D RoPE is added to both self- and cross-attention blocks to strengthen temporal alignment between the audio and video/text streams.
Training uses VGGSound for V2S (182k samples) and three VisualTTS corpora — Chem, GRID, LRS2 (162k samples combined) — jointly, for 250k steps, with classifier-free guidance (10% condition dropout) and a CFG scale of 3.0 at inference via a Dopri5 ODE solver. To extend the model to joint sound-speech generation, for which no large paired corpus exists, the authors introduce a feature-level data synthesis method that mixes independently-sourced sound (VGGSound) and speech (LRS2) samples directly in feature space, either by additive superposition at a random SNR or by in-place substitution of the sound features within the speech interval. Because the mixing happens on already-extracted features during data loading rather than on raw audio/video, it adds negligible overhead. The pretrained model is then fine-tuned on this synthetic joint data for 10k steps. Three model sizes are trained: VSSFlow-S (5 DiT blocks, 245M params), VSSFlow-M (10 blocks, 463M params), VSSFlow-L (15 blocks, 618M params).
Key Results
On the VGGSound V2S benchmark, even the smallest VSSFlow-S variant matches or exceeds larger domain-specific baselines (LoVA, 1057M params; Frieren, comparable size) on FAD, IS, KL, DeSync, and video-audio semantic alignment (VA-IB), and performance improves monotonically from S to L. On the VisualTTS benchmarks (Chem, GRID), VSSFlow-M achieves a WER of 9.4 on Chem, close to the pure-TTS baseline E2-TTS (8.7 WER) and far below dedicated VisualTTS baselines (StyleDubber 16.3, EmoDubber 16.7, DSU 37.3), while also achieving the best MCD, MCD-DTW, MCD-DTW-SL, LSE-C, and LSE-D scores across both benchmarks. For joint sound-speech generation on the V2C-Animation benchmark, VSSFlow-M (fine-tuned for only 10k steps) outperforms pipeline-based baselines that compose separate VisualTTS models (Speaker2Dubber, StyleDubber) with separate V2S models (MMAudio, LoVA): 19.4 WER and 24.7 speaker similarity versus 25.98–56.86 WER and 17.78–20.02 speaker similarity for the pipelines.
An ablation over five condition-routing variants confirms the design choice: routing temporally-dense signals (transcript, sync cues) through concatenation improves WER and lip-sync error, while routing the sparse, global video-semantic feature through cross-attention improves sound-quality and semantic-alignment metrics (FAD, VA-IB). A separate ablation compares joint (V2S+VisualTTS) training against training on each task alone and finds the joint-trained model reaches parity with the single-task models at matched training epochs on both sound and speech metrics — the paper reports this as contrary to the prior finding (from DualDub) that joint training of these two tasks degrades performance and needs curriculum staging. A data-configuration ablation on V2C shows the synthetic joint data is doing real work: fine-tuning on synthetic sound-speech mixtures alone gives the best WER (19.4) and UTMOS (2.07) of any configuration, better than fine-tuning on the small (8k-sample) real V2C corpus alone (36.5 WER).
Novelty Assessment
The architectural contribution is specific and well-isolated: rather than a new backbone, VSSFlow reuses an existing DiT block design (Stable Audio Open) and off-the-shelf pretrained feature extractors (CLIP, Synchformer, AV-HuBERT), and its novelty lies in the rule for routing each conditioning signal to cross-attention or self-attention/concatenation based on temporal density, validated with a controlled 5-variant ablation. The feature-level data synthesis for joint generation is a simple linear mixing operation, not a new algorithm, but it is shown to be more useful than the tiny available real joint corpus. The paper’s strongest and most field-relevant claim is empirical and negative in character: it directly challenges a specific prior finding (from DualDub) that joint V2S/VisualTTS training requires curriculum staging to avoid degradation, and supports the alternative with a controlled comparison at matched training epochs. The joint-generation comparison is against pipeline baselines assembled by the authors (independently-trained VisualTTS and V2S models glued together) rather than against another end-to-end unified system, since none exists in the public literature at the time of writing; this is a reasonable comparison given the state of the field but is structurally favorable to an end-to-end model.
Field Significance
Moderate — this paper offers a concrete, ablation-validated recipe for routing heterogeneous conditioning signals in a shared DiT backbone, and it demonstrates that end-to-end joint training of video-conditioned sound and speech generation does not require the multi-stage curricula that prior unified attempts used. Its scope is a comparatively narrow sub-area (video-conditioned audio-visual generation) rather than mainstream text-only TTS, and the joint-generation evaluation is bounded by the small, animation-only V2C-Animation benchmark and by pipeline rather than end-to-end baselines.
Claims
- supports: Joint training of two video-conditioned audio generation tasks with different output characteristics does not need to degrade either task’s performance, provided the conditioning pathways for each task’s signals are architecturally separated.
Evidence: Learning curves comparing joint (V2S+VisualTTS), sound-only, and speech-only training at matched epochs show the joint-trained model reaches performance parity with each single-task model on FAD/DeSync (sound) and WER/UTMOS (speech), directly contrasting with a prior report that joint training of these tasks causes mutual degradation. (§4.4, Figure 4)
- refines: The choice between cross-attention and channel-concatenation conditioning in a DiT-based generative model should be determined by whether the conditioning signal is temporally dense/aligned or sparse/global, not fixed uniformly across all condition types.
Evidence: A 5-variant ablation shows concatenation-based conditioning improves WER and lip-sync error for temporally-dense signals (transcript, sound/speech sync cues), while cross-attention conditioning improves FAD and video-audio semantic alignment for the sparse, global video-semantic feature; self-attention weight visualization shows a diagonal locality bias explaining the concatenation result. (§4.3, Figure 3)
- supports: Feature-level synthesis of paired training examples, constructed by mixing independently-sourced single-modality data in representation space, can substitute for scarce real multimodal joint-generation training data.
Evidence: Fine-tuning on synthetic sound-speech mixtures alone yields the best WER (19.4) and UTMOS (2.07) on the V2C-Animation joint-generation benchmark among all training-data configurations tested, exceeding fine-tuning on the small (8k-sample) real V2C corpus alone (36.5 WER). (§4.4, Table 4)
- complicates: Unified video-conditioned generation systems that depend on frozen pretrained feature extractors for their conditioning signals inherit those extractors’ representational limits, and systems adapted with synthetic joint data remain bounded by the distributional gap between synthetic and real joint data.
Evidence: The authors state that reliance on frozen pretrained extractors (CLIP, Synchformer, AV-HuBERT) may limit capture of fine-grained audio-visual nuances, and that generation quality on real joint scenarios is inherently constrained by the distributional gap of the synthetic training data used for joint-generation fine-tuning. (§Limitations)
Limitations and Open Questions
Beyond the frozen-extractor and synthetic-data-gap limitations the authors state directly, the joint sound-speech generation results are evaluated on a single benchmark (V2C-Animation) drawn entirely from animated content, and the joint-generation comparison is against pipeline baselines composed by the authors rather than against other end-to-end unified systems, since none are publicly available for this exact three-way task. Code was not released at the time of publication (“Demos and code will be soon released” per the paper), though a project page with generated examples is available.
Wiki Connections
- Flow Matching — VSSFlow trains a DiT-based velocity field with the standard conditional flow-matching objective, applying it to a multi-conditional video-and-speech-conditioned setting rather than the more common text-only TTS case.
- Self-Supervised Speech — the model’s speech-synchronization conditioning signal is extracted from a frozen, self-supervised AV-HuBERT encoder, making audio-visual self-supervised pretraining a load-bearing component of its lip-sync conditioning.
- GAN Vocoder — audio latents predicted by the flow-matching model are converted to waveform via a frozen HiFi-GAN vocoder, following the standard VAE-latent-plus-GAN-vocoder pipeline used in AudioLDM2-style systems.
- F5-TTS — used as a fast non-autoregressive flow-matching TTS reference point in the related-work discussion of DiT-based conditioning designs, though not directly compared as a VisualTTS baseline.