NAACL · 2025 · Conference
Yifan Liu et al. (Shanghai Jiao Tong University) · → Paper · Demo: ✗ · Code: ✓
An end-to-end video-to-speech model that predicts Mel-spectrograms directly from silent video frames, preserving speaker characteristics without any audio input or speaker embedding during training or inference.
Problem
Video-to-speech (V2S) synthesis generates speech from silent video alone, which requires recovering both linguistic content and speaker identity (timbre, pitch, tone) from visual cues only. Early V2S systems relied on auxiliary speaker embeddings extracted from reference audio clips to anchor speaker identity, but such reference audio is often unavailable in real deployment scenarios (communication aids, noisy environments). Audio-visual pre-training (notably AV-HuBERT) removed this dependency for some systems, but the paper identifies a specific remaining failure mode: ReVISE, a strong pre-training-based V2S system, eliminates the need for speaker embeddings yet still fails to preserve speaker characteristics in its output, despite achieving reasonable intelligibility. The paper traces this failure to ReVISE’s choice of vocoder rather than its visual backbone.
Method
DiVISe diagnoses the speaker-preservation failure of prior unit-based V2S pipelines by isolating the vocoder as the source of information loss. The authors compare HiFi-GAN (Mel-spectrogram-conditioned) against Unit-HiFiGAN (discrete acoustic-unit-conditioned, as used in ReVISE) when both are fed the same ground-truth acoustic signal, and find that the unit-based vocoder discards speaker-identifying information in the frequency domain, evidenced by both spectrogram visualization and a large gap in speaker cosine similarity (SECS 0.555 vs. 0.894) and speaker verification error (EER 40.52 vs. 22.96) (§3, Table 1).
Building on this diagnosis, DiVISe replaces the discrete-unit intermediate representation with direct Mel-spectrogram prediction. The architecture has two stages: a V2S frontend that maps silent video frames to Mel-spectrograms, and a vocoder that converts Mel-spectrograms to waveforms. The frontend consists of a pre-trained AV-HuBERT encoder (a modified ResNet-18 visual frontend feeding a 12-layer, 1024-dim transformer, audio-visually pre-trained following Shi et al. 2022) followed by linear upsampling to align temporal resolution with the Mel-spectrogram rate, and a 4-block conformer module that refines temporal structure before a linear projection produces the final Mel-spectrogram. The frontend is trained with an L1 reconstruction loss against the ground-truth Mel-spectrogram; no acoustic input or speaker embedding is used at any stage (§4.1, §4.2.1).
The vocoder is HiFi-GAN, first pre-trained on ground-truth audio (LJSpeech), then fine-tuned on the Mel-spectrograms actually produced by the V2S frontend to correct for the distribution shift between real and predicted spectrograms (§4.2.2). At inference, the full video clip is processed by the frontend to produce a Mel-spectrogram sequence, which the fine-tuned vocoder converts to a waveform; no gradient flows from the vocoder back into the frontend. The default configuration uses AV-HuBERT LARGE (325M parameters) as the visual backbone, yielding 350M total parameters across frontend and vocoder; a smaller AV-HuBERT BASE variant (103M backbone, 128M total) is also evaluated (§4.1, Appendix B, Table 15).
Key Results
On LRS3 (433h, full-resource), DiVISe achieves WER 35.68%, PESQ 1.17, and ESTOI 0.331, and on LRS2 (224h) WER 36.24%, PESQ 1.22, ESTOI 0.367 (§5.3, Table 2). These intelligibility numbers are competitive with or better than prior visual-only V2S systems (ReVISE, DiffV2S, VCA-GAN), and notably better than several methods that additionally use speaker embeddings or text transcriptions as extra input (Multi-Task, SVTS) (§5.3).
The larger gap is in speaker preservation. Against ReVISE specifically (controlled comparison, same data and visual backbone), DiVISe improves SECS from 0.5384 to 0.6242 and EER from 41.17 to 28.66 on LRS3 (§5.2, Table 3). Across all compared methods, DiVISe attains the best SECS score on LRS2 (0.609) and is competitive with DiffV2S on LRS3 (0.624 vs. 0.625), despite using neither audio speaker embeddings during training nor inference, unlike SVTS and Multi-Task in their speaker-embedding configurations (§5.2, Table 4). A 15-participant subjective speaker-matching test corroborates this: DiVISe scores 3.90±0.14 vs. ReVISE’s 1.80±0.18 on a 1-5 scale measuring perceived match between synthesized voice and a still image of the speaker (ground truth: 4.80±0.07) (§5.2, Table 6).
Ablations show that audio-visual pre-training is essential for DiVISe across all metrics (WER degrades from 35.68 to 93.98 without it), while pre-training mainly helps ReVISE’s intelligibility but barely improves its speaker preservation (SECS 0.5384 vs. 0.5304) (§5.4.2, Table 7) — directly supporting the paper’s claim that ReVISE’s speaker-preservation ceiling is set by its vocoder, not its visual encoder. Vocoder fine-tuning on frontend-generated spectrograms improves SECS from 0.5505 to 0.6242 with negligible effect on intelligibility (§5.4.4, Table 9). The paper also reports that DiVISe’s advantage over ReVISE widens with more training data and larger backbone size (BASE vs. LARGE AV-HuBERT) (§5.4.1, Figure 3), though absolute WER (35-40%) remains far from production-quality ASR-verifiable intelligibility, and the subjective MOS test on audio quality alone (no speaker cue) shows HiFi-GAN scoring slightly lower than Unit-HiFiGAN (4.24 vs 4.37), an inversion the authors attribute to unit-based audio sounding more “neutral” and consistent regardless of speaker identity (Appendix C, Table 16).
Novelty Assessment
The architectural components are all established: AV-HuBERT as a visual encoder, a conformer module for temporal refinement, HiFi-GAN as a vocoder, and a two-stage pre-train/fine-tune vocoder schedule are each used elsewhere in speech and V2S literature. DiVISe’s contribution is the diagnosis and architectural fix: identifying that the discrete acoustic-unit representation used by the prior state-of-the-art V2S system (ReVISE) is itself the bottleneck for speaker preservation, and demonstrating that substituting a continuous Mel-spectrogram target with a controlled, otherwise-matched architecture closes much of that gap without reintroducing any audio or speaker-embedding input. This is best characterized as a targeted engineering fix grounded in a clear empirical diagnosis (§3) rather than a structurally novel architecture; the value lies in the controlled comparison (Table 3, same backbone, same data, isolated vocoder change) and the explicit demonstration that effective speaker preservation does not require unit-based discretization in V2S.
Field Significance
Moderate — this paper provides a controlled empirical demonstration that the choice of intermediate representation (Mel-spectrogram vs. discrete acoustic units) materially affects speaker-identity preservation in video-to-speech synthesis, holding the visual backbone and training data fixed. It contributes a clear diagnostic methodology (the Table 1 vocoder-only comparison) for separating intelligibility-affecting and speaker-affecting components of a V2S pipeline, and demonstrates that a fully visual-only system can match or exceed methods that use auxiliary audio speaker embeddings on key speaker-similarity metrics.
Claims
- supports: In video-to-speech synthesis, discrete acoustic-unit intermediate representations preserve speech content but discard speaker-identifying acoustic detail relative to continuous Mel-spectrogram representations.
Evidence: With ground-truth input held fixed, Unit-HiFiGAN scores SECS 0.555 / EER 40.52 versus HiFi-GAN’s SECS 0.894 / EER 22.96, and a Mel-spectrogram visual comparison shows the unit-based vocoder output diverges from ground truth in the frequency domain. (§3, Table 1, Figure 1)
- supports: Audio-visual pre-trained visual encoders can supply enough speaker-identity information from silent video to make explicit speaker embeddings unnecessary in video-to-speech synthesis.
Evidence: DiVISe attains the best or near-best SECS scores on LRS2 (0.609) and LRS3 (0.624) among all compared methods, including several that use audio speaker embeddings during training or inference (SVTS, Multi-Task). (§5.2, Table 4)
- complicates: Gains in objective intelligibility metrics from architectural changes do not necessarily transfer to subjective audio-quality preference, especially when speaker-identity cues are removed from the listening context.
Evidence: In an audio-only MOS test without speaker reference images, Unit-HiFiGAN scores higher (4.37±0.11) than the Mel-based HiFi-GAN (4.24±0.12), the inverse of the speaker-matching and intelligibility rankings observed elsewhere in the paper. (Appendix C, Table 16)
- refines: The benefit of audio-visual pre-training for video-to-speech synthesis is not uniform across model components; it most strongly aids the component responsible for output representation choice rather than uniformly improving all downstream metrics.
Evidence: Removing pre-training degrades DiVISe across all four reported metrics (SECS, EER, ESTOI, WER), but for ReVISE removing pre-training mainly hurts intelligibility (WER 36.03 to 77.24) while leaving speaker metrics comparatively unaffected (SECS 0.5384 to 0.5304). (§5.4.2, Table 7)
Limitations and Open Questions
Warning
Reported WER remains high (35.68-36.24% in the full-resource setting), and the paper’s own subjective MOS audio-quality test (without speaker context) ranks the proposed Mel-based vocoder below the unit-based baseline, indicating the speaker-preservation gains are not accompanied by a clear audio-quality win in isolation.
The paper requires the same heavy mouth-region preprocessing pipeline as AV-HuBERT, which the authors note limits real-time applicability (§8, Limitations). Evaluation is restricted to English-only corpora (LRS2, LRS3); the authors explicitly flag multilingual generalization as untested due to compute constraints. The model’s WER substantially trails dedicated ASR or text-conditioned TTS systems, reflecting the inherent difficulty of inferring content purely from lip movements without textual or acoustic priors. Latency analysis (§6) shows the conformer module, while improving intelligibility, reduces throughput relative to a no-conformer variant, an explicit accuracy/speed trade-off the paper surfaces but does not resolve.
Wiki Connections
- Speaker Adaptation — DiVISe shows that audio-visual pre-trained visual features can substitute for explicit speaker embeddings in conditioning synthesis on speaker identity, eliminating the reference-audio requirement common in prior V2S speaker adaptation approaches.
- Disentanglement — the paper’s central diagnostic finding is that discrete acoustic-unit vocoders entangle content recovery with loss of speaker-identity information, while continuous Mel-spectrogram representations retain both more effectively.
- Self-Supervised Speech — the V2S frontend is built directly on AV-HuBERT, a self-supervised audio-visual representation learned via masked multimodal cluster prediction, and the paper’s ablations isolate its contribution to both intelligibility and speaker preservation.
- GAN Vocoder — the paper’s core architectural change is replacing a discrete-unit GAN vocoder (Unit-HiFiGAN) with a Mel-spectrogram-conditioned GAN vocoder (HiFi-GAN), with an added fine-tuning stage to address frontend-vocoder distribution mismatch.
- Evaluation Metrics — the paper combines objective speaker-similarity metrics (SECS, EER), intelligibility metrics (PESQ, STOI, ESTOI, WER), and subjective human evaluation (speaker-matching score, MOS) to separately characterize content and speaker-identity preservation in synthesized speech.