Speech Generation Research Wiki

SLD-L2S: Hierarchical Subspace Latent Diffusion for High-Fidelity Lip to Speech Synthesis

10 min read

arXiv · 2026 · Preprint

Yifan Liang et al. (Institute of Acoustics, Chinese Academy of Sciences) · → Paper · Demo: ? · Code: ?

SLD-L2S maps silent lip video directly to the continuous latent space of a pre-trained neural audio codec via a hierarchical subspace latent diffusion model, avoiding the mel-spectrogram or discrete-token intermediate representations used by prior lip-to-speech systems.

Note on task tagging: Lip-to-speech (L2S) synthesis generates speech from silent video (lip movements); the input modality is visual, not text. The canonical vocabulary has no dedicated term for this non-text-input generation task, so it is tagged TTS for lack of a better fit, consistent with the same gap flagged for interspeech-2025-1478 and interspeech-2025-1334.

Problem

Prior state-of-the-art lip-to-speech (L2S) systems generate an intermediate representation, typically mel-spectrograms or discrete self-supervised-learning (SSL) tokens, before producing a waveform. Both choices lose information that matters for high-fidelity synthesis: mel-spectrograms are a fixed, physically defined representation that cannot flexibly capture the one-to-many mapping from a single lip sequence to multiple valid prosodies, emotions, and speaking styles, while discrete SSL tokens are coarse and may discard fine acoustic detail. Neural audio codecs have proven effective as an intermediate representation in text-to-speech (e.g. VALL-E), but existing attempts to bring codecs into L2S (Uni-Dubbing) frame the task as predicting discrete residual tokens autoregressively, which the authors argue is poorly suited to L2S because the visual input is too information-sparse to support a complex language-modeling target. This leaves open whether continuous codec latents, rather than discrete tokens, can be generated directly from visual input.

Method

SLD-L2S extracts visual features from silent mouth-region video using the frozen visual frontend of AV-HuBERT LARGE (pretrained on LRS3 and VoxCeleb2), yielding 1024-dimensional per-frame representations that are temporally upsampled to match the frame rate of a pre-trained neural audio codec’s latent space (X-Codec-hubert, 16 kHz, pretrained on LibriSpeech).

These upsampled visual features are decomposed by a subspace decomposition module (SDM) into 8 parallel convolutional subspaces (each a layer norm plus 1D convolution), which are concatenated and passed to a backbone of 12 diffusion convolution blocks (DiCB). DiCB is proposed as a convolutional alternative to the self-attention-based diffusion transformer (DiT): its convolutional attention module uses a depthwise convolution (kernel 5×7, time × subspace) combined via a Hadamard product with a value projection to model local and cross-subspace dependencies, followed by a convolutional feedforward module for channel mixing. Time step and a 256-dimensional GE2E speaker embedding (extracted from a reference utterance) condition the network throughout via AdaLN-SOLA, a shared adaptive layer-norm module with a low-rank adjustment (rank 32) that reduces parameter overhead relative to per-block AdaLN. A subspace recomposition module (SRM) then projects and fuses the 8 subspace streams (128 dimensions each) back into a single 1024-dimensional vector via a 1D convolution, reshape, and a 3-layer ConvNeXt block, producing the target codec-latent format.

The model utilizes a latent flow matching approach to directly map visual features to the continuous acoustic latent space. The architecture consists of three main stages: a subspace decomposition module processes visual features into parallel pathways; a backbone of DiCBs, conditioned with AdaLN-SOLA, effectively refines these representations; and a subspace recomposition module fuses them into a final latent vector. The model is trained with a multi-objective function, including a flow matching loss for acoustic reconstruction, a semantic loss on the latent space, and an SLM loss on the final synthesized waveform. During inference, the model generates the latent vectors for the codec to synthesize into speech.

Training uses a reparameterized flow matching objective: instead of regressing the noise-dependent velocity field directly, the model predicts the target codec latent point itself, which the authors report stabilizes training and enables auxiliary losses computed in data space. Two auxiliary losses are added to the reparameterized flow matching loss: an SLM loss that decodes both the predicted and target latents to waveform via the codec decoder and penalizes the L2 distance between WavLM-derived features, and a semantic loss that penalizes the L2 distance between HuBERT-derived semantic features reconstructed (via a separately pretrained semantic decoder) from the predicted and target latents. At inference, 10 Euler ODE steps generate the latent vectors, which the codec decoder converts to the final waveform. The model is trained on LRS3-TED for 150K iterations (batch size 16, AdamW, single H100 GPU, roughly five days).

Key Results

On LRS3-TED, SLD-L2S achieves UTMOS 4.21 and SCOREQ 4.61, exceeding the strongest flow-matching baseline (V2SFlow: UTMOS 3.69, SCOREQ 4.07) by a wide margin while using only 10 inference steps versus V2SFlow’s 30. On the held-out LRS2-BBC test set (not used in training), the pattern holds: UTMOS 4.17 and SCOREQ 4.57 versus V2SFlow’s 3.46/3.92 (Table 1). WER (30.22 on LRS3-TED) is not the best among compared systems: LipVoicer, which relies on a dedicated pretrained visual-speech-recognition front end, achieves WER 20.62 but with substantially lower UTMOS (2.45) and SCOREQ (2.66). On speaker similarity (SECS), V2SFlow scores highest (0.851) because it includes an explicit speaker-embedding loss during training; SLD-L2S is second-best (0.804) without such a loss.

In a 15-participant subjective MOS test (Table 2), SLD-L2S scores 4.17 ± 0.24 on naturalness, ahead of all compared systems and close to ground truth (4.36 ± 0.15); intelligibility MOS is second-best at 3.65 ± 0.25 (behind V2SFlow’s 3.69 ± 0.25); speaker-similarity MOS is 3.77 ± 0.15, close to but below V2SFlow’s 3.90 ± 0.14. Ablations (Table 3) show that replacing DiCB with a parameter-matched DiT (57.8M vs. 57.7M parameters) collapses performance across nearly all metrics, and reverting to the standard (non-reparameterized) flow matching objective drops UTMOS from 4.21 to 3.95 and SCOREQ from 4.61 to 4.26. Removing the SLM loss slightly improves UTMOS/SCOREQ but degrades D-BERT and SECS; removing the semantic loss causes a smaller, more uniform degradation. A separate ablation over the number of subspaces (Table 5) shows a UTMOS peak at 8 subspaces (4.21), declining at both 4 subspaces (4.18) and 32 subspaces (4.13).

Novelty Assessment

The core novelty is architectural: DiCB is a genuinely new convolutional backbone for diffusion/flow-matching-based generation, motivated as a substitute for DiT-style self-attention in a setting with strong local and cross-subspace structure, and the subspace decomposition/recomposition design is specific to this paper. The reparameterized flow matching objective (predicting the data point rather than velocity) is adapted from a concurrent trajectory-forecasting paper (Fu et al. 2025) rather than invented here, but its combination with auxiliary SLM and semantic losses operating in data space is a novel training-recipe contribution for L2S. The overall framing, generating continuous codec latents directly from visual input rather than discrete tokens or mel-spectrograms, extends an approach already established in TTS (NaturalSpeech 2) to a new modality pairing (visual-to-audio) where it had not previously been demonstrated to work at this quality level. The gains are validated primarily within the narrow LRS3/LRS2 lip-to-speech benchmark setting rather than against a broader class of speech generation tasks.

Field Significance

Moderate — this paper demonstrates that generating continuous neural-audio-codec latents directly from visual input, rather than through a mel-spectrogram or discrete-token intermediate representation, produces state-of-the-art reference-free quality scores on established lip-to-speech benchmarks. It contributes a convolution-based alternative to DiT backbones for cross-modal latent generation and shows that a reparameterized flow matching objective can accommodate auxiliary perceptual losses computed in data space. Its evaluation and claims are confined to the lip-to-speech subfield rather than establishing a technique validated across the broader speech generation literature.

Claims

  • supports: Reparameterizing a flow matching objective to directly predict the target data point, rather than the noise-dependent velocity field, stabilizes training and enables auxiliary perceptual losses that operate in data space for continuous-latent speech generation.

    Evidence: Reverting SLD-L2S to the standard velocity-prediction flow matching objective drops UTMOS from 4.2096 to 3.954 and SCOREQ from 4.6075 to 4.26 on the LRS3-TED ablation, while WER improves only marginally (30.22 to 29.7). (§Ablation Study, Table 3)

  • supports: Convolution-based attention modules can substitute self-attention transformer (DiT-style) backbones in diffusion or flow-matching models operating over decomposed, cross-modal latent representations.

    Evidence: Replacing the proposed DiCB backbone with a parameter-matched DiT (57.8M vs. 57.7M parameters) causes a performance collapse across nearly all objective metrics on LRS3-TED. (§Ablation Study, Table 3)

  • complicates: Directly generating continuous neural-audio-codec latents from visual input can achieve state-of-the-art reference-free perceptual quality without matching the word-error-rate of systems that incorporate a dedicated pretrained visual-speech-recognition front end.

    Evidence: SLD-L2S outperforms all compared lip-to-speech systems in UTMOS and SCOREQ on both LRS3-TED and LRS2-BBC, but LipVoicer (which uses a pretrained visual-speech-recognition model for semantic guidance) achieves a lower WER (20.62 vs. 30.22 on LRS3-TED) despite far lower UTMOS and SCOREQ scores. (§Results and Discussion, Table 1)

  • complicates: An auxiliary perceptual loss computed via a pretrained speech language model can trade off against reference-free acoustic-quality metrics even as it improves content and speaker-identity preservation.

    Evidence: Removing the SLM loss from SLD-L2S’s training objective slightly improves UTMOS (4.2885 vs. 4.2096) and SCOREQ (4.6958 vs. 4.6075) but degrades speaker similarity (SECS 0.776 vs. 0.804) and content similarity (D-BERT 0.8207 vs. 0.8284) on LRS3-TED. (§Ablation Study, Table 3)

  • refines: In hierarchical subspace decomposition of cross-modal representations for latent generation, the number of parallel subspaces has a non-monotonic effect on synthesis quality, with both too few and too many subspaces degrading fidelity relative to an intermediate optimum.

    Evidence: On the LRS3-TED subspace-count ablation, UTMOS peaks at 4.2096 with 8 subspaces and declines to 4.1797 at 4 subspaces and 4.1288 at 32 subspaces. (§Analysis on the design choices of DiCB, Table 5)

Limitations and Open Questions

The paper’s word-error-rate results (30.22% on LRS3-TED, 39.54% on LRS2-BBC) trail the best-performing baseline (LipVoicer, WER 20.62%/24.92%) by a substantial margin, and the authors attribute this to LipVoicer’s reliance on a dedicated pretrained visual-speech-recognition model that SLD-L2S does not use; this is acknowledged as a trade-off rather than resolved. Speaker similarity (SECS 0.804 on LRS3-TED) also trails V2SFlow (0.851), which the authors attribute to V2SFlow’s explicit speaker-embedding training loss, a component SLD-L2S does not include. Evaluation is confined to two English-language benchmarks (LRS3-TED, LRS2-BBC); generalization to other languages, accents, or noisy in-the-wild video is untested. The system depends on a reference utterance at inference time to extract the GE2E speaker embedding, meaning an enrollment clip must be available. Neither code nor a demo availability is stated in the paper.

Wiki Connections

  • Flow Matching — SLD-L2S trains its DiCB backbone with a reparameterized flow matching objective that predicts the target codec latent directly rather than the velocity field, extending flow matching to accommodate auxiliary data-space losses.
  • Neural Audio Codec — the framework’s central design choice is generating continuous latent vectors of a pre-trained neural audio codec (X-Codec-hubert) directly from visual input, avoiding the discrete-token or mel-spectrogram intermediates used by prior lip-to-speech systems.
  • Self-Supervised Speech — the visual frontend is the frozen encoder of AV-HuBERT LARGE, an audio-visual self-supervised model, and auxiliary training losses draw on HuBERT and WavLM self-supervised feature spaces.
  • Subjective Evaluation — a 15-participant MOS test rates naturalness, intelligibility, and speaker similarity, revealing that SLD-L2S’s advantage in reference-free objective quality metrics does not fully translate to the largest subjective margins on intelligibility and speaker similarity.
  • VALL-E — cited as the codec-language-model precedent for using neural audio codec tokens in speech synthesis; SLD-L2S explicitly departs from VALL-E’s discrete autoregressive token-prediction paradigm, arguing it is ill-suited to the information-sparse visual input of lip-to-speech.
  • F5-TTS — cited alongside other flow-matching-based speech generation work as evidence of flow matching’s broad applicability across domains, motivating its adoption in this paper’s L2S setting.
  • CosyVoice — cited as part of recent TTS progress demonstrating that neural audio codecs are effective intermediate representations, motivating this paper’s decision to target codec latents rather than mel-spectrograms.

Backlinks

Graph View