arXiv · 2026 · Preprint

Ye et al. (Fudan University) · → Paper · Demo: ✓ · Code: ✓

HiCoDiT is a masked discrete diffusion transformer for video-to-speech that routes lip motion and facial identity to the coarse RVQ token levels and facial emotion to the fine levels, using a dual-scale AdaLN for prosody conditioning.

Problem

Video-to-speech (VTS) must recover speech from a silent video, but visual features are sparse relative to the dense acoustic signal. The paper argues that prior VTS systems inject visual cues into a holistic speech representation (mel-spectrograms or flat token sequences), so lip content, face-derived timbre, and emotion-derived prosody are not matched to the speech attributes they actually correspond to. Contemporary baselines such as FTV (flow matching over mel-spectrograms with a hierarchical visual encoder) and AlignDiT (continuous diffusion) place the hierarchy in the visual encoder rather than in the speech representation itself.

Method

HiCoDiT casts VTS as hierarchical masked token prediction. Speech is tokenized with the pretrained 12-level RVQ codec from MaskGCT (MaskGCT); tokens are corrupted with the absorbing-state discrete diffusion of SEDD and a score network predicts concrete scores to unmask them. A preliminary analysis (§4.1, Figure 2) decodes cumulative RVQ levels and reports that semantic fidelity and timbre similarity improve mostly in low levels while prosody quality improves in high levels. On that basis the tokens are split into low-level (levels 1-2) and high-level (levels 3-12) groups, handled by 8 low-level and 8 high-level transformer blocks (channel dimension 768, 12 heads), with 12 linear score heads.

Visual conditioning is disentangled by speech level. Lip features from AV-HuBERT (a self-supervised audio-visual model) are projected and concatenated with the masked low-level tokens for frame-synchronous content. Face identity from ArcFace is mapped by an MLP and trained with an L1 loss (weight 100) toward GE2E speaker embeddings, then modulates the low-level blocks through AdaLN for timbre. Facial emotion classes from Poster2 are temporally smoothed over 0.5 s windows and drive the high-level blocks through a dual-scale AdaLN: a temporal MLP predicts per-frame scale (local prosody dynamics) and a channel MLP predicts channel scale and shift from pooled emotion features (global vocal style).

Overall framework of HiCoDiT. We formulate video-to-speech generation as a hierarchical masked token prediction task. Speech is tokenized using an RVQ codec and split into low-level components x r1:r2 t and high-level components x r3:r12 t, reflecting the intrinsic hierarchy of speech tokens. Guided by this structure, we disentangle visual features from the input video V into lip motion c lip, identity c id, and emotion c emo, and inject them into the corresponding diffusion blocks. Finally, score heads take output features h low t and h high t from both blocks to predict concrete scores of all level tokens for unmasking.

Training minimizes a summed denoising score entropy loss over the 12 levels plus the identity loss, for 200k iterations at batch size 32 with AdamW, on a filtered VoxCeleb2 subset (261.5 h, 169k utterances, 3,438 speakers; English filtering, diarization, ClearerVoice enhancement, and text-speech alignment filtering). Inference uses Euler sampling with 64 steps and multi-condition classifier-free guidance. During training, ground-truth acoustic features replace the identity and emotion conditions, while only visual features are used at inference (§4.4). Model size is not reported.

Key Results

On LRS3 (Table 1, not trained on LRS3), video-only HiCoDiT reaches WER 29.41, DNSMOS 3.50, UTMOS 3.84, MCD 9.62, LSE-C 7.15, EmoAcc 79.41, and SpkSim 0.5678, against FTV at WER 30.37, UTMOS 3.99, SpkSim 0.5981 and AlignDiT at WER 31.37, UTMOS 3.76. Adding speech as identity guidance raises SpkSim to 0.6715 and lowers MCD to 8.69. On LRS2 (Table 2) the margin disappears on intelligibility: FTV has lower WER (38.09 vs 39.99) and higher UTMOS, while HiCoDiT leads DNSMOS, LSE-C, and LSE-D. In the 20-rater study (Table 3), HiCoDiT scores MOS-nat 3.17 and MOS-sync 3.50 versus FTV 2.80 and 3.48, but MOS-exp 2.88 is slightly below FTV (2.90). A/B preference is 57.0% over AlignDiT and 52.1% over FTV (Table 4). On 160 CinePile film utterances (Table 6), WER is 58.7 versus 80.8 for AlignDiT and 88.3 for EmoDubber.

Note

The baselines differ in training data and provenance: LTBS and DiffV2S numbers are copied from their papers, EmoDubber was retrained by the authors, and AlignDiT and FTV were run from released models or samples. The subjective study uses only 30 samples and 20 raters with large standard deviations (about 1 MOS point).

Novelty Assessment

The novelty is the combination of two known pieces, a masked discrete diffusion model over RVQ tokens and level-specific conditioning, together with the dual-scale AdaLN. Assigning coarse RVQ levels to content and timbre and fine levels to prosody is a reasonable prior supported by the paper’s own level-wise decoding analysis (Figure 2), and the “w/o hierarchical modeling” ablation (Table 5) degrades most metrics. The split point (2 vs 10 levels) is not ablated, and visual emotion is reduced to a smoothed class label, so the “prosody” signal is coarse. The dual-scale AdaLN ablation gives small differences (for example WER 29.60 vs 29.41) and UTMOS is actually higher without it (3.92 vs 3.84).

Field Significance

Moderate. The paper gives evidence that speech-token hierarchy can serve as a conditioning prior in VTS, a discrete-token alternative to the mel-spectrogram flow-matching and continuous-diffusion baselines it compares against. The gains are mixed across metrics and datasets, so the result is better read as a plausible design direction than a clear across-the-board advance.

Claims

  • supports: Matching conditioning signals to the speech-token hierarchy they best explain (content and timbre to coarse levels, prosody to fine levels) improves discrete-token speech generation from non-acoustic inputs.

    Evidence: Collapsing the low-level/high-level split and injecting all conditions across all tokens worsens WER (30.65 vs 29.41), DNSMOS, MCD, and EmoAcc on LRS3, with the same pattern on LRS2. (§5.4, Table 5)

  • supports: The coarse-to-fine levels of an RVQ codec carry separable speech attributes, with semantics and timbre concentrated in early levels and prosody gains in later levels.

    Evidence: Cumulative decoding across token levels shows semantic fidelity and timbre similarity improving in lower layers while prosody quality improves in higher layers (single analysis on one codec, no per-level numbers in the text). (§1, Figure 2)

  • complicates: Discrete token generation from silent video is more robust than mel-based generation on intelligibility and sync, but does not uniformly beat continuous baselines on perceptual quality and speaker similarity.

    Evidence: HiCoDiT trails FTV on UTMOS (3.84 vs 3.99) and SpkSim (0.5678 vs 0.5981) on LRS3 and on WER and UTMOS on LRS2 (39.99 vs 38.09; 3.68 vs 3.88), although it leads DNSMOS, LSE-C, and MCD on LRS3. (§5.2, Table 1, Table 2)

  • supports: Aligning a face-derived identity embedding to a speaker-verification embedding during training is necessary for preserving speaker identity in face-conditioned speech generation.

    Evidence: Removing the GE2E L1 alignment loss drops SpkSim from 56.78 to 34.10 on LRS3 with WER unchanged. (§5.4, Table 7)

  • complicates: Facial emotion class labels are a coarse proxy for speech prosody, and visual-to-acoustic conditioning quality depends on using privileged acoustic conditions during training.

    Evidence: Identity and emotion conditions are replaced by ground-truth acoustic features during training while only visual features are used at inference, and the emotion signal is a smoothed per-class sequence. (§4.2, §4.4)

Limitations and Open Questions

Training uses ground-truth acoustic identity and emotion features as conditions but inference uses visual ones, so a train-inference mismatch exists; the paper does not quantify its effect. Speaker similarity lags FTV, which the authors attribute to limited training-data diversity, and naturalness MOS on real film data is not reported (Table 6 gives objective metrics only). WER remains high in absolute terms (about 29 to 59 depending on test set; ground-truth speech is 2.29 on LRS3). The subjective study is small (20 raters, 30 samples), the RVQ split point is not ablated, and the emotion adapter relies on a seven-class facial expression recognizer that may transfer poorly to non-acted faces. Model size and inference latency are not reported.

Wiki Connections

  • Diffusion TTS — masked discrete diffusion (SEDD-style) over RVQ tokens is used as the generative backbone for speech generation from video.
  • Emotion Synthesis — facial expression classes condition the high-level token blocks through a dual-scale AdaLN (temporal and channel components), and EmoAcc is evaluated against baselines.
  • Self-Supervised Speech — AV-HuBERT features supply the lip-synchronized content conditioning.
  • Subjective Evaluation — MOS for naturalness, expressiveness, and synchronization plus A/B preference tests with 20 raters.
  • MaskGCT — provides the pretrained 12-level RVQ codec used to tokenize and decode speech.
  • EnCodec — cited as an example of RVQ-based neural codec background for the token hierarchy.
  • UTMOS — used as an objective naturalness metric in all comparison tables.
  • ECAPA-TDNN — speaker encoder used to compute SpkSim.
  • emotion2vec — cited alongside EmoBox as part of the speech emotion recognition used to compute EmoAcc.