arXiv · 2025 · Preprint
Kim, Ahn, Kwak, Chung & Watanabe (KAIST and Carnegie Mellon University) · → Paper · Demo: ✓ · Code: ?
A unified framework that jointly generates synchronized interactive talking/listening-head video and multi-turn conversational speech directly from a text dialogue and reference images, using two cross-modal mappers to exchange complementary information between the video and speech generation pipelines and avoid the error accumulation of a cascaded TTS-plus-talking-head approach.
Problem
Talking-head generation, listening-head generation, and conversational speech generation have each advanced substantially, but almost entirely in isolation: talking/listening-head systems assume a pre-recorded or TTS-cascaded audio track exists, and conversational speech generation systems have no visual dimension at all. Cascading a separate TTS system into a talking-head pipeline introduces error accumulation and requires a separate mechanism (such as acoustic prompting) to keep the synthesized voice and the visual identity consistent, and no prior text-driven approach jointly models the full dyadic structure of a two-person conversation (natural turn-taking, listener responsiveness, and speech overlaps) across both modalities simultaneously.
Method
TAVID first tokenizes an input text dialogue into dual-stream semantic tokens (one stream per participant) via an autoregressive text-to-semantic module (an encoder-decoder architecture trained with a next-token cross-entropy objective, following CoVoMix), which serve as a shared conditioning representation for both the video and speech generation pipelines. The speech pipeline then converts these semantic tokens into a mixed mel-spectrogram using a flow-matching acoustic denoiser, conditioned on per-stream speaker embeddings injected via a DSLN module; at training time these speaker embeddings are extracted directly from target audio, while at inference they are instead predicted from a reference image by the Speaker Mapper, so no reference audio is required to control voice identity. The video pipeline (built on Hallo2, a latent-diffusion UNet denoiser with a parallel ReferenceNet) generates the interactive video conditioned on reference spatial features, a face embedding, and interactive motion features produced by the Motion Mapper from the same dual-stream semantic tokens; because prosody strongly correlates with facial dynamics, the semantic tokens themselves are chosen to be prosody-aware (extracted from XLS-R rather than a purely-linguistic encoder like HuBERT). The two cross-modal mappers are the paper’s central architectural contribution: the Motion Mapper uses a joint self-attention mechanism (inspired by MMDiT) to translate the two semantic streams into interactive motion features that capture both stream-specific behavior and cross-stream turn-taking dependencies, while the Speaker Mapper combines the face embedding with intermediate ReferenceNet spatial features to predict a speaker embedding that aligns synthesized voice characteristics with the visual identity in the reference image, trained with an L2 loss against audio-extracted speaker embeddings.
Key Results
On interactive head generation (Seamless Interaction test set), TAVID substantially outperforms the only comparable public method (DIM, both audio-driven and TTS-cascaded variants) across subjective visual quality, lip-sync, and turn-taking scores (all roughly 3.75-3.84 vs. 2.0-2.4 for DIM) and every objective metric tested. On single-role talking-head generation (HDTF), TAVID achieves the best FID and FVD among SadTalker, Echomimic, and Hallo2 baselines, with lip-sync comparable to the best audio-driven system, and again shows that TTS-cascaded versions of the baselines suffer measurable degradation relative to TAVID’s direct text-driven approach. On listening-head generation (ViCo), TAVID achieves the best motion-realism (FD) and pose-variance-matching scores against L2L, RLHG, DIM, INFP, and ARIG, with a comparable (though not best) diversity score. On face-stylized speech generation (VoxCeleb2, unseen test speakers), TAVID achieves the best naturalness MOS (4.20) and UTMOS (3.53) among all compared methods, including audio-driven zero-shot TTS baselines YourTTS and CoVoMix and face-driven baselines Face-TTS and FVTTS; it trails YourTTS and CoVoMix on speaker-similarity (VoxSim), which the authors attribute to those methods extracting speaker embeddings directly from the target utterance rather than predicting them from a reference image, but achieves the best speaker similarity among face-driven methods and the best face-matching MOS of any method, including the audio-driven ones.
Novelty Assessment
TAVID’s individual pipeline components are largely adapted from prior work: the video pipeline builds directly on Hallo2’s latent-diffusion architecture, and the speech pipeline’s text-to-semantic and flow-matching acoustic denoiser design follows CoVoMix. The genuine architectural contribution is the pair of cross-modal mappers that create a bidirectional information pathway between the two otherwise-separate pipelines: the Motion Mapper’s joint self-attention design (validated against three alternative fusion strategies in ablation) and the Speaker Mapper’s use of complementary face-embedding and ReferenceNet features to predict voice characteristics from a static image rather than from reference audio. The paper’s own framing, first joint text-driven generation of both interactive video and conversational speech, appears accurate relative to the related work discussed, which treats video and speech generation as separate research threads.
Field Significance
moderate — TAVID’s main contribution is a working demonstration that video and speech generation pipelines can be coupled through learned cross-modal mappers rather than a rigid TTS-then-animate cascade, with the Speaker Mapper specifically offering a reusable mechanism for predicting voice identity from a face image alone. Its primary evaluation focus and strongest claimed advances are in the video/multimodal-interaction space (talking-head, listening-head, dyadic turn-taking realism); the speech-generation contribution, while genuinely evaluated with real TTS baselines and metrics, is one of four evaluated dimensions rather than the paper’s central focus.
Claims
- supports: A speaker’s vocal identity can be predicted directly from a static reference face image, without any reference audio, by combining a face-recognition embedding with intermediate spatial features from a face-generation network, and used to condition zero-shot speech synthesis with speaker similarity that improves on prior face-driven approaches.
Evidence: The Speaker Mapper, trained with an L2 loss against audio-extracted speaker embeddings, achieves the best VoxSim speaker-similarity score among face-driven baselines (Face-TTS, FVTTS) and the best face-matching MOS among all compared methods, including audio-driven ones, on the VoxCeleb2 test set with unseen speakers. (§5.3, §5.4, Table 4, Figure 5)
- complicates: Predicting a target speaker’s voice from a reference image rather than from reference audio trades off objective speaker-similarity accuracy relative to methods that extract speaker embeddings directly from a target utterance, even when perceived naturalness and face-matching are improved.
Evidence: TAVID achieves the best naturalness MOS and UTMOS among all compared speech generation methods, but trails audio-driven YourTTS and CoVoMix on the objective VoxSim speaker-similarity metric, since those methods inject vocal characteristics directly from the target utterance rather than predicting them from an image. (§5.3, Table 4)
- supports: Jointly conditioning video and speech generation on prosody-aware semantic tokens, rather than purely linguistic ones, improves the accuracy and expressiveness of generated speech-synchronized facial motion, including lip synchronization specifically.
Evidence: An ablation replacing the prosody-aware XLS-R-derived semantic tokens with HuBERT-based tokens (known to retain linguistic but not prosodic information) degrades lip-sync accuracy with a notable margin and overall motion realism in both talking-head and listening-head generation. (§5.4, Table 5, Table 6)
- complicates: Cascading independently-trained TTS and talking/listening-head generation systems introduces measurable quality degradation relative to a jointly-designed text-driven pipeline, even when the same underlying video generation backbone is used in both configurations.
Evidence: TTS-cascaded variants of baseline systems (DIM, SadTalker, Echomimic, Hallo2) consistently underperform their audio-driven counterparts and TAVID’s direct text-driven approach across objective video-quality metrics on both the Seamless Interaction and HDTF test sets. (§5.1, §5.2, Table 1, Table 2)
Limitations and Open Questions
The paper does not report a head-to-head comparison of overall system latency or inference cost against cascaded TTS-plus-talking-head pipelines, so the practical deployment trade-off of the joint architecture relative to cascading is not directly quantified. Speech-generation evaluation is conducted on a single dataset (VoxCeleb2) with English speakers, leaving open whether the Speaker Mapper’s face-to-voice mapping and the joint pipeline’s speech quality generalize across languages or more diverse voice populations. Because TAVID is, by the authors’ own account, the first text-driven interactive head generation system, most video-generation baselines it is compared against are audio-driven or TTS-cascaded adaptations of existing systems rather than genuinely comparable text-driven alternatives, which somewhat limits how much of its video-quality advantage can be attributed specifically to the joint cross-modal design versus simply avoiding a TTS cascade.
Wiki Connections
- Speech-to-Speech Systems — generates synchronized dyadic conversational speech with natural turn-taking and speech overlaps directly from a text dialogue script, conditioned through a shared dual-stream semantic-token representation.
- Zero-Shot TTS — the Speaker Mapper enables zero-shot voice generation for unseen speakers using only a static reference face image in place of the reference audio typically required by zero-shot TTS systems.