ACM MM · 2025 · Conference
Hu et al. (Inner Mongolia University) · → Paper · Demo: ✓ · Code: ✓
UniTalker extends conversational speech synthesis to a joint audiovisual generation task, producing both empathetic speech and temporally-aligned talking-face animations from multimodal dialogue context using a unified LLM-based prediction model with a bespoke facial landmark codec.
Problem
Conversational speech synthesis (CSS) systems respond to users with audio only, ignoring the visual channel that plays a significant role in real human communication — particularly in conveying emotion through facial expressions and eye contact. Prior CSS work extracts multi-scale textual and acoustic features from dialogue history to condition generation, but the output is unimodal speech, which limits deployment in virtual agents, digital humans, and VR applications. When talking-face animation has been added by prior systems (EmpathyEar, Empatheia), it has been cascaded from the speech signal in a secondary stage, making it unstable when speech is weak or ambiguous, and unable to use the visual dialogue history as an input cue.
Method
UniTalker frames the problem as Conversational Speech-Visual Synthesis (CSVS): given the multimodal history of a dialogue (text, speech, talking-face animations, speaker identity, and emotion labels for each prior turn) and the current turn’s speaker and text, predict the target utterance’s emotion, speech, and talking-face animations jointly.
The system is built around three components:
Multimodal Tokenization. Text is tokenized with a BPE tokenizer. Speech is tokenized using FSQ applied to intermediate encoder representations of SenseVoice-Large (following CosyVoice2), yielding semantic speech tokens at 25 Hz. Facial landmarks extracted by Mediapipe from talking-face animations are tokenized by LmkCodec, a purpose-built neural codec consisting of a convolutional encoder, an FSQ-down linear projection, a rounding quantizer, an FSQ-up linear, and a convolutional decoder. LmkCodec operates at 25 Hz with a codebook of size 1000, matching the speech tokenizer’s rate and enabling frame-level alignment. Speaker identity is encoded as a continuous embedding from a voiceprint model.
EVSLM (Emotion-Visual-Speech Language Model). Built on Qwen2.5-0.5B, EVSLM takes the serialized multimodal context and sequentially predicts: (1) the target emotion token, (2) interleaved facial expression and speech tokens via a bimodal hard alignment strategy. The interleaving enforces that visual and speech tokens are predicted in alternating frame-level pairs at 25 tokens per second for each modality, guaranteeing equal-length sequences and synchronized lip movements by construction. Training uses cross-entropy over all predicted token types jointly.
Rendering. The predicted speech tokens are decoded to mel spectrograms using emotion-conditioned Conditional Flow Matching (CFM) — a causal convolutional Transformer UNet conditioned on speaker embedding, emotion label, and a reference mel extracted from the speaker’s recent utterance in the dialogue context — followed by HiFi-GAN. The decoded facial landmarks, together with the synthesized speech and a reference portrait, are fed to EchoMimic (a diffusion U-Net portrait animation renderer) to produce the final video.
Training proceeds in four stages: LmkCodec is trained first on landmark reconstruction; EVSLM and the Speech Renderer are initialized from CosyVoice2 pretrained weights (trained on 170,000 hours of speech); then fine-tuned on the CSS audio-only dialogue data (113 hours); then further fine-tuned on visual dialogue data (307 hours of MultiDialog plus 89 hours of single-utterance visual data).
Key Results
On MultiDialog, UniTalker outperforms the strongest audiovisual CSS baselines (EmpathyEar, Empatheia) substantially. In speech naturalness (MOS_SN), UniTalker achieves 4.128 versus 4.015 for Empatheia — a +0.113 gain. In speech emotion expressiveness (MOS_SE), the margin is +0.111 over Empatheia. Speaker similarity (SIM_SPK) reaches 0.902, higher than all dialogue-context baselines.
For talking-face animations, UniTalker achieves the best FID (24.214), LPIPS (0.204), and SSIM (0.741) among dialogue-aware models, and the best MOS_VN (4.154), MOS_VE (4.213), and MOS_VC (4.135 for lip sync). The video quality also surpasses speech-only driven baselines such as Hallo on most perceptual metrics, including emotion accuracy (ACC_VE = 0.813 vs. Hallo’s 0.801).
Ablation results confirm that the bimodal hard alignment strategy is the most critical component: removing it degrades lip sync confidence (LSE-C) from 6.386 to 4.452 and FID from 24.214 to 59.127. The LmkCodec’s FSQ design with 1 token per frame also proves better than the 8-token-per-frame GRVQ variant, demonstrating that shorter sequences aid LLM contextual modeling even at some reconstruction cost.
On zero-shot speaker generalisation (IEMOCAP unseen speakers), UniTalker achieves SIM_SPK of 0.702, above EmpathyEar (0.612) and Empatheia (0.634).
Novelty Assessment
The core architectural novelty is LmkCodec — a compact FSQ-based neural codec for facial landmark sequences operating at the same token rate as the speech codec. This enables the interleaved bimodal alignment strategy, which is genuinely new in the conversational speech-visual synthesis context. The broader system architecture is an integration of existing components: CosyVoice2’s speech tokenizer and CFM renderer, Qwen2.5-0.5B as the base LLM, and EchoMimic as the portrait animation backend. The paper’s extension of the CSS task definition to include visual output (CSVS) is a reasonable conceptual framing but straightforward in scope — it formalizes a multimodal dialogue setting that prior work (EmpathyEar, Empatheia) had already partially explored.
The landmark codec itself is the most reusable contribution. Using FSQ rather than RVQ or GRVQ to achieve 1 token per frame at 25 Hz is a practical engineering choice supported by ablation evidence, though the reconstruction quality of LmkCodec (PSNR 26.143 at 1 token/frame) trails OmniTokenizer (30.054) due to the fundamentally different reconstruction path (landmark coordinates vs. full image pixels). The four-stage training curriculum, including CosyVoice2 warm-start, is sensible engineering but not novel.
Field Significance
Moderate — UniTalker proposes a plausible unification of conversational TTS with talking-face generation through a shared token-rate codec, demonstrating that LLM-based joint prediction of interleaved speech and visual tokens is feasible and outperforms cascaded baselines on emotion consistency and animation quality. Its contribution to the CSS sub-area is concrete; the LmkCodec design pattern (matching codec rates for alignment) may be adopted in future audiovisual spoken dialogue agents. The evaluation is thorough and the baselines are reasonable, though comparisons rely on a limited pool of predominantly audiovisual CSS systems.
Claims
- Matching the token rates of speech and facial landmark codecs enables frame-level synchronisation between synthesised speech and talking-face animations without post-hoc alignment. (§4.1.3, §4.2)
- LLM-based joint prediction of interleaved speech and visual tokens in dialogue context outperforms cascaded speech-then-video generation on both emotional accuracy and lip synchronisation. (§6.2, §6.3, Table 2, Table 3)
- Including visual dialogue history (talking-face animations of prior turns) improves emotion accuracy and speaker consistency in conversational speech synthesis relative to audio-text-only context. (§6.4, Table 4)
- Low-rate discrete tokenisation of facial landmarks (1 token per frame) is more effective for LLM contextual modelling than higher-rate representations, even at a cost in geometric reconstruction fidelity. (§6.1, §6.2, Table 1, Table 2)
- Emotion-guided conditioning of the speech renderer, including predicted emotion labels as explicit conditioning, improves measured emotional expressiveness over systems that rely on implicit contextual inference alone. (§6.4, Table 4)
Limitations and Open Questions
Warning
The Talking-face Animations Renderer (EchoMimic) is a pre-trained third-party module that receives no additional fine-tuning in this pipeline. Its outputs are constrained by the quality ceiling and biases of the EchoMimic base model, limiting the paper’s ability to attribute animation quality gains to UniTalker specifically vs. the renderer.
Rendering latency is notable: speech generation takes approximately 2 seconds and animation rendering takes approximately 5 seconds per 25 frames on an RTX 4080 with 32 GB RAM, making the system unsuitable for real-time interaction in its current form. The paper acknowledges this and lists streaming optimisation as future work.
The training data for dialogue context is limited to 113 hours of audio-only and 307 hours of visual dialogue, which is modest compared to the 170,000-hour pretraining corpus used for the speech renderer. Generalisation to diverse speaking styles and languages beyond the training distribution is untested. All subjective evaluations used 30 non-native English speakers recruited locally, raising questions about evaluation representativeness for naturalness judgements.
The CSVS task definition assumes that the target utterance’s text is known at inference time. This is a constrained setting (closer to expressive TTS than to open-ended dialogue response generation) and does not address the harder problem of jointly generating text, speech, and visual responses.
Wiki Connections
This paper advances emotion-synthesis and spoken-language-model by demonstrating that a unified LLM can jointly predict speech and facial expression tokens from a multimodal conversational context. The LmkCodec design relates to neural-codec, extending discrete tokenisation beyond audio to geometric facial representations. The speech rendering pipeline (FSQ tokens → CFM renderer → HiFi-GAN) connects to flow-matching and autoregressive-codec-tts. The zero-shot speaker identity rendering draws on zero-shot-tts capabilities transferred from CosyVoice2 pretraining. Emotion-guided generation relates to prosody-control.