arXiv · 2026 · Preprint
Lokesh Kumar et al. (Sony Research India) · → Paper · Demo: ✓ · Code: ?
Conditions expressive TTS prosody on hand-gesture motion extracted from video, using a multimodal Mixture-of-Experts fusion module and an explicit gesture-speech temporal alignment loss.
Problem
Neural TTS systems infer prosody from text or a reference audio prompt, but these signals under-represent the richness of how humans actually convey emphasis, rhythm, and affective intent. Prior multimodal TTS work has begun to condition on facial expressions and lip movements, and prior gesture research has focused on the reverse direction: generating gestures from speech or jointly synthesizing both modalities. Using hand gestures as an explicit conditioning signal to shape the prosody of synthesized speech, rather than the other way around, had not been directly targeted before this paper.
Method
Gesture2Speech synthesizes a waveform from three inputs: a text sequence, a reference audio sample used to establish speaker identity, and a sequence of video-derived gesture frames. Text and audio are processed through a shared encoder pathway. Reference-audio mel-spectrograms pass through a speaker encoder to produce a speaker embedding, while normalized video frames pass through a SlowFast motion encoder to produce spatiotemporal motion features. Motion features are concatenated with the broadcasted speaker embedding and fed into a Perceiver resampler to produce compressed global style tokens, projecting all modalities into a shared 1024-dimensional latent space.
Three modality-specific sparse Mixture-of-Experts (MoE) modules then transform this representation: a Speech MoE operating on the speaker embedding, a Video MoE operating on motion features, and a Global (style) MoE operating on the Perceiver-derived style tokens. Each MoE module uses top-2 gated expert routing with 8 or 16 experts (four-layer feedforward networks with Leaky ReLU), plus a Hierarchical MoE (H-MoE) variant with num_experts=(4, 4) for additional modality-specific routing. Separately, 2D gesture keypoints extracted with OpenPose are flattened and linearly projected into a gesture token sequence. The fused style representation, gesture tokens, and text embeddings are concatenated and fed to an LLM-based autoregressive decoder (30 transformer layers, hidden size 1024, 16 attention heads) that attends over all three via cross-attention; a HiFi-GAN vocoder converts the resulting token sequence into the 24kHz waveform. The architecture builds on the XTTS-V2 backbone, extending it with the multimodal MoE fusion pipeline.
Training combines a text cross-entropy loss, a mel distortion loss, a duration loss, and a novel gesture-speech alignment loss based on Cross-Modal Temporal Distance (CMTD): the mean absolute error between predicted speech durations (from decoder stop-token positions) and gesture apex times (midpoints of high-magnitude motion peaks). Models are trained from scratch on an A100 80GB GPU for 100 epochs, batch size 48, Adam optimizer at learning rate 5e-6; inference uses a condition-dropping probability of 0.1 and temperature 0.7.
Key Results
On a 90:10 split of a five-speaker PATS subset (17,747 clips, ~34.1 hours), the proposed Multimodal MoE variant outperforms both an XTTS-V2 baseline and a GPT-SoVITS baseline, as well as ablated Unimodal-MoE and Hierarchical-MoE variants, across most objective metrics in both same-text and different-text evaluation settings. In the different-text setting, Multimodal MoE improves gesture offset by 39.3% and gesture-audio mutual information by 79.9% relative to H-MoE, though H-MoE has marginally better WER (0.62%) and CER (0.40%) in the same-text setting (§Objective Evaluations, Table 1). In a 30-participant subjective study (720 rated samples per subject, 0-100 scale), Multimodal MoE improves speech quality by ~7.5% and prosodic similarity by ~9.1% over the XTTS-V2 baseline, and scores 10.8-10.9% higher than H-MoE on both dimensions (§Subjective Evaluations, Table 2). A fusion-strategy ablation shows MoE fusion outperforming both cross-attention and concatenation fusion, reducing gesture offset by 9.9% and 26.4% respectively and improving UTMOS by 8.3% and 19.7% respectively (§Ablation Experiments, Table 4).
Novelty Assessment
The core novelty is in the conditioning signal and the loss, not the backbone: the paper repurposes an existing zero-shot TTS architecture (XTTS-V2) and existing MoE machinery (drawing on Switch Transformer and V-MoE routing patterns), and its contribution is applying that machinery to a new input modality — gesture — for prosody control, together with a purpose-built cross-modal alignment loss (CMTD) that has not appeared in prior TTS work. The three custom evaluation metrics (Gesture Offset, Gesture-Audio Mutual Information) are also introduced specifically for this paper’s setting. The engineering is nontrivial (multimodal fusion of text, audio, and video inputs) but the evaluation is narrow: a single dataset, five speakers, and a non-archival workshop venue, so the generality of the approach beyond this controlled setting is not yet demonstrated.
Field Significance
moderate — Gesture2Speech opens a conditioning modality for TTS prosody control, hand gestures, that prior multimodal TTS work had not directly explored (prior work used facial expressions, lip movement, or treated gesture generation as the output rather than the input). It demonstrates that a gesture-speech alignment loss measurably improves temporal synchrony between motion and prosody on a controlled benchmark, providing an initial existence proof rather than a broadly validated result.
Claims
- supports: Conditioning TTS prosody on an auxiliary non-linguistic, non-audio modality can be integrated into an existing zero-shot TTS backbone via cross-attention without retraining the backbone from a different architecture family.
Evidence: Gesture2Speech extends the XTTS-V2 architecture by concatenating gesture tokens with text and fused style representations before the LLM decoder’s cross-attention, rather than redesigning the decoder. (§Proposed Architecture: Gesture2Speech TTS)
- supports: An explicit cross-modal temporal alignment loss between a conditioning signal and predicted speech timing can improve the measured synchrony between that signal and the generated prosody.
Evidence: Adding the CMTD-based gesture-speech alignment loss as part of the joint training objective yields a lower gesture offset and higher gesture-audio mutual information for the Multimodal MoE variant compared to ablated variants without the full multimodal fusion. (§Gesture-Speech Alignment Loss, Table 3)
- supports: Modality-specific expert routing (Mixture-of-Experts) can outperform simpler fusion strategies such as cross-attention or concatenation when combining heterogeneous conditioning modalities in TTS.
Evidence: The MoE Fusion strategy reduces gesture offset by 9.9% and increases mutual information by 171.7% relative to cross-attention fusion, and improves UTMOS by 19.7% relative to concatenation fusion, in a controlled fusion-strategy ablation. (§Ablation Experiments, Table 4)
- complicates: Improvements from richer multimodal conditioning are not uniform across all metrics; a simpler variant can still outperform a more complex one on specific dimensions.
Evidence: In the same-text evaluation setting, the Hierarchical MoE variant achieves marginally better WER (0.62%) and CER (0.40%) than the proposed Multimodal MoE variant, even though Multimodal MoE wins on gesture-alignment and perceptual metrics overall. (§Objective Evaluations)
Limitations and Open Questions
The system is trained and evaluated on a single dataset (PATS) restricted to five speakers and 4-15 second clips, so it is unclear whether the gesture-conditioning mechanism and alignment loss generalize to unseen speakers, longer utterances, or more culturally and emotionally diverse gesture styles; the authors explicitly acknowledge PATS's limited cultural and emotional scope.
The paper also notes that its framework relies on upper-limb/hand pose keypoints derived from full pose tracking, which may not always be available in real-world deployment (e.g., without full-body visibility or high-resolution hand tracking). The relationship between gesture intensity/timing and prosodic prominence is noted as complex and speaker-dependent, meaning gesture cues may not correlate with prosodic prominence for all speakers. The paper is a non-archival workshop submission (AAAI 2026 BEEU workshop), and total model parameter count is not reported.
Wiki Connections
- Prosody Control — introduces hand-gesture motion as an explicit, temporally-aligned conditioning signal for prosody, distinct from text-derived or reference-audio-derived prosody control mechanisms.
- Evaluation Metrics — proposes two new objective metrics, Gesture Offset and Gesture-Audio Mutual Information, purpose-built to quantify cross-modal temporal alignment between gesture and prosody.
- Subjective Evaluation — reports a 30-participant listening study rating speech quality and prosodic similarity on a 0-100 scale across five model variants.
- XTTS — used both as the architectural backbone that Gesture2Speech extends and as one of two baseline systems compared against.
- HiFi-GAN — used as the vocoder that converts the LLM decoder’s output token sequence into the final waveform.
- Natural TTS Synthesis — cited as an early unimodal approach to prosody control from text or reference audio that the paper positions its multimodal gesture-conditioning approach against.
- Seamless — supplies the AutoPCP prosodic-similarity metric used to evaluate the synthesized speech against reference audio.