arXiv · 2026 · Preprint
Yao Shi et al. (OPPO) · → Paper · Demo: ✓ · Code: ?
Proposes Vclip, a CLIP-style contrastive face-voice association model trained on noisy in-the-wild audio-visual data, and a retrieval-based speaker-generation procedure that uses it to produce plausible target speaker embeddings for a downstream zero-shot TTS system given only a reference face image.
Problem
Face-based speech synthesis aims to generate a voice that perceptually matches a reference face image, but training data that is both high-quality (TTS-grade audio) and paired with faces is scarce; the audio-visual corpora that do exist (e.g. LRS3) are noisy and unsuited to mainstream TTS training. Prior approaches either train TTS directly on this noisy paired data (limiting synthesis quality) or use transfer learning to map face embeddings onto a pretrained TTS speaker-embedding space, which introduces a domain gap: the predicted face-derived embedding is out-of-distribution for the downstream TTS system and accumulates error without feedback from the synthesis step. The paper also argues that deterministically predicting “the” true voice behind a face is intractable in principle, since a face alone underdetermines voice identity: several distinct voices can plausibly match a single face.
Method
The paper reframes face-based TTS as conditional speaker generation rather than direct face-to-voice-embedding prediction, via a two-part pipeline. Vclip is a CLIP-style contrastive model with two frozen, pretrained backbones: a CLIP image encoder for faces and an H/ASP-based speaker-verification network for voices, each followed by a trainable projection head (an MLP for the image side, a RealNVP normalizing-flow projection for the voice side) into a shared 512-dimensional embedding space. It is trained on co-occurring face-voice pairs from VoxCeleb2 (no speaker-identity labels required) with a symmetric CLIP contrastive loss.
Because contrastively aligned embeddings from different modalities occupy separate subspaces (the “modality gap”), the paper does not use the face embedding directly as a TTS speaker embedding. Instead, it models the distribution of known speaker embeddings in a downstream zero-shot TTS system’s embedding space (a vanilla multi-speaker VITS trained on LibriTTS-R) with a Gaussian mixture model, following prior speaker-generation work, and treats Vclip as a zero-shot scoring function to retrieve the top-k candidate embeddings from this distribution that best match a reference face. Two scoring variants are proposed: a naive version that scores candidates directly through Vclip’s voice projection, and a “TTS signature-informed” version that first distills the specific input-output speaker-embedding mismatch of the downstream TTS system (a systematic deviation inherent to any zero-shot voice-cloning system) into a small feedforward network, then scores retrieval candidates through that distilled signature rather than the raw embedding, to account for TTS-specific reconstruction error during retrieval.
The paper also introduces automatic evaluation metrics for face-based speaker generation: v2v (cosine similarity between a generated voice and the true voice behind the reference face, measuring voice reconstruction), f2v (cosine similarity between a generated voice and its reference face in Vclip’s association space, measuring face-voice matching), and the log-likelihood of generated speaker embeddings under the fitted GMM of known TTS speakers (measuring domain fit).
Key Results
On the VoxCeleb1 face-voice verification benchmark, Vclip reaches 89.63% AUC (open setting), surpassing the prior state-of-the-art Self-Lifting (89.4%) and Learnable PINs (84.7%), despite the paper’s own preliminary finding that supervised face-recognition models outperform CLIP on plain face recognition; the authors attribute Vclip’s advantage to CLIP’s semantically rich features being more concordant with voice characteristics than pure identity features. An ablation shows the CLIP contrastive loss requires large batch sizes (AUC drops from 88.78 to 86.54 at batch size 320) and outperforms a supervised SGE2E-loss alternative (87.14).
For speaker generation quality, all systems produce low v2v scores relative to true zero-shot voice cloning (0.13-0.25 vs. a 0.588 reference value), confirming that generated voices are not literal voice reconstructions. However, f2v (face-voice match quality) for the generate-and-retrieve approach is comparable to or exceeds the direct feature-mapping baseline (0.301-0.313 vs. 0.305 vs. a 0.307 reference), and its speaker-embedding likelihood (516-519) is far closer to real TTS speakers than the baseline’s (328.82), indicating the retrieval strategy substantially closes the domain gap that direct feature mapping leaves open. The TTS-signature-informed scoring variant achieves the best f2v score (0.313) among all tested configurations. Subjectively, naturalness MOS for the proposed system (4.14) is close to true zero-shot voice cloning (4.01-4.15) and far above an end-to-end baseline (Face-TTS) trained directly on the lower-quality LRS3 corpus (2.46); face-voice match ratings (m-MOS) for the proposed system (3.26) approach the oracle voice-clone baseline (3.33) and both exceed random face-voice pairing (2.31).
Novelty Assessment
The core novelty is narrow but genuine: applying CLIP-style contrastive learning to face-voice association (extending the Wav2CLIP/AudioCLIP lineage of CLIP-adjacent cross-modal models to a new pair of modalities), and specifically the TTS-signature-informed retrieval mechanism, which is a reasonably creative way to fold downstream-system-specific error into an otherwise-generic retrieval scoring function. The generate-and-retrieve framing itself is not new (VoiceMe used a similar idea with human-in-the-loop scoring); this paper’s contribution is replacing costly human judgment with a learned, scalable scoring function. The evaluation is thorough for the scale of the problem (AUC verification, three automatic generation-quality metrics, two subjective studies), though the paper is honest that its own baseline comparisons are not fully controlled (“works on face-based TTS has been conducted under vastly different settings, fair comparisons cannot be performed for accurately reproduced implementations”), and the reported gains over the feature-mapping baseline are real but not dramatic (f2v 0.313 vs. 0.305).
Field Significance
Moderate, this is a solid, well-evaluated contribution to a narrow, low-adoption sub-area (face-conditioned speaker generation), advancing a genuine and previously under-addressed technical problem (the domain gap between contrastively learned cross-modal embeddings and a target TTS system’s speaker-embedding manifold) with a reusable idea (distilling a downstream system’s own input-output mismatch into a scoring signal). Its direct applicability is limited to systems that specifically need face-conditioned voice personalization.
Claims
- supports: A CLIP-derived semantic image encoder, contrastively aligned with a speaker-verification voice encoder on noisy in-the-wild audio-visual data, learns face-voice association more effectively than encoders built on supervised face-recognition features.
Evidence: Vclip, using a frozen CLIP image encoder, reaches 89.63% AUC on the VoxCeleb1 face-voice verification benchmark, surpassing the prior state-of-the-art Self-Lifting (89.4% open-set AUC), despite CLIP features underperforming supervised face-recognition models on plain face recognition in the authors’ own preliminary experiments. (§4.2, Table 1)
- complicates: Directly substituting a cross-modally learned face embedding for a speaker embedding in a zero-shot TTS system produces out-of-domain speaker representations, due to a persistent modality gap between contrastively learned face and voice feature spaces.
Evidence: Using the face-projected embedding directly as the TTS speaker embedding produces a markedly lower speaker-embedding log-likelihood (328.82) relative to known TTS speakers than a generate-and-retrieval strategy sampling from the true speaker-embedding distribution (516-519), despite comparable face-voice matching (f2v) scores. (§4.3, Table 3)
- supports: Distilling a specific zero-shot TTS system’s characteristic input-output speaker-embedding mismatch into an auxiliary network, and incorporating it into cross-modal retrieval scoring, improves the correspondence between generated voices and the conditioning signal beyond what the raw cross-modal association model alone provides.
Evidence: TTS-signature-informed retrieval scoring achieves the highest face-to-voice (f2v) matching score (0.313) among tested configurations, exceeding both naive retrieval scoring (0.301) and the direct feature-mapping baseline (0.305), while maintaining a comparably high speaker-embedding likelihood. (§4.3, Table 3)
- complicates: Face-conditioned speech synthesis trained via transfer learning from noisy in-the-wild data can match the subjective naturalness of standard zero-shot voice cloning, while systems trained end-to-end on lower-quality dedicated audio-visual corpora cannot.
Evidence: Subjective naturalness MOS for the proposed transfer-learning-based system (4.14) is comparable to known-speaker zero-shot voice cloning (4.01-4.15), while a baseline trained end-to-end on the LRS3 audio-visual corpus (Face-TTS) scores markedly lower (2.46). (§4.4, Table 4)
Limitations and Open Questions
- All v2v (voice-reconstruction) scores remain far below the true zero-shot voice-cloning reference (0.13-0.25 vs. 0.588), confirming generated voices are plausible matches rather than reconstructions of the true voice behind a face; the paper frames this as expected given the underdetermined nature of the task rather than as a failure.
- An “attribute mismatch” is noted as unresolved: in-the-wild audio-visual data captures variation in language, ethnicity, and environmental factors that the LibriTTS-R-trained downstream TTS system cannot reproduce, since only attributes overlapping with LibriTTS-R’s own coverage are expressible through the speaker-embedding-centric TTS paradigm used here.
- Baseline comparisons for face-based speaker generation quality are explicitly acknowledged by the authors as not fully controlled, since prior work in this area uses varying datasets, TTS backbones, and evaluation protocols.
- Preliminary experiments on text-based voice-attribute control (extending Vclip toward text prompts) found controllability limited to simple concepts like gender, which the authors flag as needing further study rather than reporting as a solved capability.
Wiki Connections
- Zero-Shot TTS — uses a zero-shot multi-speaker VITS as the downstream synthesis engine and frames its own retrieval mechanism as a zero-shot scoring function generalizing to unseen reference faces at inference.
- Subjective Evaluation — conducts separate human studies for synthesis naturalness (MOS) and face-voice matching (m-MOS), each with at least 12 evaluators per sample.