arXiv · 2026 · Preprint

Carlos Muñoz-Romero et al. (Universitat Oberta de Catalunya / Monoceros Labs / University of Granada) · → Paper · Demo: ✗ · Code: ✗

Generates a plausible voice for an unseen speaker from a single face image by training a small adapter that maps face-recognition embeddings into the style space of a frozen StyleTTS 2, so the TTS model itself is never retrained.

Problem

Zero-shot TTS clones a voice from a short audio prompt, which fails when no recording of the target exists (historical figures, fictional or video-game characters). Face-to-speech (F2S) synthesis predicts a voice from a face instead, but the correlation between facial and vocal traits is weak. Prior F2S systems trained with regression objectives tend to collapse toward generic, gender-averaged voices, and alignment methods such as Deep CCA capture coarse shared attributes (gender) without separating individuals. Prior systems also fine-tune the whole TTS model end-to-end, and F2S outside English is barely explored because paired audiovisual data in other languages is scarce.

Method

The “Freeze-Align” framework has three parts: a face encoder, a trainable Face Adapter, and a frozen StyleTTS 2 acoustic model used as a teacher. An InceptionResnetV1 pre-trained on VGGFace2 turns a static face image into a 512-d embedding. The Face Adapter, an MLP (512 → 1024 → 1024 → 128, ≈1.7M parameters, ReLU, 0.3 dropout), projects it into the 128-d StyleTTS 2 style space. The upper blocks of the face encoder (≈18.3M parameters, about 78% of the backbone) are also updated (“soft-tuning”), while the lower blocks stay frozen. The training target is the style vector that the frozen StyleTTS 2 style encoder extracts from the speaker’s real audio. StyleTTS 2 is chosen because its style-diffusion design condenses timbre and prosody into a compact style vector separate from content. The English StyleTTS 2 is the LibriTTS model; for Spanish, the authors fine-tune StyleTTS 2 on a private audio-only corpus and reuse the English-trained adapter unchanged.

Figure 1: Proposed Freeze-Align F2S architecture used during training. An InceptionResnetV1 backbone maps the input face to a 512-d embedding h vis, which the trainable Face Adapter (MLP, 512 → 1024 → 1024 → 128) projects into the 128-d style space z face. The frozen StyleTTS 2 style encoder provides the acoustic target z audio. Training aligns the two modalities through the hybrid loss (InfoNCE, RKD, variance, and demographic auxiliary heads). The acoustic teacher is frozen; the Face Adapter is trainable and the upper blocks of the face encoder are soft-tuned.

The training objective combines four terms instead of an MSE regression:

  • Supervised multi-positive InfoNCE: each projected face is pulled toward the voice embeddings of the same speaker and pushed from the other speakers in the batch (CLIP-style learnable temperature). A balanced sampler puts 4 utterances per speaker in each batch of 1024.
  • Relational knowledge distillation (RKD, distance term only): pairwise distances between projected faces must match the distances between the corresponding audio style vectors (Huber loss on batch-normalized distances).
  • Variance regularization: per-dimension standard deviation of the face projections is matched to that of the audio embeddings, to discourage collapse.
  • Demographic auxiliary heads: gender (BCE) and age (MSE) predictions from InsightFace pseudo-labels.

Four loss weightings are compared (Balanced, Identity-priority, +variance, Pure-contrastive). Training uses AdamW for 200 epochs on one A100.

At inference, the StyleTTS 2 style vector is split into a timbre part (used by the waveform decoder) and a prosody part (used by the duration and prosody predictor). The face supplies the timbre; each part can be interpolated with a vector sampled from StyleTTS 2’s text-conditioned style diffusion: timbre = (1−α)·z_face + α·z_diff, prosody = (1−β)·z_ref + β·z_diff, with z_ref defaulting to z_face when no reference audio is given. Low α preserves face-derived identity; high α moves toward the diffusion prior. The paper treats α as an explicit identity-vs-naturalness control.

Key Results

All evaluation uses 24 LRS3 speakers removed from training, so every test identity is unseen. Face-to-voice retrieval ranks the 24 real-audio style embeddings for each predicted face embedding (chance: Top-1 4.17%, Top-5 20.8%).

  • Retrieval is above chance but weak. English Top-1 ranges 7.1–9.3% and Top-5 32.6–39.2% across the four loss settings (Table 3). A McNemar test finds no significant difference among the four configurations (all p > 0.1), so no loss weighting is better than the others on this test set.
  • Soft-tuning the face encoder matters for embedding-level fidelity. SECS-emb (cosine between predicted face embedding and the speaker’s mean audio style vector) is 0.40–0.42 with soft-tuning versus 0.29 with a frozen encoder. Top-1 does not differ significantly (6.8% frozen; p = 0.25 against the soft-tuned setting).
  • Naturalness is not degraded by face conditioning. English UTMOS is 3.72–3.98 and NISQA 3.84–4.22 at α = 0.5, against 3.61 and 3.63 for the ground-truth recordings (Table 4). Both are automatic MOS predictors; there is no listening test.
  • Speaker similarity of the synthesized audio is lower than earlier end-to-end F2S systems. SECS-audio (Resemblyzer) is 0.61–0.62 in English and 0.57 in Spanish, against 0.754 (FVTTS) and 0.748 (Face-TTS) reported by prior work on LRS3. The paper states these figures are not directly comparable: prior systems fine-tune the full TTS model, and SECS-audio here partly reflects the StyleTTS 2 voice prior.
  • The face contributes only part of the identity. Sweeping α (Table 5, English, β = 0.9) moves SECS-audio between 0.56 (α = 0.9) and 0.64 (α = 0.3, the peak), while UTMOS rises from 1.82 (α = 0.1) to 4.17 (α = 0.7). A nearly pure face vector (α = 0.1) lands off the acoustic manifold and has very low naturalness. The authors read the narrow SECS range as evidence that most absolute identity comes from the TTS prior; no mismatched-face baseline is run.
  • Cross-lingual transfer. With the Spanish-fine-tuned StyleTTS 2 and the English-trained adapter, Spanish Top-1 is 7.1–12.7% and Top-5 35.5–44.0% (Table 3). Spanish UTMOS is 2.67–2.88 and NISQA 3.66–3.85; the authors note both predictors are English-biased, so these values are indicative only. Spanish and English embedding metrics are computed in different style spaces and are not cross-comparable.

The evaluation has no matched baseline: no prior F2S system is re-run under the same protocol, and the checkpoint was selected on the same unseen split used for the quality numbers in Table 4, which the authors declare as a mild optimism.

Novelty Assessment

The main idea is to leave a pretrained style-based TTS model untouched and learn only a face-to-style mapping, with a hybrid contrastive and relational-distillation objective in place of regression. Each loss term is an established technique (CLIP-style InfoNCE, RKD, variance matching, auxiliary attribute heads); the new part is their combination as a training objective for F2S on a frozen acoustic teacher. The prior F2S systems it cites for comparison (FVTTS, Face-TTS) fine-tune the full TTS model end-to-end. The inference-time timbre/prosody split reuses StyleTTS 2’s existing style-vector structure rather than adding a new mechanism.

The evaluation is careful about its limits: it reports chance levels, a significance test, a frozen-encoder ablation, and an explicit statement that the SECS comparison with prior work is not like-for-like. It is also small. The test set has 24 speakers, there is no human listening test, and no baseline is reproduced. The Spanish result shows that an English face-to-style mapping can drive a separately fine-tuned Spanish backbone. The adapter itself is trained only on English.

Field Significance

Low — This paper shows that face-to-voice generation can work as a lightweight adapter on a frozen style-diffusion TTS model, with naturalness kept at ground-truth level by automatic predictors. The evidence is automatic-only (no listening test) on 24 held-out English speakers, which limits how far the result generalizes. It is also a cautionary data point: its own α sweep suggests the face accounts for only a modest share of speaker identity, with most coming from the TTS prior. That finding matters for interpreting speaker-similarity numbers in face-conditioned TTS.

Claims

  • supports: Voice identity for unseen speakers can be predicted from a face image by mapping visual embeddings into the style space of a frozen pretrained TTS model, without retraining the acoustic model.

    Evidence: A ≈1.7M-parameter Face Adapter plus soft-tuned face-encoder blocks, trained against a frozen StyleTTS 2 style encoder, gives above-chance face-to-voice retrieval on 24 held-out LRS3 speakers (English Top-1 7.1–9.3% vs. 4.17% chance; Top-5 32.6–39.2% vs. 20.8%) with UTMOS 3.72–3.98 against 3.61 for ground truth. (§2.1, §4.2, §4.3, Tables 3–4)

  • complicates: Speaker-similarity scores for face-conditioned speech synthesis can mostly reflect the TTS model’s voice prior rather than identity information extracted from the face.

    Evidence: Sweeping the face/diffusion interpolation weight α changes SECS-audio only between 0.56 and 0.64, and the authors conclude the face supplies a modest part of absolute identity, with most coming from the StyleTTS 2 prior; no mismatched-face control is run. (§4.4, Table 5)

  • complicates: Conditioning a style-based TTS model on an out-of-distribution predicted style vector trades speaker identity against naturalness.

    Evidence: At α = 0.1 (nearly pure face vector) UTMOS is 1.82; it rises to 4.17 at α = 0.7, while SECS-audio peaks at α = 0.3 (0.64) and falls to 0.56 at α = 0.9. The authors attribute the low-α drop to the face vector landing off the acoustic manifold. (§4.4, Table 5)

  • refines: Adapting the upper layers of a pretrained face encoder, not only a projection head, is needed for close alignment between face embeddings and a speech style space.

    Evidence: Soft-tuning the upper face-encoder blocks raises embedding-level SECS from 0.29 (frozen encoder) to 0.40–0.42, though Top-1 retrieval does not differ significantly (p = 0.25); the frozen encoder is reported to place styles off the acoustic manifold. (§4.2, Table 3)

  • supports: A face-to-voice mapping learned in one language can condition a TTS backbone in another language without paired data in the target language.

    Evidence: The English-trained adapter, applied without retraining to a StyleTTS 2 fine-tuned on a private Spanish corpus, gives Spanish Top-1 7.1–12.7% and Top-5 35.5–44.0% on the unseen LRS3 faces; Spanish UTMOS (2.67–2.88) is lower than English, and the authors treat it as indicative because the predictors are English-biased. (§3.1, §4.2, §4.3, Tables 3–4)

Limitations and Open Questions

Warning

All quality evidence is automatic (UTMOS, NISQA, Resemblyzer SECS) on 24 unseen speakers. There is no human listening test and no prior F2S system re-run under the same protocol. The checkpoint was selected on the same unseen split used for the Table 4 quality numbers.

  • Modest face contribution. The α sweep bounds the face’s effect on speaker similarity to a narrow range (0.56–0.64). Without a mismatched-face baseline, the net identity gain from the face is not isolated.
  • Small test set. With 24 identities, Top-1 differences are noisy, and McNemar tests find no significant differences among the loss configurations. The four configurations differ in several weights at once (Table 2), so they compare operating points, not single-factor ablations.
  • No matched comparison with prior F2S. SECS-audio (0.61–0.62 English) is below the 0.754 and 0.748 reported for FVTTS and Face-TTS, and the paper leaves a protocol-matched end-to-end StyleTTS 2 comparison to future work.
  • Cross-lingual scope. The adapter is trained on English only; the Spanish StyleTTS 2 is fine-tuned on a private audio-only corpus of 48 speakers, so this setting cannot be reproduced. Spanish quality scores come from English-biased predictors.
  • Prosody from a static image. Prosody comes from the diffusion sampler or optional reference audio, not from the face. β is fixed at 0.9 in the α sweep and its effect is not evaluated.
  • Demographic pseudo-labels. Gender and age labels for the auxiliary heads are automatic InsightFace estimates, available for 1,449 of 2,007 speakers.
  • Code and audio samples are not released; the paper says they will follow acceptance.

Wiki Connections

  • Zero-Shot TTS — Generates voices for LRS3 speakers held out from training using a face image instead of an audio prompt, following the corpus convention for non-audio zero-shot voice generation.
  • Diffusion TTS — The method steers a frozen StyleTTS 2 style-diffusion model, interpolating face-predicted style vectors with samples from its style diffusion to trade identity for naturalness.
  • ZipVoice — Cited as an audio-prompt zero-shot TTS system; this paper targets the case where no reference audio exists.
  • VoxCPM — Cited as an audio-prompt voice-cloning system whose reliance on reference audio motivates face-based voice generation.
  • VibeVoice — Cited as a large-corpus audio-prompt zero-shot TTS system in the motivation for face-to-speech.
  • LibriTTS — The corpus on which the frozen English StyleTTS 2 backbone is trained.
  • UTMOS — Used as the automatic naturalness proxy in place of a listening test.