EMNLP · 2025 · Conference
Jeon et al. (POSTECH) · → Paper · Demo: ? · Code: ?
An end-to-end face-to-voice (FTV) synthesis framework that replaces multi-stage audio-alignment pipelines with a hierarchical facial encoder and bilateral demographic attribute supervision, enabling personalized voice generation without any audio recordings.
Problem
Existing face-to-voice (FTV) synthesis systems operate in multiple stages: a face encoder is first trained (or fine-tuned) to align its outputs with embeddings from a separately trained audio-based speaker encoder, and the face encoder then substitutes the audio encoder at inference. This pipeline introduces training complexity and, critically, forces the face representation into audio-embedding space — stripping fine-grained facial information such as gender or ethnic cues that correlate with perceived vocal characteristics. Datasets with static speaker-to-image mappings (one face per speaker) further limit robustness to natural variations in pose, lighting, and expression.
Method
The system is built on the VITS (Kim et al., 2021) backbone — a conditional variational autoencoder with adversarial training that directly generates waveforms from text — and replaces the standard audio-based speaker encoder with a newly designed facial identity module.
The Progressive Facial Granularity Aggregation (FGA) encoder decomposes an input image into non-overlapping patches of resolution P×P, linearly projects each patch to a fixed embedding, and refines those patch tokens through standard transformer blocks (multi-head self-attention and feed-forward layers). After each transformer stage, a convolutional aggregation layer groups four adjacent patch representations using 3×3 convolution followed by layer normalization and 3×3 max pooling, progressively expanding the receptive field. The cycle repeats until a single spatial token remains, yielding a 128-dimensional facial identity vector F. The encoder is trained from scratch — no pretrained visual backbone is used.
F is injected into the VITS text encoder via adaptive conditioning (following Yoon et al., 2023): F is projected to gain, direction, and bias parameters that modulate phoneme hidden states through a 1D convolutional layer.
Bilateral Attribute-based Enhancement adds multi-task supervision over demographic attributes. A pretrained DeepFace model provides pseudo-labels for gender and ethnicity from face images. Two prediction heads — one from the facial embedding and one from the synthesized audio (temporally resampled to a fixed size) — each predict these attributes via cross-entropy loss. This bilateral objective encourages face-voice alignment at the semantic level rather than the signal level. The combined attribute loss is weighted and added to the VITS training objective (reconstruction, KL-divergence, adversarial, and duration losses).
A multi-view augmentation strategy addresses static dataset limitations by randomly selecting five face frames per speaker from video, covering diverse poses and lighting conditions, each paired with the same ground-truth audio.
Total model size is approximately 43.6 million parameters. Training runs for 350,000 iterations on a single A6000 GPU for roughly two days. Audio is produced at 16 kHz.
Key Results
Evaluated on LRS3-TED against three baselines — Plüster et al. (2021), FaceTTS (Lee et al., 2023), and FVTTS (Lee et al., 2024) — with 21 annotators recruited via Amazon Mechanical Turk (Table 2).
The proposed model scores MOS 3.51 versus 3.25 (Plüster), 2.98 (FaceTTS), and 3.20 (FVTTS). In ABX preference (1-of-4 selection matching ground truth), it achieves 31.07% vs. 28.45% (FaceTTS, best baseline). On the face-to-voice (F2V) alignment task, where raters select which audio best matches a target face image, the proposed model scores 30.36% versus 27.38% (FVTTS, best baseline).
Speaker embedding cosine similarity (SECS via Resemblyzer) shows the largest gains: 79.96 for seen speakers and 71.39 for unseen, against Plüster’s 67.04 / 64.81 (nearest competitor). The gap for seen speakers (+12.92) suggests the architectural improvements address the fine-grained facial information loss of prior pipelines. CER is 0.1302, slightly above FaceTTS’s 0.107 but substantially better than Plüster (0.2381) and FVTTS (0.2743). UTMOS is 3.2218, highest among all systems.
Age alignment (Table 3) favors the Identical category at 0.649 versus 0.538 (Plüster). Gender classification accuracy improves by 10% and agreement scores by 2.4 points relative to the best baseline (Figure 2).
Out-of-domain evaluation on the GRID dataset shows the model maintains the top ABX preference score (+10% margin) and highest SECS (47.38, +3.16 over next best).
Ablation studies (Table 5) show removing multi-view augmentation causes the largest SECS drop (−12.45 points overall), followed by removing FGA (−10.40), visual attribute enhancement (−4.14), and audio attribute enhancement (−1.05).
Novelty Assessment
The primary contributions are the progressive hierarchical facial encoder and the bilateral attribute enhancement mechanism. Both are new for the FTV synthesis task, though the individual components draw on established techniques: hierarchical patch-based vision encoders are common in image generation (Zhang et al., 2022), and multi-task attribute prediction has precedent in speaker modeling. What is genuinely novel is the combination applied end-to-end to FTV synthesis, avoiding the audio-space alignment target that characterizes prior work. The VITS backbone and adaptive conditioning mechanism are adopted unchanged from existing work, so the speech generation architecture itself is not new.
The evaluation protocol is more thorough than typical FTV papers, introducing F2V alignment as a face-image-referenced ABX task alongside age and gender classification assessments — a methodological contribution to how FTV systems are compared.
The baseline set is reasonable (three published FTV systems), and performance gains are large and consistent across metrics, which mitigates concerns about evaluation sensitivity. However, the test set comprises only 40 speakers from LRS3-TED, and the dataset is heavily skewed toward Caucasian male speakers (74.66% male, 72.23% Caucasian per Table 6). SECS is lower for Middle Eastern (69.25) and African American (71.65) speakers, leaving demographic generalization as an open question.
Field Significance
Moderate — This paper advances a specialized sub-area of personalized TTS by demonstrating that direct hierarchical facial feature learning outperforms audio-space alignment targets for face-voice congruence. The bilateral attribute supervision technique, combining cross-modal demographic prediction with the speech generation objective, provides a concrete mechanism others can adopt for identity-consistent synthesis. The contribution is focused on the FTV synthesis niche and does not directly alter mainstream TTS methodology.
Claims
-
supports: Hierarchical, locally-aware facial feature extraction substantially improves voice-face identity congruence compared to global visual representations in face-to-speech synthesis.
Evidence: Progressive Facial Granularity Aggregation (FGA) — hierarchical patch decomposition with progressive convolutional aggregation — yields SECS of 79.96 (seen) vs. 66.25 when replaced with a single vanilla transformer over the full image, a gap of 13.71 points. (§5.3, Table 5)
-
supports: Cross-modal multi-task supervision over demographic attributes improves face-voice alignment in end-to-end synthesis systems.
Evidence: Bilateral attribute enhancement (predicting gender and ethnicity from both the facial embedding and synthesized audio) produces the highest SECS configuration at 79.96 seen / 71.39 unseen; applying supervision to only one modality yields lower scores (77.76 / 70.27 for visual-only, 75.84 / 69.52 for audio-only). (§5.2, Table 4)
-
supports: Multi-view face augmentation — pairing multiple facial frames per speaker across varying poses and lighting — is critical for robust speaker identity generalization in face-to-speech systems.
Evidence: Removing the multi-view augmentation strategy causes the largest single-component SECS drop in ablation studies (from 79.96 to 64.43, a decrease of 15.53 on seen speakers and 9.35 on unseen), exceeding the contribution of any individual architectural module. (§5.3, Table 5)
-
complicates: End-to-end face-to-voice synthesis without audio-space alignment targets still requires explicit cross-modal semantic supervision to maintain speaker identity coherence.
Evidence: Training the progressive facial encoder end-to-end without audio-alignment targets is the paper’s stated improvement over prior multi-stage pipelines, but removing bilateral attribute supervision (even partially) consistently degrades SECS — visual attribute enhancement removal alone costs 5.23 SECS points. (§5.2, §5.3, Table 4, Table 5)
-
complicates: Face-to-voice synthesis systems trained on demographically skewed corpora may produce uneven speaker similarity across underrepresented groups.
Evidence: The LRS3-TED training set is 74.66% male and 72.23% Caucasian; the proposed model achieves SECS of 75.97 for male and 75.03 for female speakers, while Middle Eastern (69.25) and African American (71.65) speakers show measurably lower scores despite the bilateral attribute supervision. (§Appendix B, Table 6, Table 7)
Limitations and Open Questions
The framework focuses on overall vocal timbre and does not capture emotional prosody: facial expressions that correlate with affective speech are excluded from the current conditioning design. Age-conditioned synthesis is also limited by dataset skew; the LRS3-TED distribution is not representative of older speakers, leaving age alignment as partial.
Demographic coverage in the training data is heavily imbalanced, with Middle Eastern and African American speakers showing noticeably lower SECS scores despite the bilateral supervision. Whether this gap can be closed by dataset rebalancing or additional attribute supervision is an open question.
The evaluation uses 40 speakers and 21 annotators, which, while reasonable for an FTV study, limits statistical power for cross-demographic comparisons. The comparison baselines do not include the most recent large-scale zero-shot TTS systems, so absolute quality relative to the broader field is unclear.
Wiki Connections
- Zero-Shot TTS — this paper extends zero-shot voice generation to the setting where no audio reference exists, using a facial image to infer speaker identity at inference time.
- Speaker Adaptation — the progressive facial encoder and bilateral attribute supervision serve as a substitute for speaker-encoder-based adaptation, targeting identity-consistent synthesis without audio samples.
- Voice Conversion — face-to-voice synthesis is closely related to VC in that both tasks target a specific voice identity; this paper addresses the harder case where the identity reference is visual rather than acoustic.
- Subjective Evaluation — the paper conducts a five-task human evaluation (MOS, ABX, F2V alignment, age alignment, gender classification) with 21 annotators, extending standard FTV evaluation protocols.
- Evaluation Metrics — introduces face-to-voice (F2V) alignment as a structured subjective task alongside age and gender classification dimensions, broadening the evaluation toolkit for FTV synthesis.