arXiv · 2025 · Preprint
Carbonneau et al. (Ubisoft La Forge) · → Paper · Demo: ? · Code: ✓
Analyzes which voice identity markers ASV embeddings capture, identifies confounding factors that corrupt speaker similarity measurements in synthesis evaluation, and proposes U3D: a language-agnostic rhythm metric based on phoneme-like unit duration distributions from self-supervised speech representations.
Problem
Speaker similarity in generative speech systems is routinely assessed using automatic speaker verification (ASV) embeddings, but these representations were designed to discriminate between speakers rather than to characterize the full range of voice identity. This design mismatch raises an underexplored question: do ASV embeddings actually capture all the identity markers that matter for synthesis evaluation? Prior work has largely treated ASV cosine similarity and equal error rate (EER) as proxies for perceptual identity preservation without systematically examining what these representations encode or how sensitive they are to non-identity factors in the recording signal.
Method
The paper proceeds in three parts. First, it examines which identity markers ASV embeddings encode. Using a lasso regressor trained on LibriSpeech, ARCTIC, and L2-ARCTIC data, the authors predict handcrafted acoustic features from embeddings of seven ASV models (GE2E, X-Vectors, ResNet-TDNN, ECAPA-TDNN, WavLM+ECAPA-TDNN, WavLM Large+ECAPA-TDNN, WavLM+X-Vector) and measure the coefficient of determination (r²) as a proxy for which features each embedding encodes.
Second, it investigates confounding factors in EER-based evaluation. Experiments on ARCTIC and L2-ARCTIC measure how utterance duration differences, channel noise (varying SNR), and equalization shift EER away from its expected value of 50% for same-speaker comparisons, revealing systematic distortions unrelated to voice identity. The paper also demonstrates that re-equalization using a 16-band FIR graphic equalizer matched to the genuine audio spectral balance restores correct EER.
Third, it introduces U3D (Unit Duration Distribution Distance). U3D operates in three steps: HuBERT features are extracted from all utterances; hierarchical clustering groups discrete speech units into phoneme-like categories (sonorants, obstruents, silences); and the Wasserstein distance between synthesized and genuine speech duration distributions is computed for each phoneme-like group. Because U3D uses self-supervised speech units, it requires no forced aligner and applies across languages without modification.
Key Results
The embedding analysis (Figure 1) shows that static spectral features — mean pitch, HNR, α-ratio, shimmer — are strongly encoded across all ASV models (high r²), while dynamic behavioral features — speech rate, voiced and unvoiced segment lengths, pitch and loudness standard deviations — are poorly represented (near-zero r²). Notably, older X-Vector embeddings capture a slightly wider range of markers than more recent WavLM-based models, suggesting that optimization pressure for speaker verification does not align with what synthesis evaluation requires.
The confounding factor experiments (Table 2) reveal that grouping same-speaker utterances by duration drives EER down from the expected 50% to 30–39% across all models, showing that ASV systems can discriminate utterance duration independent of speaker identity. Channel noise at SNR 20 dB reduces EER to 15–38% depending on model; at SNR 0 dB, all models fail near-completely. Equalization filters cause catastrophic failure in GE2E (EER drops to 1–7%), while re-equalization restores EER to approximately 50% for all models.
For U3D validation (Table 3), same-speaker rhythm distances average 2.15 (Wasserstein); nearest-neighbor speaker pairs matched by speech rate average 18.40; and random pairs average 21.53. The forced-alignment baseline produces nearly identical separation (2.48, 18.37, 24.43), confirming that unsupervised HuBERT-based grouping approximates phoneme structure with comparable discriminative power.
Novelty Assessment
The finding that ASV embeddings fail to capture dynamic identity markers is not unexpected given the discrimination objective of ASV training, but the paper provides the first systematic quantification of this gap across multiple ASV architectures in the synthesis evaluation context. The confounding factor analysis — particularly duration sensitivity and equalization failure — is practically significant and addresses a concrete methodological gap that affects published results in the field. U3D itself is a straightforward combination of existing tools (HuBERT, hierarchical clustering, Wasserstein distance), building directly on prior rhythm modeling work; the novelty lies in packaging these into a validated, language-agnostic metric. The contribution is primarily evaluation-methodology rather than algorithmic or architectural.
Field Significance
Moderate — The paper addresses a genuine gap in synthesis evaluation by demonstrating that the field’s default speaker identity metric (ASV EER) is both incomplete and sensitive to experimental confounders. U3D provides a complementary rhythm dimension that existing metrics miss, and the practical mitigation recommendations on duration matching and re-equalization are immediately actionable for researchers designing voice cloning or zero-shot TTS evaluation protocols.
Claims
-
supports: ASV embeddings encode static anatomical speech features but systematically fail to represent dynamic behavioral identity markers such as rhythm and timing patterns.
Evidence: Lasso regression predicting handcrafted features from ASV embeddings across LibriSpeech, ARCTIC, and L2-ARCTIC shows high r² for mean pitch, HNR, shimmer, and α-ratio, but near-zero r² for speech rate, voiced and unvoiced segment lengths, and pitch standard deviation across all seven tested ASV models. (§3.2, Figure 1)
-
complicates: EER-based speaker similarity measurements in speech synthesis evaluation are susceptible to confounding factors unrelated to voice identity, which can invalidate comparisons between synthesis systems.
Evidence: Duration-sorting same-speaker utterances depresses EER from 50% to 30–39% across all ASV models; SNR 20 dB noise reduces EER to 15–38%; equalization shifts cause near-total failure in GE2E. Re-equalization and duration matching restore correct EER. (§3.3, Table 2)
-
supports: Characterizing speaker rhythm for identity assessment requires modeling phoneme-duration distributions rather than aggregate measures such as mean speech rate.
Evidence: Many L2-ARCTIC speakers share similar syllable rates but show substantially different voiced segment duration patterns; U3D Wasserstein distances clearly separate same-speaker pairs (avg. 2.15) from nearest-by-speech-rate pairs (18.40) and random pairs (21.53). (§3.4, Figure 2, Table 3)
-
refines: Self-supervised speech unit representations serve as language-agnostic substitutes for phoneme labels in rhythm analysis, avoiding the need for forced alignment.
Evidence: U3D using HuBERT-derived unsupervised clusters achieves Wasserstein distance separation between speaker conditions nearly identical to forced-alignment-based phoneme rhythm analysis (same: 2.15 vs. 2.48; nearest: 18.40 vs. 18.37; random: 21.53 vs. 24.43). (§3.4, Table 3)
Limitations and Open Questions
Warning
U3D is validated as a discriminative metric (separating speaker pairs) but has not been validated against human perceptual judgments of rhythm similarity. Whether Wasserstein distances correlate with listeners’ perception of rhythmic difference between voices remains untested.
The paper is limited to neutral speech, with the authors explicitly noting extension to expressive or conversational speech as future work. Experiments use clean studio-quality recordings (ARCTIC, L2-ARCTIC), so the behavior of confounding factors in in-the-wild multi-condition data remains untested. Recommended mitigation strategies (duration matching, re-equalization) assume access to the same text prompts used in genuine recordings, which may not be feasible in all evaluation settings. U3D’s language-agnosticism is argued theoretically but not empirically tested on non-English languages.
Wiki Connections
- Evaluation Metrics — proposes U3D as a new complementary metric and demonstrates that widely used ASV-based EER is incomplete and confounded by non-identity factors in synthesis evaluation.
- Subjective Evaluation — notes that behavioral identity markers such as rhythm are also difficult for human evaluators unfamiliar with the speaker, motivating automated metrics that go beyond static spectral features.
- Self-Supervised Speech — uses HuBERT discrete speech units as the core feature extraction step in U3D, enabling phoneme-like grouping without supervised forced alignment.
- Voice Conversion — ASV-based EER is a standard evaluation metric in voice conversion; the confounding factor analysis and mitigation strategies apply directly to VC evaluation pipelines.
- Zero-Shot TTS — situates the work in voice cloning evaluation, noting that high-quality synthesis systems can produce cleaner output than enrollment recordings, causing systematic EER measurement distortion.