arXiv · 2025 · Preprint
Feng et al. · → Paper · Demo: ? · Code: ✓
Vox-Profile introduces a multi-dimensional speaker and speech trait benchmark spanning eight trait categories (age, sex, accent, voice quality, emotion, arousal/valence, speech flow, and expressiveness), grounded in speech science taxonomy, and demonstrates its utility for evaluating speech generation systems.
Problem
Most existing speaker characterization efforts focus on a single trait dimension such as accent or emotion, each adopting inconsistent taxonomies that cannot be compared across studies. The field lacks a unified, linguistically principled framework that simultaneously addresses static speaker traits (those tied to the speaker’s identity) and dynamic speech traits (those that vary with context and affect). This fragmentation makes it difficult to systematically audit speech models for demographic fairness, to evaluate how well speech generation systems reproduce target speaker characteristics, or to automatically generate rich speaking style descriptions for instruction-conditioned TTS.
Method
Vox-Profile defines a two-tier taxonomy distinguishing static traits (age, sex, accent, voice quality) from dynamic traits (categorical emotion, arousal/valence, speech flow, speech expressiveness). The taxonomy is grounded in speech science literature rather than computational convenience: age is framed as a three-way classification (young adults under 30, adults 30-60, senior adults over 60) because perceived vocal age maps more naturally to intervals than to regression targets; accents are organized by regional origin and first-language typology rather than nationality alone.
All trait predictors share a common downstream architecture applied on top of frozen pretrained encoders. Hidden representations from convolutional and Transformer encoder layers are combined via weighted averaging, processed through 1D pointwise convolution, and pooled before a final classification or regression head. Models evaluated include HuBERT Large, WavLM Large, ECAPA-TDNN, and the Whisper family (Tiny, Small, Large-V3). The benchmark uses over 15 publicly available datasets covering diverse recording conditions, speaking styles, and accent backgrounds.
For multitask settings such as age-sex joint modeling and speech flow (fluency vs. disfluency type), the paper employs multiple classification heads on a shared representation. Voice quality and speech expressiveness are formulated as multilabel classification tasks. Emotion is trained with KL divergence loss over soft label distributions; arousal and valence use CCC loss with sigmoid normalization.
Key Results
Across static trait prediction (Table 3), WavLM Large leads on accent tasks (broad accent macro F1 of 0.884; narrow accent macro F1 of 0.617) while Whisper Large achieves the strongest overall profile, including the best narrow accent F1 (0.724), speaker age accuracy (69.5%), and speaker sex accuracy (98%). Speaker sex is relatively easy for all models, with most exceeding 95% accuracy. Narrow accent classification remains challenging, with Whisper Large the only model to exceed macro F1 of 0.70.
For dynamic traits (Table 4), speech flow classification is tractable, with WavLM Large reaching 80.7% fluency accuracy. Speech emotion recognition is uniformly difficult: even the top models reach only around macro F1 of 0.40, including the authors’ top-ranked IS25-SER challenge system which uses both text and speech modalities.
Compared to prior work (Table 5b), Vox-Profile’s accent classifiers outperform CommonAccent on both VCTK (macro F1 0.423 vs. 0.379) and British Isles (macro F1 0.770 vs. 0.581). Age and sex systems match a Wav2Vec 2.0-based baseline on VoxCeleb and TIMIT.
As an evaluation tool for speech generation, Vox-Profile is applied to compare FreeVC and VALL-E X on accent preservation. Accent similarity scores confirm that FreeVC preserves source accent rather than reference accent, while VALL-E X through its ASR-TTS pipeline more closely matches the reference, consistent with prior findings.
Human evaluation of synthetic speaking style prompts generated from Vox-Profile predictions shows comparable or preferred quality relative to human-annotated ParaSpeechCaps prompts on emotion, age, and speech flow descriptions, while accent descriptions from ground-truth labels remain superior.
Novelty Assessment
The contribution is primarily taxonomic and evaluative rather than architectural. The underlying downstream model architecture is borrowed from existing work. The genuine contributions are: (1) a linguistically principled unification of eight speaker and speech trait categories under a single benchmark that no prior work covers comprehensively; (2) a coherent accent taxonomy integrating 11 datasets with divergent label conventions; and (3) demonstration that automated Vox-Profile predictions can substitute for manual metadata in downstream speech model analysis with similar statistical conclusions. The comparison of textless voice conversion (FreeVC) versus ASR-TTS voice cloning (VALL-E X) on accent fidelity is a useful secondary application but not the paper’s main claim. The benchmark’s English-only scope and reliance on publicly available datasets with their existing label limitations are notable constraints on generalizability.
Field Significance
Moderate — Vox-Profile provides the field with a structured, multi-dimensional framework for characterizing speaker and speech traits that can serve as a common evaluation substrate for speech generation systems. Its primary value is enabling systematic auditing of TTS and voice conversion models for speaker characteristic fidelity, complementing existing naturalness (MOS) and intelligibility (WER) metrics with richer attribute-level assessments.
Claims
- Speech emotion recognition remains the most difficult speaker/speech trait to predict from speech alone, with state-of-the-art models plateauing around macro F1 of 0.40 even when trained on large naturalistic corpora. (§5, Table 4)
- Narrow accent classification is substantially harder than broad regional accent grouping, with the performance gap between coarse and fine-grained accent labels exceeding 0.20 macro F1 points for all tested models. (§5, Table 3)
- ASR-TTS voice cloning pipelines preserve target speaker accent more faithfully than latent-space textless voice conversion, because the intermediate transcription stage retains accent-relevant phonetic patterns. (§6.2, Table 6)
- Automated speech trait predictions can substitute for manual ground-truth labels in statistical analysis of ASR performance variability across demographic groups, with similar trend conclusions. (§6.1, Figure 2)
- Model scale correlates positively with speaker and speech trait prediction accuracy across both self-supervised and supervised speech foundation models. (§5, Tables 3–4)
Limitations and Open Questions
Warning
The benchmark covers English only. All accent, emotion, and speech flow models are trained on English-language datasets; generalizability to other languages is not assessed and is acknowledged as a primary limitation.
The emotion models are evaluated on the MSP-Podcast development set rather than the held-out test set, because challenge submission opportunities were limited. Comparisons of SER performance in Table 5b are therefore not fully equivalent to published challenge results. Voice quality and speech expressiveness training relies heavily on ParaSpeechCaps, which itself covers a limited speaker pool from VoxCeleb and Expresso. Human evaluation of speaking style prompts involved approximately 10 pilot participants, predominantly male and non-native English speakers, limiting the generalizability of that comparison. The benchmark does not yet cover multilingual speech attributes such as language identification or code-switching, which the authors flag as planned extensions.
Wiki Connections
- evaluation-metrics — Vox-Profile introduces multi-dimensional speaker trait metrics (accent similarity, emotion F1, speech flow accuracy) for evaluating speech generation beyond standard MOS and WER.
- 2503.04713 (ParaSpeechCaps) — Vox-Profile adopts ParaSpeechCaps voice quality and expressiveness taxonomy and extends it with age, accent, emotion, and speech flow dimensions.
- 2303.03926 (VALL-E X) — Vox-Profile evaluates VALL-E X’s accent preservation in cross-accent voice conversion, finding it superior to FreeVC due to its intermediate text representation.
- 2402.01912 (ParlerTTS) — Vox-Profile complements ParlerTTS-style heuristic speaking style annotation with richer probabilistic trait profiles that include emotion, arousal/valence, and speech flow.