arXiv · 2025 · Preprint
Lameris et al. (KTH Royal Institute of Technology) · → Paper · Demo: ? · Code: ✓
Introduces VQ-Bench, a controlled parallel benchmark of synthesized modal, breathy, creaky, and end-creak speech prompts, and uses it to show that speech foundation models systematically change their open-ended generation and emotion-recognition behaviour depending on phonation type alone.
Problem
Speech foundation models (SFMs) increasingly process raw audio directly, which exposes them to paralinguistic cues beyond lexical content, including voice quality (phonation type). Prior work on SFM paralinguistic sensitivity has relied almost entirely on multiple-choice question answering (MCQA) benchmarks, which constrain model outputs and can mask how non-lexical variation actually shapes generation. Voice quality specifically, phonation types such as breathy and creaky voice, carries documented social and affective meaning for human listeners (e.g. breathy voice linked to intimacy, creaky voice to authority or disengagement, with gendered asymmetries in how these are perceived), yet no controlled framework existed to test whether SFMs reproduce or amplify these associations. The paper’s goal is to isolate voice quality as a variable while holding speaker identity and lexical content constant, and to probe SFM behaviour through open-ended tasks rather than constrained-choice formats.
Method
The authors construct VQ-Bench, a parallel dataset of speech prompts synthesized in four phonation conditions: modal, breathy, creaky, and end-creak (a modal-to-creak transition marking phrase or turn finality). Reference speech is drawn from the Buckeye Corpus (spontaneous conversational American English interviews) and the VCTK Corpus (read speech from over 100 speakers of different English dialects). For each reference speaker, 12-second audio segments are used as zero-shot speaker prompts for F5-TTS F5-TTS, which synthesizes the base utterances. The resulting prompts are then converted to breathy, creaky, and end-creak variants using VoiceQualityVC VoiceQualityVC, which manipulates glottal source (acoustic) parameters while preserving speaker identity and lexical content. Target acoustic parameter shifts (Creak, CPPS, H1-H2, H1-A3) for each condition are fixed relative to corpus means (Table 1) and verified post-hoc by measuring H1-H2 and H1-A3 on the synthesized outputs, confirming the four conditions are acoustically separated.
Two evaluation settings use this dataset. First, a long-form open-ended generation task presents 20 prompts per speaker (five each across therapy, career advice, interview screening, and storytelling categories, extended from an existing prompt set) to two SFMs: the OpenAI real-time speech-to-speech API and LFMAudio2-1.5B. Responses are scored on a 1-5 scale across multiple task-specific evaluation dimensions (e.g. agency of advice, STEM vs. care orientation, shortlist decision, heroic agency) by an LLM judge (gemini-2.5-flash-lite), and effects of voice quality and gender are estimated with a cumulative link mixed model with speaker as a random intercept. Second, a speech emotion recognition (SER) task applies xlsr-en-speech-emotion-recognition, a Wav2Vec 2.0 model fine-tuned to predict eight emotion classes, to the Buckeye subset only, analyzed with Bayesian multilevel categorical regression to detect shifts in the full emotion probability distribution (not just the top class) as a function of voice quality and gender.
Key Results
An initial validation check found the OpenAI real-time speech-to-speech API unreliable at detecting speaker gender from audio, defaulting to classifying nearly all samples as male; as a consequence it showed no systematic response variation across voice qualities and was excluded from further analysis, leaving LFMAudio2-1.5B as the sole model analyzed for the long-form task. For LFMAudio2-1.5B, voice quality had a significant effect on nearly every evaluation dimension in both the VCTK and Buckeye subsets, with the exceptions of “Role status” and “Emotional validation” (Table 2). Effects were direction-dependent by task: breathy and end-creak voice raised STEM-oriented career ratings while creaky voice raised care-oriented ratings; all non-modal voice qualities lowered interview scores except creaky voice for shortlist decisions; therapy ratings rose for all non-modal qualities where effects were significant; and female voices were rated systematically lower than male voices on salary offer and leadership endorsement in the interview task, regardless of voice quality. In the SER task, breathy voice increased predicted probability of calm (+1.17) and decreased fearful (-1.21), creaky voice decreased fearful (-0.94) and happy, and end-creak decreased fearful (-0.60) relative to modal voice, all with 95% credible intervals excluding zero (Table 4). No comparison to prior benchmarks is made since VQ-Bench is a new evaluation suite rather than a leaderboard entry; the paper’s comparisons are internal, across the four phonation conditions and two source corpora.
Novelty Assessment
The contribution is empirical and infrastructural rather than architectural: no new model or training method is proposed. The novelty lies in (1) constructing a controlled, parallel voice-quality benchmark (VQ-Bench) that isolates phonation type from speaker identity and lexical content by chaining two existing systems (zero-shot TTS plus a dedicated voice-quality conversion tool), and (2) applying open-ended generation and SER probing, rather than MCQA, to reveal how SFMs respond differently to a paralinguistic dimension prior benchmarks did not systematically control for. The finding that a leading commercial SFM (the OpenAI speech-to-speech API) fails a prerequisite gender-detection check is itself a notable diagnostic result, though it also narrows the paper’s own coverage to a single SFM for the generation task.
Field Significance
moderate — This paper opens phonation type (voice quality) as a distinct, previously under-tested evaluation dimension for speech foundation models, and provides a reusable parallel dataset and open-ended evaluation protocol for future studies. Its own empirical scope is narrow: two SFMs are tested (one of which was excluded after failing a basic gender-detection check), only English speech is used, and gender is treated as binary, following the source corpora.
Claims
- supports: Controlled variation in phonation type (voice quality), applied while holding speaker identity and lexical content constant, can measurably shift a speech foundation model’s open-ended generation behaviour.
Evidence: Across therapy, career advice, interview screening, and storytelling tasks, voice quality had a statistically significant effect on nearly every LLM-judge-rated evaluation dimension for LFMAudio2-1.5B, with effect direction varying by task (e.g. breathy/end-creak raising STEM-oriented ratings, creaky raising care-oriented ratings). (§4, Table 2)
- supports: Phonation type systematically shifts the emotion-probability distribution produced by speech emotion recognition models, independent of lexical content.
Evidence: Bayesian multilevel regression on a Wav2Vec 2.0-based SER model found breathy voice increased calm (+1.17) and decreased fearful (-1.21) probabilities, and creaky/end-creak voice decreased fearful probabilities, all with 95% credible intervals excluding zero. (§4, Table 4)
- complicates: Multiple-choice question-answering formats used in existing paralinguistic evaluation benchmarks can mask model sensitivity to non-lexical speech variation that only becomes visible under open-ended generation tasks.
Evidence: The paper motivates its open-ended generation and SER probing design by noting existing SFM paralinguistic benchmarks depend on MCQA formats that are “prone to failure” and constrain how nuanced paralinguistic effects can surface. (§1)
- complicates: Not all deployed speech foundation models reliably extract basic paralinguistic attributes such as speaker gender from raw audio, which can confound or invalidate downstream voice-quality sensitivity measurements for that model.
Evidence: The OpenAI real-time speech-to-speech API defaulted to classifying nearly all samples as male and showed no systematic response variation across voice qualities, leading the authors to exclude it from further analysis. (§4)
- supports: Chaining a zero-shot TTS system with a dedicated voice-quality conversion system can produce acoustically controlled phonation variants (e.g. modal, breathy, creaky) suitable for isolating paralinguistic effects in model evaluation.
Evidence: F5-TTS-synthesized prompts converted with VoiceQualityVC’s glottal-source parameter manipulation showed distinct, verified separations in H1-H2 and H1-A3 acoustic measurements across the four voice-quality conditions, for speakers drawn from two disparate corpora. (§2.1.3, Figure 1, Figure 2, Table 1)
Limitations and Open Questions
The long-form generation results rest on a single SFM (LFMAudio2-1.5B); the second model tested, the OpenAI real-time speech-to-speech API, was excluded after failing a basic gender-detection check, so the paper's central open-ended generation findings are not yet cross-validated across models.
Additional limitations acknowledged by the authors: the analysis is restricted to binary gender categories, reflecting the structure of the source corpora (Buckeye, VCTK), and does not address gender-ambiguous or nonbinary voices. The study covers English speech only. The SER analysis uses only the Buckeye subset, on the grounds that VCTK is designed to be affectively neutral, which limits the SER findings to conversational (not read) speech. LLM-judge scoring (gemini-2.5-flash-lite) is used in place of human listening tests for the long-form task, so judged dimensions reflect the judge model’s own biases and calibration rather than independently verified human perception.
Wiki Connections
- Spoken Language Model — evaluates two audio-native speech foundation models (OpenAI’s speech-to-speech API and LFMAudio2-1.5B) for paralinguistic sensitivity, treating them as speech LMs that process raw audio directly rather than through an ASR-to-text pipeline.
- Speech-to-Speech — probes speech-in, speech/text-out foundation models on open-ended dialogue-style tasks (therapy, career advice, interview screening, storytelling) to test whether phonation type biases their responses.
- Evaluation Metrics — introduces voice quality (phonation type) as a new controlled evaluation dimension and open-ended evaluation protocol, arguing existing MCQA-based paralinguistic benchmarks are insufficient.
- Zero-Shot TTS — uses F5-TTS’s zero-shot speaker-prompt synthesis to generate the base utterances that make up the VQ-Bench dataset.
- Voice Conversion — applies the VoiceQualityVC voice conversion system to convert synthesized speech into breathy, creaky, and end-creak variants while holding speaker identity and content fixed.
- F5-TTS — used as the zero-shot TTS system to synthesize the base speech prompts from 12-second reference clips of each source speaker.
- VoiceQualityVC — used directly as the voice conversion system that produces the four phonation-type variants (modal, breathy, creaky, end-creak) forming VQ-Bench.