arXiv · 2025 · Preprint
Zhan et al. · → Paper · Demo: ✓ · Code: ✓
VStyle introduces a bilingual benchmark for voice style adaptation in spoken language models, paired with an LALM-as-a-Judge evaluation framework that progressively assesses content faithfulness, style adherence, and naturalness across 1,523 spoken instruction prompts.
Problem
While spoken language models have made rapid progress in semantic accuracy and instruction following, their capacity for expressive style adaptation remains poorly understood and under-evaluated. Standard TTS metrics like WER and speaker similarity do not capture timbre, prosody, role-play competence, or the ability to infer appropriate affect from contextual cues. Existing benchmarks focus on content correctness, are limited by small test sets with poor reproducibility, or rely on costly human evaluations that do not scale. VStyle addresses this gap by formalizing Voice Style Adaptation (VSA) as a distinct evaluation task and providing a reproducible, multi-dimensional benchmark.
Method
VStyle constructs 1,523 bilingual (Chinese and English) speech instruction prompts organised across four categories. Acoustic attributes prompts specify explicit constraints on age, gender, speaking rate, pitch, loudness, and emotion, testing fine-grained acoustic controllability. Natural language instruction prompts use open-ended style descriptions including single-emotion, global style, and temporal style variation within a single utterance. Role-play prompts ask systems to adopt scene-based or character-based personas. Implicit empathy prompts do not specify style directly but require the model to infer appropriate affect from a friend-like conversational context.
Instructions were constructed with a hybrid human-LLM pipeline: seed instructions were manually designed, then expanded and refined iteratively. Audio instructions were synthesized using commercial voice cloning for non-emotional categories, and Gemini-TTS for empathy-related prompts requiring aligned emotional expression.
Evaluation uses the LALM-as-a-Judge framework, which leverages large audio-language models (specifically Gemini-2.5-pro) as automated assessors. The framework is hierarchically structured: first assessing content faithfulness (failure yields score 1), then style adherence (non-compliance gives 2, partial gives 3), and finally naturalness (unnatural gives 4, highly natural gives 5). This staged 5-point MOS scheme distinguishes systems that achieve content correctness but fail on expressiveness from those achieving both.
Key Results
GPT-4o achieves the highest English overall score (4.05), while Doubao leads in Chinese (4.10). Open-source models cluster between 2 and 3: Kimi-Audio reaches 3.11 in Chinese and Step-Audio 2.77 in English, the best open-source results per language. The performance gap is attributed to two factors: commercial models benefit from larger training corpora and stronger instruction-following robustness, while open-source models rely primarily on semantic-level representations that under-model acoustic features. Baichuan-Audio is an exception, using a unified codec representation that yields stronger performance on age control and some timbre attributes.
Models show pronounced language preference: Doubao, Kimi-Audio, and Qwen2.5-Omni perform substantially better in Chinese than English, while the GPT-4o series shows the reverse pattern. Composite acoustic tasks (controlling multiple attributes simultaneously) are harder than single-attribute tasks for all models. Style variation (temporal changes in style within an utterance) is the hardest subtype for all models except GPT-4o.
The LALM-as-a-Judge framework achieves Spearman correlation of 77.01% with consensus human evaluation in English and 73.03% in Chinese, close to inter-human agreement levels of 78.58% and 70.54%, confirming that automated LALM-based evaluation is a practical and scalable substitute for manual assessment on this benchmark.
Novelty Assessment
The contribution is primarily in evaluation methodology and dataset construction. VStyle fills a genuine gap by explicitly targeting expressive style dimensions (particularly role-play, implicit empathy, and temporal style variation) that prior benchmarks overlook. The LALM-as-a-Judge framework adapts the model-as-a-judge pattern to the audio modality with a staged, hierarchically structured scoring scheme. Neither the benchmark categories nor the judge architecture are individually novel, but combining them systematically for bilingual spoken style adaptation is new. The finding that commercial systems substantially outperform open-source models on expressive style tasks is a useful diagnostic, though it reflects the current capability gap rather than constituting a research contribution in itself.
Field Significance
Moderate - VStyle provides a reproducible, bilingual evaluation foundation for a dimension of speech generation that existing benchmarks systematically overlook. The LALM-as-a-Judge methodology demonstrates that large audio-language models can approximate human assessors on style evaluation tasks, enabling scalable evaluation of the expressive capabilities of spoken language models without relying on costly human panels.
Claims
-
supports: Large audio-language model judges can approximate human evaluation of speaking style adherence with near-human-level reliability.
Evidence: LALM-as-a-Judge (Gemini-2.5-pro) achieves Spearman correlations of 77.01% (English) and 73.03% (Chinese) with consensus human scores, comparable to inter-human agreement of 78.58% and 70.54%. (§3.3, Table 2)
-
complicates: Current open-source spoken language models exhibit substantial deficits in controllable expressive speech generation compared to commercial systems.
Evidence: Open-source models (Step-Audio, Kimi-Audio, Qwen2.5-Omni, Baichuan-Audio) score 2.46-3.11 overall on VStyle, while commercial systems (GPT-4o, Doubao) reach 3.74-4.10, with the gap attributed to weak acoustic feature modeling and lower instruction-following robustness. (§3.2, Table 1)
-
complicates: Speaking style control does not transfer evenly across languages even within the same system, indicating language-specific rather than language-general style representations in current models.
Evidence: Doubao and Kimi-Audio score substantially higher in Chinese than English (e.g., Doubao: 4.10 vs. 3.88), while the GPT-4o series shows the reverse, attributed to imbalanced training data distributions and differences in pronunciation habits. (§3.2)
-
supports: Staged hierarchical evaluation frameworks capture finer distinctions in speech generation quality than flat or single-dimension scoring approaches.
Evidence: VStyle’s LALM-as-a-Judge separates content correctness, style adherence, and naturalness into sequential assessment gates, enabling discrimination between systems that achieve content but fail on style versus those achieving both; this structure is validated against human evaluation at near-human-level correlation. (§2.3, Figure 2, §3.3)
Limitations and Open Questions
The instruction dataset is built from manually designed seeds expanded by LLMs, meaning its distribution reflects annotator preferences and model-driven patterns that may diverge from real user interaction needs. Despite strong aggregate correlation between LALM scoring and human judgment, large audio-language model evaluators remain prone to hallucinations; the paper mitigates this with step-by-step prompts but does not eliminate the risk. The benchmark synthesizes audio instructions using commercial voice cloning, meaning style signals in the input prompts may exhibit a narrower range of speaking styles than naturally produced speech.
Wiki Connections
- Spoken Language Model - VStyle targets spoken language models and provides the first systematic evaluation of their voice style adaptation capabilities across four distinct style dimensions.
- Instruction-Conditioned TTS - the benchmark evaluates whether systems can follow spoken natural language style instructions to modify timbre, prosody, and persona in generated speech.
- Evaluation Metrics - the LALM-as-a-Judge framework introduces a hierarchical 5-point MOS scheme for automated evaluation of style adherence and naturalness at scale.
- Subjective Evaluation - human evaluation with expert annotators is used to validate the automated LALM judge, achieving near-human-level Spearman correlations in both English and Chinese.
- Emotion Synthesis - the implicit empathy and emotion subcategories test systems’ ability to infer and produce appropriate affective speech from conversational context cues.