arXiv · 2026 · Preprint
Zhao et al. (Northeastern University / NiuTrans Research) · → Paper · Demo: ? · Code: ?
Introduces StyleBench, a multi-turn dialogue benchmark that quantifies how well speech language models can interpret and follow natural-language instructions to intensify or attenuate conversational speaking style along four dimensions: emotion, speed, volume, and pitch.
Problem
Speech language models (SLMs) increasingly claim the ability to adjust speaking style in response to user instructions during dialogue, following the interactive paradigm popularized by GPT-4o. However, prior benchmarking work for SLMs concentrated on single-turn conversations and conventional tasks such as ASR and spoken question answering, or, where style was considered at all (e.g. AudioBench, SpeechFeedback), was limited to distinguishing coarse emotional categories rather than measuring graded intensity. No existing benchmark systematically tested whether a model can track and vary the degree of a style attribute across a multi-turn conversation, or covered non-emotional prosodic dimensions such as speed, volume, and pitch under natural-language instruction. This left a gap in assessing whether SLMs faithfully follow stylistic prompts rather than merely producing style-adjacent output.
Method
StyleBench evaluates existing SLMs rather than proposing a new model; its contribution is the benchmark dataset, evaluation toolkit, and resulting empirical study. Each of the 14.4K samples is structured as a three-turn, bilingual (English/Chinese) dialogue with progressively varying style intensity: turn one elicits a neutral baseline response, and turns two and three re-synthesize the same underlying content under increasingly strong or increasingly weak stylistic instructions, forming either an intensifying or attenuating trajectory. Only one style dimension is varied per sample while semantic content is held fixed, and prompts are phrased conversationally rather than as templated commands or categorical labels. For the emotion subset, textual QA content conveys the target emotion contextually while avoiding explicit affective vocabulary, to prevent label leakage; for speed, volume, and pitch, a shared pool of semantically neutral dialogues is reused so that evaluation isolates acoustic variation. All prompt and reference audio is synthesized with CosyVoice2, with the emotion subset additionally referencing RAVDESS recordings for emotional intensity and the other three dimensions post-processed with FFmpeg; eight speaker timbres (4 English, 4 Chinese) are assigned across each subset for timbre diversity.
For evaluation, the authors define a three-stage protocol. The first two stages check single-turn instruction-following (Single-turn Relevance Degree, SRD) and multi-turn semantic consistency (Multi-turn Relevance Degree, MRD), both scored by Qwen3-4B-Instruct as an LLM judge of question-answer relevance; only models exceeding an MRD threshold of 40% are carried forward to style evaluation, since coherent multi-turn tracking is treated as a prerequisite for meaningful style-control assessment. The third stage introduces two custom metrics: Valid Sample Percentage (VSP), the proportion of samples for which a model produces a distinct, intended stylistic output, and Style Variation Degree (SVD), the magnitude of style-score shift between consecutive turns for quantifiable dimensions (speed via Whisper-large-v3-derived syllables-per-minute, volume via waveform RMS energy, pitch via average F0 over voiced frames, following the FastSpeech2 definition). Emotional intensity, which the authors find unreliable to score automatically with classifiers such as Emotion2Vec, is instead judged by human raters. Ten open-source end-to-end SLMs and omni language models (OLMs) are evaluated: Mini-Omni, Mini-Omni2, SLAM-Omni, Freeze-Omni, MiniCPM-o 2.6, Qwen2.5-Omni, Baichuan-Omni-1.5, LLaMA-Omni2, GLM-4-Voice, and Kimi-Audio.
Key Results
Semantic instruction-following diverges sharply from multi-turn consistency: while most models achieve high single-turn relevance (SRD > 89% for 7 of 10 models), only Qwen2.5-Omni (64.51%), GLM-4-Voice (69.31%), and Kimi-Audio (67.43%) exceed 60% MRD, and the smallest models (Mini-Omni, Mini-Omni2) fall well below the 40% eligibility threshold and are excluded from style evaluation entirely (Table 2). Among the three eligible large-scale models plus GLM-4-Voice and Kimi-Audio evaluated on emotion, Kimi-Audio leads every emotion category in Turn-2 VSP (e.g. 76.25% on Disgusted, 73.13% on Surprised) but its intensity effect diminishes by Turn 3, falling behind GLM-4-Voice; LLaMA-Omni2 and Baichuan-Omni-1.5 are almost entirely unresponsive to emotional instructions (Table 3). For speed, volume, and pitch, Kimi-Audio and GLM-4-Voice again lead on both VSP and SVD (e.g. GLM-4-Voice reaches 31.96% SVD on Volume Turn1-to-Turn2, versus 7.5% for Qwen2.5-Omni), while the remaining models frequently fail to produce a valid stylistic response or show negligible intensity variation (Table 4). No commercial or closed-source omni models (e.g. GPT-4o) are evaluated; the study is confined to the 10 open-source systems.
Novelty Assessment
The architectural and modeling components (CosyVoice2 for synthesis, RAVDESS and FFmpeg for reference/post-processing, existing open-source SLMs as subjects) are all pre-existing; StyleBench’s contribution is the benchmark construction and evaluation methodology, not a new generative model. The dataset design (bilingual, three-turn, single-dimension-isolated, four-dimension coverage of emotion/speed/volume/pitch) and the VSP/SVD metric pair for jointly measuring response validity and intensity magnitude are the genuinely new elements. The diagnostic analysis linking performance gaps to training-data composition and speech-tokenizer design is an empirical observation rather than a controlled ablation, since it draws on existing published differences between models rather than new intervention experiments.
Field Significance
moderate — StyleBench fills a genuine gap in evaluating graded, multi-turn style-intensity control for speech language models, an axis that prior single-turn or coarse-emotion-only benchmarks did not measure. Its main value to the field is the empirical characterization of large, consistent performance gaps among size-comparable open-source SLMs and OLMs, together with the VSP/SVD metric pair as a reusable measurement toolkit for this specific capability.
Claims
- supports: Multi-turn semantic coherence is a necessary precondition for evaluating a conversational speech system’s style-control ability, and should be measured and gated before finer-grained style metrics are applied.
Evidence: Only 3 of 10 evaluated SLMs exceeded a 60% Multi-turn Relevance Degree despite most scoring highly on single-turn relevance; the study applies an MRD > 40% threshold to filter which models are eligible for style-control evaluation at all. (§3.3, Table 2)
- supports: Differences in pretraining data composition, specifically whether a model is trained mainly on conventional ASR/spoken-QA-style corpora versus unsupervised natural dialogue or style-targeted data, are a primary driver of a speech language model’s ability to follow conversational style-intensity instructions, independent of parameter count.
Evidence: Among similarly-sized 7-9B models, GLM-4-Voice (trained with unsupervised speech data) and Kimi-Audio (trained with a dataset explicitly designed for style control) substantially outperform LLaMA-Omni2 and Baichuan-Omni-1.5, which were curated mainly for ASR and spoken QA and show almost no response to emotional-style instructions. (§3.4.1, Table 3)
- supports: The design of the discrete speech tokenizer independently shapes how well paralinguistic style cues are preserved and reproduced, separate from the waveform decoder used downstream.
Evidence: Most evaluated SLMs share a flow-matching-based decoder, yet performance still diverges sharply; GLM-4-Voice’s independently trained tokenizer is found to retain more semantic and acoustic style information than tokenizers built on SpeechTokenizer or Whisper-large-v3, and speech tokens synthesized from identical text under different styles are shown to differ, indicating tokens inherently carry paralinguistic cues. (§3.4.2)
- complicates: Automatic classifier-based scoring is unreliable for quantifying fine-grained, graded emotional style intensity in synthesized conversational speech, limiting how much of this evaluation dimension can be fully automated.
Evidence: Emotion2Vec-based classification reliability drops when scoring style intensity in synthesized data, and its softmax confidence score saturates above 0.95, failing to capture fine-grained turn-to-turn changes; the authors fall back on human evaluation for the emotion dimension while using automatic acoustic metrics (SPM, RMS, F0) for speed, volume, and pitch. (§3.1)
- complicates: Strong single-turn instruction-following performance does not reliably predict a speech language model’s ability to sustain consistent semantic and stylistic control across multi-turn dialogue.
Evidence: Several models with high Single-turn Relevance Degree (e.g. Freeze-Omni at 91.94%, MiniCPM-o 2.6 at 95.07%) drop to Multi-turn Relevance Degree scores of 26.46% and 26.18% respectively, well below the study’s 40% eligibility threshold for style-control evaluation. (§3.3, Table 2)
Limitations and Open Questions
The benchmark evaluates only 10 open-source SLMs and OLMs; no commercial or closed-source omni model (e.g. GPT-4o, which the paper cites as having established the interactive paradigm) is included, so the study cannot establish where the current state of the art actually stands on style-intensity control.
The emotion dimension depends entirely on human evaluation because the authors found no automated classifier reliable at this granularity, which limits the scalability and exact reproducibility of that portion of the benchmark relative to the automatically-scored speed, volume, and pitch dimensions. Coverage is restricted to two languages (English and Chinese) and eight speaker timbres. The diagnostic explanations offered for the observed performance gaps (training data composition, tokenizer design) are drawn from comparing published model differences rather than from controlled ablations run by the authors themselves, so they should be read as plausible explanatory hypotheses rather than causally isolated findings.
Wiki Connections
- Spoken Language Model — StyleBench is a benchmark built specifically to stress-test the conversational style-control capabilities that speech language models claim to offer.
- Speech-to-Speech — The evaluated systems are end-to-end spoken dialogue agents that take and produce speech across multi-turn conversations, the setting StyleBench is designed to probe.
- Evaluation Metrics — Introduces two new automatic metrics, Valid Sample Percentage and Style Variation Degree, for jointly measuring whether a stylistic instruction was followed and by how much.
- Subjective Evaluation — Relies on human raters to judge emotional intensity variation after finding automatic classifiers unreliable at fine granularity.
- Emotion Synthesis — Directly evaluates and compares how well existing SLMs can intensify or attenuate emotional expression in speech responses across dialogue turns.
- Kimi-Audio — Evaluated as one of the two strongest models on style-intensity control across all four dimensions, attributed partly to its style-control-specific training data.
- GLM-4-Voice — Evaluated as the other top-performing model; its independently trained speech tokenizer is analyzed as a likely contributor to its superior style retention.
- Qwen2.5-Omni — One of only three models to clear the multi-turn relevance threshold, used as a mid-tier comparison point on the style-control dimensions.
- LLaMA-Omni2 — Evaluated model shown to be almost entirely unresponsive to emotional-style instructions, illustrating the effect of ASR/QA-focused training data.
- Freeze-Omni — Evaluated model with high single-turn relevance but low multi-turn relevance, excluded from style-control evaluation by the paper’s eligibility threshold.
- CosyVoice 2 — Used as the speech synthesizer that generates all prompt and reference audio in the StyleBench dataset.
- SpeechTokenizer — Cited as a baseline tokenizer design against which GLM-4-Voice’s independently trained tokenizer is compared for style-cue retention.