arXiv · 2026 · Preprint
Ruohan Liu et al. (Nanjing University / Xiaomi) · → Paper · Demo: ? · Code: ?
SpeechParaling-Bench is a bilingual (English-Chinese) benchmark of 1,001 spoken queries spanning 13 paralinguistic dimensions and over 100 features. It tests whether large audio-language models can produce speech with the requested style, vary it within an utterance, and adapt it to a user’s implicit vocal cues. Scoring uses a baseline-relative pairwise LALM judge, validated against human preference.
Problem
Large audio-language models (LALMs) and voice assistants now generate speech directly, and frontier systems claim expressive, style-aware output. Existing evaluation for this capability is thin. General audio benchmarks such as AIR-Bench and MMAU test understanding with text answers. Paralinguistic dialogue benchmarks (StepEval-Audio-Paralinguistic, EChat-eval, ParaS2SBench) cover a small set of cues, mostly emotion, age and gender, and fewer than 50 features (Table 2). Most do not test whether a model can control paralinguistic features on request, or vary them within one utterance. Paralinguistic quality is also hard to score reliably: absolute ratings are subjective, transcription-based scoring discards the acoustic signal, and human listening tests are expensive.
Method
Task design. The benchmark has three tasks of increasing difficulty (§3.1):
- Paralanguage Control. A spoken instruction asks the model to repeat a sentence with specified features. It covers 12 common dimensions in single- and multi-dimension settings: expressive (emotion, attitude, cognitive state, non-linguistic vocalizations), prosodic (pace, pause, stress, rhythm) and acoustic (pitch, timbre, volume, age). It also includes holistic “abstract style” prompts (e.g. “a lively and mischievous voice”).
- Dynamic Variation. The model must change one of 8 dimensions within a single utterance. Changes are either transitional (Happy → Sad) or progressive (whisper rising to normal volume).
- Situational Adaptation. The model hears a user utterance whose age, emotion, attitude or non-linguistic vocalization (laughter, sighs) carries implicit cues. It must reply with appropriate content and speaking style, and there is no reference answer.
The 13 dimensions and their feature values are listed in Appendix A (Table 6).
Data engine. Gemini 2.5 Flash writes the text instructions and their dimension labels (§3.2). The prompts cover five real-world settings (campus, workplace, daily life, family, entertainment) with five scenarios each, and include in-context demonstrations. Dimension sets are fed in batches to keep coverage balanced. Control and variation items use the fixed template “Please read this sentence {Dimension(s)}: {Sentence}”. IndexTTS2 (IndexTTS2) turns the text into spoken queries. Control and variation items use one fixed neutral male reference voice. Situational items encode age and attitude through timbre and style reference clips, emotion through IndexTTS2’s emotion vector, and vocalizations through textual hints. All samples are checked manually for intelligibility and plausibility. The final set has 1,001 samples. English queries average 27.4 words and 10.3 s; Chinese queries average 35.3 characters and 8.5 s (Table 1).
Pairwise evaluation pipeline. Gemini 3 Pro acts as the judge (§3.3). Following EmergentTTS-Eval, it compares each candidate’s audio response against a fixed baseline’s response to the same query. The baseline is Doubao Realtime Voice for Chinese and Gemini Audio for English. The judge receives both audio responses, the query transcript, the target dimensions and the criteria. It reasons step by step, citing timestamps, and scores each response 0-3 per aspect:
- Control and variation: Content Accuracy, Fluency and Naturalness, Paralinguistic Compliance.
- Situational Adaptation: Content Relevance, Fluency and Naturalness, Paralinguistic Alignment.
It then names a winner or a tie. Presentation order is randomised to reduce position bias. A win earns 1 point and a tie 0.5 each. The baseline’s own score is a weighted average of its pairwise results against all candidates, weighted by each candidate’s relative strength (Eq. 3, motivated by Chatbot Arena). Scores are normalised to 0-100 per task.
The benchmark proposes no generation model. Five API-accessed systems are evaluated with default decoding (§4): gpt-audio-2025-08-28, Gemini 2.5 Flash Audio, Doubao Realtime Voice, Qwen3-Omni-Flash-2025-12-01 and Qwen3-Omni-Realtime (Qwen3-Omni).
Key Results
- Language split in overall rankings (Table 3). On Chinese, Doubao Realtime Voice leads overall (70.84) and on Paralanguage Control (71.86); GPT Audio is next (39.09 overall). On English, Gemini Audio leads (64.97 overall, 66.49 Paralanguage Control); GPT Audio is next (49.39) and Doubao drops to 31.39. The top system in each language is also that language’s pairwise baseline, so these baseline scores come from the weighted scheme in Eq. 3 rather than from direct head-to-head comparison.
- Qwen3-Omni models fail on control but not on dialogue. Both score in single digits on Paralanguage Control (4.72 to 14.16 Total) and very low on English Dynamic Variation (5.00 and 7.92). Qwen3-Omni-Realtime nevertheless places among the top three on Situational Adaptation in both languages (49.29 ZH, 48.57 EN), so instruction-following for style and empathetic reply generation separate clearly.
- Different models are strong on different dimensions (Figure 5, Chinese Paralanguage Control). Doubao is stronger on expressive and acoustic features but weaker on prosody. GPT and Gemini do better on pause and stress but lag on expressive dimensions. The authors attribute the split to training-corpus and language differences.
- Dynamic Variation is the hardest task (Table 4). Averaged over all models, the judge’s initial 0-100 scores are 56.51 for Dynamic Variation, 66.01 for Paralanguage Control and 68.64 for Situational Adaptation.
- Judge agreement with humans (§4.2). On 416 response pairs (5% of each task), judged by human raters with criteria matching the judge prompt, Spearman rank correlation between the judge-derived and human-derived model rankings is 0.90 (Chinese) and 1.00 (English). Both are reported as statistically significant. These correlations are over the rankings of five systems, so they say little about agreement on individual items.
- Failure analysis (§4.3, Figure 6, Table 5). Gemini Audio fails 67 of 190 Chinese Situational Adaptation samples. The failures fall into role-play failure, intent misunderstanding, template response and neglecting paralanguage. Ignoring the user’s paralinguistic cues accounts for 43.3% of them.
Novelty Assessment
The contribution is a benchmark, not a model. The main advances are breadth and task structure. Coverage roughly doubles relative to prior paralinguistic benchmarks. Within-utterance Dynamic Variation is separated out as its own task, which only VStyle (VStyle) also covers among the compared benchmarks. Control, variation and situational adaptation are scored together with one pipeline.
The evaluation method mostly adapts existing work. Baseline-candidate pairwise judging with randomised order comes from EmergentTTS-Eval, and the strength-weighted baseline score borrows from Chatbot Arena-style aggregation. Applying it to direct audio judgment of speech-to-speech responses, with timestamp-grounded chain-of-thought, is a reasonable engineering step rather than a methodological break.
The data are fully synthetic. Instructions come from an LLM and audio queries from a single TTS system, so the benchmark measures responses to TTS-rendered prompts rather than natural human speech. Feature counts are reported inconsistently: Table 1 lists 104 features, Table 2 lists 101 for this benchmark, and the text says “over 100”.
Field Significance
Moderate — The benchmark breaks expressive speech generation by voice assistants into separately measurable skills: static control, within-utterance modulation, and cue-sensitive response. It shows that current proprietary and open models handle them very unevenly. It is a useful diagnostic resource, though its judge-based scores depend on a single proprietary judge and a small human validation.
Claims
- complicates: Fine-grained paralinguistic control in end-to-end spoken assistants is uneven across feature types and languages: no current system controls expressive, prosodic and acoustic dimensions equally well.
Evidence: On Chinese Paralanguage Control, Doubao is stronger on expressive and acoustic features but weaker on prosodic ones, while GPT and Gemini lead on pause and stress but trail on expressive dimensions. Model rankings flip between the Chinese and English subsets (Doubao 71.86 vs. 28.05 control score). (§4.1, Table 3, Figure 5)
- supports: Changing paralinguistic features within a single utterance is harder for speech-generating language models than holding one style across an utterance.
Evidence: Averaged across five evaluated LALMs, the judge’s initial scores are lowest for Dynamic Variation (56.51/100), compared with 66.01 for Paralanguage Control and 68.64 for Situational Adaptation. The Qwen3-Omni models often fail to turn variation instructions into acoustic change (5.00 to 7.92 on English Dynamic Variation). (§4.1, Table 3, Table 4)
- complicates: Spoken dialogue models often fail to respond appropriately to paralinguistic cues in user speech, such as sarcasm or laughter, even when the lexical content is understood.
Evidence: Manual analysis of Gemini Audio on Chinese Situational Adaptation finds 67 failures out of 190 samples. 43.3% of these come from ignoring paralinguistic information, e.g. reading a sarcastic “truly ‘perfect’” literally as praise. (§4.3, Figure 6, Table 5)
- supports: Pairwise comparison against a fixed baseline, judged by an audio-capable LLM, can produce system rankings for paralinguistic speech quality that closely match human pairwise preferences.
Evidence: Human raters judged 416 response pairs (5% of each task). Spearman correlation between judge-derived and human-derived model rankings is 0.90 on Chinese and 1.00 on English, both statistically significant. (§4.2)
- complicates: In speech-generating language models, following explicit style instructions and producing context-appropriate empathetic replies are separable abilities, so one cannot stand in for the other.
Evidence: Qwen3-Omni-Realtime scores 4.72 (ZH) and 6.75 (EN) on Paralanguage Control yet places in the top three on Situational Adaptation in both languages (49.29 ZH, 48.57 EN). (§4.1, Table 3)
Limitations and Open Questions
Warning
All scores come from one proprietary judge, Gemini 3 Pro, and the English baseline is Gemini Audio, from the same model family. Gemini 2.5 Flash also wrote all benchmark instructions. The human validation is a rank correlation over five systems, so it does not rule out item-level or same-family judge bias.
- Queries are synthesised by a single TTS system (IndexTTS2), and Control and Variation items use one fixed male reference voice. Robustness to natural human speech, other speakers and recording conditions is untested.
- Scores are relative to a per-language baseline, so absolute numbers are not comparable across the Chinese and English subsets, and the baseline’s score depends on the candidate pool (Eq. 3).
- Only five API-accessed systems are evaluated, with default decoding. No open research checkpoints are included beyond the Qwen3-Omni product variants.
- The failure analysis covers only Gemini Audio on the Chinese Situational Adaptation subset.
- Feature totals differ between Table 1 (104) and Table 2 (101).
- Code and data release are not described in the paper beyond a project page.
Wiki Connections
- Instruction-Conditioned TTS — Paralanguage Control and Dynamic Variation test whether speech LMs follow natural-language style instructions such as “read this sentence with a sad emotion”.
- Emotion Synthesis — Emotion, attitude and cognitive state of the generated speech are scored directly, and emotion transitions within an utterance form part of Dynamic Variation.
- Prosody Control — Pace, pause, stress, rhythm, pitch and volume are explicitly requested and judged, both statically and as within-utterance changes.
- Speech-to-Speech — Situational Adaptation evaluates spoken dialogue responses to speech queries carrying implicit paralinguistic cues.
- Evaluation Metrics — Introduces a baseline-relative pairwise LALM-as-judge protocol with a strength-weighted baseline score for paralinguistic generation.
- Subjective Evaluation — Human pairwise preference judgments on 416 response pairs are used to validate the automatic judge’s system rankings.
- VStyle — The closest prior benchmark, and the only compared one that also covers control and dynamic variation; this paper extends feature coverage and adds pairwise judging.
- IndexTTS2 — Used as the TTS engine for all benchmark speech queries, providing timbre, style and emotion-vector control.
- Qwen3-Omni — Two Qwen3-Omni variants (Flash and Realtime) are among the evaluated systems and show the largest control-versus-dialogue gap.
- Step-Audio 2 — Source of StepEval-Audio-Paralinguistic, a prior paralinguistic benchmark this paper contrasts with on feature coverage.
- OSUM-EChat — Source of EChat-eval, an empathetic-dialogue benchmark dominated by emotion that this paper contrasts with.
- AIR-Bench — Listed in the benchmark comparison as an audio-understanding benchmark with none of the three generation-side evaluation aspects.
- VoiceBench — Cited as a speech-based instruction-following benchmark for voice assistants, assessed through content rather than paralinguistic generation.