arXiv · 2025 · Preprint
Huang et al. (Fudan University) · → Paper · Demo: ? · Code: ?
InstructTTSEval is a hierarchical benchmark of 6,000 test cases (English and Chinese) that measures how well instruction-conditioned TTS systems follow complex natural-language paralinguistic descriptions, exposing large gaps between commercial and open-source models.
Problem
Existing evaluation frameworks for TTS measure intelligibility (WER) and speaker similarity, but provide no standardised way to measure whether a system correctly follows natural-language style instructions specifying acoustic properties such as pitch, speed, emotion, or timbre. Subjective MOS evaluations are expensive and inconsistently administered. Prior datasets with style descriptions are limited to a small number of fixed categorical labels (five to eleven features) and rely on manual annotation or narrow classifiers, leaving dynamic acoustic events — transitions between emotional states, vocal events like sighs or laughter — entirely uncovered.
Method
InstructTTSEval constructs its test set bottom-up from expressive speech in film, television, and variety shows. Audio segments are cleaned with speaker diarization (pyannote.audio), transcribed with Whisper-large-v3, and filtered by DNSMOS (threshold 2.8) and the DVA arousal/dominance toolkit to retain only high-expressiveness, single-speaker clips longer than 3 seconds. This yields 2,000 reference audio segments (1,000 English, 1,000 Chinese) drawn from the NCSSD dataset and additional self-collected material.
The benchmark covers 12 acoustic features organised into four tiers: physiological (gender, pitch, texture), linguistic (clarity, fluency, speed), social (accent, age, volume), and psychological or pragmatic (emotion, tone, personality). Three task types are defined at increasing levels of abstraction:
- Acoustic-Parameter Specification (APS): structured, fine-grained instructions covering all 12 features, generated by prompting Gemini to caption each reference clip.
- Descriptive-Style Directive (DSD): free-form prose descriptions rewritten from APS captions by GPT-4o, with random attribute dropout to simulate underspecified prompts.
- Role-Play (RP): high-level role or scenario descriptions (e.g., “a teacher scolding a student”) that require the TTS system to infer appropriate acoustic properties from world knowledge, using chain-of-thought GPT-4o generation.
Evaluation uses Gemini as a judge in a binary pass/fail setup: a synthesised output is scored “true” if its primary style attributes align with the instruction without conflict, and “false” if at least one key attribute clearly deviates. Human-Gemini consistency is validated on 50 reference samples per task and language, achieving 79% average agreement across tasks and languages (87% on APS, 79% on DSD, 71% on RP).
Key Results
Five TTS systems are evaluated on the English subset (Table 5): Gemini-2.5-flash-preview-tts achieves the highest average at 88.7%, followed by Hume (71.1%) and GPT-4o-mini-TTS (68.5%). Among open-source systems, VoxInstruct leads at 50.4%, with Parler-TTS-mini (46.9%), Parler-TTS-large (45.7%), and PromptTTS/PromptStyle below 48%. On the Chinese subset (Table 6), VoxInstruct is the only open-source system evaluated; closed-source Gemini models reach 84–85% while GPT-4o-mini-TTS drops sharply to 51.1%, suggesting weaker Chinese prosodic control.
Reference audio itself does not score perfectly: APS accuracy is 96.2% (EN) / 90.9% (ZH), confirming that RP instructions are inherently ambiguous and the benchmark imposes a realistic upper bound rather than a trivially achievable one.
Case studies reveal three systematic failure modes across all evaluated systems: reproduction of paralinguistic sound events (sighs, laughter, screams), extreme emotional intensity and rapid affective transitions, and singing or melodic vocal production.
Novelty Assessment
The benchmark’s primary technical contribution is the automated caption-then-instruction pipeline using Gemini for audio captioning and GPT-4o for diverse instruction generation, enabling scalable, annotation-free dataset construction that captures dynamic vocal transitions. This is a genuine improvement over prior datasets relying on fixed categorical labels. The three-tier task design (APS to DSD to RP) that maps low-level acoustic attributes to high-level social roles is a useful framing for evaluating generalisation. However, the evaluation methodology is straightforward (binary LLM-as-judge), and the main limitations — reliance on a proprietary LLM as judge, self-evaluator bias (Gemini scoring Gemini outputs), and class imbalance in the mined data — are acknowledged but not resolved.
Field Significance
Moderate — InstructTTSEval fills a concrete gap: no prior benchmark measures instruction-following in the paralinguistic dimension with this breadth or scale. The finding that even top commercial systems fall well short of reference audio on fine-grained acoustic control, and that open-source systems lag by 20–40 percentage points, provides actionable evidence of where the field stands. The LLM-as-judge setup, while cost-efficient, introduces self-evaluator bias that limits the reliability of scores for Gemini models specifically.
Claims
- Fine-grained paralinguistic control — independent manipulation of multiple acoustic attributes from natural-language instructions — remains an open challenge for current TTS systems, including commercial offerings. (§4.3, Table 5)
- A substantial performance gap exists between closed-source commercial TTS systems and open-source alternatives on instruction-following benchmarks, widening as instruction abstraction increases. (§4.3, Tables 5–6)
- Timbre controllability and emotional expressiveness represent orthogonal capabilities in current TTS systems, as models that succeed on one dimension frequently fail on the other. (§4.4, Table 7)
- Automatic LLM-based evaluation of instruction-following in speech achieves approximately 79% agreement with human annotators, with lower reliability on subjective role-play tasks. (§4.1, Table 4)
- Paralinguistic vocal events such as laughter, sighs, and screams are not reliably reproducible by any current TTS system when specified in natural-language instructions. (§4.4, Table 7)
Limitations and Open Questions
Warning
Gemini is used both as the caption generator for benchmark construction and as the judge evaluating model outputs. Gemini TTS models score highest on the benchmark, and the authors acknowledge LLM self-evaluator bias (Panickssery et al., 2024) as a confound. Gemini scores should be treated with caution.
The RP task is inherently subjective: a scenario description like “nervous applicant in an interview” admits many valid acoustic realisations, and lower human-Gemini agreement (71% average) reflects this ambiguity rather than evaluator failure. The dataset construction pipeline draws from film and television sources, so emotional and stylistic distributions reflect those domains; certain categories (specific emotions, role archetypes) are under-represented by design. Continuous large-scale evaluation using Gemini-as-a-judge costs roughly $25 per full evaluation run, which may limit iterative use.
Wiki Connections
- instruction-conditioned-tts — InstructTTSEval is the first benchmark specifically designed to measure how well TTS systems follow complex, free-form paralinguistic instructions, providing a standardised reference for this capability.
- evaluation-metrics — the paper introduces a hierarchical evaluation framework using LLM-as-a-judge, adding a scalable alternative to human MOS for style-instruction compliance.
- emotion-synthesis — case studies surface which aspects of emotional expressiveness (vocal events, extreme intensity, affective transitions) remain unsolved across evaluated systems.
- 2402.01912 (ParlerTTS) — Parler-TTS-mini and -large are among the open-source systems evaluated; results show limited performance on DSD and RP tasks despite natural-language conditioning.
- 2407.05407 (CosyVoice) — cited as representative of special-token-based controllable TTS; contrasted with free-form instruction approaches in the related work.
- 2412.10117 (CosyVoice 2) — cited as a system that extends instruction-following to natural-language style prompts, representing the benchmark’s target capability class.
- 2409.00750 (MaskGCT) — cited as a state-of-the-art zero-shot TTS system benchmarked in this evaluation context.
- 2410.06885 (F5-TTS) — cited among recent high-performing TTS systems surveyed in the introduction.
- 2502.04128 (LLaSA) — cited as a scaling study for LLM-based speech synthesis relevant to the systems being benchmarked.
- 2503.01710 (SparkTTS) — cited among recent LLM-based TTS systems in the introduction.
- 2503.04713 (ParaSpeechCraft) — compared directly in Table 1 as a prior style-description dataset with 11 labels, which InstructTTSEval extends.
- 2006.04558 (FastSpeech 2) — cited as a representative early TTS baseline in the background discussion.