arXiv · 2026 · Preprint
Huakang Chen et al. (Northwestern Polytechnical University) · → Paper · Demo: ✓ · Code: ?
MINT-Bench is a ten-language benchmark for instruction-following TTS. It is organised by a four-axis taxonomy of control cases, and its staged protocol checks content consistency with ASR, judges instruction following with an audio LLM, and then awards perceptual-quality bonuses only to samples that fully follow the instruction.
Problem
Instruction-following TTS lets users describe the voice they want in free-form natural language (timbre, style, persona, dynamic changes, nonverbal events) instead of picking from fixed tags. Standard TTS metrics (WER, speaker similarity, MOS) do not measure whether such instructions were carried out. Earlier instruction-following benchmarks have three gaps, according to the authors. Their coverage is loosely organised, especially for timbre, composition and extra-vocal behaviour. Their judgments are too coarse to tell content errors, failed instruction following and poor audio quality apart. And they offer little support for languages beyond English and Chinese. Speech also has many valid realisations for one instruction, so evaluation cannot simply compare outputs with a reference.
Method
MINT-Bench has three parts: a taxonomy, a data-construction pipeline, and an evaluation protocol. The paper trains no TTS model of its own.
Hierarchical multi-axis taxonomy. The benchmark is built on ten atomic attributes. Four are timbre attributes (age, gender, pitch, texture) and six are style attributes (speed, volume, emotion, accent, personality, tone), each with a discrete set of values (Table 1, Table 6). A benchmark case (“taxonomy node”) is one point in a four-axis space:
- difficulty level: Easy, Hard, or Special;
- control domain: timbre, style, composition, or extra-vocal;
- control specification: tag, direct description, simple or complex composition, implicit or explicit cue;
- fine-grained control pattern: static, dynamic, layered, conflictive, scenario-based, or character-based.
Easy cases are single attributes or simple pairs. Hard cases are complex compositions of three or more attributes, attributes that change over the utterance (dynamic), emotional subtext (layered), mismatched cues (conflict), and scenario or character personas. Special cases cover disfluency, dysphonia, and implicit or explicit nonverbal vocalisations.
Three-stage data construction. An LLM (Gemini 2.5 Pro) builds each item in three steps: (1) node specification with an item budget per language, (2) a structured label plan that fixes the target values, text semantics, text length, and anti-leakage constraints, and (3) realisation as a natural-language instruction plus a synthesis text. The text is written so that it does not lexically give away the target; for example, a cough instruction cannot be paired with text about illness. Separating planning from realisation is meant to prevent attribute omission, semantic drift and target leakage, which the authors observe in one-shot prompt generation. The large Chinese and English splits have 890 instruction-text pairs each (Appendix A.2). The other eight languages (German, Spanish, French, Italian, Japanese, Korean, Portuguese, Russian) use mini splits of 274 pairs, sampled at about 30% of the node budget.
Hierarchical hybrid evaluation. Evaluation runs in three stages.
- Content consistency. ASR (Paraformer-zh for Chinese, Whisper large-v3 for the other languages) gives a WER for each sample. The WER threshold for each language is the trimmed mean of the system-level WERs across all evaluated systems. The coefficient p is the fraction of samples below the threshold, and it scales both later scores. Accent and Special subsets skip this check because ASR is unreliable on them.
- Instruction following. Gemini 3.1 Pro Preview acts as the audio-LLM judge, with a separate prompt family for Easy, Hard and Special cases. It scores each sample on a three-level scale (1 to 3). The system IF score is p times the mean judge score.
- Perceptual quality and timbre diversity. Samples scoring 3 can earn two binary bonuses, one for naturalness and one for expressiveness, which gives a 1 to 5 PE score. For timbre diversity, each system generates several candidates per instruction. Using WavLM-Large speaker embeddings, the Timbre Diversity Score (TDS) is (1 − mean pairwise similarity) × (fraction of candidates that fully follow the instruction).
The judge was validated with a human study across all ten languages: 30 native-speaker annotators each for English and Chinese, and 10 per language for the rest. They rated four systems on a web interface using the same three-level instruction-following scale and the same bonuses (§4.4, Appendix C.3).
Key Results
The benchmark evaluates eight open-source systems (VoiceSculptor, MiMo-Audio-7B-Instruct, Qwen3TTS-12Hz-1.7B-VD, Ming-omni-tts 16.8B-A3B and 0.5B, Parler-TTS Mini and Large, MOSS-VoiceGenerator) and six commercial systems (Gemini 2.5-Flash and 2.5-Pro, GPT-4o-mini-TTS, ElevenLabs-ttv-v3, MiniMax-Speech-2.7, Hume-Octave1). Each system is evaluated only on the languages it supports.
- English and Chinese (Table 3). Commercial systems lead on English: Gemini 2.5-Flash scores IF/PE 2.44/3.66 and Gemini 2.5-Pro 2.39/3.45, ahead of the best open-source system, Qwen3TTS-12Hz-1.7B-VD (2.12/2.94). On Chinese the order flips: Qwen3TTS has the best overall PE (3.12), above Gemini 2.5-Flash (2.95) and 2.5-Pro (2.93). Most systems’ WERs fall between 0.6% and 3.5%, yet their IF/PE scores vary widely. The outliers are VoiceSculptor (9.2% on Chinese) and Parler-TTS Mini (17.5% on English, with an IF/PE of only 1.04/1.13).
- Difficulty strata. Scores drop from Easy to the Hard and Special subsets. Disfluency control is the lowest subset score for most systems; for example, ElevenLabs-ttv-v3 English scores 0.89 on disfluency against 3.40 on explicit nonverbal events. Dysphonia and explicit nonverbal control remain partly achievable for the strongest systems.
- Timbre diversity. TDS does not follow the main ranking. Gemini 2.5-Pro has the highest commercial TDS (0.57 on English). MiMo-Audio-7B-Instruct has a fairly high TDS (0.52 on English) despite weak IF/PE scores.
- Ten languages (Table 4). Gemini 2.5-Flash leads the aggregate (IF/PE 2.50/3.75), followed by Gemini 2.5-Pro (2.48/3.63). Qwen3TTS is the strongest open-source system (2.29/3.19). Other open-source systems fall sharply outside Chinese and English. MiMo-Audio, for example, has per-language PE scores between 0.19 and 0.59 on the eight non-primary languages, against 2.22 on Chinese and English. Gemini 2.5-Pro beats 2.5-Flash on Japanese and Korean.
- Judge validity (Table 5). Spearman correlation between the audio-LLM judge and the mean human score ranges from 67.12 to 77.35 (×100) across languages. Agreement between human annotators ranges from 69.45 to 79.12. The judge stays about 2 to 3 points below the human ceiling in every language. The analysis uses one-fifth of the samples that pass the WER check.
Novelty Assessment
The contribution is evaluation infrastructure, not a model. Three parts are new compared with earlier instruction-following benchmarks such as InstructTTSEval and VStyle:
- The taxonomy is explicit and combinatorial: it separates difficulty, control domain, how the instruction is phrased, and the structural pattern. This allows per-subset diagnosis instead of a single flat score.
- The scoring is gated. WER-based content consistency scales the judge scores, and perceptual bonuses go only to samples that fully follow the instruction, so content, control and quality are kept separate.
- The benchmark covers ten languages, and the judge is validated by native-speaker raters in each one.
The parts themselves are established: audio-LLM-as-judge scoring, LLM-generated prompts, ASR-based WER, and speaker-embedding diversity. The novelty is in how they are combined and in the coverage design. The trimmed-mean WER threshold makes the content coefficient depend on which systems are in the comparison pool. A system’s IF/PE score can therefore change when other systems are added or removed, which works against using the benchmark as a stable leaderboard. The multilingual design is uneven: eight of the ten languages have only 274 items each.
Field Significance
Moderate. The paper provides a structured, multilingual diagnostic benchmark for instruction-following TTS, and a fairly broad comparison of fourteen open-source and commercial systems on it. Its main empirical findings are threefold. Low WER does not imply good instruction following. Dynamic, layered, conflictive and extra-vocal (especially disfluency) controls are where current systems fail. And open-source multilingual instruction-following lags far behind the best commercial systems outside Chinese and English. Its value as a durable reference depends on the release of the toolkit and on how stable the proprietary judge turns out to be.
Claims
- complicates: Low word error rate is not a reliable indicator of instruction-following ability in controllable TTS. Content fidelity and controllability should be measured separately.
Evidence: On the English and Chinese large splits, most systems reach WERs between 0.6% and 3.5%, yet their overall IF/PE scores range from about 1.4/1.7 to 2.4/3.7. Similar WERs do not translate into similar instruction-following scores. (§4.2, Table 3)
- supports: Instruction-following TTS systems handle single-attribute instructions much better than dynamic, layered, conflictive or extra-vocal ones. Disfluency control is especially hard.
Evidence: Across fourteen systems, scores fall from the Easy subsets to the Hard (dynamic, layered, conflict, scenario, character) and Special subsets. Disfluency is the lowest subset for most systems, e.g. ElevenLabs-ttv-v3 English 0.89 on disfluency vs 3.40 on explicit nonverbal events. (§4.2, Table 3)
- supports: Audio-language-model judges, given task-specific rubrics, can approach human agreement levels when scoring whether synthesised speech follows a natural-language style instruction.
Evidence: The Spearman correlation between the Gemini 3.1 Pro Preview judge and the mean human rating (67.12 to 77.35) stays within about 2 to 3 points of inter-human agreement (69.45 to 79.12) in all ten languages. The study used 30 native annotators each for English and Chinese and 10 for each other language, on four systems. (§4.4, Table 5)
- complicates: Instruction-following ability in multilingual TTS is uneven across languages. Strength in the primary training languages does not carry over to other languages, particularly for open-source systems.
Evidence: Four of the five open-source systems in the ten-language table score PE above 2.2 on Chinese but below 1.4 on each of the eight languages other than Chinese and English; MiMo-Audio-7B-Instruct, for example, scores between 0.19 and 0.59 there. Even Gemini 2.5-Pro beats the higher-ranked 2.5-Flash on Japanese and Korean. (§4.3, Table 4)
- refines: Generation diversity under a fixed voice-design instruction is a separate axis from controllability, and should be reported as its own diagnostic.
Evidence: The timbre diversity score (TDS) counts only candidates that fully follow the instruction, yet it does not track the IF/PE ranking. MiMo-Audio-7B-Instruct has a fairly high English TDS (0.52) despite low IF/PE (1.75/2.22). (§3.4, §4.2, Table 3)
Limitations and Open Questions
Warning
The instruction-following judge (Gemini 3.1 Pro Preview) and the benchmark-construction LLM (Gemini 2.5 Pro) come from the same vendor as the two top-ranked systems (Gemini 2.5-Flash and 2.5-Pro). The judge is also proprietary and its version can change. The human-agreement study covers only four systems, and its correlations are pooled rather than computed per system, so it does not rule out same-vendor preference in the commercial ranking.
- Pool-dependent content gate. Each language’s WER threshold is the trimmed mean of the evaluated systems’ WERs, so a system’s scores depend on which other systems are included (§3.4, Appendix C.2).
- Uneven multilingual coverage. Only Chinese and English have large splits (890 items); the other eight languages have 274-item mini splits, so cross-language comparisons are less reliable (§6). Fine-grained accent control is defined mainly for English and Chinese (Appendix A.1).
- Not every system covers every language. Systems are evaluated only on the languages they support (§4.1), so multilingual aggregates are not over identical language sets for every system.
- Bounded scope. Long-form generation, dialogue context, richer discourse and code-switching are excluded (§6).
- Coarse scoring. Judge scores are three-level, plus two binary bonuses. The Special and accent subsets skip the WER check, so content errors there go unpenalised.
- Release status. The paper says the toolkit is or will be released, but it gives no code repository link beyond the project page.
Wiki Connections
- Instruction-Conditioned TTS — Directly benchmarks natural-language instruction-following TTS across a four-axis taxonomy of timbre, style, composition and extra-vocal control in ten languages.
- Evaluation Metrics — Proposes a gated hybrid scoring protocol (WER content coefficient, three-level audio-LLM instruction-following score, conditional naturalness/expressiveness bonuses) and a timbre diversity score.
- Subjective Evaluation — Validates the audio-LLM judge against native-speaker human ratings in all ten languages, reporting inter-human and judge-to-human Spearman agreement.
- InstructTTSEval — The closest earlier instruction-following TTS benchmark; MINT-Bench argues it lacks diagnostic breakdown by control pattern and broad multilingual coverage.
- VStyle — An earlier voice-style benchmark that MINT-Bench sets apart as aimed at spoken language models and dependent on a single LALM judge.
- Speech-DRAME — A role-play-oriented, human-aligned speech evaluation framework that MINT-Bench positions as narrower in scope.
- Qwen3-TTS — Its voice-design variant (Qwen3TTS-12Hz-1.7B-VD) is the strongest open-source system evaluated and leads all systems on Chinese.
- Gemini 2.5 — Gemini 2.5-Flash and 2.5-Pro TTS are the top-ranked commercial systems; Gemini 2.5 Pro also generates the benchmark items.
- Parler-TTS — Parler-TTS Mini and Large are evaluated as description-conditioned open-source baselines and score near the bottom.
- VoiceSculptor — An evaluated open-source voice-design system from the same lab, tested on Chinese only.
- MiMo-Audio — The evaluated MiMo-Audio-7B-Instruct shows a high timbre diversity score despite weak instruction following, which illustrates that TDS is a separate diagnostic.