Interspeech · 2026 · Conference
Liumeng Xue et al. (Nanjing University / HKUST) · → Paper · Demo: ? · Code: ?
NVBench is a bilingual (English/Chinese) benchmark for TTS with non-verbal vocalizations. It combines a 45-type NVV taxonomy, a class-balanced 4,500-item evaluation set, and a multi-axis protocol that scores NVV controllability, placement, and salience separately from speech naturalness and quality. It is applied to 15 commercial and open-source systems.
Problem
Non-verbal vocalizations (NVVs) such as laughter, sighs, sobs, coughs, and lip smacks carry affective and interactional meaning that lexical content does not. Many current TTS systems expose NVV control, either through inline tags ([laugh], [sigh]) or through natural-language captions. Their NVV inventories are small, skewed toward a few types such as laughter, and labelled inconsistently: across eight tag-based systems and six NVV datasets, supported inventories range from 1 to 17 types (Table 2). Standard TTS evaluation (WER, DNSMOS, naturalness MOS) cannot tell whether the requested NVV was produced, placed at the right point, and audible without harming the surrounding speech. So systems with different control interfaces could not be compared on a common footing.
Method
NVBench has three components.
Taxonomy. The authors surveyed the NVV inventories of representative tag-based systems and datasets. From these they built a unified 45-type taxonomy with six top-level categories: respiratory (10), throat/physiological (7), laughter spectrum (7), crying spectrum (5), emotional vocalizations (7), and oral/miscellaneous (9) (Table 1). Fine-grained subtypes are added only where the variation is perceptually salient and already distinguished in prior datasets or control interfaces. For example, laughter is split by intensity and function, while reflexes like cough stay coarse.
Data construction. A three-stage pipeline builds the evaluation set:
- Seed mining. Gemini 2.5 Pro labels, localizes, and re-captions NVVs in expressive bilingual recordings from InstructTTSEval. Three human annotators audit each candidate, with a fourth reviewer resolving disagreements. This yields about 80 English and 30 Chinese seeds.
- Taxonomy-driven generation. Conditioned on one target NVV type at a time, Gemini 2.5 Pro generates text-only items in a four-field schema:
text,text_with_nvv,caption_with_nvv, andnvv_list. - Validation and replenishment. Automatic schema and consistency checks run first, then manual quality control, which also removes items containing interjection tokens like “ah” or “hmm”. Generation repeats until each type has 50 validated items.
The result is 45 × 50 = 2,250 items per language, or 4,500 in total. Tag-based systems receive text_with_nvv; prompt-based systems receive caption_with_nvv.
Evaluation protocol. The protocol has three parts.
- Objective metrics, averaged over three synthesis runs:
- WER (Whisper-large-v3) for English and CER (paraformer-zh) for Chinese.
- DNSMOS P.835 (SIG/BAK/OVRL).
- CLAP caption-speech similarity, for prompt-based systems only.
- For tag-based systems, coverage (the fraction of the 45 types the system supports) plus NVV precision, recall, F1, and normalized tag distance (NTD).
- NVV detection for the objective metrics. Detection uses a “GT-conditioned” Gemini 2.5 Pro verifier, because open-ended detection hallucinated events. The verifier receives the synthesized audio, the reference transcript, and the target type. It returns a presence decision and inserts one inline marker, which gives an onset index. A match requires the correct type and an onset within a fixed tolerance δ of the ground truth.
- Human listening tests on Prolific with 97 raters, covering 450 samples per language (10 per type). Raters score Overall Naturalness, Overall Quality, and NVV Perceptual Effect (PE) for all systems. Prompt-based systems also get Caption-Speech Match and NVV Instruction Following (IF); tag-based systems also get NVV Accuracy and Overall Expression. All criteria use a 1-5 Likert scale, and NVV-specific criteria get a 0 when the NVV is absent.
- LLM multi-rater protocol. This repeats the human rubric with anonymized, order-randomized comparative judging at temperature 0.2. It uses four LLM rater instances (Gemini 2.5 Pro per §4.3), run in three rounds with three-fold partitions.
Fifteen systems are evaluated:
- Seven prompt-based: Parler-TTS Mini/Large, CapSpeech, Qwen3-TTS, GPT-4o mini TTS, Gemini 2.5 Flash, and Gemini 2.5 Pro.
- Eight tag-based: ChatTTS, Higgs-Audio, Bark, Fish-Speech, Orpheus TTS, CosyVoice 2, Dia, and ElevenLabs.
Parler-TTS, CapSpeech, and Dia are evaluated in English only.
Key Results
Objective results (Table 3).
- Prompt-based intelligibility and quality. Among prompt-based systems, Qwen3-TTS has the best intelligibility (WER 2.06 EN, CER 4.08 ZH) and the highest English CLAP score (0.45). GPT-4o mini TTS leads DNSMOS (OVRL 3.35 EN, 3.40 ZH).
- Gemini WER inflation. The Gemini models score competitively on DNSMOS but have badly inflated WER: 58.80 ± 19.52 for Gemini 2.5 Flash EN, and 16.45 for Flash ZH. The authors attribute this to content-boundary violations under prompt conditioning, such as prompt leakage, repeated phrases, and long laughter transcribed as repeated “ha ha”.
- Tag-based coverage vs. correctness. Tag-based systems cover only 0.02 (ChatTTS) to 0.29 (Dia) of the taxonomy. In English, Orpheus TTS has the best precision and F1 (0.687 / 0.728), and ElevenLabs the best recall (0.787) along with 0.27 coverage. ChatTTS tops all three correctness scores in Chinese (F1 0.703), even though it supports only one NVV type. The authors call this “selective compliance”, where narrow inventories inflate correctness metrics.
Human listening (Table 4).
- Tag-based. ElevenLabs is the strongest tag-based system in both languages. In English it scores naturalness 4.60, NVV Accuracy 4.21, and NVV PE 3.92.
- Prompt-based, English. Gemini 2.5 Pro leads (naturalness 4.07, NVV IF 2.74).
- Prompt-based, Chinese. Qwen3-TTS leads naturalness, quality, and caption match, while Gemini 2.5 Flash leads NVV IF and PE.
- Fidelity vs. NVV realization. CosyVoice 2 achieves the best Chinese perceived quality (4.35) but lags on NVV correctness (1.65). The authors read this as evidence that speech fidelity and NVV realization are separable.
- Score dispersion. Standard deviations on NVV-specific criteria are often larger than the means (e.g., Gemini 2.5 Pro EN NVV PE 2.68 ± 3.49), because the 0 score for absent NVVs makes the distributions strongly bimodal.
LLM judge (Table 5). The authors state that the LLM judge “generally tracks” human judgments, but no correlation coefficient is reported. Rankings diverge in places: the LLM ranks Gemini 2.5 Flash first on English prompt-based naturalness, while humans rank Gemini 2.5 Pro first.
With vs. without NVV control (Table 6, Table 3 w/o rows). Adding NVV control raises WER/CER for both ElevenLabs and Gemini 2.5 Pro (e.g., ElevenLabs ZH CER 2.38 to 4.13). Human CMOS diverges between the two:
- ElevenLabs (tag-based) gains on expressiveness (+0.93 EN, +0.52 ZH), naturalness, and quality.
- Gemini 2.5 Pro (caption-based) gains almost no expressiveness (+0.05) and loses naturalness and quality.
Per-type analysis (Figure 2). High-salience bursts are easiest across both paradigms: laughter, breath, cough, sigh, and gasp. Low-SNR oral cues (tsk, sss, lipsmack, swallow) and long-duration affective NVVs (crying, sobbing, wailing) are the hardest. Tag-based systems show a partial “frequency advantage” for types that appear in many inventories.
Novelty Assessment
The contribution is benchmark design, not modelling.
What is new:
- Joint protocol. It scores type correctness, onset placement, and coverage jointly, and adds coverage awareness to precision/recall. This exposes the selective-compliance artefact (ChatTTS scoring high F1 on a single supported type), which a plain F1 would hide.
- Interface span. It evaluates both tag-based and caption-based control under one taxonomy and dataset.
- Inventory breadth. At 45 types, the taxonomy is wider than any single system or dataset inventory it surveys.
Limits on novelty:
- Concurrent benchmark. A concurrent NVV synthesis benchmark (NV-Bench) exists. The authors acknowledge it and argue its types are largely covered by their taxonomy, but they make no head-to-head methodological comparison.
- Standard metric stack. The objective stack is standard: Whisper/Paraformer ASR, DNSMOS, and CLAP.
- Gemini dependence. The NVV detector is a prompted Gemini 2.5 Pro verifier rather than a dedicated, validated recognizer. The same model family generates the data, verifies NVVs, judges quality, and appears among the benchmarked systems.
- Synthetic text. The dataset text is LLM-generated, anchored on a small human-audited seed set (~110 items).
Field Significance
Moderate. NVBench provides a structured, coverage-aware way to compare NVV-capable TTS systems across tag and caption interfaces. Its main empirical lessons are:
- NVV controllability and speech fidelity vary largely independently.
- Standard ASR- and DNSMOS-based metrics penalize NVV-conditioned speech that listeners prefer.
- Low-SNR oral cues and sustained affective vocalizations remain unsolved across commercial and open-source systems.
Its value as a reference benchmark depends on the release of the data and verifier prompts, and on independent validation of its Gemini-based detector.
Claims
- complicates: Standard ASR-based intelligibility and non-intrusive quality predictors penalize speech containing deliberately synthesized non-verbal vocalizations, even when listeners prefer it.
Evidence: Enabling NVV tags in ElevenLabs raises WER/CER (EN 2.16 to 2.31, ZH 2.38 to 4.13) and lowers DNSMOS, yet human CMOS favors the NVV-conditioned output on naturalness, quality, and expressiveness (expressiveness +0.93 EN, +0.52 ZH). (§4.4, Table 3, Table 6)
- supports: Non-verbal vocalization controllability in TTS is largely separable from overall speech fidelity, so high naturalness or signal quality does not imply reliable NVV realization.
Evidence: CosyVoice 2 attains the best perceived quality among Chinese tag-based systems (4.35) but scores only 1.65 on NVV Accuracy and 1.56 on NVV PE. Qwen3-TTS has the best prompt-based intelligibility yet trails Gemini on NVV instruction following. (§4.1, §4.2, Table 3, Table 4)
- complicates: Tag-level NVV correctness metrics overstate controllability when a system supports only a narrow vocalization inventory, so coverage must be reported alongside precision and recall.
Evidence: ChatTTS supports a single NVV type (coverage 0.02) yet achieves the best Chinese precision, recall, and F1 (0.644 / 0.773 / 0.703). Tag-based systems overall cover only 0.02 to 0.29 of the 45-type taxonomy. (§4.1, §4.5, Table 3)
- supports: Low-energy oral cues and long-duration affective vocalizations remain the hardest non-verbal vocalization types for current TTS systems, while brief high-salience events such as laughter, coughs, and sighs are realized far more reliably.
Evidence: Per-type NVV PE heatmaps across all 15 systems show consistently low scores or missing support for tsk, sss, lipsmack, swallow, and mumble, and for crying, sobbing, wailing, and whimpering. Laughter, breath, cough, sigh, and gasp score higher when supported. (§4.5, Figure 2)
- complicates: Caption-based natural-language control of non-verbal vocalizations can increase content errors and reduce naturalness without a matching expressiveness gain, compared with inline tag control.
Evidence: Gemini 2.5 Pro with NVV-aware captions gains only +0.05 CMOS expressiveness in both languages, with negative naturalness and quality CMOS. The Gemini models also show inflated WER (up to 58.80 for Gemini 2.5 Flash EN) from prompt leakage, repetition, and over-long NVV segments. (§4.1, §4.4, Table 3, Table 6)
Limitations and Open Questions
Warning
The benchmark relies heavily on one proprietary model family. Gemini 2.5 Pro generates the evaluation texts and captions, serves as the NVV verifier behind all objective controllability metrics, and acts as the LLM judge. Gemini 2.5 Pro and Flash are also among the benchmarked systems. The verifier’s accuracy against human NVV annotation is not reported, and neither is the LLM judge’s correlation with human ratings.
- Synthetic text. The evaluation text is LLM-generated with a single target NVV type per item. The ~110 human-audited seeds calibrate the generation but do not guarantee naturalistic discourse for all 45 types.
- Dispersion and statistics. Because absent NVVs are scored 0, the human NVV scores are strongly bimodal, with standard deviations often exceeding the means. No significance testing is reported, so close rankings between systems are not resolved.
- Unequal comparisons. Comparisons across interfaces are not perfectly matched:
- Commercial prompt-based systems (Gemini, GPT-4o mini TTS) use predefined voices that may not match the caption’s speaker description, which depresses caption-match scores.
- Several systems are evaluated in English only.
- Coarse localization. Onset localization is measured in transcript indices within a tolerance δ, not on the audio time axis. Fine-grained timing errors are therefore invisible.
- Availability. Neither the parsed text nor this page records whether the data, verifier prompts, or code are released. The footnote link is not preserved in the parsed paper.
Wiki Connections
- Evaluation Metrics — Introduces coverage-aware NVV precision/recall/F1 and normalized tag distance, and shows that WER/CER and DNSMOS penalize NVV-conditioned speech that listeners prefer.
- Subjective Evaluation — Runs Prolific listening tests with 97 raters, NVV-specific criteria with a 0 score for absent vocalizations, and CMOS ablations, and compares them against an LLM multi-rater protocol.
- Instruction-Conditioned TTS — Evaluates seven caption-prompted systems on following NVV instructions and finds that caption-only NVV prompting can cost naturalness without improving expressiveness.
- NV-Bench — A concurrent NVV synthesis benchmark that the authors cite and argue is largely covered by their 45-type taxonomy.
- InstructTTSEval — Source of the bilingual expressive recordings and captions used for NVBench’s seed mining stage.
- CosyVoice 2 — Benchmarked tag-based system with the best Chinese perceived quality but weak NVV correctness and salience.
- Qwen3-TTS — Benchmarked prompt-based system with the best intelligibility and Chinese naturalness, but weaker on subtle NVV cues.
- Fish-Speech — Benchmarked tag-based system whose NVV perceptual effect is near zero in Chinese despite seven supported tags.
- CapSpeech — Benchmarked open-source caption-based system that scores low on NVV instruction following.
- NVSpeech — NVV dataset whose label inventory is surveyed when building the unified taxonomy.
- SynParaSpeech — Paralinguistic dataset whose NVV inventory informs the taxonomy and illustrates the skewed coverage of existing resources.