arXiv · 2026 · Preprint
Qinke Ni et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ? · Code: ?
Introduces NV-Bench, a paired ground-truth, functionally-taxonomized benchmark for evaluating how well TTS systems generate nonverbal vocalizations (breaths, laughter, filled pauses, and similar communicative cues) as controllable, realistic events rather than generic sound effects.
Problem
Expressive TTS systems increasingly try to reproduce nonverbal vocalizations (NVs) such as breathing, laughter, sighs, and filled pauses, but existing evaluation practice has no reliable way to check whether they succeed. Prior corpora that pair NVs with training data (Emilia-NV, SMIIP-NV, NVTTS, DisfluencySpeech, NVS, SynParaSpeech) either lack a held-out testset, lack ground-truth (GT) human recordings to compare against, or leave the label distribution unbalanced, so aggregated scores can be dominated by whichever NV category happens to be over-represented. Without paired GT audio, evaluation collapses to coarse presence/absence checks of an NV event, which cannot quantify how far a synthesized NV is from a real one, and cannot separate “the model didn’t attempt the event” from “the model attempted it but sounds unnatural.” The paper also argues that most prior work treats NVs as acoustic textures bolted onto linguistic content, rather than as communicative acts (physiological, affective, or discourse-management signals) that need to be evaluated for pragmatic appropriateness, not just acoustic presence.
Method
NV-Bench is built on the functional taxonomy of Batliner et al., which splits NVs into three levels: vegetative sounds (breathing, coughing), affect bursts (laughter, surprise, dissatisfaction), and conversational grunts (filled pauses and prosodic particles that manage turn-taking or signal confirmation/hesitation). The authors construct a unified, multi-lingual (Mandarin/English) label inventory across these three levels by consolidating and re-normalizing labels from six existing NV corpora, mapping non-speech labels to Level 3 of the AudioSet ontology and manually re-annotating English subsets to capture pragmatic distinctions (e.g. “[Question-huh]”) missing from the source label sets.
To construct the benchmark itself, the authors first train a multilingual NV-capable ASR model (NVASR) by fine-tuning SenseVoice-Small with a CTC objective on the consolidated, re-normalized corpus, so that transcription and NV-label recognition can be automated at scale. They then curate benchmark audio from roughly 565,000 in-the-wild audiovisual clips (about 1,560 hours) published in 2025, run it through the Emilia-Pipeline for standardization and initial diarization, use MiMo-Audio-7B-Instruct to catch residual multi-speaker segments that diarization missed, and have ten expert annotators verify NVASR transcripts and NV-label pragmatic appropriateness (5% cross-annotated, Cohen’s kappa above 0.85). This yields 1,651 prompt/GT-audio pairs (7.9 hours) split into a strictly balanced single-label subset (one NV event per utterance, 50 samples per category) and a relatively balanced multi-label subset (2+ co-occurring NV events per utterance) to test robustness under denser paralinguistic conditions.
Evaluation is dual-dimensional. Instruction Alignment measures whether target NV events appear at the correct textual position without omission or hallucination, using the NVASR to compute standard CER, an Overall CER (OCER) that folds NV symbols into the edit-distance calculation alongside text, and a paralinguistic CER (PCER) computed only over the NV symbols to isolate paralinguistic control accuracy from lexical transcription accuracy. Acoustic Fidelity measures the distributional gap to real recordings via Fréchet Audio Distance and a PANNs-based Fréchet Distance, timbre consistency via WavLM-based speaker similarity, and perceptual quality via DNSMOS. Ten human annotators additionally rate 100 utterances per system on naturalness (NMOS) and instruction accuracy (IMOS) on a 5-point scale. The benchmark evaluates five existing zero-shot/instruction-following TTS systems (Orpheus-TTS, SMIIP-NV-CV2, Emilia-NV-CV2, CosyVoice3) plus two reference baselines the authors fine-tune themselves on the consolidated NV corpus: NV-FlexiVoice (from a 0.5B FlexiVoice checkpoint pretrained on Emilia) and NV-CV3 (from CosyVoice3), optimized with AdamW at a learning rate of 1e-5 on 4 A800 GPUs.
Key Results
The fine-tuned NVASR evaluator matches or slightly improves on SenseVoice-Small’s plain-speech CER on WenetSpeech test-net and LibriSpeech test-other, while substantially outperforming a Qwen2.5-Omni-7B checkpoint fine-tuned on MNV-17 for NV recognition (e.g. 1.29% CER / 1.36% OCER for NVASR versus 3.59% CER / 4.17% OCER for Qwen2.5-Omni on the SMIIP-NV testset), supporting its use as an automated evaluator (Table 3).
On the benchmark itself, NV-CV3 (the authors’ own CosyVoice3 fine-tune on the consolidated corpus) achieves the best instruction alignment on the Mandarin single-label subset (27.69% PCER, 4.90% OCER) and the best subjective NMOS/IMOS scores overall, while NV-FlexiVoice achieves the lowest FAD (0.29) and FD (2.72), indicating its output distribution is closest to real recordings even though it trails NV-CV3 on controllability. Off-the-shelf systems not specifically adapted to NVs (Orpheus-TTS, CosyVoice3 zero-shot) show much higher PCER (often 50-90%) despite competitive plain CER, showing they frequently fail to realize the correct NV event even when overall transcription accuracy looks acceptable. Ground-truth recordings retain a substantial residual gap to even the best system (GT PCER of 9.38% vs. NV-CV3’s 27.69% on the Mandarin single-label subset), and English-subset scores are uniformly worse than Mandarin across systems, reflecting the smaller English NV training data available. Crucially, IMOS (human-rated instruction accuracy) correlates strongly and negatively with PCER (Spearman ρ = -0.65, p < 0.001), and NMOS aligns with the FD acoustic-distance metric, which the authors use to validate that their objective metrics track human judgment (§4.2.4).
Novelty Assessment
The core contribution is a benchmark and evaluation protocol, not a new generative architecture: the two “new” systems (NV-CV3, NV-FlexiVoice) are fine-tunes of existing zero-shot TTS backbones (CosyVoice3, FlexiVoice) built to give the benchmark strong reference points, and the NVASR evaluator is a fine-tune of an existing ASR model (SenseVoice-Small). What is genuinely new is the combination of (1) a functional taxonomy applied consistently across languages and corpora, (2) a paired, in-the-wild, human-verified testset with balanced single- and multi-label subsets curated specifically to avoid data leakage (media from 2025), and (3) a dual-dimensional evaluation protocol (PCER/OCER for controllability, FAD/FD/DNSMOS/SIM for fidelity) that disentangles two failure modes previous NV evaluations conflated: not attempting an event versus attempting it poorly. The correlation analysis against human ratings is a genuine methodological contribution that most prior NV-TTS papers, which rely on internal testsets or coarse event-presence checks, did not provide.
Field Significance
moderate — NV-Bench addresses a real methodological gap in a fast-growing but still narrow corner of expressive TTS: nonverbal vocalization synthesis lacked any standardized, paired-ground-truth benchmark before this work, and the paper demonstrates that its proposed metrics correlate with human perception. Its significance is bounded by the narrowness of the sub-task it targets (14 NV categories in two languages) and by being a fresh preprint without demonstrated downstream adoption; within that scope it provides a concrete, reusable evaluation protocol and a public testset that future NV-capable TTS work can benchmark against.
Claims
- supports: A paired ground-truth, in-the-wild testset for nonverbal vocalizations enables objective evaluation metrics that track human perceptual judgments of both instruction-following and naturalness.
Evidence: Human-rated IMOS shows a significant negative Spearman correlation with the paper’s PCER metric (ρ = -0.65, p < 0.001), and human-rated NMOS aligns with the Fréchet Distance acoustic-fidelity metric. (§4.2.4)
- supports: Decoupling instruction-following accuracy from acoustic realism reveals that TTS systems fail via two distinct, largely independent failure modes when generating nonverbal vocalizations.
Evidence: Systems with competitive plain CER (e.g. CosyVoice3 at 3.85% CER on the Mandarin single-label subset) can still show very high paralinguistic error (57.69% PCER), showing lexical accuracy does not imply correct NV realization. (§3, Table 4)
- complicates: Fine-tuning a zero-shot TTS backbone on a large consolidated multi-corpus nonverbal-vocalization dataset substantially improves controllability relative to off-the-shelf systems, but a wide gap to ground-truth control accuracy persists.
Evidence: The best fine-tuned system (NV-CV3) achieves 27.69% PCER on the Mandarin single-label subset, more than double the ground-truth recording’s own measured PCER of 9.38%, indicating the metric floor set by ASR/annotation noise is far from reached by any evaluated TTS system. (§4.2.4, Table 4)
- refines: Standard character error rate computed only over lexical text substantially undercounts a TTS system’s paralinguistic control errors and should be supplemented by metrics computed specifically over nonverbal-vocalization symbols.
Evidence: The paper introduces Overall CER (which folds NV symbols into the edit-distance calculation) and paralinguistic CER (computed only over NV symbols) because plain CER on lexical text alone is shown to obscure large differences in NV controllability across systems (e.g. NV-FlexiVoice and CosyVoice3 have similar plain CER but a 20+ point PCER gap on the Mandarin single-label subset). (§4.1.2, §4.2.3, Table 4)
Limitations and Open Questions
The benchmark's English coverage is narrower than its Mandarin coverage (fewer NV categories per level, smaller per-category sample counts, and consistently worse scores across all evaluated systems), so cross-lingual comparisons and any claims about English NV synthesis quality should be read with that imbalance in mind.
The evaluated TTS systems that lack native support for certain NV symbols (e.g. “[Question-huh]”) are scored after mapping the unsupported interjection to its nearest lexical equivalent with punctuation, which is an approximation that could understate or overstate a system’s true paralinguistic capability. The benchmark also depends on the paper’s own fine-tuned NVASR model as the automated evaluator for CER/OCER/PCER; while its reliability is cross-checked against SMIIP-NV and NVTTS testsets and against human ratings, any residual bias in NVASR’s own recognition of specific NV categories would propagate into every system’s reported scores. Finally, the testset is curated from Mandarin- and English-language 2025 web media, so its generalization to other languages, domains, or older/newer speech styles is untested.
Wiki Connections
- Evaluation Metrics — introduces paralinguistic CER and Overall CER as CER-family extensions purpose-built to isolate nonverbal-vocalization control accuracy from lexical transcription accuracy.
- Subjective Evaluation — validates its objective metrics against real human ratings (NMOS, IMOS) collected from ten annotators, showing strong correlation with the proposed PCER and FD metrics.
- Zero-Shot TTS — benchmarks several zero-shot TTS systems (CosyVoice2/3-based models, Orpheus-TTS) specifically on their ability to realize nonverbal vocalizations under prompt control.
- CosyVoice 3 — used as the foundational zero-shot TTS backbone that the authors fine-tune into NV-CV3, their best-performing reference baseline.
- CosyVoice 2 — the backbone fine-tuned into the SMIIP-NV-CV2 and Emilia-NV-CV2 baselines evaluated on NV-Bench.
- Qwen2.5-Omni — used as a multimodal-LLM ASR baseline for NV recognition, which NV-Bench’s fine-tuned NVASR is shown to substantially outperform.
- NVSpeech — one of six source corpora (as Emilia-NV) consolidated to train NV-Bench’s NVASR model and the NV-CV3/NV-FlexiVoice reference baselines.
- Scalable Pipeline for Non-Verbal Speech Generation and Understanding — source of the NVS corpus used in NV-Bench’s training data consolidation and compared against in the paper’s dataset landscape table.
- SynParaSpeech — cited as a comparable large-scale paralinguistic corpus lacking a balanced, paired-ground-truth testset, motivating NV-Bench’s design.
- CapSpeech — cited as an example of the “overlapping layers” approach to NV integration that NV-Bench’s functional-taxonomy framing is positioned against.