arXiv · 2025 · Preprint
Liao et al. (The Chinese University of Hong Kong, Shenzhen) · → Paper · Demo: ✓ · Code: ?
NVSpeech introduces an end-to-end pipeline for paralinguistic vocalization modeling in Mandarin, combining a word-level annotated corpus of 573 hours, a paralinguistic-aware ASR model that treats non-verbal cues as inline decodable tokens, and fine-tuned zero-shot TTS capable of inserting those cues at arbitrary token positions.
Problem
Conventional ASR and TTS systems treat paralinguistic vocalizations (laughter, breathing, hesitation markers, expressive interjections) as noise or ignore them entirely. While recent instruction-conditioned TTS models have begun incorporating non-verbal sounds, they rely on proprietary datasets with limited coverage and coarse control. No open, large-scale Mandarin corpus exists with word-level alignment of both lexical and paralinguistic content, making it difficult to train or evaluate models that jointly handle verbal and non-verbal signals.
Method
NVSpeech proceeds in three stages. First, a human-annotated corpus of 48,430 utterances (76 hours) is constructed from Chinese game voice lines (Genshin Impact and Star Rail datasets) supplemented with coughing and crying clips and a small synthesized set. Ten trained annotators label 18 paralinguistic vocalization categories at the word level, including physiological sounds ([Laughter], [Cough], [Breathing]), discourse markers ([Uhm]), and tonal interjections ([Confirmation-en], [Question-ah], [Surprise-oh]). Inter-annotator agreement exceeds Cohen’s kappa 0.85 on major categories.
Second, the human-annotated data is used to fine-tune a paralinguistic-aware ASR model. The approach treats paralinguistic events as special tokens in the output vocabulary, extending standard CTC and autoregressive decoding. Four backbone models are evaluated (Paraformer, SenseVoice-Small, Qwen-Audio, Whisper); SenseVoice-Small achieves the best overall results (in-domain F1 0.83, open-domain F1 0.85) and is used as the automatic annotator.
Third, the ASR model is applied to a larger unlabeled corpus (Emilia dataset clips plus additional game data) to produce an auto-labeled set of 174,179 utterances (573 hours). CosyVoice and CosyVoice2 are then fine-tuned on subsets of this corpus with their vocabularies extended to include paralinguistic tags, enabling token-level insertion of non-verbal vocalizations at arbitrary positions in the target sequence.
Key Results
On paralinguistic tagging, SenseVoice achieves F1 0.73 and PANNs 0.72 on the NVSpeech test set (Table 3). On the paralinguistic-aware ASR task, SenseVoice leads on both in-domain (CER 4.61%, F1 0.83) and open-domain sets (CER 3.79%, F1 0.85), outperforming Paraformer, Whisper, and Qwen-Audio (Table 4).
For TTS, scaling from human-annotated to large-scale auto-labeled data consistently improves results. Fine-tuning CosyVoice2 on the full auto-labeled corpus achieves 7.51% in-domain CER (vs. 8.61% on human data alone) and UTMOS 2.67 (vs. 2.54), with speaker similarity preserved at 0.700 (Table 5). Human evaluation finds listener preference for para-enhanced output at win rates of 78.7% (CosyVoice) and 75.4% (CosyVoice2), with naturalness MOS (NMOS) of 3.9-4.0 and quality MOS (QMOS) of 3.96-4.04 (Table 6). Paralinguistic tag recall reaches 61.9%.
The comparisons are conducted on the paper’s own test sets rather than a publicly established TTS benchmark, and no comparison with other paralinguistic TTS systems using identical test conditions is included.
Novelty Assessment
The primary contribution is the dataset: NVSpeech is the first open, large-scale Mandarin corpus with word-level annotations across 18 paralinguistic vocalization categories. The ASR approach (treating PV events as inline vocabulary tokens) extends standard fine-tuning of existing models and is not architecturally novel. The TTS component is similarly based on vocabulary extension and fine-tuning of CosyVoice and CosyVoice2, two existing systems. The pipeline’s value lies in the automated annotation loop, which enables scaling without proportional human annotation cost, but this data-flywheel pattern itself is well established. The paper’s honest novelty claim is the corpus and the integrated pipeline, not a new model architecture or training objective.
Field Significance
Moderate — NVSpeech provides the first open Mandarin corpus with word-level paralinguistic annotations at meaningful scale (573 hours), filling a data gap that has limited expressiveness research in non-English TTS. The engineering pipeline demonstrates that ASR-assisted auto-labeling can scale paralinguistic supervision at low cost, and the results support the viability of token-level paralinguistic control within existing autoregressive TTS frameworks. The contribution is primarily a resource and methodology for expressive speech modeling, rather than a novel architecture.
Claims
- Treating paralinguistic vocalizations as inline vocabulary tokens in ASR decoding enables joint transcription of lexical and non-verbal content within a single unified sequence, without requiring a separate event detection module. (§Paralinguistic Aware Speech Recognition)
- Automatically annotated data at scale can match or exceed the training effectiveness of smaller human-annotated sets for paralinguistic-enhanced TTS, measured by CER and perceptual quality. (§Main Results, Table 5)
- Fine-tuning zero-shot TTS models with paralinguistic vocabulary extensions enables token-level control over non-verbal vocalization placement without material degradation of lexical quality or speaker similarity. (§Paralinguistic-enhanced TTS Experiments, Table 5)
- Listener preference for para-enhanced TTS is strong even when paralinguistic tag recall remains below 65%, suggesting that the mere presence of contextually appropriate non-verbal cues improves perceived naturalness. (§Human Evaluation, Table 6)
Limitations and Open Questions
Warning
The NVSpeech corpus is restricted to Chinese (Mandarin) and draws primarily from game-style voice acting rather than spontaneous conversational speech, which may limit generalization to natural dialogue scenarios. The auto-labeled data quality depends entirely on the SenseVoice ASR model’s precision, and label errors accumulate at scale without a second annotation pass.
Paralinguistic tag recall in TTS reaches only 61.9%, leaving substantial room for improvement in ensuring that specified non-verbal cues are faithfully rendered. The paper evaluates TTS on its own in-domain and open-domain splits rather than a shared benchmark, making direct comparison with other paralinguistic TTS systems (such as EmoCtrl-TTS or ELaTE) difficult. Model size and inference cost are not reported. English experiments (Appendix D) use a small consolidated dataset and a single speaker source, limiting cross-lingual generalization claims. Potential misuse risks from highly naturalistic TTS with non-verbal behaviors (deepfakes, audio manipulation) are acknowledged but not technically mitigated.
Wiki Connections
- zero-shot-tts — fine-tunes CosyVoice and CosyVoice2 with paralinguistic vocabulary extensions, demonstrating that zero-shot speaker generalization is preserved under paralinguistic conditioning.
- multilingual-tts — the Mandarin-focused corpus and English appendix experiments position NVSpeech as a resource for multilingual paralinguistic TTS, though cross-lingual capability is not the primary contribution.
- emotion-synthesis — models 18 categories of paralinguistic vocalizations including expressive interjections and affective markers, directly extending the scope of controllable emotion and affect in TTS.
- evaluation-metrics — introduces paralinguistic-specific evaluation metrics (Para Detection Rate, paralinguistic F1, NMOS, QMOS) that complement standard MOS and CER for expressive speech assessment.
- 2407.04051 (FunAudioLLM / CosyVoice) — fine-tuned on NVSpeech data, extending CosyVoice’s instruction-following capability to paralinguistic vocalization control.
- 2412.10117 (CosyVoice2) — used as a second TTS backbone for fine-tuning and comparison, demonstrating paralinguistic control within the streaming LLM-based TTS framework.
- 2311.07919 (Qwen-Audio) — evaluated as an ASR baseline for paralinguistic tagging; underperforms SenseVoice and Paraformer on fine-grained non-verbal event detection.
- 2206.08317 (Paraformer) — the non-autoregressive ASR baseline, achieving the highest paralinguistic detection rate (96.1%) on the in-domain set despite lower overall CER performance.