O-COCOSDA · 2025 · Conference

Wangzixi Zhou et al. (Nara Institute of Science and Technology) · → Paper · Demo: ✓ · Code: ?

Re-annotates non-verbal vocalizations from the EARS corpus with structured tags that encode vocalization type, count, and duration, and shows in a small listening test that an emotional Grad-TTS trained on these fine-grained tags conveys emotion more recognizably than verbal-only or coarse-tag training, at a small cost in naturalness.

Problem

Emotional TTS systems usually express emotion only through verbal prosody (pitch, duration, energy driven by emotion embeddings), leaving out non-verbal vocalizations (NVs) such as laughter, crying, and screams that carry much of the affective signal in real speech. Existing NV corpora annotate these events with coarse tags like <laugh>, which only mark presence, so a model cannot be asked for a longer cry or three laughs instead of two. The authors also argue that available NV resources are noisy (the NonverbalTTS corpus, with 1525 of 3642 utterances noise-free per the authors), narrow in NV inventory (AMI: mostly laughter and coughs), or language-specific (the Japanese JNV corpus), and that NV-capable systems built on large LLM-based TTS rely on proprietary instruction data or on fine-tuning a large pretrained model.

Method

Data. The authors take the non-verbal recordings of 60 female speakers from EARS, covering six categories (laughter-open, laughter-closed, cheering, yelling, crying, screaming). The 360 source files (about 10 to 12 s each) are split on silence with pydub (-40 dBFS threshold, 200 ms minimum silence, 100 ms padding), giving 739 utterances of roughly 2 to 6 s. Whisper produces initial transcripts, which are manually corrected into a fine-grained tag format (Table 1, Table 2):

  • a style-level tag naming the NV category, e.g. <(crying) ...>;
  • discrete vocalizations (ha, whep) repeated to encode how many occur, so <(Laughter-open) ha ha ha> means three open laughs;
  • continuous vocalizations (wuu, ah, Wo ho) with the final character repeated to encode length; for a crying wuu, each extra u adds about 0.2 s to a base of about 1 s.

Model. The backbone is Grad-TTS (a score-based diffusion acoustic model predicting 80-bin mel-spectrograms) extended with an emotion encoder conditioned on continuous arousal and valence values, following Russell’s circumplex model. An NV processor is inserted at the start of the text-cleaning pipeline: a style parser, a discrete-unit parser, and a duration parser convert each tagged NV segment into NV tokens that the emotional Grad-TTS consumes alongside the ordinary text (§4, Figure 1). Waveforms are produced with an off-the-shelf HiFi-GAN vocoder (§5.1). Model size is not reported.

Training. Verbal emotional speech comes from a 9-hour mix of English female speech from Expresso, SEMAINE, and ESD at 22.05 kHz; arousal and valence labels missing from Expresso and ESD are predicted with a pretrained speech emotion recognition model. Two NV variants are trained for comparison: one with the curated fine-grained EARS data and one with NonverbalTTS as the coarse-grained baseline. Training runs 400k iterations on a single RTX A6000 (§5.1).

Key Results

All evaluation is subjective: 15 participants rate 60 samples built from 20 verbally ambiguous sentences (e.g. “what did you do”) for naturalness MOS (nMOS), expressiveness MOS (eMOS), and a four-way emotion identification task (happy, sad, fear, anger) (§5.2.1).

  • Naturalness: verbal-only 3.54, coarse NV 3.46, fine-grained NV 3.40 (Figure 2). Adding NVs lowers naturalness slightly; the per-emotion breakdown shows fine-grained NV below verbal-only nMOS for all four emotions, most clearly for anger (3.34 vs 3.61) and happy (3.43 vs 3.67) (Table 3).
  • Expressiveness: fine-grained NV 4.20, coarse NV 4.00, verbal-only 3.81 (Figure 2). Per emotion, fine-grained eMOS exceeds verbal-only in every category and exceeds coarse NV in both categories the coarse system covers (happy 4.21 vs 3.85, sad 4.25 vs 4.15) (Table 3).
  • Emotion recognition: fine-grained NV 82.0%, verbal-only 62.1%, coarse NV 58.7% (Figure 2). Per-emotion accuracy for the fine-grained system is happy 82.50%, anger 64.35%, fear 82.69%, sad 98.28%, against verbal-only 62.24%, 53.45%, 46.36%, and 86.67% (Figure 3). The largest gain is for fear (+36.3 points), attributed to the screaming NV; anger gains least, which the authors attribute to yelling being a generic high-arousal cue rather than a specifically angry one (§5.3.1). Coarse-NV happy samples are often misheard as sad (25.74% correct), which the authors attribute to quiet, subdued laughter in that data.
  • Preference test: for happy, cheering (Wo ho, Yo) is preferred over laughter, with closed laughter ranked third by 62% of participants; for sad, the combined wuuuuuuu whep is ranked first by 56% (§5.3.2, Figure 4).

The comparison is uneven: the coarse-grained system was only evaluated on happy and sad (NonverbalTTS lacks fear and anger NV cues), so its averages cover a different emotion subset than the other two systems. No statistical significance tests are reported despite the paper’s “significantly” wording. The overall accuracy for the proposed system is reported as 78.8% in the abstract and §5.3 text but as 82.0% in Figure 2 and §6; the 82.0% figure matches the mean of the four diagonal cells in Figure 3(c).

Novelty Assessment

The contribution is mainly the annotation scheme and the curated data: encoding NV count by repeating discrete syllables and NV length by repeating the final character is a simple, interpretable way to make NV properties controllable through text, which coarse presence tags do not allow. The modeling side is engineering integration: Grad-TTS, an arousal/valence emotion encoder, and HiFi-GAN are established components, and the NV processor is a rule-based text front-end rather than a learned module. The dataset is small (739 utterances, female speakers only), the coarse baseline differs from the proposed system in both annotation granularity and source data quality (NonverbalTTS is noted as noisy), so the gain cannot be attributed to annotation granularity alone. The paper does not objectively verify that the duration or count encoded in the tags is realized in the generated audio.

Field Significance

Low. The paper offers a small, concrete data point that explicit non-verbal vocalizations raise listener-perceived expressiveness and emotion identifiability in emotional TTS while costing a little naturalness, and it proposes a text-tag convention for controlling NV count and duration that can be applied to other corpora. The evidence comes from a single backbone, one speaker gender, and a 15-listener test with an uneven baseline.

Claims

  • supports: Inserting non-verbal vocalizations into emotional speech synthesis improves listeners’ ability to identify the intended emotion beyond what verbal prosody alone conveys.

    Evidence: On verbally ambiguous sentences, human four-way emotion identification rises from 62.1% for verbal-only emotional Grad-TTS to 82.0% with fine-grained NV tags, with fear rising from 46.36% to 82.69% when screaming is added. (§5.3, §5.3.1, Figure 2, Figure 3)

  • complicates: Adding non-verbal vocalizations to synthetic speech trades a small loss in perceived naturalness for gains in expressiveness.

    Evidence: Naturalness MOS falls from 3.54 (verbal-only) to 3.40 (fine-grained NV) while expressiveness MOS rises from 3.81 to 4.20; per emotion, the NV system has lower nMOS than verbal-only in all four categories. (§5.3, Figure 2, Table 3)

  • refines: The emotional benefit of a non-verbal vocalization depends on how specifically that vocalization is associated with the target emotion.

    Evidence: Emotion-specific cues (screaming for fear, crying for sadness) yield 82.69% and 98.28% recognition, while yelling, a generic high-arousal cue used for anger, reaches only 64.35% with eMOS 4.06, the lowest of the four emotions; coarse laughter for happy is often misheard as sad. (§5.3.1, Figure 3, Table 3)

  • supports: Annotating non-verbal vocalizations with type, count, and duration information yields more recognizable emotional synthesis than presence-only tags.

    Evidence: A Grad-TTS trained with the fine-grained EARS tags outperforms one trained on coarse NonverbalTTS tags in eMOS (4.21 vs 3.85 happy, 4.25 vs 4.15 sad) and happy recognition (82.50% vs 25.74%), though the two variants also differ in source data and recording quality. (§5.1, §5.3.1, Table 3, Figure 3)

Limitations and Open Questions

Warning

The fine-grained vs coarse comparison confounds annotation granularity with source corpus: the coarse system is trained on a different, noisier NV corpus and is evaluated only on happy and sad, so the evidence that granularity itself drives the gain is indirect.

  • The listening test is small (15 participants, 60 samples), reports no confidence intervals or significance tests, and gives two different overall accuracies for the proposed system (78.8% in the text, 82.0% in Figure 2 and the conclusion).
  • Controllability of NV count and duration is not measured objectively; there is no check that a longer wuu tag yields a longer cry in the output.
  • All data are female English speakers, and arousal and valence labels for two of the three verbal corpora come from a pretrained SER model rather than human annotation.
  • No objective metrics (intelligibility, speaker similarity, automatic quality) are reported, and it is untested whether the tag scheme transfers to larger LLM-based TTS backbones.

Wiki Connections

  • Emotion Synthesis — Adds non-verbal vocalizations with controllable type, count, and duration to an arousal/valence-conditioned emotional TTS and evaluates emotion expressiveness and recognition with listeners.
  • Diffusion TTS — The system is built on the Grad-TTS diffusion acoustic model, extended with an emotion encoder and NV tokens.
  • Subjective Evaluation — All results come from human listening tests: naturalness MOS, expressiveness MOS, four-way emotion identification, and NV preference ranking.
  • Grad-TTS — Serves as the backbone acoustic model that the paper extends with emotion conditioning and a non-verbal token front-end.
  • HiFi-GAN — Used off the shelf as the mel-to-waveform vocoder.
  • Expresso — One of the three corpora forming the 9-hour emotional verbal training mix.
  • Whisper — Provides initial transcripts of the segmented EARS non-verbal clips before manual correction into fine-grained tags.
  • CosyVoice 2 — Cited as an LLM-based TTS with NV capabilities whose instruction data is proprietary, motivating a data-centric approach for conventional TTS backbones.