arXiv · 2026 · Preprint

Vinh Dang Quang et al. (Aimesoft JSC) · → Paper · Demo: ? · Code: ?

A VLSP 2022 shared-task system report that adds lookup-table emotion and speaker embeddings, plus a prosody bottleneck for cross-speaker emotion transfer, to FastSpeech 2 for Vietnamese emotional TTS, with organizer-run listening scores that remain low on both sub-tasks.

Problem

The VLSP 2022 Emotional Speech Synthesis task asks for Vietnamese TTS systems that produce speech with a requested emotion in two settings: (1) a single speaker whose training data is emotional, and (2) a second speaker who has only neutral, non-expressive training data, so that emotion must be transferred from the first speaker while keeping the target speaker’s timbre. The provided emotional corpus is small and collected from films and interviews, which makes both settings data-limited and noisy.

Method

Data preprocessing. The emotional corpus (VLSP-EMO) contains background noise, music, other speakers’ voices and recording-condition shifts. The authors run Facebook Denoiser (Defossez et al., 2020) over it, fix transcript typos, respell English words as similar-sounding Vietnamese syllables, transcribe all audio with an ASR tool and correct or drop utterances whose transcripts have high CER against the ASR output, resample to 22050 Hz and remove clips they judge unintelligible. After cleaning, 3.8 h of VLSP-EMO and 11.89 h of VLSP-NEU remain.

Acoustic model. Both sub-tasks use a modified FastSpeech 2 that maps phoneme sequences to log-mel spectrograms, with phoneme durations extracted by Montreal Forced Aligner. A pretrained HiFi-GAN V1 vocoder (used frozen, not retrained) converts mel spectrograms to waveforms.

  • Sub-task 1 (single speaker): an emotion lookup table maps an emotion ID to a fixed-length vector, which is broadcast-added to the encoder output before the variance adaptor and decoder.
  • Sub-task 2 (emotion transfer to a neutral speaker): following Pan and He (2021), a speaker lookup embedding is broadcast-concatenated with the encoder output and projected back to the encoder dimension; an emotion lookup embedding is passed through a linear layer with tanh and added; the combined representation then passes through a “prosody bottleneck” whose output is added back via a residual connection before the variance adaptor and decoder. The authors expect the bottleneck to retain emotional information while preserving the target speaker’s timbre, but no separate loss enforces this.

Training. Each model is trained from scratch for 40,000 steps with batch size 16 using Adam (β1 = 0.9, β2 = 0.98, ε = 10^-9), 3,000 warm-up steps, and learning-rate decay by 0.3 at steps 5,000, 9,000 and 17,000. Sub-task 1 trains on VLSP-EMO only; sub-task 2 trains on the union of VLSP-EMO and VLSP-NEU. Training takes about one hour (sub-task 1) and more than three hours (sub-task 2) on one RTX 2080 Ti. Model size is not reported.

Key Results

All numbers are the organizers’ announced scores (§3.3); no baselines, competing systems, ablations or confidence intervals are reported.

Sub-taskNaturalness MOS (/5)SUS syllable error rateSpeaker similarity (/4)
1: single-speaker emotional TTS2.71972.40%not reported
2: emotion transfer to neutral speaker1.62264.80%1.543

Naturalness is below the midpoint of the 5-point scale in both sub-tasks and markedly lower in sub-task 2, and the semantically-unpredictable-sentence intelligibility test shows syllable error rates of 64.80% to 72.40%. Sub-task 2 has the lower (better) syllable error rate but the lower MOS; the two sub-tasks use different target speakers and test conditions, so this difference is not a controlled comparison. Emotion expressiveness itself is not scored in the reported results. The authors nonetheless describe the synthesized audio as understandable, natural-sounding and clearly expressive (§1, §4); the reported scores do not quantify those properties and do not include comparison points that would place the system relative to other entries.

Novelty Assessment

The contribution is engineering integration: FastSpeech 2, lookup-table emotion and speaker embeddings, the prosody bottleneck of Pan and He (2021), a pretrained HiFi-GAN vocoder and an off-the-shelf denoiser are combined for a Vietnamese shared task. The data-cleaning pipeline for found emotional speech (denoising plus ASR-based transcript filtering) is the most task-specific element. The paper does not ablate the prosody bottleneck, the denoising step or the emotion embedding, so the effect of each component is not isolated.

Field Significance

Low — a short shared-task system description that provides a low-resource Vietnamese data point for embedding-based emotional TTS on top of a non-autoregressive acoustic model. Its organizer-run listening scores serve mainly as a cautionary reference for how far a few hours of cleaned found emotional data and lookup-table conditioning reach.

Claims

  • complicates: Conditioning a non-autoregressive acoustic model on lookup-table emotion embeddings learned from a few hours of found, noisy emotional speech yields low naturalness and intelligibility.

    Evidence: FastSpeech 2 with an added emotion lookup table, trained on 3.8 h of denoised VLSP-EMO film and interview speech, scores MOS 2.719/5 and a 72.40% SUS syllable error rate in the organizers’ evaluation. (§2.1, §2.2.1, §3.1, §3.3)

  • complicates: Transferring emotion to a target speaker who has only neutral training data, while preserving that speaker’s identity, remains difficult in low-resource settings even with a prosody bottleneck.

    Evidence: The sub-task 2 system (speaker and emotion embeddings plus a prosody bottleneck, trained on the union of 3.8 h VLSP-EMO and 11.89 h VLSP-NEU) scores MOS 1.622/5, speaker similarity 1.543/4 and a 64.80% SUS syllable error rate; no ablation isolates the bottleneck’s contribution. (§2.2.2, §3.1, §3.3)

  • supports: Emotional speech harvested from films and interviews needs enhancement and transcript verification before it is usable for TTS training.

    Evidence: VLSP-EMO contains background noise, music, other speakers and recording-condition shifts, plus typos, English words and mistranscribed utterances; the authors apply a speech-enhancement model, ASR-based CER filtering and manual exclusion of unintelligible clips before training. (§2.1)

Limitations and Open Questions

Warning

Results are organizer-announced scores only, with no baselines, no other team’s results, no ablations and no emotion-accuracy or expressiveness measure, so neither the benefit of the prosody bottleneck nor the system’s relative standing can be assessed from the paper.

  • The claimed disentanglement of prosody from content and speaker timbre by the bottleneck is asserted from prior work, not trained with an explicit objective or tested here.
  • The paper does not report the effect of the denoising and transcript-filtering pipeline on synthesis quality.
  • Evaluation covers only Vietnamese and the two VLSP speakers; the conclusion that the method adapts straightforwardly to other languages is not tested.
  • Model size, test-set size and listener counts are not reported; code and samples are not linked.

Wiki Connections

  • Transformer Encoder-Decoder TTS — Uses a modified FastSpeech 2 non-autoregressive Transformer acoustic model as the backbone for both emotional TTS sub-tasks.
  • Emotion Synthesis — Provides a low-resource Vietnamese data point for emotion-ID embedding conditioning and cross-speaker emotion transfer.
  • Subjective Evaluation — Reports organizer-run naturalness MOS, SUS intelligibility and speaker-similarity listening tests as its only evaluation.
  • FastSpeech 2 — The acoustic model the system modifies with emotion and speaker embeddings and a prosody bottleneck.
  • HiFi-GAN — Pretrained V1 vocoder used, without retraining, to generate waveforms.
  • Adam — Optimizer used to train both acoustic models.