arXiv · 2026 · Preprint

Nina Hosseini-Kivanani et al. (RTL / University of Luxembourg / Tallinn University of Technology) · → Paper · Demo: ✓ · Code: ?

Builds a 21-hour spontaneous emotional speech corpus for Luxembourgish from youth broadcasts and benchmarks five expressive TTS strategies on it with objective metrics and a 20-listener native test, finding that waveform quality, ASR-based intelligibility, prosody and listener-rated emotional quality rank systems differently.

Problem

Expressive TTS corpora are mostly acted, studio-recorded and in high-resource languages. Luxembourgish is technologically low-resource. Its everyday speech mixes in German, French and English, and no expressive TTS corpus exists for it. Prior Luxembourgish speech work covered ASR and phonological description. Real broadcast speech adds residual music, overlapping talk, uneven speaker participation, missing transcripts and skewed emotion classes. The paper asks how to build a usable expressive TTS resource under those conditions, and how current TTS strategies (German-proxy transfer, multilingual phoneme coverage, Luxembourgish adaptation, retrieval-based prosody transfer) behave on it.

Method

Corpus construction. Audio comes from RTL Youth video programs (studio discussions, interviews, on-location reports). The pipeline applies VAD (segments under 200 ms dropped), DeepFilterNet denoising, and segmentation into 1-10 s mono 22.05 kHz chunks. Ground-truth transcripts were unavailable, so transcriptions come from LuxASR TextGrids aligned to the VAD segments, and LuxASR diarization maps single-speaker segments to four recurring speakers (1 male, 3 female). The result is 7,562 segments, 1,260.4 minutes (21.0 h), mean segment duration 9.87 s (Table I).

Annotation. Utterance-level language ID uses the Wav2Vec2-based MMS LID model (mms-lid-4017): 83.8% Luxembourgish, 6.5% English, 2.2% German, 7.5% minority codes. A native speaker manually reviewed 1,135 segments (about 15%), focused on language assignments. Emotion labels (neutral, happy, sad, angry) come from a HuBERT-based classifier plus a lexical model over transcripts. A manually annotated subset tunes the decision thresholds, and each segment gets a confidence score. The authors state that these are weakly supervised labels, not independently validated gold labels. The distribution is heavily skewed: 2,968 neutral, 3,660 happy, 896 sad and 38 angry segments (Table I). Train, development and test splits are speaker- and episode-aware (all segments of one speaker within one episode go to one split), use a fixed seed, and guarantee all four emotions in the test set.

Benchmark. Five systems are evaluated. Three are used off the shelf (zero-shot): GradTTS (Grad-TTS) and XTTS (XTTS) with German as a proxy language, since neither supports Luxembourgish, and Toucan, whose multilingual phoneme inventory covers Luxembourgish. Two are adapted by the authors to LuxEmo. Qwen3 FT is a prompt-controlled language-model-based TTS fine-tuned per speaker. kNN TTS is a GlowTTS-based system that transfers prosody by retrieving k nearest-neighbour LuxEmo segments (the approach of kNN-TTS). As an acted German reference, GradTTS synthesizes 40 EmoDB sentences with German settings. Objective metrics are WV-MOS, NISQA-TTS, NISQA v2.0, DNSMOS (OVRL/SIG/BAK), Whisper-base WER (Whisper), LuxASR WER/CER, ECAPA-TDNN speaker similarity and EER via SpeechBrain (ECAPA-TDNN, SpeechBrain), and pYIN F0 RMSE and correlation against the original recordings. The listening test uses 20 native Luxembourgish listeners aged 20-50. Each rates 20 stimuli (five systems by four target emotions, one text per pair, not fully crossed). For each stimulus, listeners pick the perceived emotion and rate emotion intensity and naturalness/emotion appropriateness on 5-point scales, giving 1,200 responses.

Key Results

Corpus quality. The processed corpus scores NISQA 3.38, DNSMOS OVRL 2.91 and BAK 3.73 (Table II), which the authors read as denoised broadcast speech with residual music and noise.

Non-intrusive quality of synthesized speech (Table II). Toucan (lb) has the highest NISQA-TTS (2.95) and NISQA overall (3.94) among synthesized systems. XTTS (de) has the highest DNSMOS OVRL (2.87) and SIG (3.22), and Toucan (lb) has the highest BAK (3.80). Both LuxEmo-adapted systems score lower than all three zero-shot systems on NISQA overall (Qwen3 FT 1.75, kNN TTS 2.15) and DNSMOS OVRL (1.95, 1.86). The systems are scored on different sample counts: 612 each for the zero-shot systems, 53 for Qwen3 FT, and 280 for kNN TTS.

Objective metrics (Figure 2, values printed on the bars).

SystemWV-MOSSpk-SimF0 RMSE (Hz)Whisper WERLuxASR WER
GradTTS on EmoDB (acted German ref.)4.170.2344.19N/AN/A
GradTTS (de), zero-shot3.890.1762.992.031.72
XTTS (de), zero-shot3.520.3280.461.751.11
Toucan (lb), zero-shot3.500.2279.801.230.83
Qwen3 FT, LuxEmo-adapted3.360.31133.043.7113.26
kNN TTS, LuxEmo-adapted0.920.1164.211.051.81

The paper does not state the units of the WER values. Several values exceed 1, so they are reproduced here as plotted. On the Luxembourgish conditions, GradTTS (de) has the highest WV-MOS (3.89) and Toucan (lb) the lowest LuxASR WER (0.83). XTTS (de) has the highest speaker similarity (0.32, with Qwen3 FT at 0.31). The text says GradTTS (de) has the highest LuxASR WER among Luxembourgish conditions. Figure 2 supports this only among the zero-shot systems (1.72): kNN TTS (1.81) and Qwen3 FT (13.26) are higher. kNN TTS has the lowest Whisper WER (1.05). Its F0 RMSE (64.21 Hz) is close to GradTTS (de) (62.99 Hz), but its WV-MOS is the lowest (0.92). On the same text, Toucan (lb) improves on XTTS (de) for Whisper WER (1.23 vs 1.75), LuxASR WER (0.83 vs 1.11) and, marginally, F0 RMSE (79.80 vs 80.46 Hz).

Listening test (Table II). Qwen3 FT has the highest mean naturalness (3.9, CI 3.6-4.2) and emotion appropriateness (4.0, CI 3.7-4.3). Toucan (lb) follows (3.7 and 3.8), then GradTTS (de) (3.7 and 3.6), XTTS (de) (3.4 and 3.4) and kNN TTS (2.3 and 2.4). On naturalness, Toucan (lb) and GradTTS (de) tie. The authors note that confidence intervals overlap among the top systems, so the ranking is descriptive. Emotion recognition accuracy and Krippendorff’s alpha are announced in §III-B but are not reported in the results. Emotion intensity ratings are not reported either.

Per-emotion behaviour (Figure 3, values shown graphically). Angry is consistently hardest for XTTS and Toucan, which matches the corpus imbalance (anger is 0.5% of segments). Toucan (lb) keeps LuxASR WER in a narrow band across emotions.

The authors frame the benchmark as a usability comparison, not a controlled ablation. The systems differ in architecture, training data, language coverage, vocoder and adaptation procedure.

Novelty Assessment

The main contribution is the resource. As far as the paper reports, LuxEmo is the first expressive TTS corpus for Luxembourgish, and it comes with a documented semi-automatic curation recipe for broadcast archives. Every curation component is an existing tool: VAD, DeepFilterNet, MMS LID, LuxASR, a HuBERT emotion classifier and a lexical model. The novelty is in combining them for a code-switching, low-resource broadcast setting. The benchmark is modest: five heterogeneous systems with mismatched sample counts (53 to 612 per system), a non-crossed 20-listener test, and no statistical tests. No new TTS method is proposed. The two adapted systems are fine-tuning and retrieval applications of existing models.

Field Significance

Low — LuxEmo provides a first expressive speech resource and benchmark for Luxembourgish, with a native-listener evaluation of emotional TTS output. Its reach is limited by four speakers, weakly supervised emotion labels and an almost empty anger class. Its broader value is as a cautionary data point: on realistic, code-switching speech, quality predictors, ASR intelligibility and listener-rated emotional quality rank systems differently.

Claims

  • complicates: Non-intrusive quality predictors disagree with each other on synthesized speech, so a single predicted-MOS metric can misrank expressive TTS systems.

    Evidence: Toucan (lb) leads NISQA-TTS (2.95) and NISQA overall (3.94), while GradTTS (de) leads WV-MOS (3.89 vs 3.50 for Toucan) and XTTS (de) leads DNSMOS OVRL (2.87). (§IV-A, Table II, Figure 2)

  • complicates: ASR-based intelligibility and predicted waveform quality do not reliably track listener-rated naturalness and emotional appropriateness for expressive TTS in a low-resource language.

    Evidence: Qwen3 FT has the highest listener naturalness (3.9) and emotion appropriateness (4.0) but by far the highest LuxASR WER (13.26 as plotted) and the lowest NISQA overall (1.75). The authors note that confidence intervals overlap among the top systems. (§IV-B, §IV-C, Table II, Figure 2)

  • supports: Cross-lingual transfer from a closely related high-resource proxy language can yield smooth synthesized speech while leaving target-language segmental accuracy worse than a model with target-language phoneme coverage.

    Evidence: XTTS (de) and Toucan (lb) have similar WV-MOS (3.52 vs 3.50), but Toucan (lb) has lower Whisper WER (1.23 vs 1.75) and LuxASR WER (0.83 vs 1.11). Qualitative inspection finds German prosody leaking into Luxembourgish utterances and mispronounced Luxembourgish-specific words. (§IV-B, §IV-D, Figure 2)

  • complicates: Adapting expressive TTS to a small, noisy, spontaneous target-language corpus does not uniformly improve objective metrics, and gains on one axis can come with losses on another.

    Evidence: kNN TTS retrieves LuxEmo prosody (F0 RMSE 64.21 Hz, near GradTTS (de) at 62.99 Hz) but has very low WV-MOS (0.92). Qwen3 FT keeps high speaker similarity (0.31) but has the worst F0 RMSE (133.04 Hz) and LuxASR WER (13.26). (§IV-B, Figure 2)

  • complicates: Emotion classes that are rare in spontaneous broadcast speech remain the hardest to synthesize, which limits how far found-data corpora can support high-arousal expressive TTS.

    Evidence: Anger covers 38 of 7,562 segments (0.5%) and is consistently the hardest emotion for XTTS and Toucan. Confusions are also frequent for low-intensity sadness. (§IV-C, §IV-D, Table I, Figure 3)

Limitations and Open Questions

Warning

Emotion labels are weakly supervised (a HuBERT classifier plus lexical cues, with threshold tuning on a manually annotated subset) and are explicitly not validated as gold labels, and anger has only 38 segments. Claims about emotion-specific TTS behaviour on LuxEmo inherit this label uncertainty.

  • Only four recurring presenters (1 male, 3 female), set by the source material. The authors state that the corpus is not demographically representative.
  • Transcripts come from LuxASR, not human transcription, and LuxASR is also used to score intelligibility. Manual review covered about 15% of segments and focused on language labels.
  • The benchmark is not controlled. Systems differ in training data, vocoder and adaptation, and they are evaluated on different sample counts (612, 53 and 280). The listening test is not fully crossed, has no subjective intelligibility rating, and reports no significance testing. The announced emotion recognition accuracy and inter-rater agreement are not reported.
  • WER units are not stated, and several plotted values exceed 1.
  • The Qwen3 FT and Toucan systems are not identified by citation, and model sizes are not reported.
  • The anonymous sample repository linked in the paper is a sample page. Release of the full corpus and code is not stated explicitly in the text.

Wiki Connections

  • Emotion Synthesis — Benchmarks emotional TTS output on a spontaneous Luxembourgish corpus with listener emotion appropriateness ratings and per-emotion objective breakdowns.
  • Subjective Evaluation — Runs a 20-listener native test of naturalness and emotion appropriateness and finds that it ranks systems differently from ASR and quality predictors.
  • Speaker Adaptation — Fine-tunes a prompt-controlled LM-based TTS per speaker on LuxEmo and evaluates its speaker similarity and listener ratings.
  • Grad-TTS — Evaluated zero-shot with German as a proxy language, and also used for the acted EmoDB German reference condition.
  • XTTS — Evaluated zero-shot with German as a proxy language. It has the highest speaker similarity but intermediate Luxembourgish intelligibility.
  • kNN-TTS — The kNN-retrieval GlowTTS approach adapted to LuxEmo for prosody transfer. It is close to Grad-TTS (de) on F0 RMSE but has poor waveform quality.
  • Whisper — Whisper base provides the generic WER metric, and LuxASR is a Luxembourgish fine-tune of Whisper large-v3.
  • SpeechBrain — Toolkit used for the ECAPA-TDNN speaker-similarity and EER computation.
  • ECAPA-TDNN — Speaker embedding model behind the speaker-similarity metric.