Interspeech · 2025 · Conference
Bogdan Vlasenko et al. (Idiap Research Institute) · → Paper · Demo: ✗ · Code: ✓
An empirical study showing that early fusion of multilingual SSL acoustic embeddings (WavLM Large) with word-level multilingual text embeddings (XLM-RoBERTa) significantly outperforms unimodal and knowledge-based features for sentence mode prediction (declarative/interrogative/exclamatory) across Italian, French, and German, with implications for multilingual prosody modeling in TTS.
Problem
Sentence mode (declarative, interrogative, exclamatory) is a suprasegmental prosodic property that TTS systems must model to generate contextually appropriate speech. TOBI-based prosodic annotation is language-specific and costly, limiting scalable multilingual prosody modeling. Prior work relies on knowledge-based features (F0 modeling) or unimodal SSL representations. It remains unclear how well multilingual latent spaces preserve sentence-mode information, and whether acoustic or linguistic features are more discriminative for cross-lingual transfer.
Method
The study evaluates three systems for sentence mode prediction (3-class: ’.’, ’!’, ’?’) on Italian, French, and German:
System 1 (main comparison): An MLP classifier (100 hidden units, ReLU) applied to a variety of feature representations:
- Acoustic: (a) knowledge-based (COMPARE 2016, 6373 features, openSMILE) and (b) data-driven (WavLM Large final-layer embeddings, mean-pooled to sentence level)
- Linguistic: (a) phoneme-level (XPhoneBERT, multilingual phoneme BERT) and (b) word-level (XLM-RoBERTa, trained on 2.5TB of 100-language CommonCrawl)
- Early fusion (EF): (a) data-driven acoustic + word-level linguistic, and (b) all four feature types combined
Both intra-language (5-fold cross-validation) and inter-language (train on one, test on another) protocols are evaluated.
System 2 (ASR baseline): WHISPER Large V3 transcripts with punctuation detection.
System 3 (cross-corpora): MLP trained on CML-TTS German acoustic features, evaluated on EMO-DB (German emotional speech, neutral vs. emotional conditions).
Data: CML-TTS audiobook recordings (Italian: 6226, French: 7160, German: 45718 samples). Labels derived automatically from terminal punctuation marks.
Key Results
Best intra-language UAR (System 1, early fusion of all features):
- German: 74.22% (data-driven EF) / 73.40% (all EF)
- French: 67.10% (data-driven EF)
- Italian: 64.14% (data-driven EF)
Key findings:
- Linguistic FRS (word-level XLM-RoBERTa) consistently outperform acoustic FRS for interrogative sentence discrimination.
- Acoustic FRS (knowledge-based) capture exclamatory sentences better.
- Early fusion outperforms unimodal features in intra-language settings.
- Cross-lingual transfer is feasible: models trained on German achieve UAR up to 63.62% on French (inter-language).
- WHISPER (System 2) performs poorly on exclamatory sentences (recall well below chance for all languages), despite strong declarative/interrogative recall, suggesting current ASR systems cannot reliably generate prosodically informative punctuation.
- Emotional speech (EMO-DB) causes significant degradation: most emotional samples are misclassified as exclamatory, reflecting the definitional overlap between exclamatory tone and emotional speech.
Novelty Assessment
This is primarily an empirical study rather than a new model. Its contribution is a systematic, multilingual evaluation of multimodal feature fusion for sentence mode prediction, using recent SSL embeddings not previously benchmarked for this task. The finding that WHISPER transcripts fail to capture exclamatory sentence prosody is practically important for TTS pipelines that rely on ASR-derived text labels. The cross-corpora emotional speech experiment is a useful stress test. The lack of a dedicated TTS synthesis component means this paper’s direct application to TTS output quality is indirect — it informs feature engineering for prosody prediction models that could be integrated into TTS frontends.
Field Significance
Moderate — This paper provides a useful empirical baseline for multilingual prosody feature evaluation, demonstrating that multimodal fusion of SSL acoustic and text embeddings outperforms unimodal approaches for sentence mode classification. Its practical significance for TTS developers lies in the finding that ASR-derived punctuation (WHISPER) is an unreliable proxy for exclamatory prosody, a caveat directly relevant to pipelines that use ASR transcripts as prosody labels.
Claims
- Multimodal fusion of data-driven acoustic and word-level linguistic embeddings outperforms unimodal and knowledge-based features for multilingual sentence mode classification. (§4, Table 2)
- Multilingual SSL representations trained on large corpora can transfer sentence-mode discriminative information across languages without language-specific training. (§5)
- State-of-the-art ASR systems are unable to reliably detect exclamatory sentence mode from speech, producing recall rates below chance level for that class. (§4, Table 3)
- Sentence mode prediction performance degrades substantially on emotional speech due to definitional overlap between exclamatory sentence mode and emotional expressiveness. (§5, Table 4)
Limitations and Open Questions
- No end-to-end TTS experiment; the study measures sentence mode prediction accuracy, not synthesized prosody quality.
- Labels derived from punctuation marks are a proxy for sentence mode; they may not reflect actual prosodic realization (especially for audiobooks where speakers may monotonize exclamatory passages).
- Only three languages evaluated; coverage of typologically diverse languages is absent.
- MLP classifier is simple; more powerful sequence models may perform better.
- UAR is used rather than accuracy due to class imbalance, but class sizes differ substantially across languages.
Wiki Connections
- prosody-control — the paper establishes that multilingual SSL embeddings encode sentence-mode information useful for TTS prosody frontends
- self-supervised-speech — WavLM Large and XLM-RoBERTa are core feature extractors in the experimental system
- multilingual-tts — cross-lingual transfer experiments directly address shared prosodic representations across language families
- evaluation-metrics — the WHISPER ASR baseline result raises questions about reliability of text-derived prosody labels