arXiv · 2025 · Preprint
Woszczyk et al. (Amazon Alexa / Imperial College London) · → Paper · Demo: ? · Code: ?
A voice conversion system for Lombard speaking style is evaluated with implicit style reconstruction loss and explicit acoustic feature conditioning, showing that the implicit approach matches intelligibility gains from explicit conditioning while better preserving target speaker similarity.
Problem
Training TTS systems in Lombard speaking style (the natural speech modifications people produce in noisy environments) requires recordings of each target speaker producing Lombard speech, which is difficult to collect at scale due to variability in noise conditions, speaker-specific Lombard effect intensity, and tiring recording conditions. Voice conversion offers a route to augmenting Lombard TTS datasets by transferring Lombard prosodic characteristics from existing recordings to target voices. However, most VC models focus on speaker identity transfer and discard speaking style features, making it necessary to design explicit mechanisms that retain Lombard characteristics during conversion.
Method
The VC model is built on CopyCat, a non-parallel many-to-many prosody transfer model with a VAE-based reference encoder. Mel-spectrograms and speaker embeddings from a speaker verification model are encoded together with frame-level phonemes, and the concatenated representation is passed to a decoder that outputs a mel-spectrogram for vocodering. The baseline is trained with a KL-divergence loss on the VAE component and an L1 reconstruction loss on mel-spectrograms.
Two conditioning strategies are compared. In the explicit conditioning approach, f0, spectral energy (mgc0), and spectral tilt (mgc1) are extracted from the source Lombard recording using the WORLD vocoder and fed directly to the decoder, conditioning the model to follow the source feature distributions. In the implicit conditioning approach, an auxiliary style reconstruction loss is added using a Lombard/neutral binary classifier pre-trained on oracle mel-spectrograms. Training proceeds in two stages: the classifier is trained first and its weights are then frozen while the VC model is trained with the combined reconstruction, KL-divergence, and binary cross-entropy style loss. At inference, the classifier is discarded. A fusion system combining the style reconstruction loss with explicit features is also evaluated.
Key Results
A pilot study on the Audio-Visual Lombard Grid dataset confirms that Lombard speech is rated more intelligible than neutral speech in noisy conditions (MUSHRA subjective intelligibility 65.66 vs. 57.26 at SNR-1) and that speaker similarity remains perceptible to listeners even in noise, supporting the value of preserving speaker identity during style conversion (Table 1).
For the main VC experiments, spectral energy and tilt (mgc0+mgc1) are the most effective explicit features across genders, while f0 shows gender-specific behaviour: it improves SIIB for male target speakers but is detrimental for female target speakers (Table 2). The implicit conditioning model achieves SIIB scores of 86.79 and 107.26 for female and male target speakers at SNR-1, comparable to the best explicit conditioning variants, while performing closer to the baseline on speaker similarity (Figure 6). The fusion of the style reconstruction loss with all explicit features (Ls+f0+mgc0+mgc1) achieves the best combined objective scores in some conditions, but the marginal improvement over implicit conditioning alone is inconsistent across genders and noise levels (Figure 5).
Novelty Assessment
The core contribution is applying a style reconstruction loss (an established technique from expressive TTS) to Lombard-style VC for the first time. Both the CopyCat backbone and the principle of using a frozen auxiliary classifier to enforce implicit style preservation are prior work; this paper combines them in a new domain. The result is a useful engineering integration confirming that implicit style learning avoids manual acoustic feature selection while matching intelligibility gains from explicit conditioning. The evaluation is constrained to a single small dataset (54 speakers, 100 utterances each) with two VC target speakers, which limits generalizability. The paper was originally presented at the Clarity Challenge 2023, reflecting its applied scope.
Field Significance
Low — Applies known voice conversion and style reconstruction techniques to Lombard speaking style for the first time, demonstrating that implicit conditioning via an auxiliary loss can substitute for hand-crafted acoustic feature extraction in this domain. The evaluation scope is narrow (single dataset, two target speakers), and the findings are primarily confirmatory of established methods in a new application setting. The paper provides useful evidence for VC-based data augmentation for intelligibility-enhancing TTS, but the contribution does not extend beyond this specific speaking style context.
Claims
-
supports: Implicit style modeling via an auxiliary reconstruction loss can preserve speaking style characteristics in voice conversion without requiring explicit acoustic feature extraction.
Evidence: A style reconstruction loss using a frozen Lombard/neutral classifier achieves SIIB scores comparable to explicit conditioning on f0, mgc0, and mgc1 across male and female target speakers, with no statistically significant differences in most conditions. (§3.3, §3.4, Table 2)
-
complicates: The effectiveness of explicit prosodic features for style-preserving voice conversion depends on speaker gender, limiting the generalizability of feature-based conditioning approaches.
Evidence: F0 conditioning improves SIIB for male target speakers but is detrimental for female target speakers at both SNR-1 and SNR-3; spectral tilt and energy (mgc0+mgc1) are beneficial across genders. (§3.4, Table 2)
-
supports: Speaker identity remains perceptible to listeners in noisy conditions, confirming that speaker similarity is a relevant evaluation dimension for intelligibility-enhancing voice conversion.
Evidence: Pilot study results show that listeners maintain consistent similarity rankings between speakers at SNR-1 and SNR-3 noise levels, and the implicit conditioning VC model preserves speaker similarity comparably to the CopyCat baseline. (§2.3, Figure 1, §3.4, Figure 6)
-
complicates: Combining implicit and explicit style conditioning does not consistently improve intelligibility over either approach alone, and can degrade performance in some configurations.
Evidence: Fusion of Ls with individual explicit features (e.g., Ls+f0 for female targets) reduces SIIB compared to Ls alone; only Ls+mgc0+mgc1 reliably improves over Ls in most conditions, and gains are inconsistent across genders. (§3.4, Figure 5, Table 2)
Limitations and Open Questions
The model does not modify duration, which is an observed acoustic correlate of the Lombard effect, potentially leaving an intelligibility benefit unexploited. The dataset is small (54 speakers, 100 short utterances each) and recorded under a constrained sentence task that produces somewhat robotic prosody, limiting the naturalness of synthesised samples. Only two target speakers are evaluated in the VC experiments, making it difficult to assess robustness across a wider voice population. The authors propose extending to larger conversational datasets, evaluating downstream Lombard TTS intelligibility using transcriptions and hearing-loss-targeted metrics, and exploring adversarial disentanglement to control Lombard intensity independently of speaker identity.
Wiki Connections
- Voice Conversion — This paper addresses Lombard-style voice conversion, targeting speaker identity transfer that also preserves the acoustic characteristics defining Lombard speech.
- Prosody Control — Explicit conditioning on f0, spectral energy, and spectral tilt provides a mechanism for preserving source Lombard prosody during voice conversion, with gender-dependent effectiveness.
- Subjective Evaluation — MUSHRA-like listening tests for perceived intelligibility, naturalness, and speaker similarity are used to validate both oracle Lombard speech and the VC systems in noisy conditions.
- Evaluation Metrics — SIIB (speech intelligibility in bits) is used as the primary objective intelligibility metric alongside subjective MUSHRA scores, chosen for its suitability in noise-degraded speech assessment.