arXiv · 2026 · Preprint
Benedikt Mayrhofer et al. (Graz University of Technology) · → Paper · Demo: ✓ · Code: ?
Adapts the lightweight StreamVC architecture, with its pitch and energy modules removed, to convert electro-laryngeal (post-laryngectomy) speech into healthy-sounding speech, using perceptually and intelligibility-guided losses to raise naturalness MOS from 1.1 to 3.3.
Problem
Post-laryngectomy patients who use an electrolarynx device produce electro-laryngeal (EL) speech characterized by constant, monotonic pitch, reduced prosody, and mechanical buzzer noise, which degrades naturalness, expressiveness, and intelligibility. Prior real-time voice conversion (VC) approaches for EL-to-healthy (HE) speech (e.g. causal CLDNN-based systems) reported limited modeling accuracy and reduced naturalness. Meanwhile, recent streaming VC architectures for healthy speech, such as StreamVC, achieve high quality at low latency, but StreamVC’s pitch (F0) conditioning module is unsuited to EL speech, since there is no meaningful F0 variation to condition on, and EL speech’s slower speaking rate and different acoustic characteristics complicate aligning EL and HE speech for supervised training.
Method
The system adapts StreamVC’s fully causal convolutional architecture (content encoder, speaker encoder with learnable pooling, and a FiLM-conditioned decoder), removing the pitch and energy estimation modules entirely (since EL speech lacks F0 variation and removing them also reduces model complexity) and replacing StreamVC’s HuBERT-base distillation teacher with a multilingual mHuBERT-147 teacher better suited to German. The resulting ~30M-parameter (18M content encoder, 4M speaker encoder, 8M decoder), 123MB model is trained in a GAN framework with a HiFi-GAN-style multi-period/multi-scale discriminator design.
Training has two stages. Pretraining is self-supervised on 542 hours of healthy (HE) German speech (from quality-filtered Common Voice, HUI-Audio-Corpus-German, and the German portion of MLS): the content encoder is first trained via k-means-clustered discrete-unit distillation from the mHuBERT-147 teacher, then frozen while the decoder and speaker encoder are jointly trained to reconstruct the waveform (mel-spectral loss first, then adversarial and feature-matching losses added). Fine-tuning uses the parallel Austrian-German ELHE corpus (8 pseudo-EL, 8 real-EL, and 8 HE speakers, 2.75 hours each of EL and HE speech), with the content encoder unfrozen. Because standard low-level acoustic alignment (mel-spectra, MFCC) proved unsuitable for the large EL-HE acoustic mismatch, the authors built a custom alignment pipeline: silence removal, PSOLA time-stretching of HE utterances to approximate EL speaking rate, extraction of Whisper-small encoder output (WEO) features from both signals as a shared content-oriented representation, dynamic time warping between EL and HE WEO sequences, and frame-wise PSOLA alignment of the EL waveform along the resulting warping path (keeping HE targets artifact-free). Aligning each EL utterance against all available HE realizations of the same linguistic content, rather than one-to-one pairs, expanded the training set from 3,298 to 19,592 EL-HE combinations.
Fine-tuning combines the original StreamVC losses with several perceptually guided auxiliary losses evaluated in different combinations: a WavLM-feature perceptual loss, a human-feedback (HF) loss defined as negative mean UTMOS score, an F0-contour loss (log-F0 MSE against pitch predicted by an external estimator), and two intelligibility losses (MSE against pre-softmax bottleneck features from a Conformer-CTC ASR model fine-tuned on HE speech, and MSE against Whisper-encoder hidden states from a Whisper model fine-tuned on HE speech). Noise-injection augmentation (30% of samples, SNR 3-30dB) is used for robustness.
Key Results
Objectively (160 unmodified test utterances), the best configuration (+WavLM+HF: WavLM perceptual loss plus human-feedback loss) reduces CER from 88.2% (unconverted EL) to 41.9%, versus 140.3% and 61.2% for offline VC baselines FreeVC and XVC fine-tuned on the same HE data; it also achieves the best or near-best DNSMOS (SIG/BAK/OVRL), wvMOS, and log-F0 RMSE, approaching ground-truth (GT) healthy speech on nearly all metrics except F0 RMSE and CER. Adding more than two auxiliary losses (e.g. +WavLM+HF+F0, +WEO+WavLM+HF) tends to degrade both intelligibility and quality relative to the two-loss combinations, and the more content-constrained bottleneck-feature intelligibility loss (+BNF+HF) improves wvMOS and speaker similarity but clearly worsens CER (55.4% vs. 41.9%). Subjectively (22 listeners, three configurations tested), +WavLM+HF achieves the best naturalness MOS (3.3, up from 1.1 for unconverted EL) and highest speaker similarity (0.77), while reducing WER and CER relative to EL input; +BNF+HF again shows the opposite pattern, higher naturalness and similarity but markedly worse intelligibility (WER 68.4% vs. 36.4% for +WavLM+HF). Under added background noise, CER degrades as SNR decreases for both quasi-stationary and non-stationary conditions, with non-stationary noise more harmful; below approximately 5dB SNR, converted speech becomes less intelligible than the unconverted EL input.
Novelty Assessment
The core VC architecture (StreamVC) is not modified beyond removing its pitch/energy modules and swapping the distillation teacher, so the paper’s contribution is an application-domain adaptation and a systematic empirical study of loss configurations, not a new architecture. The two more genuinely novel elements are the Whisper-encoder-output-based DTW alignment pipeline for a source-target pair with severe acoustic mismatch (a real methodological problem specific to disordered-speech VC that standard mel/MFCC alignment does not solve), and the transparent ablation across five perceptual/intelligibility loss types, which yields a non-obvious and useful negative result: more auxiliary losses is not better, and the choice of loss (WavLM/WEO versus BNF) trades intelligibility against naturalness and speaker similarity in opposite directions. The absence of standardized open-source EL-HE VC baselines limits the FreeVC/XVC comparison to an indirect reference point rather than a head-to-head benchmark against systems designed for this exact task.
Field Significance
Moderate, this is a well-executed, clinically motivated adaptation of an existing lightweight VC architecture to a genuine unmet need (electro-laryngeal voice rehabilitation), with rigorous objective and subjective evaluation and an honest identification of remaining bottlenecks (prosody generation, intelligibility, and noise robustness) rather than an overstated success claim. Its contribution is domain-specific rather than architecturally general, and the underlying dataset (Austrian-German, 8+8+8 speakers) is small and language-specific, limiting immediate generalization.
Claims
- supports: A GAN-based lightweight streaming voice conversion architecture designed for healthy speaker-to-speaker conversion can be adapted to disordered, non-prosodic source speech (electro-laryngeal speech) by removing its pitch and energy conditioning modules and adding perceptually guided auxiliary losses during fine-tuning.
Evidence: Adapting StreamVC with pitch/energy modules removed and fine-tuning with a WavLM-feature perceptual loss plus a human-feedback (UTMOS-based) loss reduces CER from 88.2% (unconverted EL speech) to 41.9% and raises subjective naturalness MOS from 1.1 to 3.3, in a ~30M-parameter, 123MB model. (§5, Table 1, Table 2)
- complicates: Adding more than a small number of auxiliary perceptual and intelligibility losses to a GAN-based voice conversion training objective does not reliably improve, and can degrade, both intelligibility and perceptual quality.
Evidence: Combining a single perceptual loss (WavLM or Whisper-encoder-output) with the human-feedback loss achieves the best results, while adding further terms (F0 contour, additional intelligibility losses) or using a more content-constrained bottleneck-feature intelligibility loss increases CER despite comparable or better naturalness and speaker-similarity scores. (§5, Table 1, Table 2)
- supports: Aligning severely acoustically mismatched parallel speech pairs is more effective using content-oriented ASR encoder features than low-level acoustic features, and expanding alignment beyond strict one-to-one utterance pairs meaningfully increases usable training data.
Evidence: A Whisper-encoder-output DTW alignment pipeline is used because standard mel-spectral or MFCC-based alignment proved suboptimal for the EL-HE acoustic mismatch, and aligning each EL utterance against all available HE realizations of the same content, rather than one-to-one, increased the training pair count from 3,298 to 19,592. (§3.2)
- complicates: Perceptually guided voice conversion of disordered source speech remains substantially more vulnerable to background noise than clean-condition intelligibility gains would suggest, with converted speech becoming less intelligible than the unconverted source below a moderate noise threshold.
Evidence: CER for the best model configuration degrades as SNR decreases under both quasi-stationary and non-stationary noise, and below approximately 5dB SNR the converted speech shows lower intelligibility than the unconverted electro-laryngeal input. (§5, Figure 5)
Limitations and Open Questions
- Prosody remains a stated open problem: the converted speech’s F0 contour approximates ground truth closely except for flattened pitch excursions at sentence endings, and log-F0 RMSE is one of only two metrics where the best model does not approach ground-truth performance.
- Evaluation is limited to German (Austrian-German ELHE corpus, 8 pseudo-EL, 8 real-EL, 8 HE speakers, 2.75 hours each), so generalization to other languages or larger, more diverse EL speaker populations is untested.
- No standardized open-source EL-HE VC baseline exists, so the FreeVC and XVC comparisons are indirect (general-purpose VC models fine-tuned on the HE portion of the same corpus) rather than systems designed for this specific task.
- Below approximately 5dB SNR, the proposed system’s converted output becomes less intelligible than leaving the electro-laryngeal input unconverted, a practical failure mode for real-world noisy-environment deployment that the authors flag but do not resolve.
Wiki Connections
- Voice Conversion — adapts a lightweight streaming VC architecture to convert disordered electro-laryngeal speech into healthy-sounding speech, evaluated with dedicated speaker-similarity and intelligibility metrics against real VC baselines.
- GAN Vocoder — trains waveform synthesis in a GAN framework with a HiFi-GAN-style multi-period/multi-scale discriminator design, following established GAN vocoder practice.
- Subjective Evaluation — conducts a 22-participant listening survey measuring transcription accuracy (WER/CER/WIP/WIL), naturalness MOS, and speaker-similarity ratings across three loss configurations.
- Self-Supervised Speech — the content encoder is trained via k-means-clustered discrete-unit distillation from a self-supervised mHuBERT-147 teacher during pretraining.
- HiFi-GAN — the discriminator design for the adapted StreamVC model follows this paper’s multi-period and multi-scale discriminator architecture.
- Whisper — used both as the zero-shot transcription model for CER evaluation and, fine-tuned on the HE portion of the corpus, as the source of content-oriented features for the EL-HE time-alignment pipeline.