Interspeech · 2025 · Conference
Yejin Jeon et al. (POSTECH) · → Paper · Demo: ? · Code: ?
A teacher-student knowledge anchoring framework with curriculum learning enables zero-shot multi-speaker TTS to generate intelligible, speaker-consistent speech from short dysarthric reference audio, achieving a 50+ point reduction in phoneme error rate over previous baselines.
Problem
Personalized TTS for dysarthric speakers faces two compounding difficulties. First, dysarthric speech contains articulation errors (slurring, incorrect phoneme production) that, if used as training data directly, cause models to reproduce those errors in synthesis. Second, dysarthric speakers can typically only produce very short utterances (single words) due to the physical demands of speech production, whereas standard TTS training relies on long, well-articulated sentences. This creates a domain gap between training data (normal speech) and inference conditions (short, dysarthric reference audio), both in articulation quality and in utterance duration. Prior hybrid approaches (single-speaker TTS followed by voice conversion) address intelligibility at the cost of speaker identity.
Method
The system uses a FastSpeech2 backbone acoustic model with HiFi-GAN vocoder for mel-to-waveform conversion. The key innovation is in the speaker encoder, which is split into a teacher-student pair:
Knowledge anchoring. Both teacher (θ_t) and student (θ_s) speaker encoders share an identical architecture: a pretrained speaker verification network computes per-utterance embeddings, which are averaged into a speaker centroid, then passed through a linear projection, FC block with Mish activation, and a GLU-gated convolution stack. The teacher receives full-length reference audio and produces a speaker representation that conditions the backbone TTS (text encoder and decoder via adaptive instance normalization). The student receives shorter (cropped) audio and is trained to match the teacher’s output via an MAE loss (L_MAE between teacher and student style vectors). The teacher thus acts as an anchor: the student learns to extract timbre-specific features robust to articulation distortions, while being guided by the cleaner teacher representation as a filter.
Curriculum learning. The student is progressively trained on shorter audio segments over the course of training. With C=3 cropping stages over S=500,000 steps, each stage of ~160,000 steps trains on audio cropped to (1 - k/(C+1)) of the full length, transitioning from 75% to 50% to 25% of the full reference length. This gradual exposure bridges the gap from training-time long utterances to inference-time single-word inputs. Random cropping is shown to be inferior to progressive cropping in ablations. At inference, only the student encoder is used.
The total training loss combines mel reconstruction MAE and the teacher-student MAE loss on style vectors. All training uses LibriSpeech (normal speech); zero-shot synthesis is then applied to UASpeech dysarthric recordings.
Key Results
Evaluated on UASpeech against three baselines: Adaptive (style prototype/episodic training), Conditional (pretrained speaker verification encoder), and Hybrid (FastPitch + FreeVC).
The proposed method achieves PER 14.254% vs. 31.017% for the best baseline (Hybrid), a >50% relative reduction. Speaker similarity is 0.619 (comparable to Conditional at 0.647, better than Hybrid at 0.534). MOS-Nat: 3.601 vs. 3.371 (Hybrid); MOS-Spk: 3.909 vs. 3.731 (Hybrid).
Across dysarthric severity groups (Very Low / Low / Middle / High intelligibility), the proposed model consistently achieves the lowest PER and comparable or superior speaker similarity. The benefit is most pronounced for Low intelligibility speakers (PER drops from ~33 to ~15).
Ablation confirms both knowledge anchoring and curriculum learning are necessary: removing the student model raises PER from 14.254 to 22.846; removing curriculum learning raises it further to 26.428.
Novelty Assessment
The application of teacher-student knowledge distillation to speaker encoder learning for dysarthric TTS is new. The framing of dysarthric TTS as a dual domain-transfer problem (articulation distortion + utterance length mismatch) is clear and well-motivated. The curriculum learning strategy — progressive shortening of student inputs — is a practical and effective solution to the length mismatch problem. The backbone (FastSpeech2 + HiFi-GAN + speaker conditioning) is standard; the novelty lies entirely in the training strategy for the speaker encoder. The 33M-parameter model is compact and trains on a single GPU.
Field Significance
Moderate — This paper introduces a practical training strategy for adapting zero-shot multi-speaker TTS to pathological speech, a domain where the standard assumption of well-articulated reference audio breaks down. The teacher-student knowledge anchoring and curriculum learning approach provides a reusable template for any task where inference-time inputs differ systematically from training-time inputs in both quality and duration. The specific gains in phoneme error rate are substantial and demonstrate that the dual domain-transfer challenge in dysarthric TTS is tractable with targeted architectural choices.
Claims
- A teacher-student speaker encoder architecture, where the teacher conditions the TTS backbone during training and the student handles inference-time pathological inputs, substantially reduces phoneme articulation errors compared to single-encoder approaches in dysarthric TTS. (§4.2, Table 3)
- Progressive curriculum learning, which gradually reduces the duration of student encoder inputs during training, outperforms random cropping and substantially improves phoneme error rate over training without structured audio augmentation. (§4.3, Figure 3)
- Zero-shot multi-speaker TTS trained on normal speech can generalize to dysarthric speakers at inference time without any dysarthric training data, provided the speaker encoder is made robust to articulation distortions and short reference lengths. (§1, §3)
- Speaker similarity and phoneme intelligibility can be simultaneously improved through knowledge anchoring, avoiding the trade-off seen in prior hybrid approaches where intelligibility gains came at the cost of speaker identity. (§4.1, Table 1)
Limitations and Open Questions
The system is English-only. The backbone is non-autoregressive (FastSpeech2) with mel-spectrogram output, which imposes a quality ceiling compared to codec-based or flow-matching systems. The approach requires labeled phoneme data to compute PER, which is available for UASpeech but may not generalize. Speaker similarity (0.619) remains below what might be needed for truly personalized assistive use. The method has not been tested on languages with non-Latin scripts or very different phoneme inventories. Future work could explore curriculum learning with codec-based or diffusion-based TTS backbones.
Wiki Connections
- zero-shot-tts — extends zero-shot multi-speaker TTS into the assistive technology domain, where the input reference audio is pathological rather than normal
- disentanglement — the knowledge anchoring objective explicitly separates speaker timbre from articulation distortions, preventing content leakage into the speaker conditioning signal
- speaker-adaptation — the student encoder must learn a robust speaker representation from minimal, low-quality reference audio, a harder variant of the few-shot speaker adaptation problem