arXiv · 2025 · Preprint
Karen Rosero et al. (Carnegie Mellon University) · → Paper · Demo: ? · Code: ?
Adapts a style-based TTS reconstruction framework to children’s disordered speech, preserving speaker identity while suppressing mispronunciations, and validates an automatic clinical intelligibility metric against expert annotations.
Problem
Disordered speech reconstruction, generating a more intelligible version of an utterance while preserving the speaker’s identity, has been explored mainly for adult dysarthric speech using voice conversion or, more recently, pre-trained TTS adapted with speaker style. Two gaps motivate this paper. First, reconstruction models trained on healthy adult speech degrade when applied to disordered speech, since acoustic and prosodic patterns differ substantially. Second, most prior work targets adults; children with speech sound disorders (SSD) are underexplored despite having articulation and prosodic patterns that diverge even further from adult norms, and despite holistic per-speaker adaptation risking the reproduction of the child’s own mispronunciation patterns rather than suppressing them.
Method
The paper proposes ChiReSSD, an adaptation of StyleTTS2 that separates pronunciation-linked acoustic style from prosodic and speaker-identity style so the two can be manipulated independently at inference. StyleTTS2 already factorizes generation into acoustic and prosodic style encoders, a style diffusion sampler, text encoders operating on IPA phoneme sequences, an audio-text aligner, duration/prosody predictors, and a HiFi-GAN decoder. ChiReSSD fine-tunes the acoustic and prosodic style encoders and the pitch extractor on child SSD data, increasing the weight of the pitch reconstruction loss and partially freezing the style diffusion model, since StyleTTS2’s original pitch ranges were calibrated to adult speech. Text is phonemized with a British English phonemizer to match the Central Scottish accent of the training population.
At inference, style embeddings are extracted in a single pass from a four-second reference recording of an unseen child, requiring no parallel data and no reference of the target utterance. Two guidance parameters control generation: α scales the contribution of the acoustic text encoder (set to 0.8, favoring canonical pronunciation over the reference’s mispronunciation patterns), and β regulates how closely the generated prosody follows the source style (set to 0.6). Generation uses 10 diffusion steps. Separately, the paper proposes an automated clinical evaluation pipeline: a universal phone recognizer (wav2vec2-lv-60-espeak-cv-ft) transcribes both the original and reconstructed utterances into IPA, consonants are extracted, and the Levenshtein distance between the two consonant sequences is used to estimate percentage of correct consonants (PCC), a standard clinical severity metric, without requiring manual phonetic transcription.
Key Results
On the STAR dataset (six children, 234 utterances), ChiReSSD achieves a speaker-similarity score of 0.62 (Resemblyzer embeddings), exceeding a one-shot StyleTTS2 baseline (0.52) and a single-speaker Standard TTS baseline trained on LJSpeech (0.40); all differences are statistically significant (§5.1, Table 1). ChiReSSD also has the smallest pitch deviation from the original (19.98% F0 difference, 3.83 semitones) versus 22.58%/4.14 semitones for one-shot TTS and 39.95%/6.09 semitones for Standard TTS. For lexical accuracy (§5.2, Table 2), ChiReSSD reduces CER from 0.56 (no reconstruction) to 0.29 and WER from 0.84 to 0.49 on STAR; Standard TTS scores lower error (CER 0.11, WER 0.34) because it discards style transfer and produces adult-like, more ASR-friendly speech, illustrating a direct trade-off between intelligibility-to-an-ASR and identity preservation. On synthetic LibriTTS sentences generated with STAR-derived style embeddings, ChiReSSD reaches CER 0.06 and WER 0.15, indicating that STAR’s residual error is driven largely by its non-sentential utterance content rather than by the reconstruction pipeline itself.
For the automated clinical evaluation (§5.3), the automatic pipeline estimates PCC improving from 68.14% (original) to 87.44% (reconstructed) across the full STAR set; on a 21-pair subset independently annotated by a certified speech-language therapist, clinical PCC improves from 70.13% to 94.81% while the automatic estimate improves from 50.65% to 92.21%. The Pearson correlation between automatic and expert consonant-difference measures is ρ = 0.63. On TORGO (adult dysarthria, §5.4), ChiReSSD reduces CER to below 0.03 across all severity levels (from 0.40 to 0.02 in the severe group) and WER from 0.61 to 0.06 in the severe group, while maintaining speaker similarity of 0.74-0.77, above the 0.6 threshold commonly cited for speaker verification and clearly higher than Standard TTS (0.43-0.55) at matched severity.
Novelty Assessment
The architecture is not new: ChiReSSD is a fine-tuning adaptation of the existing StyleTTS2 model, and the loss-reweighting and partial-freezing recipe used to retarget its pitch range to children’s voices is an engineering adaptation rather than a structural innovation. The paper’s more distinctive contribution is the automated PCC estimation pipeline, which repurposes a universal phone recognizer and edit-distance comparison to approximate a clinical severity metric without manual transcription, and validates it against real expert annotations rather than only against automatic metrics. The domain itself, style-based TTS reconstruction targeted specifically at pediatric SSD with cross-population generalization to adult dysarthria, is underexplored, so the paper’s evidence is a genuine (if narrow) addition even though the underlying method is incremental.
Field Significance
Moderate — the paper demonstrates that a pretrained style-disentangled TTS system can be adapted with targeted loss reweighting to a population (children with SSD) whose acoustic and prosodic ranges differ substantially from its original training data, and it provides one of the first systematic comparisons between an automatic and a clinician-derived intelligibility metric for reconstructed disordered speech. Its contribution is primarily an application and evaluation-methodology one rather than an architectural advance.
Claims
- supports: Disentangling acoustic (pronunciation-linked) style from prosodic and speaker-identity style in a TTS-based reconstruction system allows mispronunciation patterns to be suppressed while speaker identity is preserved.
Evidence: ChiReSSD, which selectively fine-tunes acoustic/prosodic style encoders and the pitch extractor while reusing StyleTTS2’s disentangled style representations, achieves higher speaker similarity (0.62) and lower pitch deviation (19.98%) than a one-shot baseline that transfers style holistically from an unseen reference without domain adaptation (0.52 similarity, 22.58% F0 difference). (§5.1, Table 1)
- complicates: Optimizing a speech reconstruction system for automatic-speech-recognition-based intelligibility metrics can conflict with optimizing for speaker-identity preservation.
Evidence: A single-speaker adult-voice TTS baseline with no style transfer obtains the lowest WER/CER on STAR (0.34/0.11) precisely because it discards the child speaker’s identity, while the identity-preserving ChiReSSD model scores worse on the same ASR-based metrics (WER 0.49, CER 0.29) despite being judged clinically more accurate. (§5.2, Table 2)
- supports: An automatic phone-recognition-based estimate of consonant production accuracy can approximate expert clinical judgments of speech intelligibility well enough to reduce, though not eliminate, the need for manual phonetic transcription.
Evidence: Automatic PCC estimates derived from a universal phone recognizer and Levenshtein distance correlate with a certified speech-language therapist’s manual PCC annotations at Pearson ρ = 0.63 across 21 original/reconstructed sample pairs. (§5.3)
- supports: A style-based TTS reconstruction approach adapted for one clinical population of disordered speech can generalize to a structurally different disorder and age group without population-specific retraining.
Evidence: ChiReSSD, trained only on pediatric SSD data, reduces CER to below 0.03 and WER to as low as 0.06 on the TORGO adult dysarthria dataset across mild-to-severe severity levels, while keeping speaker similarity at 0.74-0.77, versus 0.43-0.55 for a non-personalized TTS baseline. (§5.4, Table 3)
Limitations and Open Questions
Warning
Clinical validation of the automatic PCC metric rests on only 21 annotated sample pairs from a single speech-language therapist, with no inter-rater reliability reported; the ρ = 0.63 correlation should be treated as a preliminary estimate, not a validated clinical tool.
The STAR evaluation set itself is small (234 utterances from six children), and reconstruction quality is partly assessed with ASR models (whisperX) that are known to be poorly calibrated for child and disordered speech, which the paper acknowledges as a likely source of residual error inflation on STAR relative to LibriTTS. Generalization beyond the two evaluated accents/populations (Central Scottish child SSD, Canadian English adult dysarthria) is untested, and the guidance parameters (α, β) were tuned rather than learned, leaving open whether they transfer to other disorder types or severities without manual retuning.
Wiki Connections
- Voice Conversion — frames disordered speech reconstruction as a style-transfer problem structurally similar to VC, reconstructing intelligible speech while preserving the source speaker’s identity rather than converting to a different target speaker.
- Zero-Shot TTS — extracts style embeddings from a four-second reference of an unseen child speaker in a single pass, enabling reconstruction for any target text without speaker-specific training or parallel data.
- Disentanglement — relies on and extends StyleTTS2’s separation of acoustic (pronunciation-linked) and prosodic style embeddings, selectively suppressing the acoustic component while retaining prosody and identity.
- Prosody Control — introduces a guidance parameter (β) that explicitly trades off prosodic fidelity to the reference speaker against naturalness during generation.
- Evaluation Metrics — proposes an automatic phone-recognition-based estimator of percentage of correct consonants (PCC) as a lower-cost substitute for manual clinical phonetic transcription.
- Subjective Evaluation — validates its automatic clinical metric against real expert annotations from a certified speech-language therapist rather than relying solely on automated proxies.