arXiv · 2025 · Preprint
Valdivia et al. (California State University Long Beach) · → Paper · Demo: ? · Code: ?
A mispronunciation detection system that uses ElevenLabs voice cloning to generate a speaker-matched, correctly-pronounced reference, then compares original and clone frame-by-frame with MFCC features and DTW to identify pronunciation deviations.
Problem
Existing computer-assisted pronunciation training (CAPT) systems typically rely on generic reference models — such as GOP (Goodness of Pronunciation) scoring — that do not account for individual speaker variability or L1 phonological transfer effects. Because these systems use speaker-independent baselines, they tend to miss nuanced mispronunciations tied to a learner’s idiolect, accent, or specific native-language interference patterns. Training speaker-specific models requires substantial labeled data that is rarely available for less common L1 backgrounds, leaving CAPT systems poorly calibrated for the breadth of real-world learner populations.
Method
The system operates in two phases over six steps. During calibration, learner utterances are passed through the ElevenLabs voice cloning API to produce a speaker-matched synthetic version with corrected pronunciation. Word-level alignments are extracted from TextGrid annotation files, and for each aligned word pair, 13-dimensional MFCC envelopes are computed for both the real and cloned segments. Both sequences are resampled to a common length, and a normalized DTW distance is computed per MFCC coefficient, then averaged across all 13 dimensions to yield a scalar deviation score per word. Calibration builds two distance distributions — one for correctly pronounced words and one for mispronounced words — and fits kernel density estimators (KDE) to each. Class-specific decision thresholds are set at the 90th percentile of each distribution.
At runtime, a new utterance is cloned, word distances are computed, and each word is classified by comparing KDE likelihoods: if the incorrectness density exceeds the correctness density and the distance exceeds the incorrect-class threshold, the word is flagged as INCORRECT; if the reverse holds and the distance falls below the correct-class threshold, it is CORRECT; otherwise the prediction is AMBIGUOUS.
Key Results
Tested on four L2-ARCTIC speakers (EBVS, ERMS, MBMPS, NJS), classification accuracy ranges from 0.461 (MBMPS) to 0.646 (EBVS), with F1-scores for the mispronounced class ranging from 0.409 to 0.684 (Table 1). Across all four speakers, mispronounced words consistently show higher mean MFCC-DTW distances than correctly pronounced words (Table 2), providing empirical support for the acoustic deviation hypothesis. Performance varies substantially across speakers, with EBVS yielding noticeably stronger results than the other three, suggesting that per-speaker calibration could improve robustness. A case study on a Korean speaker demonstrates successful detection of the caught-cot vowel merger (/AW1/ versus /AO1/) in the word “Fraud.” No comparison to GOP scoring or other established CAPT baselines is provided.
Novelty Assessment
The framing is original: using a personalized voice clone as a pronunciation reference for acoustic deviation analysis had not been explicitly proposed in the literature reviewed by the authors. However, the technical components are all classical: MFCC features, DTW alignment, and KDE-based threshold estimation are established signal processing tools. The synthesis step relies entirely on the proprietary ElevenLabs API with no new model. The paper makes no architectural contribution to TTS or voice cloning; it is an application integration. The evaluation is limited to four speakers from a single corpus, with no comparison to standard CAPT baselines, making it difficult to assess whether the approach is competitive with existing mispronunciation detection methods.
Field Significance
Low — This paper applies existing voice cloning services and classical acoustic analysis to a CAPT task. It contributes a plausible hypothesis (that acoustic deviation between learner speech and a personalized voice-cloned reference localizes mispronunciations) and provides initial empirical support on a small subset of L2-ARCTIC, but the results are modest and the evaluation is narrow. The approach is not competitive with CAPT methods in the absence of baseline comparisons, and reproducibility is constrained by dependence on a proprietary API.
Claims
-
supports: Acoustic deviation between a learner utterance and a personalized voice-cloned reference correlates with mispronunciation at the word level.
Evidence: MFCC-DTW distances are consistently higher for mispronounced words than correctly pronounced words across all four L2-ARCTIC speakers tested (mean deviation of 0.027–0.028 units on average), confirming the core deviation hypothesis. (§4, Table 2)
-
complicates: Acoustic deviation metrics derived from voice cloning exhibit high per-speaker variability, making universal calibration insufficient for reliable mispronunciation detection.
Evidence: Classification accuracy ranges from 0.461 to 0.646 across four speakers; the authors note that per-individual models would likely improve results. (§4, Table 1)
-
complicates: Mispronunciation detection systems based on acoustic distance alone require an explicit ambiguity class when threshold-based decision rules cannot confidently separate correct from incorrect pronunciations.
Evidence: The system defines an AMBIGUOUS output for samples falling between class-specific KDE thresholds, but the paper does not report how frequently this category occurs in practice. (§3.6)
Limitations and Open Questions
Warning
No comparison against established CAPT baselines (e.g., GOP scoring, end-to-end neural classifiers) is provided, making it impossible to assess whether the proposed approach is competitive. All experiments use only four speakers from L2-ARCTIC, and the reliance on the proprietary ElevenLabs API prevents reproducibility.
The system produces an AMBIGUOUS output category but does not quantify its frequency, leaving the practical coverage of the detector unclear. Precision and recall trade-offs across speakers suggest that a single model calibrated on all speakers is suboptimal; speaker-specific models are proposed as future work. Extension to under-resourced languages and real-time implementation remain open.
Wiki Connections
- Zero-Shot TTS — The ElevenLabs voice cloning API, used here as the core reference-generation component, is a zero-shot TTS application; the quality and speaker fidelity of that synthesis directly determines the accuracy of the deviation signal.
- Voice Conversion — Personalized pronunciation-corrected synthesis shares objectives with accent conversion research; the paper evaluates on L2-ARCTIC, a corpus designed specifically for voice conversion and accent modification tasks.
- Evaluation Metrics — The paper introduces a normalized MFCC-DTW distance as a frame-level deviation score for mispronunciation classification, proposing an acoustic comparison methodology not previously applied in CAPT with personalized voice cloning.