arXiv · 2025 · Preprint
Wang et al. (National Taiwan Normal University) · → Paper · Demo: ✗ · Code: ?
Combines LLM-driven text synthesis with speaker-conditioned voice cloning (XTTSv2) and a dynamic importance reweighting loss to expand training data for low-resource automated speaking assessment of L2 opinion expressions.
Problem
Automated speaking assessment (ASA) for second-language (L2) opinion-expression tasks suffers from scarcity of labeled recordings. Few annotated prompts are available, which limits diversity during training and causes models to overfit. Standard TTS augmentation exacerbates the problem by producing over-clean speech lacking the disfluencies, mispronunciations, and prosodic variability characteristic of real non-native speakers, creating a domain mismatch between synthetic and authentic data.
Method
The system integrates three components within a unified low-resource training framework. First, an LLM (OpenAI o4-mini) generates proficiency-conditioned text responses via in-context learning: k real transcripts at the target proficiency level are concatenated into a prompt and the model decodes at temperature 1.5 to produce diverse synthetic responses aligned with each level’s language characteristics.
Second, a speaker-conditioned voice cloning stage converts these texts into learner-like audio. A speaker encoder extracts a 512-dimensional embedding from a real reference recording at the same proficiency level, capturing voice identity, speaking rate, rhythm, and characteristic disfluencies. A multi-speaker TTS model (Coqui-ai XTTSv2) then synthesises audio conditioned on this embedding, producing speech that preserves non-native prosodic traits rather than the overly fluent output of conventional single-speaker systems. Pairing each generated text with different reference embeddings across proficiency levels further diversifies the synthetic corpus.
Third, a dynamic importance reweighting loss (DIMP) bridges the remaining domain gap in mixed-data training. A quality model (Phi-4 multimodal fine-tuned on real data with LoRA) establishes a reference confidence distribution. A separate target model trained on the combined real and synthetic corpus computes per-instance importance ratios from the two models’ confidence scores, amplifying synthetic instances that most closely resemble real learner speech while applying an elevated base weight to real instances. The target model backbone is also Phi-4 multimodal, fine-tuned with LoRA, and directly ingests the audio recording alongside its ASR transcript and task prompt to predict a binary pass/fail proficiency score.
Key Results
Results are reported on the LTTC GEPT Intermediate-Advanced dataset (1,200 prompt-response-score triples; 900 for training, 300 held out as an unseen prompt test set). Baselines include wav2vec 2.0 (audio-only), BERT (transcript-only), their combination, and Phi-4 multimodal fine-tuned on real data only.
On seen-prompt test sets, the proposed approach achieves 76.67% overall accuracy and 81.11% pass-fail accuracy, compared to 73.33%/77.78% for the real-data-only Phi-4 baseline and 61.80%/65.10% for wav2vec 2.0 (Table 1). On unseen prompts, the model reaches 64.00% overall accuracy and 68.33% pass-fail accuracy, versus 63.33%/68.00% for the real-data-only baseline.
Ablation results confirm that synthesised data alone is insufficient: training only on synthetic speech yields 42.22% seen-test accuracy, while mixing real and synthetic at equal weight recovers most performance (72.22%), and the dynamic real-sample upweighting (alpha=2) yields the best result (76.67%) (Table 2). The gains on unseen prompts are modest (less than 1 percentage point over the real-only baseline), indicating the augmentation helps primarily in familiar prompt domains.
Novelty Assessment
The contribution is engineering integration. XTTSv2, OpenAI o4-mini, Phi-4 multimodal, and LoRA fine-tuning are all pre-existing systems used without architectural modification. The dynamic importance reweighting principle adapts a data-weighting technique from text classification (Kuo et al., ICLR 2025) to a mixed real-synthetic speech training setting. The application of speaker-conditioned TTS to generate non-native-sounding augmentation data is a practical insight, but the mechanism (speaker embedding from reference clip into a multi-speaker TTS model) is standard in zero-shot TTS. The paper’s genuine contribution is demonstrating that this specific combination improves ASA scores on the LTTC GEPT dataset and generalises moderately to unseen prompts. The evaluation is limited to a single language (English) and a single proprietary dataset.
Field Significance
Low — This paper addresses automated speaking assessment for language learning, with TTS serving as a data augmentation tool rather than the primary research target. Its relevance to the speech generation field is indirect: the voice cloning pipeline illustrates a practical application of speaker-conditioned synthesis to replicate non-native speech characteristics, but it introduces no new TTS architecture or training method. The marginal accuracy gains over a real-data-only baseline on unseen prompts suggest the augmentation strategy provides limited generalisation benefit beyond familiar training domains.
Claims
-
supports: Speaker-conditioned TTS synthesis, when conditioned on reference embeddings from real non-native learners, can produce augmentation data that preserves proficiency-level-appropriate disfluencies and prosodic variability.
Evidence: XTTSv2 conditioned on 512-dimensional speaker embeddings from real learner recordings produces speech retaining speaking rate, rhythm, and characteristic disfluencies; this contrasts with standard single-speaker TTS, which generates uniformly fluent output incompatible with non-native learner characteristics. (§2.2, Figure 1)
-
complicates: Synthetic speech from standard TTS pipelines cannot substitute for authentic learner recordings in isolation, even when speaker conditioning is applied to mimic non-native characteristics.
Evidence: Training exclusively on synthesised data achieves only 42.22% seen-test accuracy versus 73.33% for a real-data-only baseline, demonstrating that the acoustic domain gap between synthetic and authentic speech is not fully bridged by speaker-conditioned voice cloning alone. (§3.4, Table 2)
-
supports: Mixed real-plus-synthetic training with adaptive importance reweighting outperforms training on real data alone for low-resource speech understanding tasks.
Evidence: Adding synthesised augmentation with dynamic real-sample upweighting raises seen-test overall accuracy from 73.33% to 76.67% and pass-fail accuracy from 77.78% to 81.11% on the LTTC GEPT dataset, outperforming both single-modality and multimodal baselines trained on real data only. (§3.3, Table 1)
-
complicates: Gains from generative data augmentation in speech assessment diminish substantially on out-of-distribution prompt types, limiting the generalisation benefit of synthetic speech augmentation.
Evidence: Overall accuracy on unseen-prompt test data improves by only 0.67 percentage points (63.33% to 64.00%) compared to 3.34 points on seen prompts, indicating that augmentation-driven gains do not transfer robustly across different prompt domains. (§3.3, Table 1)
Limitations and Open Questions
Warning
The evaluation is restricted to a single proprietary dataset (LTTC GEPT Intermediate-Advanced) in English only, with 1,200 samples total. The extent to which the augmentation approach generalises across languages, accent groups, or assessment rubrics is not tested.
Accent and hesitation mismatches between synthesised and real non-native speech persist despite speaker conditioning, as acknowledged in the paper. The skewed mid-range score distribution in the dataset also impairs discrimination at extreme proficiency levels. The paper does not report any measure of the TTS output’s perceptual quality (e.g., MOS or speaker similarity scores), making it difficult to assess whether the voice cloning pipeline actually produces convincing non-native-sounding speech or whether the scoring model is robust to low-fidelity synthesis.
Wiki Connections
- Zero-Shot TTS — the voice cloning pipeline uses reference speaker embeddings extracted from real learner recordings to condition XTTSv2 without speaker-specific fine-tuning, a zero-shot conditioning approach.
- Speaker Adaptation — speaker embeddings from real non-native learners guide synthesis to replicate proficiency-level-appropriate prosody and disfluencies rather than using a fixed generic voice.
- Evaluation Metrics — the paper’s primary concern is automated scoring of spoken opinion expressions, using pass/fail classification accuracy as the evaluation criterion for an ASA system built on synthesised speech augmentation.