arXiv · 2025 · Preprint
Gao et al. (A*STAR Institute for Infocomm Research) · → Paper · Demo: ✓ · Code: ?
A fine-tuned CosyVoice-based TTS system for child-friendly speech generation in three low-resource Southeast Asian languages — Singaporean-accented Mandarin, Malay, and Tamil — using x-vector speaker conditioning to capture age-appropriate vocal characteristics.
Problem
Multilingual TTS for low-resource Southeast Asian languages — Malay, Tamil, and Singaporean-accented Mandarin — remains underserved, especially in child-centric contexts. Prior work applying LLM-based architectures to such languages is absent; existing educational speech systems for Singapore (such as the authors’ SingaKids platform) relied on conventional architectures like VITS or FastSpeech that lack the acoustic fidelity and prosodic naturalness needed to model children’s speech. Children’s voices present specific challenges — higher pitch, smaller vocal tracts, evolving pronunciation — that standard adult-focused TTS pipelines do not address, and annotated children’s speech data for these languages is scarce.
Method
MultiGen fine-tunes CosyVoice-300M, an existing multilingual zero-shot TTS model, for five epochs on domain-specific data covering three target languages. The architecture inherits CosyVoice’s three-component pipeline: a transformer-based autoregressive LM that generates speech tokens conditioned on text and language identifiers (zh, ma, ta for Mandarin, Malay, Tamil), a pretrained flow-matching model that maps speech tokens to continuous acoustic features, and a pretrained HiFi-GAN vocoder for waveform synthesis. The model size is 300M parameters (matching the CosyVoice-300M base).
Child-friendliness is introduced through speaker conditioning: x-vector embeddings extracted from children’s speech in each of the three languages are incorporated as speaker representations, steering the model toward age-specific vocal characteristics including higher pitch and distinctive prosodic patterns. The training objective minimises KL divergence between predicted and ground-truth speech token distributions, guided by language identity. Dynamic batching is applied, with the best checkpoint selected per language on validation performance. No architectural modifications are made to the base CosyVoice model — the contribution is the fine-tuning regime, the child-speaker conditioning, and the curated multilingual data.
Key Results
MultiGen substantially outperforms baselines across all three languages on subjective MOS and on objective CER. For Tamil, speech quality MOS improves from 2.51 (VITS baseline) to 4.05, and naturalness from 2.19 to 3.72. For Malay, quality rises from 2.62 to 3.61 and naturalness from 2.46 to 3.63. Gains over CosyBase (CosyVoice-300M without fine-tuning) for Singaporean-accented Mandarin are smaller: quality improves from 3.05 to 3.97 and naturalness from 3.85 to 4.03.
On preference tests, 91.33% of listeners preferred MultiGen over VITS for Tamil, 83.33% for Malay, and 77.33% over CosyBase for Mandarin. CER improvements are large for Tamil (9.60% → 2.10%) and Mandarin (10.70% → 4.00%), and modest for Malay (3.40% → 2.30%, where the VITS baseline was already competitive on intelligibility).
Note
Baselines differ by language: VITS is compared for Malay and Tamil while CosyBase (the same underlying model as MultiGen before fine-tuning) is the Mandarin baseline. MOS scores are not cross-language comparable, and test sets involve only 30 samples per language rated by 10 listeners each — the small scale limits statistical confidence beyond the reported 95% CIs.
Novelty Assessment
The contribution is an engineering integration. CosyVoice-300M’s LLM + flow-matching + HiFi-GAN architecture is adopted wholesale; the paper does not modify the model structure. The novelty lies in: (1) applying this architecture to three previously unaddressed low-resource Southeast Asian languages; (2) the age-appropriate fine-tuning strategy using child-speaker x-vectors; and (3) demonstrating that large multilingual foundation models can be adapted to child-speech domains with limited data (as few as 1,400 utterances for Singaporean-accented Mandarin). The data curation and collection for the Malay children’s corpus (30,000 utterances from 104 speakers) is a meaningful practical contribution, though the corpus is not released. Compared to VITS, the gains are very large and expected given the capability gap between architectures; the more informative comparison is against CosyBase, where improvements are real but moderate.
Field Significance
Moderate — This paper extends a capable multilingual foundation model into a domain — child-friendly speech for low-resource Southeast Asian languages — that has received little systematic attention. The gains over VITS are unsurprising given the architecture gap, but the adaptation approach with child-speaker conditioning and the demonstrated low-data effectiveness (particularly the Mandarin case with 1,400 utterances) provide useful evidence for practitioners targeting similar low-resource, domain-specific TTS scenarios. The work is narrow in scope and the data is proprietary, limiting reproducibility.
Claims
- LLM-based TTS architectures, when fine-tuned with appropriate speaker conditioning, substantially outperform conventional non-autoregressive models on low-resource language naturalness as measured by subjective MOS. (§4.2)
- Child-specific vocal characteristics can be transferred to multilingual TTS through x-vector speaker conditioning without architectural modifications to the base model. (§2.2, §3.2)
- Foundation TTS models originally trained on high-resource languages can be adapted to low-resource language targets with limited domain data while retaining high intelligibility. (§3.1, §4.3, Table 1)
- Intelligibility gains from modern TTS architectures in low-resource settings are more consistently captured by ASR-based CER than by subjective listener ratings, particularly when native annotators are limited. (§4.3, Table 1)
Limitations and Open Questions
Warning
Evaluation uses only 30 speech samples per language rated by 10 listeners each. The small scale means reported MOS differences, while large, should be treated cautiously — and no significance testing distinguishes genuine quality gaps from listener variability. All data is proprietary and not released, preventing independent replication.
The Singaporean-accented Mandarin training set contains only 1,400 utterances from a single child speaker, raising questions about voice diversity and generalisation to other child speakers. The paper does not evaluate speaker similarity to the target child voices, making it difficult to assess how faithfully the age-appropriate characteristics are reproduced. Malay and Tamil models are not evaluated against CosyBase, so it is unclear whether fine-tuning adds value beyond what a capable pretrained model already provides for those languages. All evaluation is limited to Singapore’s specific linguistic and cultural context; the approach’s generalisability to other low-resource child-speech settings is undemonstrated.
Wiki Connections
Concept pages most informed by this paper: multilingual-tts (low-resource Southeast Asian language adaptation), autoregressive-codec-tts (CosyVoice fine-tuning), flow-matching (inherited inference component), speaker-adaptation (child x-vector conditioning), subjective-evaluation (small-scale MOS methodology).