arXiv · 2026 · Preprint

Phannet Pov et al. (Chungbuk National University / Cambodia Academy of Digital Technology / BigData Labs Co., Ltd.) · → Paper · Demo: ? · Code: ?

A single zero-initialised LoRA adapter, trained jointly on language-tagged Khmer and Korean data and attached to both the language model and the flow-matching decoder of VoxCPM2, significantly raises Khmer MOS while giving no significant gain for Korean, and the rank that minimises validation loss is not the rank listeners prefer.

Problem

Large pretrained TTS models approach natural quality for high-resource languages but lag for languages that are rare in pretraining. Khmer, which writes words without spaces, is the paper’s primary low-resource case; Korean, which the base model already handles well, is included as a contrast to separate genuine gap-filling from a gain any language would show. Full fine-tuning would require a separate multi-billion-parameter checkpoint per language and risks forgetting, so the paper asks how far parameter-efficient adaptation can close the gap and whether one shared adapter can serve two very different scripts.

Method

The base model is VoxCPM2 (about 2.39B parameters), a tokenizer-free TTS system that combines a MiniCPM4 language-model backbone (text-semantic stage, FSQ stage, and residual-acoustic stage; hidden size 2048, 28 plus 8 residual layers) with a local diffusion transformer (LocDiT) decoder trained with a flow-matching objective, followed by the AudioVAE V2 decoder that renders 48 kHz audio from continuous latents. All base weights stay frozen.

A single LoRA adapter is added to the query, key, value, and output attention projections of both the language model (base and residual layers) and the DiT decoder; feed-forward layers and the audio VAE are untouched. The B matrix is zero-initialised, so training starts exactly at the unadapted model. With alpha set to 2r, ranks 8, 16, 32, 64, and 128 give 4.5M to 72.4M trainable parameters (0.19% to 3.03% of the base; adapter files of 18 MB to 277 MB).

The training corpus pools a private Khmer corpus from IDRI Cambodia (4,000 utterances, 3.96 h) with Korean KSS and Common Voice/FLEURS data (15,769 utterances, 22.21 h). Each transcript is prefixed with a language tag ([km] or [ko]), clips outside 0.5 to 20 s or above 256 text tokens are dropped, and Khmer training clips are upsampled to 40% of the training mix (23,487 training clips; the validation split keeps the natural ratio). One adapter is trained on both languages together, with the language tag signalling which script is being read. Each rank trains for 10,000 steps with AdamW (peak learning rate 1e-4, warmup plus cosine decay, effective batch 16, bf16) on one H200 GPU, about 2.6 hours per rank.

Evaluation uses the validation flow-matching loss and, for each language, a native-speaker MOS test: five raters score 20 sentences per system on naturalness, prosody, and pronunciation (5-point scale), with overall MOS as the mean of the three axes. Systems are compared with the unadapted base (which the paper calls the “zero-shot base”, meaning no adaptation, not voice cloning) using a paired Wilcoxon signed-rank test, with bootstrap confidence intervals and Krippendorff’s alpha. The two languages used different rater panels, so comparisons are within-language only.

Key Results

Validation loss. Loss falls from about 0.83 for the unadapted model to between 0.71 and 0.73 for every rank. It is not monotonic in rank: rank 128 is lowest (0.7094), rank 8 second (0.7243), then rank 64 (0.7266), rank 16 (0.7334), and rank 32 (0.7344) (§IV-A, Table II).

Khmer. Overall MOS rises from 3.85 (base) to 3.91 (r=8), 3.94 (r=16), 4.03 (r=32), 4.23 (r=64), and 4.08 (r=128). Rank 64 has the highest mean on every axis (naturalness 4.25, prosody 4.36, pronunciation 4.07) and overall. Gains are significant at ranks 32 (p=.001), 64 (p<.001), and 128 (p=.001), but not at ranks 8 (p=.19) or 16 (p=.07). The largest single-axis change is prosody, from 3.76 to 4.36 at rank 64 (§IV-A, Table IV).

Korean. No rank significantly improves overall MOS. The base scores 3.65; ranks 8, 16, and 32 score 3.73, 3.71, and 3.76 (rank 32 the highest mean, p=.49), while ranks 64 and 128 score 3.48 and 3.54. Rank 64 is significantly worse than the base (p=.02), driven by naturalness falling from 3.67 to 3.23. Pronunciation is the only axis that rises for every rank (from 3.64 to between 3.72 and 4.03, highest at rank 32), whereas naturalness is highest for the base (3.67) and prosody peaks at 3.64, tied between the base and rank 8 (§IV-A, Table III).

Loss versus listeners. The rank with the lowest validation loss (128) is not the one listeners prefer for Khmer (64), and for Korean the highest-capacity settings are the worst rated. The authors read this as evidence that loss overstates the value of extra capacity and that rank should be chosen by listening tests (§IV-C).

Comparisons are only against the unadapted base model: there is no full fine-tuning upper bound, no separate per-language adapter, and no objective intelligibility or speaker-similarity metric. Inter-rater agreement is low (Krippendorff’s alpha 0.31 for Khmer and 0.26 for Korean) with five raters and 20 sentences per system (§IV-D).

Novelty Assessment

The contribution is engineering integration and an empirical rank sweep: LoRA is applied off the shelf to an existing foundation TTS model, with the specific choices of a single jointly trained, language-tagged adapter and placement in both the LM and the flow-matching decoder. The authors describe it as the first parameter-efficient adaptation of a foundation TTS model for Khmer. The more interesting observations are that validation loss and perceived quality pick different ranks, and that the same adapter helps the weakly covered language while doing nothing or harm for the well covered one. Both rest on small, low-agreement listening tests, and the Khmer/Korean contrast is confounded by different fine-tuning data sources (private Khmer corpus vs. public Korean corpora). The paper does not test whether sharing one adapter across languages beats per-language adapters, so the “shared adapter” design is not isolated as a benefit.

Field Significance

Low. The paper adds a confirmatory data point that low-rank adaptation of a large pretrained TTS model can improve a low-resource language with a few hours of data and a small parameter budget, plus a cautionary observation that training loss is a poor guide to adapter capacity and that adaptation can degrade a language the base model already covers well. Its evidence is limited to two languages, one base model, and small listening panels.

Claims

  • supports: Parameter-efficient low-rank adaptation of a large pretrained TTS model can significantly improve perceived quality for a language that is poorly covered in pretraining, using only a few hours of target-language data.

    Evidence: A single LoRA adapter on VoxCPM2, trained on about 4 h of Khmer (upsampled) pooled with Korean, raises Khmer overall MOS from 3.85 to 4.23 at rank 64 (p<.001), with prosody showing the largest gain (3.76 to 4.36), while training 1.51% of parameters. (§IV-A, Table IV)

  • complicates: Adapting a pretrained TTS model to a language it already covers well yields little benefit and can degrade quality at higher adapter capacity.

    Evidence: The same jointly trained adapter gives no significant Korean overall MOS gain at any rank (best 3.76 vs. 3.65 base, p=.49), and rank 64 is significantly worse (3.48, p=.02), with naturalness falling from 3.67 to 3.23. (§IV-A, Table III)

  • complicates: Validation loss on the generative objective is an unreliable criterion for choosing adaptation capacity in TTS, because the loss-optimal setting need not match listener preferences.

    Evidence: Validation flow-matching loss is lowest at rank 128 (0.7094), but Khmer MOS peaks at rank 64 (4.23 vs. 4.08 at rank 128) and Korean MOS is lowest at the higher ranks; loss is also non-monotonic in rank. (§IV-A, §IV-C, Tables II to IV)

  • complicates: Cross-language comparisons of adaptation gains are confounded when languages differ in both pretraining coverage and fine-tuning data source.

    Evidence: Khmer uses a private in-house corpus while Korean uses KSS and Common Voice/FLEURS, the two MOS panels differ, and inter-rater agreement is low (alpha 0.31 and 0.26), so the authors restrict conclusions to within-language trends. (§Limitations)

Limitations and Open Questions

Warning

The listening tests are small and noisy (five raters, 20 sentences per system, Krippendorff’s alpha 0.31 for Khmer and 0.26 for Korean), use different panels per language, and the Khmer and Korean fine-tuning data come from different kinds of source, so the central “adaptation helps only where the base is weak” contrast is not a controlled cross-language comparison.

  • No full fine-tuning upper bound and no per-language adapter at matched parameter budget, so the benefit of sharing one adapter across languages is untested (§Limitations).
  • No objective metrics (WER/CER, speaker similarity, UTMOS) are reported; the only automatic measure is the validation flow-matching loss.
  • The corpus is small (about 26 h), Korean is largely single-speaker KSS, and part of the Khmer gain comes from upsampling rather than more unique data.
  • The source of the 20 evaluation sentences, and whether they overlap the training domain, is not reported; no code or sample URL is given despite a reference to released artifacts.
  • Whether the rank-quality relationship generalises to other low-resource languages or base models is left open; the authors propose per-language rank selection as future work.

Wiki Connections

  • Fine-Tuning — The paper’s method is LoRA fine-tuning of a frozen foundation TTS model, with a rank sweep from 8 to 128 evaluated by native-speaker MOS.
  • Multilingual TTS — One adapter is trained jointly on language-tagged Khmer and Korean data and evaluated separately on each language.
  • Flow Matching — The LoRA adapter is inserted into VoxCPM2’s frozen flow-matching DiT decoder, and the validation flow-matching loss is the main automatic metric.
  • Subjective Evaluation — Native-speaker MOS on three axes with Wilcoxon tests, bootstrap intervals, and Krippendorff’s alpha drives all conclusions, and disagrees with the loss-based ranking.
  • VoxCPM2 — The frozen base model adapted here; its LM backbone and DiT decoder both receive LoRA updates.
  • VoxCPM — Predecessor tokenizer-free TTS model whose continuous-representation design VoxCPM2 inherits.
  • Flow Matching — The generative objective of the adapted decoder and of the validation loss used for rank comparison.
  • Common Voice — Source of part of the Korean fine-tuning data.
  • XTTS — Cited as a cross-lingual zero-shot system that transfers to new languages from limited data, motivating targeted adaptation.
  • VALL-E — Cited as the discrete codec-token approach that VoxCPM2’s continuous-feature design departs from.