arXiv · 2025 · Preprint
Shuhei Kato · → Paper · Demo: ? · Code: ✓
LoRA fine-tuning on CosyVoice 2 with two special phoneme-tag tokens enables explicit Japanese segmental pronunciation and pitch accent control in a multilingual LLM-TTS, raising naturalness MOS from 3.44 to 3.88 and accent correctness from 0.498 to 0.899 while consuming less than 0.5% of the base model’s parameter count.
Problem
Multilingual LLM-based TTS systems such as CosyVoice 2 process raw text through byte-pair encoding (BPE) rather than an explicit grapheme-to-phoneme (G2P) front-end, which simplifies training but forces the model to learn pronunciation entirely from data. For Japanese this is especially problematic: kanji have multiple readings, more than 2,000 characters are in everyday use, and pitch accent is lexically contrastive and context-sensitive. The result is that CosyVoice 2 — acknowledged in its own paper — lags on Japanese relative to English and Chinese. Users who need to correct a mispronunciation or specify an accent pattern have no mechanism to do so short of full retraining.
Method
UtterTune layers LoRA onto CosyVoice 2’s text-speech language model (a Qwen2.5-0.5B autoregressive LM that generates discrete semantic codec tokens from BPE-tokenised text). All original weights are frozen; rank-16 LoRA matrices (alpha 64, dropout 0.05) are injected into every Q/K/V/O projection in the self-attention blocks. The trainable footprint is less than 0.5% of the base model.
Alongside the LoRA layers, two special tag tokens — <PHON_START> and <PHON_END> — are added to the vocabulary. When the model encounters these tags, the enclosed span is interpreted as a phonemic transcription in katakana with accent nuclei marked by apostrophes and phrase boundaries by slashes (following the standard Japanese speech synthesis notation). Outside the tags, the model behaves as the original CosyVoice 2 would. Only the embeddings of the two new tokens and the LoRA matrices are updated during fine-tuning; the original embedding matrix remains frozen.
Training uses 15,097 Japanese speech-transcription pairs drawn from JSUT and JVS. Each transcript has one noun replaced with its phonemic-plus-accent representation and wrapped in the tag tokens, providing explicit pronunciation supervision. Fine-tuning runs for 20,000 steps with AdamW on a single RTX 4090 in under one hour. Because the LoRA module is language-specific and optional, inference on other languages simply omits it, leaving cross-lingual performance unaffected in principle.
Key Results
On a 50-sentence general Japanese test set (Test Set 1) using 42 zero-shot prompt speakers, UtterTune raises crowd-sourced naturalness MOS from 3.44 to 3.88 (statistically significant, p < 0.0001, Wilcoxon test). UTMOS increases marginally from 3.25 to 3.28, and ERes2Net speaker similarity is unchanged at approximately 0.69 across all systems.
On the accent stress-test (Test Set 2), the baseline CosyVoice 2 achieves accent correctness of only 0.498 — near chance — even when kana transcription is supplied to resolve kanji ambiguity. Feeding the kana via the LoRA phoneme-tag mechanism raises accent correctness to 0.899. CER increases slightly relative to the kana-only baseline (0.0626 vs. 0.0595), but the effect size is negligible (Cliff’s delta = 0.115). A follow-up probe using 50 katakana-only sentences confirms that the phoneme-tag mechanism does not bleed pronunciation control into spans outside the tags: accent correctness is 0.829 for UtterTune vs. 0.796 for the baseline, with no significant difference.
Novelty Assessment
The contribution is engineering integration: LoRA applied to a pre-trained TTS model is an established pattern (applied previously for speaker adaptation and style control), and phoneme-level input with special tokens is a natural extension of existing tagging schemes in sequence models. The genuinely new element is the combination of both, applied specifically to pitch accent control in Japanese for a BPE-input multilingual LLM-TTS — a gap that does exist and that the paper addresses cleanly. The evaluation is narrow in scope (a single base model, one language, custom test sets with no public baseline) and the paper is short (5 pages), but the results are internally consistent and the experimental methodology is careful: native-speaker crowdworker evaluations, statistical significance testing, and a dedicated probe for tag-boundary leakage.
Field Significance
Moderate — UtterTune demonstrates that parameter-efficient fine-tuning can restore user-controllable pronunciation correction in multilingual LLM-TTS systems that omit explicit G2P front-ends, addressing a practical usability gap that scales with the number of typologically complex languages a system targets. The approach is unlikely to be the final answer — it requires language-specific phonemic transcription from users — but it points toward a broader pattern: as LLM-TTS covers more languages, targeted LoRA modules per language may become a lightweight corrective layer.
Claims
- Parameter-efficient fine-tuning with LoRA can restore explicit pronunciation controllability to LLM-TTS systems that omit G2P front-ends, without degrading cross-lingual performance. (§3.2, §5.1)
- BPE-based multilingual TTS models systematically underperform on phonologically complex languages where reading ambiguity is high and pitch accent is lexically contrastive. (§1, §5.2)
- Phoneme-tag injection via special tokens can provide localised pronunciation control without leaking into untagged text spans. (§5.2)
- Speaker similarity in zero-shot TTS is robust to targeted LoRA fine-tuning on language-specific pronunciation data. (§5.1, Table 1)
Limitations and Open Questions
Warning
Evaluation uses two custom test sets authored or curated by the single researcher-author, with no publicly available benchmark comparison. The listening test crowd cohort (16 workers for MOS, 15 for accent correctness) is small for a task with high inter-rater variance. Results should be treated as promising but not yet independently validated.
The approach requires users to supply phonemic transcription and accent markup, which demands linguistic expertise most end users lack. The paper targets only standard Tokyo Japanese; dialectal variation and code-switching remain untested, as the author acknowledges. The LoRA module is trained on a single female-heavy corpus (JSUT) and 100-speaker JVS, which may limit generalisation to speaker types not well represented in that data. Applicability to other prosodically complex languages — Mandarin tones, Scandinavian pitch accent — is left for future work.
Wiki Connections
- multilingual-tts — the core challenge addressed: Japanese pronunciation in a multilingual BPE-input LLM-TTS
- prosody-control — pitch accent control via phoneme tags is the primary novel mechanism
- speaker-adaptation — LoRA fine-tuning pattern shared with speaker adaptation literature
- autoregressive-codec-tts — builds on CosyVoice 2 as the base model
- VALL-E — cited as a foundational LLM-TTS system that the BPE-input paradigm descends from