arXiv · 2026 · Preprint
Lianbo Liu et al. (SB Intuitions) · → Paper · Demo: ? · Code: ✓
Builds an open-source Japanese-centric LLM-based TTS system whose kanji reading accuracy comes from data rather than architecture (a 361k-hour, roughly half-Japanese corpus plus synthetic sentences covering every regular-use kanji reading), and introduces a kanji-level pronunciation benchmark and a kana-space CER metric to measure it.
Problem
Japanese text mixes kanji with kana, and most kanji have several context-dependent readings: the 2,136 Joyo (regular-use) kanji carry 4,378 recognized readings, some more than ten. Picking the right reading (kanji polyphony disambiguation) is the authors’ central challenge for Japanese TTS. Multilingual LLM-based TTS systems that support Japanese give it only a small share of their training data and have no data strategy aimed at rare readings, which the authors argue hurts both basic pronunciation and robustness to non-Japanese voice prompts (§1).
Evaluation is the second gap. Sentence-level CER/WER benchmarks cannot attribute an error to a particular kanji or reading. Standard ASR-based CER is also distorted by Japanese orthographic variation (one pronunciation can be written several ways, e.g. 行う / おこなう / 行なう), so the same correct pronunciation can score as an error. The authors point to this as the reason Japanese looks like an outlier in multilingual TTS evaluations (§1, §4.1).
Method
The system follows the two-stage semantic/acoustic split of CosyVoice-style LLM-TTS (§2, Figure 1):
- Speech tokenizer. The S3Tokenizer V2 from CosyVoice 2, reused as is: an FSQ module inside an ASR encoder trained with an ASR objective, producing single-codebook semantic tokens at 25 Hz. The authors argue that ASR supervision makes the token space sensitive to fine reading differences.
- Backbone LLM. A 24-layer decoder-only Transformer initialized from Sarashina2.2-0.5B-Instruct, a text LLM pretrained mostly on Japanese. Its vocabulary is extended with 6,561 semantic tokens plus BOS, EOS and a speech-start token. Inputs are arranged for in-context zero-shot cloning: BOS, prompt transcript, target text, speech-start, prompt speech tokens, then the predicted target speech tokens and EOS. Training uses teacher-forced next-token cross-entropy over the semantic-token positions only, on naturally contiguous multi-sentence speech (§2).
- Acoustic stage (inherited, not trained here). The open-source release takes the conditional flow-matching decoder (convolutional Transformer UNet, conditioned on semantic tokens, a reference mel-spectrogram and a speaker embedding) and the HiFi-GAN vocoder directly from CosyVoice 2. The backbone LLM is the only component trained in this work, and the authors state that the acoustic components do not affect kanji reading accuracy (§2, §5.1).
Training data. About 361k hours of licensed or public-domain speech-text pairs: 194,051 h of Japanese (53.7%) across podcast, audiobook/narration, customer service, TV/broadcast, public speech, conversation and language-learning domains, plus 167,153 h of English (46.3%) (§3.1.1, Table 1). The roughly even language split is intentional. It is meant to support bilingual synthesis, code-switched English words inside Japanese text, and cross-lingual prompting (an English reference voice speaking Japanese). Preprocessing is a standard pipeline (source separation, diarization, VAD segmentation, enhancement, language ID, DNSMOS filtering) with dual-ASR transcript verification: Whisper large-v3-turbo transcribes, OWSM-CTC-v4 cross-checks, and samples where the two disagree strongly are dropped (§3.1.2).
PronSteering and targeted synthesis. An internal variant of the backbone (Pronunciation Steering) adds two delimiter tokens. These let a target kanji in the input text be replaced by its kana reading plus pitch-accent rise/fall tags. The variant is obtained by fine-tuning the Stage 1 model on about 4,000 h of broadcast/audiobook data whose noun spans were auto-annotated with dictionary-predicted readings and accents (§3.2.1). PronSteering is not part of the open-source release. It is used to generate data in three steps (§3.2.2):
- An LLM writes sentences in which only the target reading is valid, with UniDic morphological verification.
- Dictionary tools annotate the target morpheme with reading and pitch accent (97.5% get prosody tags; the rest get reading control only), and the sentences are synthesized with diverse speaker prompts.
- Kana-ASR filters the outputs at both the kanji-segment and sentence level.
Run over all 2,136 Joyo kanji and 4,378 readings, this gives about 280k samples (about 320 h), with a 95.1% retention rate after filtering.
Two-stage training (§5.1):
- Stage 1: full-parameter training on the 361k-hour corpus at a constant learning rate of 1e-4.
- Stage 2: continued training with the learning rate decayed linearly from 1e-4 to 1e-6, on a re-filtered, higher-quality subset of the Stage 1 data mixed with the synthetic kanji data.
Evaluation framework (§4):
- Kana-CER. A Whisper large-v3-turbo model fine-tuned to output kana (Kana-ASR, trained on the Corpus of Spontaneous Japanese, which is not in the TTS training data) transcribes the synthesized speech. CER is then computed against kana reference readings, which removes orthographic variation. On real JSUT recordings, Kana-ASR’s own Kana-CER is 0.979% (§4.1.1). It comes in two forms: Kana-CER sent over the whole sentence, and Kana-CER kanji on the target-kanji substring, which is extracted by Levenshtein alignment. A clipped variant (Kana-CER† kanji) caps each sample’s CER at 1.0, because hallucinations on 1-2 character segments can push per-sample CER to several hundred percent (§4.2).
- Joyo Kanji Yomi Benchmark. 13,095 LLM-generated sentences, three per kanji-reading pair, in which context uniquely determines the target reading. The target reading is marked in a full-sentence kana annotation. All sentences were reviewed by 35 native speakers in three stages (correctness, disambiguation, kana correction). Kanji-reading pairs that cannot be disambiguated from context alone are excluded (§4.2).
Key Results
All systems run with 5 random seeds (mean ± std). Baselines are four LLM-TTS systems that support Japanese: T5Gemma-TTS, Qwen3-TTS, FishAudio S1-mini and FireRedTTS-2 (§5.1).
Kanji reading (Table 2).
| System | Joyo Kana-CER kanji | Joyo Kana-CER† kanji | Joyo Kana-CER sent | Joyo CER | JSUT Kana-CER sent | JSUT CER |
|---|---|---|---|---|---|---|
| T5Gemma-TTS | 13.81 | 8.55 | 3.69 | 5.68 | 2.80 | 7.63 |
| Qwen3-TTS | 185.89 | 21.70 | 23.20 | 13.26 | 15.58 | 14.87 |
| FishAudio S1-mini | 33.43 | 20.19 | 5.46 | 6.15 | 5.16 | 9.03 |
| FireRedTTS-2 | 27.82 | 16.39 | 4.28 | 5.32 | 5.26 | 9.33 |
| Sarashina2.2-TTS Stage 1 | 11.06 | 6.94 | 4.59 | 6.36 | 3.04 | 8.08 |
| Sarashina2.2-TTS Stage 2 | 7.83 | 5.45 | 3.41 | 5.28 | 2.91 | 8.02 |
On all four Joyo benchmark metrics, Stage 2 has the lowest error of any system. The margin over the best baseline is large at the kanji level (Kana-CER† kanji 5.45 vs 8.55 for T5Gemma-TTS; the authors’ “57% higher” is 8.55/5.45) and small on sentence-level standard CER (5.28 vs 5.32 for FireRedTTS-2). On JSUT, T5Gemma-TTS is lower than Stage 2 on both Kana-CER sent (2.80 vs 2.91) and CER (7.63 vs 8.02). The authors describe this as matching the best baseline, and the proposed system is not best there. Stage 2 improves on Stage 1 in every Table 2 column (§5.2.1).
Per-reading analysis. In the text’s summary of the cumulative per-reading curve, Stage 2 pronounces 81.5% of the 4,378 readings correctly in all 15 trials, against 79.8% for Stage 1, 79.7% for T5Gemma-TTS, 66.3% for FireRedTTS-2, 63.6% for FishAudio S1-mini and 58.3% for Qwen3-TTS (§5.2.2, Figure 2). Stage 1 errors often fall back to a more frequent reading of the same kanji (e.g. 坂 always read as サカ rather than the rare ハン). For the rare reading 従-ショウ, Stage 2 drops to 2/15 errors while Stage 1 and every baseline err 15/15. For the rare reading 生-オウ, every system, Stage 2 included, errs 14-15/15 (§5.2.2, Table 4).
Kana-CER vs standard CER. Standard CER exceeds Kana-CER sent for every system on both benchmarks except Qwen3-TTS, whose hallucinated output inflates Kana-CER more than CER (Kana-ASR has no language-model compensation). For Stage 2 on JSUT the gap is 5.11 points (8.02 vs 2.91) (§5.2.1, Table 2).
Cross-style robustness (Table 5). These are JSUT CERs (Whisper) averaged over 9 Japanese style prompts (narration, news, podcast, rakugo, horse-race commentary, etc.). FishAudio S1-mini is best on both mean CER (9.44) and cross-style STD (0.71). Stage 2 is second (9.55, 1.32), ahead of T5Gemma-TTS (10.88, 2.69), FireRedTTS-2 (11.05, 2.06) and Qwen3-TTS (63.11, 125). The authors hypothesize that the gap to FishAudio partly reflects Whisper struggling with expressive styles that their system reproduces faithfully, but they do not test this (§5.3.1).
Cross-lingual prompts (Table 6). Here the comparison is Japanese output with English-accented (American, British, Indian) prompts against Japanese prompts. Degradation is defined as (non-Japanese − Japanese) / Japanese, so positive values mean worse. Qwen3-TTS degrades by +328.1%, FireRedTTS-2 by +168.3%, FishAudio S1-mini by +113.4% and T5Gemma-TTS by +18.4%. Stage 1 changes by −1.1% and Stage 2 by −0.2% (9.55 → 9.53), so they are the only systems that do not degrade (§5.3.2).
Speaker similarity (Table 7). CAM++ SIM on the Japanese subset of CV3-Eval: Stage 1 75.64, Stage 2 74.75, then Qwen3-TTS 69.86, FireRedTTS-2 66.20, FishAudio S1-mini 61.38 and T5Gemma-TTS 50.59. Both stages are above every baseline, and Stage 2 is slightly below Stage 1 (§5.4).
Automatic quality (Table 8, CV3-Ja). Stage 2 has the highest DNSMOS (3.824). On UTMOS v2, Stage 1 is highest (2.888) and Stage 2 second (2.877). Qwen3-TTS is highest on UTMOS Strong (3.406 vs 3.174 for Stage 2) and DNSMOS P.835 OVRL (3.263; Stage 2 3.242, FishAudio 3.244). Stages 1 and 2 are close on every quality metric, and all systems score above the prompt recordings (§5.5). There are no human listening tests.
Novelty Assessment
There is little architectural novelty. The tokenizer, flow-matching decoder and vocoder are taken unchanged from CosyVoice 2, and the backbone is an existing Japanese text LLM with an extended vocabulary, trained with the standard in-context continuation objective. The contribution is in data and evaluation:
- a large, roughly balanced Japanese/English corpus;
- a reading-controlled synthesis loop (PronSteering) that covers every Joyo kanji reading, including rare readings missing from natural speech;
- a kanji-level benchmark with native-speaker-verified kana annotations;
- a kana-space CER that removes orthographic noise from Japanese pronunciation scoring.
PronSteering’s delimiter-token reading and pitch-accent override is a small mechanism, and in this paper it serves only as a data generator. The benchmark, Kana-ASR model and evaluation scripts are released alongside the model weights, so the evaluation side is likely the most reusable part.
Field Significance
Low. The paper gives a concrete account of how a speech-token LLM-TTS system can be specialized for a language with heavy logographic polyphony through data scale, language balance and targeted synthetic coverage, without changing the architecture. It also supplies a kanji-level benchmark and a kana-space metric that make Japanese pronunciation errors attributable and comparable across systems. The evidence is entirely automatic (ASR-based CER, embedding SIM, MOS predictors) and limited to Japanese output, and no human listening test checks the gains, so it sits at low on the precedent for single-language, automatic-only evaluation (PSP, PashtoTTS-Bench); its value is in the measurement tools and the data recipe rather than in perceptual quality claims.
Claims
- supports: For a speech-token LLM-TTS system, reading-controlled synthetic data that covers rare grapheme-to-pronunciation mappings missing from natural speech can reduce polyphone reading errors without degrading speaker similarity or predicted quality much.
Evidence: Stage 2 (Stage 1 continued on a re-filtered natural subset plus about 320 h of PronSteering-synthesized sentences covering all 4,378 Joyo readings) cuts Joyo Kana-CER kanji from 11.06 to 7.83 and Kana-CER† kanji from 6.94 to 5.45, with CV3-Ja SIM going from 75.64 to 74.75 and UTMOS v2 from 2.888 to 2.877. (§5.2.1, Table 2; §5.4, Table 7; §5.5, Table 8)
- complicates: Orthographic CER from a general ASR model overstates pronunciation error in languages with multiple valid spellings per pronunciation, while phonological-transcription CER is more sensitive to hallucinated or highly expressive output.
Evidence: Standard CER exceeds kana-space CER for every system on both benchmarks except Qwen3-TTS (e.g. Stage 2 on JSUT: 8.02 vs 2.91). Qwen3-TTS’s hallucinations inflate Kana-CER above CER, and the authors fall back to Whisper CER for the stylistically diverse prompt evaluation because Kana-ASR becomes unstable there. (§5.2.1, Table 2; §4.1.1; §5.3)
- supports: In zero-shot LLM-TTS, a target language that makes up only a small share of the training data can let the prompt’s language bias target-language pronunciation. A roughly balanced bilingual training mix can remove this sensitivity.
Evidence: Japanese-output CER on JSUT degrades by +328.1% (Qwen3-TTS), +168.3% (FireRedTTS-2), +113.4% (FishAudio S1-mini) and +18.4% (T5Gemma-TTS, about 20k of 170k h Japanese) with English-accented prompts. It changes by −0.2% for Stage 2, trained on 53.7% Japanese / 46.3% English. The causal link to language balance is the authors’ hypothesis and is not ablated. (§5.3.2, Table 6; §3.1.1, Table 1)
- complicates: Large-scale multi-domain training alone gets most of the way on context-dependent reading disambiguation, and some rare readings stay unlearned even after targeted synthetic coverage.
Evidence: Stage 1, trained without the synthetic kanji data, already pronounces 79.8% of readings correctly in all 15 trials, against 81.5% for Stage 2. The rare reading 生-オウ is still mispronounced in 14/15 trials by Stage 2, as by every baseline. (§5.2.2, Figure 2; Table 4)
- complicates: Specializing an LLM-TTS system for pronunciation accuracy in one language does not guarantee the best robustness across expressive prompt styles.
Evidence: Across 9 Japanese style prompts, FishAudio S1-mini has lower mean CER (9.44 vs 9.55) and lower cross-style STD (0.71 vs 1.32) than Stage 2. The authors attribute part of this to Whisper transcription difficulty on faithfully cloned expressive styles, which they do not verify. (§5.3.1, Table 5)
Limitations and Open Questions
All evaluation is automatic and the Stage 1 → Stage 2 comparison is confounded
There are no human listening tests: naturalness relies on UTMOS/DNSMOS predictors and pronunciation on ASR transcription. Stage 2 differs from Stage 1 in three ways at once: the added synthetic kanji data, a re-filtered higher-quality subset of the natural data, and a decayed learning rate. Its gains therefore cannot be attributed to the synthetic data alone, and no ablation isolates it (§5.1).
- Benchmark/augmentation overlap. The synthetic training data and the benchmark were built the same way: both use LLM-generated sentences that target exactly the same 2,136 kanji and 4,378 readings. The Stage 2 benchmark gains are therefore partly in-distribution. The authors’ safeguard is to cover all readings rather than only error-prone ones (§3.2.2, §5.2.2), but sentence-level separation between the synthetic set and the test set is not reported. The JSUT gains from Stage 2 are smaller (Kana-CER sent 3.04 → 2.91).
- Monolingual output. Training covers Japanese and English, but every evaluation synthesizes Japanese. English synthesis and code-switched input, both cited as reasons for including English, are not evaluated.
- Metric depends on an in-house ASR. Kana-CER depends on the authors’ own Kana-ASR model. Its 0.979% error on real JSUT speech is low, but the authors acknowledge that it mis-transcribes reduced or expressive pronunciation. To keep it usable, all systems are evaluated with a fixed reading-style prompt, which may favour systems tuned for read speech (§4.1.1).
- PronSteering not released or evaluated. PronSteering is absent from the open-source release, and its reading and pitch-accent control is never evaluated directly. Its effect is visible only through the synthetic data.
- Inherited acoustic stage. The decoder and vocoder come from CosyVoice 2 and may differ in internal deployments. All reported results use the open-source configuration (§5.1).
- Narrow baseline set. Only four baselines, all LLM-TTS. Non-LLM Japanese-specialized TTS or systems with explicit G2P front-ends are not compared.
Wiki Connections
- Autoregressive Codec TTS — The backbone is a decoder-only LLM that autoregressively predicts single-codebook S3Tokenizer semantic tokens. The paper’s contribution is how this backbone is trained and evaluated for Japanese.
- Flow Matching — The acoustic stage is a conditional flow-matching decoder inherited unchanged (not retrained) from CosyVoice 2, which renders all evaluated waveforms.
- Zero-Shot TTS — Evaluated on prompt-based cloning on the CV3-Eval Japanese subset (highest SIM among the systems compared) and on robustness to cross-style and English-language prompts.
- Evaluation Metrics — Defines Kana-CER (sentence- and kanji-level, plus a clipped variant), validates its ASR on real JSUT recordings, and introduces the kanji-level Joyo Kanji Yomi Benchmark.
- CosyVoice 2 — Source of the S3Tokenizer V2, flow-matching decoder and HiFi-GAN vocoder, all reused as is in the open-source release.
- CosyVoice 3 — Provides CV3-Eval, whose Japanese subset is used for the zero-shot speaker-similarity and automatic-quality evaluation.
- T5Gemma-TTS — Japanese-capable baseline. It is best on JSUT sentence-level CER but trails on kanji-level accuracy, speaker similarity and cross-lingual prompt robustness.
- Qwen3-TTS — Multilingual baseline with the highest UTMOS Strong and DNSMOS P.835 but frequent Japanese hallucinations and the largest cross-lingual prompt degradation.
- FireRedTTS-2 — Multilingual baseline, close on Joyo sentence-level CER but far behind on kanji-level metrics and degrading heavily under English prompts.
- Whisper — Whisper large-v3-turbo transcribes the training data, computes the standard CER, and is the base model fine-tuned into the Kana-ASR.
- UTMOS — UTMOS Strong is one of the automatic MOS predictors used to check that pronunciation-focused training does not reduce predicted quality.
- HiFi-GAN — The inherited vocoder that converts the decoder’s mel-spectrograms to waveforms.