arXiv · 2026 · Preprint
Jullajak Karnjanaekarin et al. (Jasmine Technology Solution / Chulalongkorn University / SIIT) · → Paper · Demo: ✓ · Code: ✓
Continues training the tokenizer-free VoxCPM architecture on about 10,000 hours of Thai-centric speech to produce a Thai zero-shot voice cloning model that reads raw text with Arabic numerals and Thai-English code-switching without a text normalization front end, and evaluates it on new short- and long-duration Thai benchmarks.
Problem
Zero-shot voice cloning systems are largely optimized for English. Open multilingual models such as Qwen3-TTS cover Thai but, according to the authors, still show occasional pronunciation and prosodic errors, plausibly because Thai is a small share of their training data. The Thai-specific ThonburianTTS (an F5-TTS derivative) has limited zero-shot speaker cloning and struggles on long utterances because it was trained on the short-utterance-dominated GigaSpeech2. Conventional Thai pipelines also depend on complex text normalization for numbers and for the frequent Thai-English code-switching of everyday text. The authors further argue that codec-token AR systems built on X-codec2 (e.g. Llasa, TTS-1) rely on a tokenizer whose reported training data does not appear to include Thai, leaving its handling of Thai tones and consonant clusters unverified (§1).
Method
JaiTTS-v1.0 reuses the VoxCPM architecture unchanged and adapts it by continual training (author term; mapped here to fine-tuning) on Thai data (§2). No architectural modification is described. Generation is autoregressive over patches of continuous latents produced by a separately trained causal audio VAE, so no discrete speech codec is involved. For each patch, a Local Audio Encoder (LocEnc) compresses the history of VAE latent patches into acoustic embeddings. A Text-Semantic Language Model (TSLM), a decoder-only Transformer initialized from MiniCPM-4, consumes BPE text tokens plus this acoustic context to produce a semantic-prosodic hidden state. A Finite Scalar Quantization (FSQ) layer quantizes that state into a “semi-discrete skeleton”, trained through the straight-through estimator as a regularizer rather than as a token vocabulary. A Residual Acoustic Language Model (RALM) adds a continuous residual carrying speaker and acoustic detail, and a bidirectional Local Diffusion Transformer (LocDiT) decodes the next latent patch with conditional flow matching, conditioned on the sum of the FSQ skeleton and RALM residual plus the previous patch (framed as outpainting for cross-patch continuity). A lightweight stop head reads the FSQ state (§2).
The training loss is the flow-matching loss plus a weighted binary cross-entropy stop loss, optimized jointly through LocEnc, TSLM, FSQ, RALM, and LocDiT; language-model conditioning to LocDiT is dropped with probability 0.1 to enable classifier-free guidance (§2). The paper lists only these modules as jointly optimized; the audio VAE is described as separately trained and is not described as being updated during Thai training, so it is treated here as a frozen inherited component. Model size is not reported. Inference uses CFG 2.5 and 10 timesteps (§3.3).
The training corpus is about 10,000 hours of internal Thai-centric speech, combining general sources such as podcasts with four domain verticals (finance, healthcare, education, law), both studio and crowdsourced, with ASR-derived transcripts after multi-step post-processing (§3.1). Its English or code-switched proportion is not reported.
Key Results
Evaluation uses two sets (§3.1): a short-duration set of 1,000 Thai Common Voice test utterances filtered by DNSMOS Pro above 3.9, denoised and silence-trimmed (Seed-TTS-style curation), and a new long-duration set of 231 manually verified YouTube cases of 16-30 s. In both, transliterated English terms are restored to English spelling and Thai number words are rewritten as Arabic numerals, so the systems receive raw code-switched and numeric text. CER uses Typhoon-Whisper-Large-v3 against a reference text normalized to the ASR’s output form (numerals to Thai words, English transliterated to Thai script, mai yamok expanded); SIM uses a WavLM-Large speaker-verification model; scores are means over five runs (§3.3).
Table 1 (checked against table-1.png):
| System | Short CER (%) | Short SIM | Long CER (%) | Long SIM |
|---|---|---|---|---|
| Human ground truth | 1.98 | 0.61 | 2.47 | 0.83 |
| Qwen3-TTS-0.6B | 3.14 | 0.62 | 6.10 | 0.79 |
| Qwen3-TTS-1.7B | 2.56 | 0.62 | 3.64 | 0.78 |
| ThonburianTTS | 6.26 | 0.48 | omitted | omitted |
| Kaitom Voice V3 (Alpha, commercial) | 2.34 | 0.59 | 5.81 | 0.79 |
| JaiTTS-v1.0 | 1.94 | 0.62 | 2.55 | 0.76 |
JaiTTS-v1.0 has the lowest CER among synthesized systems on both sets, and its long-duration CER (2.55%) is close to the human recording (2.47%), whereas Qwen3-TTS-0.6B and Kaitom degrade sharply on long inputs. Speaker similarity is a different picture: on the short set JaiTTS ties both Qwen3-TTS models at 0.62, and on the long set it has the lowest SIM of the synthesized systems reported (0.76 versus 0.79 for Qwen3-TTS-0.6B and Kaitom and 0.78 for Qwen3-TTS-1.7B; human 0.83). ThonburianTTS produced no sensible long-form output and was omitted (§4, Table 1).
RTF (Table 2, checked against table-2.png): JaiTTS-v1.0 0.1136, ThonburianTTS (NAR) 0.1150, Qwen3-TTS-0.6B 1.5092, Qwen3-TTS-1.7B 1.5409, measured on the same hardware, which is not specified (§4).
In a blind pairwise listening test with 20 native Thai raters, 30 YouTube reference speakers (gender balanced) and 30 texts mixing code-switching and numerals, raters chose one winner per pair considering naturalness/intelligibility and speaker similarity (§5). Against ElevenLabs eleven_v3 JaiTTS won 161, tied 19, and lost 20 of 200; against MiniMax speech-2.8-hd it won 122, tied 40, and lost 38 of 200, for 283/59/58 over 400 (Figure 2, checked against figure-2.png). The human test did not include the open-source baselines or Kaitom, and no significance test is reported.
Novelty Assessment
The contribution is a language adaptation, not a new model: the VoxCPM architecture and objectives are used as published, and the new elements are the roughly 10,000-hour Thai-centric training corpus (internal, not described as released), the decision to train directly on raw code-switched and numeric text rather than a normalization front end, and a pair of Thai evaluation sets including a long-duration one. The most informative empirical finding is the long-duration robustness gap in CER between JaiTTS and the other systems, but the paper does not evaluate the untuned VoxCPM base model, so how much of that comes from Thai continual training versus the inherited architecture cannot be separated. There are no ablations of data composition, data scale, or the raw-text strategy.
Field Significance
moderate — The paper provides a data point that a tokenizer-free continuous-latent AR architecture can be adapted to a lower-resource tonal language by continual training and, on its own benchmarks, reach near-recording CER on both short and long inputs while reading unnormalized code-switched text. Its value is mainly as a Thai system and benchmark reference; the evidence for the approach’s generality is limited by in-house data, a single base model, and no base-model comparison.
Claims
- supports: Continual training of a pretrained zero-shot TTS model on a large single-language corpus can bring intelligibility for that language close to the level of real recordings, including on long utterances where general multilingual models degrade.
Evidence: JaiTTS-v1.0 reaches 1.94% CER on 1-15 s Thai inputs (human 1.98%) and 2.55% on 16-30 s inputs (human 2.47%), while Qwen3-TTS-0.6B rises from 3.14% to 6.10% and the commercial Kaitom Voice V3 from 2.34% to 5.81%. (§4, Table 1)
- complicates: Gains in intelligibility from language-specific adaptation do not necessarily carry over to speaker similarity, especially on long-form generation.
Evidence: JaiTTS-v1.0 ties both Qwen3-TTS models at 0.62 SIM on the short set and has the lowest SIM among synthesized systems on the long set (0.76 versus 0.78-0.79). (§4, Table 1)
- supports: Training a TTS model end to end on raw text containing digits and code-switched foreign words can replace an explicit text normalization front end for languages where such inputs are frequent.
Evidence: Evaluation inputs keep Arabic numerals and English spellings, yet JaiTTS achieves the lowest CER of all synthesized systems, and the human-judgment texts (23 of 30 with numbers and/or code-switching) yield 283 wins, 59 ties, and 58 losses against ElevenLabs eleven_v3 and MiniMax speech-2.8-hd. (§3.1, §5, Table 1, Figure 2)
- complicates: Comparing synthesized speech CER to “human ground truth” CER is sensitive to reference text normalization and ASR choice, so small margins over human recordings are weak evidence of superiority.
Evidence: CER is computed with a Thai Whisper model against a reference rewritten into the ASR’s normalized form, and the authors attribute the 1.94% versus 1.98% margin to synthesized audio being cleaner than natural speech. (§3.3, §4)
- complicates: Short-utterance training data limits a TTS model’s ability to generate long utterances.
Evidence: ThonburianTTS, which the authors note was trained on short-utterance-dominated GigaSpeech2, produces 6.26% CER on 1-15 s inputs and fails to produce sensible output on 16-30 s inputs, so it is omitted from the long-duration results. (§1, §4, Table 1)
Limitations and Open Questions
Warning
The base VoxCPM model is not evaluated, and there are no ablations, so the effect of the 10,000-hour Thai continual training (and of training on raw text) is not isolated from what the architecture already provides. The training corpus is internal and its composition (including the share of English or code-switched speech) is not quantified.
- Speaker similarity is not a strength on long inputs (lowest among synthesized systems), which the “competitive SIM” framing in §1 does not make explicit.
- The human test compares only against two commercial systems, combines naturalness/intelligibility and similarity into one vote per pair, and reports no confidence intervals or significance tests; the open-source baselines and Kaitom are absent from it.
- RTF hardware is not reported, and per-run variance for the five-run averages is not given.
- The long-duration set is small (231 cases) and the paper does not state whether either benchmark is released.
- Kaitom Voice V3 was tested in an alpha phase, and ThonburianTTS has no long-duration numbers.
- Model size is not reported.
Wiki Connections
- Autoregressive Codec TTS — JaiTTS is an autoregressive TTS model, though it predicts continuous VAE latents with an FSQ-regularized planner rather than discrete codec tokens, and the authors motivate this against X-codec2-based codec LMs.
- Flow Matching — each latent patch is decoded by a local diffusion transformer trained with conditional flow matching, fine-tuned on Thai along with the rest of the model.
- Zero-Shot TTS — the system clones unseen speakers from a reference clip and is evaluated on speaker similarity for unseen Common Voice and YouTube speakers.
- Multilingual TTS — continually trains a model pretrained on other languages for Thai and evaluates it on Thai-English code-switched input, so its own output spans two languages.
- Subjective Evaluation — reports a blind pairwise preference test with 20 native Thai raters against two commercial systems.
- VoxCPM — the base architecture and starting checkpoint that JaiTTS continues training on Thai.
- Qwen3-TTS — the 0.6B and 1.7B models are the main open-source multilingual baselines on CER, SIM, and RTF.
- Seed-TTS — the short-duration test-set curation follows its evaluation methodology.
- Common Voice — the Thai test split is the source of the short-duration benchmark.
- MiniCPM4 — the TSLM backbone is initialized from this LLM via VoxCPM.
- Finite Scalar Quantization — supplies the FSQ bottleneck used to regularize the semantic planner.
- Llasa — cited as a codec-token AR system whose X-codec2 tokenizer may not cover Thai.
- TTS-1 — cited alongside Llasa as an X-codec2-based AR TTS system, motivating the tokenizer-free choice.