arXiv · 2026 · Preprint
Venkata Pushpak Teja Menta (Praxel Ventures) · → Paper · Demo: ✓ · Code: ✓
Extends a frozen multilingual TTS base (Chatterbox Multilingual) that lacks Telugu and Tamil to those languages by romanising Brahmic scripts to ISO-15919, LoRA-adapting only the text-token predictor, and recovering native-sounding acoustics at inference with a same-language voice prompt and tuned sampling settings, all evaluated on 10-utterance pilot sets.
Problem
Open-source multilingual TTS bases trail commercial systems on Indic phonology, and the base studied here (Chatterbox Multilingual, 23 languages) cannot tokenise Telugu or Tamil at all: passing those scripts either raises an unsupported-language error or produces byte-level junk tokens. The two established routes to Indic TTS are training an Indic-native model from scratch, which the author frames as costing hundreds to thousands of GPU-hours, or paying for closed commercial APIs. The paper asks what minimum intervention on a frozen non-Indic base reaches commercial-class output on Telugu, Tamil, and Hindi without training a new acoustic decoder and without any commercial TTS training data.
Method
The system wraps Chatterbox Multilingual (810M parameters across a text-token predictor t3, a flow-matching acoustic generator s3gen, and a voice encoder) with three components.
BUPS (Brahmic Unified Phoneme Space). A deterministic preprocessing layer segments input into single-script runs by Unicode block, transliterates every Brahmic run (Devanagari, Telugu, Tamil, Kannada, Bengali, Gujarati, Malayalam) to ISO-15919 Latin with diacritics using the indic-transliteration package, and passes non-Brahmic runs through unchanged. The result can be consumed by Chatterbox’s existing Latin-script tokeniser paths. There is no learned component.
LoRA on the text head only. A LoRA adapter (rank 32, alpha 64) is applied to the attention projections of t3; s3gen and the voice encoder stay frozen, giving 7.86M trainable parameters (0.97% of the base). Training text is BUPS-romanised Indic transcripts, conditioned with Hindi as a proxy language_id because the base rejects te/ta and earlier rounds using them were unstable. The R6 adapter is trained for 8,000 steps on roughly 1,220 h of licensed speech (IndicTTS, Rasa, FLEURS, and Shrutilipi subsets, UTMOS-filtered), on a single A100-80GB in about 11 h at a peak learning rate of 3e-6; an earlier peak of 2e-5 diverged.
Voice-prompt recovery with “Config B”. At inference the system supplies an 8 to 11 s same-language reference clip through Chatterbox’s zero-shot voice-prompt interface and overrides three sampling parameters (exaggeration 0.7, temperature 0.6, min_p 0.1). The motivation is that the LoRA’d text head emits reasonable tokens while the frozen acoustic decoder lacks an Indic prior, so the prompt supplies native acoustic context. Config B was chosen from a three-configuration sweep plus one native Telugu listener’s informal ear test, which the author explicitly frames as an engineering finding rather than hyperparameter optimisation. Adapting s3gen itself was attempted but did not fit on an A100-80GB at a usable batch size.
Routing. Deployment is a rule-based router: Telugu and Tamil pure-script input goes to the LoRA + BUPS branch; Hindi goes to unmodified Chatterbox with the same voice-prompt and Config B recipe; any input containing a Latin word of two or more characters goes to a third code-mix branch. The code-mix branch uses AI4Bharat IndicF5 (a frozen flow-matching model with a character-level tokeniser) zero-shot, after an instruction-tuned LLM (Claude Haiku 4.5, temperature 0, cached) rewrites English words into native-script phonetic spellings, because IndicF5 silently drops Latin spans.
Key Results
Evaluation uses the companion PSP benchmark, which scores per-phoneme collapse via XLS-R embeddings against native prototypes plus corpus-level FAD and a prosodic Fréchet distance (PSD), together with LLM-WER from a Qwen-2.5-72B semantic judge over Whisper-family transcripts. All test sets are 10-utterance pilots per language (Table I):
- Telugu: retroflex collapse 26.7% for Praxy versus 33.3% for Sarvam Bulbul, 40.0% for ElevenLabs v3, and 50.0% for Cartesia Sonic-3. The author notes this is a one-token difference on 15 retroflex tokens, within small-sample noise. LLM-WER is 0.033 versus 0.029 for Sarvam and Cartesia; FAD is 291.3 versus 250.4 for Sarvam.
- Tamil: Tamil-zha collapse 71.4% versus 85.7% for each of the three commercial systems (a one-token difference on 7 tokens). The open-source Indic Parler-TTS baseline scores lower (61.5%) on this dimension and also on retroflex collapse (64.3% versus 69.2%).
- Hindi: vanilla Chatterbox with the recipe ties Cartesia Sonic-3 on LLM-WER (0.025), but FAD (439.3) trails Sarvam (211.8) and Cartesia (267.4) substantially.
Ablations, all on n=10:
- Config B sweep (Table II, Telugu): Config B reaches LLM-WER 0.034 versus 0.159 (Config A) and 0.061 (Config C), and FAD 291.3 versus 534.4 and 355.0. Note that the caption names a Cartesia-Te reference while the FAD/PSD values match the Sarvam-Te row of Tables I and IV; the paper does not explain the overlap.
- Hindi scope control (Table III): applying the R6 LoRA + BUPS to Hindi raises LLM-WER from 0.025 to 0.334 (0.204 with LoRA but without BUPS), while retroflex and aspiration collapse stay at 0% for all variants.
- Reference source (Table IV, Telugu): a same-language Sarvam-Te reference gives FAD 291.3 and PSD 13.1 versus 355.0 and 61.7 with no reference; a 49 s English reference gives FAD 448.2.
- Data scale (§V-E): moving from 85 h (R5) to about 1,220 h (R6) lowered Telugu FAD from 534 to 355 and LLM-WER from 0.171 to 0.034, but raised PSD from 14 to 62.
- Code-mix branch (Table V): native-script transliteration drops IndicF5 LLM-WER from 0.855 to 0.198 (Hindi), 0.798 to 0.142 (Telugu), and 0.745 to 0.268 (Tamil). Cartesia Sonic-3 remains ahead on Hindi (0.000) and Telugu (0.106), which the author partly attributes to the Whisper-based judge favouring American-English pronunciation of embedded English words.
Comparisons are confounded in ways the paper partly acknowledges: Praxy rows use one voice (n=10) while commercial rows use two (n=20), the voice-prompt references are clips generated by the very commercial systems being compared against, several commercial cells are unmeasured, and no human listening test was run.
Novelty Assessment
The individual components are existing tools: ISO-15919 transliteration via an off-the-shelf package, LoRA via PEFT on an existing Chatterbox fine-tuning framework, the base model’s own voice-prompt interface, sampling-parameter changes, and LLM-based transliteration in front of a frozen IndicF5. The contribution is the integration: using script romanisation as a routing layer so a Latin-tokenised base can be LoRA-extended to languages it does not cover, and pairing this with inference-time prompting rather than acoustic-decoder training. The Hindi control, showing that the same adapter harms a language the base already supports, is a useful scoping observation. The work is engineering-integration with a small evaluation footprint rather than an architectural advance.
Field Significance
Low: the paper provides a low-cost recipe and released artefacts (LoRA weights, router, demo) for extending a frozen multilingual TTS base to unsupported Brahmic-script languages, with an informative negative control on a natively supported language. Its headline comparisons with commercial systems rest on 10-utterance pilots with token counts the author concedes are within noise, references drawn from the competitor systems, and no formal listening test, so it serves mainly as a preliminary data point on language extension of frozen TTS bases.
Claims
- supports: Adapting only the text-to-token stage of a frozen multilingual TTS model can extend it to languages whose scripts the base tokeniser does not cover, provided the text is routed through a script the tokeniser already handles.
Evidence: BUPS romanisation to ISO-15919 plus a 7.86M-parameter LoRA on Chatterbox’s t3 (acoustic decoder frozen, Hindi-proxy language ID) yields Telugu LLM-WER 0.033 and Tamil LLM-WER 0.041 on 10-utterance pilots, where the unmodified base cannot accept these scripts. (§III-A, §III-B, Table I)
- complicates: Language-extension fine-tuning of a shared text front end can degrade a language the base model already supported.
Evidence: Applying the Telugu/Tamil-trained LoRA to Hindi raises LLM-WER from 0.025 (vanilla) to 0.334 (0.204 without BUPS), while per-phoneme retroflex and aspiration collapse stay at 0%, so the damage is at the token level; deployment therefore routes Hindi to the unmodified base. (§V-C, Table III)
- supports: In zero-shot TTS with a frozen acoustic decoder, a same-language reference prompt improves target-language acoustic and prosodic fidelity, while a cross-language prompt degrades it.
Evidence: On Telugu with the LoRA branch and fixed sampling, a same-language Sarvam-Te reference gives FAD 291.3 and PSD 13.1 versus 355.0 and 61.7 with no reference, while a 49 s English reference worsens FAD to 448.2. (§V-D, Table IV)
- complicates: Scaling adaptation data for a new language can improve intelligibility and distributional similarity while worsening prosodic similarity to native speech.
Evidence: Moving from 85 h (R5) to about 1,220 h (R6) of Indic data reduced Telugu FAD from 534 to 355 and LLM-WER from 0.171 to 0.034, but increased PSD from 14 to 62. (§V-E)
- complicates: ASR-based intelligibility metrics can penalise nativised pronunciation of embedded foreign words in code-switched TTS, so they may disagree with native-listener judgements.
Evidence: The author attributes Cartesia Sonic-3’s 0.000 LLM-WER on Hindi code-mix (versus 0.198 for the transliterate-then-IndicF5 branch) partly to Whisper-based transcription favouring American-English pronunciation of English spans; no listening test was run to confirm this. (§VI-B, Table V)
Limitations and Open Questions
Warning
Every comparison uses 10-utterance pilot sets (n=10 for Praxy, n=20 for commercial systems), with as few as 7 to 15 target tokens per per-phoneme cell; the author states that 5-point differences are not statistically separable, and there is no formal MOS or listening test.
- The voice-prompt references for the headline Praxy rows are clips synthesised by Sarvam and Cartesia, the same commercial systems used as baselines, which entangles the comparison.
- The evaluation benchmark (PSP) and the system come from the same author; per-phoneme probes have a high native-audio noise floor (the paper cites about 46% retroflex collapse on native Telugu audio; the PSP paper’s own calibration table reports 0.43), so these numbers are only interpretable as relative rankings.
- The acoustic decoder was never adapted, and Hindi FAD remains well behind commercial systems.
- The code-mix branch depends on a proprietary LLM at inference and is independent of the LoRA contribution; Tamil commercial code-mix cells were not measured.
- Config B was selected from three configurations with one listener, and the reference used in the Config B ablation is not consistently labelled between Table II and Table IV.
- Whether BUPS-style routing transfers to the other four Brahmic scripts it supports (Kannada, Bengali, Gujarati, Malayalam) is untested.
Wiki Connections
- Multilingual TTS — extends a frozen 23-language base to Telugu and Tamil through script romanisation and text-head LoRA, and shows the adapter harms Hindi, which the base already supports.
- Zero-Shot TTS — relies on the base model’s reference-prompted cloning as the main way to recover native acoustics, with ablations on prompt language and source.
- OmniVoice — cited as the from-scratch omnilingual alternative that this minimum-intervention approach is positioned against.
- A2TTS — cited as a diffusion-based Indic TTS trained for low-resource speaker adaptation that does not cover Telugu, contrasted with adapting an existing non-Indic base.
- F5-TTS — cited as a contemporary voice-prompt TTS system; the code-mix branch uses IndicF5, an Indic flow-matching model released by AI4Bharat.
- Speaker Embedding Phonological Rule Interactions — cited as a contemporary rule-based phonological accent benchmark for English, alongside the companion PSP benchmark used for all evaluation here.
- PSP — companion benchmark by the same author; supplies every per-phoneme, FAD and PSD number on this page, so system and evaluation share an author.