arXiv · 2025 · Preprint
Singh et al. (Dubverse AI) · → Paper · Demo: ? · Code: ✓
MahaTTS-v2 is a two-stage multilingual TTS system combining a Gemma-based autoregressive LM for text-to-semantic modeling with a conditional flow matching model for acoustics, trained on 20K hours of Indic language data across 22 languages.
Problem
Most high-performing TTS systems are developed primarily for English and European languages, leaving Indic languages — a group representing hundreds of millions of speakers — with no unified, high-quality TTS coverage. Existing multilingual systems like XTTS cover some Indic languages incidentally, but none are purpose-built with large-scale, high-quality Indic training data and cross-lingual transfer as a primary design goal.
Method
MahaTTS-v2 follows a two-stage pipeline: M1 converts text to semantic tokens, and M2 converts semantic tokens to an audio waveform — the two stages are trained independently with no joint optimisation required.
For semantic representations, the system uses a pretrained XLS-R wav2vec2.0 model (1B parameters, trained on 436K hours across 128 languages) with k-means clustering to produce a 10,000-token discrete phoneme dictionary at approximately 25 tokens per second. This choice avoids the need for joint training and generalises out of the box to unseen languages.
M1 is a 0.5B Gemma-based decoder-only language model fine-tuned for text-to-semantic generation. The model is conditioned on language embeddings, speaker embeddings (derived from the nearest three reference clips via a six-layer attention encoder), and character tokens. The total vocabulary is 14,894 tokens (10,001 semantic + 4,893 character). RoPE positional encoding is used; special care is taken during fine-tuning — only the classification heads are frozen — to prevent the hallucination that arises when speaker encoder weights or full model weights are frozen.
M2 is a 300M parameter conditional flow matching model (Matcha-TTS-inspired) that maps semantic tokens to mel spectrograms. The model uses flat alternating attention and convolution layers with residual connections, and is conditioned on semantic tokens plus the nearest three reference speaker clips. It is trained with an optimal-transport flow objective and uses RelativePositionBias to generalise beyond the 10-second training context to 30-second outputs at inference. BigVGAN converts mel spectrograms to the final audio waveform at 24 kHz.
M1 is trained on 8 A100 40GB GPUs with a 1,024 sample batch size in bfloat16 for 7 epochs. Data augmentation brings the effective training set to 50K hours. The flow model is trained on 4 A5000 GPUs on a 1K-hour subset for 700K iterations. Adding a new language requires approximately 200 hours of data. The M2 stage alone can also function as a voice conversion system by encoding input speech to semantic tokens and regenerating with a different speaker conditioning.
Key Results
WER (word error rate via Scribe ASR) is the primary evaluation metric, computed over 10 sentences per language. MahaTTS-v2 achieves competitive or best results on the majority of tested Indic languages. On English, it reaches 2% WER, matching XTTS (0%) and outperforming Indic-Parler-TTS (2%). For Hindi, it achieves 4% WER versus Indic-Parler-TTS (6%), IndicF5 (10%), and XTTS (42%). On Odia, MahaTTS-v2 reports the highest WER of tested systems (73%), while it achieves best or near-best on Bengali (7% vs. 39% for Indic-Parler-TTS), Gujarati (36% vs. 30%), and several other Indic languages.
Note
WER comparisons are not always directly comparable across systems — some baselines have no coverage for certain languages (marked as ”-”), and the test set is small (10 sentences per language). No MOS or naturalness evaluations are reported; intelligibility via WER is the sole quantitative metric.
Performance is uneven: results on Assamese (42%) and Odia (73%) are substantially weaker than the leading baselines, suggesting that low-resource languages with limited training hours remain difficult even within this framework.
Novelty Assessment
The architecture assembles established components — wav2vec2.0 semantic tokens, a Gemma-based LM, conditional flow matching (adapted from Matcha-TTS), and BigVGAN — into a single pipeline targeting Indic TTS. None of the individual components are novel; the contribution is the combination applied to a new language domain, the Indic data collection effort (20K hours across 22 languages), and the explicit cross-lingual transfer capability.
The paper’s most defensible novelty is the dataset scale for Indic languages and the observation that 200 hours suffice to add a new language to the system. The architectural decision to use wav2vec2.0 k-means tokens over VQVAE (Tortoise-TTS style) is sensibly motivated — avoiding joint training — but is not itself a research contribution.
The evaluation is limited to WER over a small test set; no listening tests, speaker similarity scores, or UTMOS evaluations are reported, making it difficult to assess audio quality and speaker fidelity relative to baselines.
Field Significance
Moderate — MahaTTS-v2 demonstrates that a Tortoise-TTS-style two-stage pipeline, updated with a modern LM backbone and flow matching acoustics, can be adapted to support 22 Indic languages at useful quality with open-source data. The primary contribution is the data pipeline and cross-lingual coverage rather than architectural innovation. It provides a reproducible baseline for Indic TTS research and releases both code and model weights.
Claims
- Large multilingual TTS systems built on semantic token intermediaries transfer to low-resource languages more readily than end-to-end spectrogram models. (§2.1, §5.1)
- Decoupling the text-to-semantic and semantic-to-acoustic stages enables independent training and simplifies the addition of new languages without full system retraining. (§2, §4.1)
- Flow matching is a viable replacement for diffusion in the acoustic generation stage of two-stage TTS pipelines, maintaining competitive quality at lower training complexity. (§2.3, §5.1)
- Intelligibility in low-resource languages with limited training data remains markedly worse than high-resource languages within the same multilingual system. (§5.1, Table 2)
Limitations and Open Questions
Warning
The evaluation relies exclusively on WER over 10 sentences per language with no MOS, SMOS, or naturalness scores. This makes it impossible to assess audio quality, expressiveness, or speaker similarity relative to baselines — core dimensions for a TTS system.
English dominates the training set at 58%, which may explain strong English results but raises questions about whether true cross-lingual transfer or data dominance is responsible. The system lacks prosody and pace control conditioning in M1, which the authors flag as future work. Zero-shot speaker fidelity for M2 is acknowledged as limited compared to infilling-based approaches like Seamless. Fine-tuning introduces hallucination that requires careful intervention (freezing classification heads only), suggesting the LM component is sensitive to distribution shift. Languages with fewer training hours (Assamese: 48h, Dogri: 8h, Rajasthani: 20h) show substantially weaker results.
Wiki Connections
multilingual-tts — the paper directly targets Indic language coverage as its primary motivation.
autoregressive-codec-tts — M1 follows the text-to-semantic-token LM paradigm pioneered by VALL-E and Tortoise-TTS.
flow-matching — M2 uses OT-CFM (Matcha-TTS style) for the semantic-to-mel stage.
self-supervised-speech — wav2vec2.0 (XLS-R) is used for the semantic token front-end; the pretrained model’s cross-lingual generalisation is central to the design.
zero-shot-tts — the speaker encoder enables zero-shot inference from reference clips; the authors note this is not fully zero-shot in the sense of Seamless-style infilling.
speaker-adaptation — the fine-tuning recipe for new speaker adaptation is described in detail, with specific findings on which components to freeze.