NAACL · 2025 · Conference
Basher et al. (Hishab Singapore / University of Central Florida / Qatar Computing Research Institute) · → Paper · Demo: ? · Code: ?
Adapts the XTTS zero-shot multilingual TTS architecture to Bangla, pretraining on 3.85k hours of Bengali speech and demonstrating that few-shot fine-tuning on 20 minutes per speaker closes most of the gap to ground truth on naturalness, clarity, and speaker similarity.
Problem
Zero-shot and few-shot speaker-adaptive TTS has advanced substantially for high- and medium-resource languages, but systems such as XTTS, YourTTS, and VALL-E X remain focused on languages with abundant training data. Bangla, despite having a large speaker population, has lacked a TTS system capable of instant speaker adaptation: prior Bangla systems (Katha, Subhachan, LSTM-based synthesizers, Indic-TTS variants) are either limited to a small fixed set of static speakers, produce robotic-sounding output, or are not open-source. No existing Bangla TTS framework supports few-shot voice cloning from minimal target-speaker data.
Method
BnTTS adapts the XTTS architecture (a GPT-2-style autoregressive LLM backbone over discrete VQ-VAE speech tokens, paired with a HiFi-GAN vocoder) to Bangla through continual pretraining rather than training from scratch. The audio encoder discretizes mel-spectrogram frames into VQ-VAE tokens; a Conditioning Encoder (six attention blocks, 32 heads) followed by a Perceiver Resampler compresses a variable-length speaker mel-spectrogram prompt into a fixed-length speaker embedding. Text tokens, speaker embeddings, and ground-truth spectrogram embeddings are concatenated and fed to the LLM backbone (GPT-2 dimensionality: hidden size 1024, intermediate size 3072, max sequence length 400), which is trained with a joint text-generation and audio-generation cross-entropy loss (weighted 0.01:1.0). At inference, only text and speaker embeddings are given as input, and the model autoregressively generates spectrogram embeddings, which the HiFi-GAN decoder converts into the final waveform conditioned on the resized speaker embedding.
Training proceeds in two pretraining stages: Partial Audio Prompting (random audio segment as speaker prompt, 5 epochs) followed by Complete Audio Prompting (full-duration audio prompt, 1 epoch, continuing from the prior checkpoint). The HiFi-GAN vocoder is fine-tuned separately for three days using GPT-2 embeddings from the second pretraining stage; the audio encoder and speaker encoder remain frozen throughout. Pretraining uses 3.85k hours of Bengali speech assembled from open-source corpora, synthetic TTS output, and a custom-built pseudo-labeling pipeline (ASR transcription, LLM-based punctuation restoration, diarization-based speaker filtering, noise suppression, and audio super-resolution). For few-shot speaker adaptation, the pretrained model is fine-tuned on just 20 minutes of studio-quality audio per speaker across four speakers, jointly, for 10 epochs.
Key Results
On the reference-aware BnStudioEval benchmark, the few-shot-adapted model (BnTTS-n) closely approaches ground truth: SMOS 4.624 vs. 4.809, Naturalness 4.600 vs. 4.798, and Clarity 4.869 vs. 4.913 (Table 1). BnTTS-n also outperforms the zero-shot pretrained model (BnTTS-0) on every subjective and speaker-similarity metric, including SECS (reference) 0.548 vs. 0.529 and SpeechBERTScore 0.791 vs. 0.789. Against three prior Bangla/Indic systems — IndicTTS, Google Cloud TTS, and Azure TTS — BnTTS-n achieves substantially higher SMOS, Naturalness, and Clarity on both BnStudioEval and the text-only BnTTSTextEval benchmark (overall SMOS 4.601 vs. 4.223 for the best commercial baseline, Table 2), although neither commercial system supports speaker adaptation, so the comparison is not apples-to-apples on that dimension.
The paper also reports a CER trade-off: both BnTTS variants show higher character error rate than Google/Azure TTS, which the authors attribute to BnTTS’s more conversational, expressive prosody producing less ASR-friendly output rather than genuine intelligibility loss. A targeted ablation (Table 3) on a 23-sample short-text subset shows that combining short audio prompts with adjusted sampling (T=1.0, TopK=2) reduces CER from 0.081 to 0.015 and improves a duration-matching score from 0.699 to 0.827, addressing a specific failure mode in generating short utterances.
Novelty Assessment
The contribution is engineering integration, not architectural novelty: BnTTS reuses the XTTS architecture (audio VQ-VAE encoder, Conditioning Encoder with Perceiver Resampler, GPT-2 backbone, HiFi-GAN decoder) without structural modification, adapting it to a new language through continual pretraining and a curated data pipeline. The genuine new artifacts are the 3.85k-hour Bangla pretraining corpus (much of it pseudo-labeled via an in-house acquisition framework), the 4.22-hour in-house high-quality speaker-adaptation set, and two new public evaluation datasets (BnStudioEval, BnTTSTextEval). The paper is explicit that it does not train XTTS from scratch for Bangla and relies on continual pretraining due to resource constraints, leaving open whether a from-scratch Bangla model would perform differently.
Field Significance
Moderate — this paper provides the first published few-shot speaker-adaptive TTS system for Bangla and contributes a sizeable Bangla training corpus plus two new public evaluation sets. Its significance is primarily as a data point demonstrating that an existing zero-shot multilingual architecture (XTTS) can be extended to a low-resource language via continual pretraining and a modest amount of speaker-adaptation data, rather than as a methodological advance.
Claims
- supports: Few-shot fine-tuning on a small amount of target-speaker studio audio substantially improves naturalness, clarity, and speaker similarity over a zero-shot pretrained multilingual TTS model, even when the zero-shot model was pretrained on thousands of hours of in-language speech.
Evidence: Fine-tuning on 20 minutes per speaker (4 speakers, 10 epochs) raises SMOS from 4.456 (BnTTS-0, zero-shot) to 4.624 (BnTTS-n, few-shot) and SECS (reference) from 0.529 to 0.548 on BnStudioEval. (§4, Table 1)
- complicates: Expressive, conversationally-prosodic TTS output can score worse on ASR-based intelligibility metrics than flatter-sounding commercial systems, even when subjective listening scores favor the expressive system.
Evidence: BnTTS-n and BnTTS-0 show higher CER than Google Cloud TTS and Azure TTS on both BnStudioEval and BnTTSTextEval despite BnTTS-n achieving the highest SMOS, Naturalness, and Clarity scores overall; the authors attribute this to ASR systems being better suited to standardized, less expressive speech patterns. (§4, “High CER in Text Generation”, Tables 1–2)
- complicates: Autoregressive codec-based TTS models trained primarily on longer utterances can fail to generate short sequences reliably under default sampling settings.
Evidence: Under default generation settings (T=0.85, TopK=50), the model achieves only 0.699 DurationEquality and 0.081 CER on a 23-sample short-text subset (under 30 characters); adjusting prompt length and sampling temperature/top-K improves this to 0.827 DurationEquality and 0.015 CER. (§4, “Effect of Sampling and Prompt Length on Short Speech Generation”, Table 3)
- refines: Adapting a multilingual zero-shot TTS architecture to a new low-resource language via continual pretraining can match or exceed existing language-specific commercial and open-source systems without requiring training from scratch.
Evidence: BnTTS-n, initialized from the pretrained XTTS checkpoint and continually pretrained on 3.85k hours of Bengali data, outperforms IndicTTS, Google Cloud TTS, and Azure TTS on SMOS, Naturalness, and Clarity across both evaluation datasets, while the authors note training from scratch was not attempted due to resource constraints. (§3 “BnTTS model”, §7 “Limitations”, Tables 1–2)
Limitations and Open Questions
Warning
The system struggles to adapt to speakers with unique vocal traits absent from training, and fails to generate short sequences (under two words or 20 characters) in some cases even after mitigation; these failures stem from limitations inherited from the underlying XTTS foundation model rather than being fully resolved by the Bangla adaptation.
The authors did not train XTTS from scratch for Bangla, relying instead on continual pretraining due to resource constraints, leaving the ceiling of a from-scratch Bangla model untested. A substantial portion of the pretraining corpus (3636 of 3856 hours) is pseudo-labeled and not manually reviewed, which may introduce label noise not fully captured by the reported metrics. The evaluation is limited to four in-house speakers for the few-shot speaker-adaptation setting, and dialectal variation within Bangla is not addressed.
Wiki Connections
- Zero-Shot TTS — BnTTS evaluates both a zero-shot pretrained variant and a few-shot fine-tuned variant of the same XTTS-derived architecture, directly comparing the two adaptation regimes for a low-resource language.
- Speaker Adaptation — the paper’s core contribution is a few-shot speaker-adaptation recipe (20 minutes per speaker, multi-speaker joint fine-tuning) layered on top of a pretrained zero-shot backbone.
- Autoregressive Codec TTS — the model follows the autoregressive LLM-over-discrete-audio-token paradigm established by XTTS, generating VQ-VAE-derived spectrogram embeddings token by token.
- GAN Vocoder — a separately fine-tuned HiFi-GAN vocoder converts the LLM’s spectrogram-embedding output into the final waveform.