arXiv · 2025 · Preprint
Bhadoriya et al. (Indian Institute of Technology Bombay) · → Paper · Demo: ? · Code: ?
A2TTS extends the Grad-TTS diffusion framework with a cross-attention duration predictor conditioned on reference mel-spectrograms, enabling zero-shot multispeaker TTS across seven low-resource Indian languages trained on the IndicSUPERB dataset.
Problem
Most neural TTS systems are developed for high-resource languages, and direct application to low-resource Indian languages is complicated by limited training data, high speaker and dialectal variability, and poor duration modeling when transferring speaker identity to unseen voices. Standard duration predictors in diffusion TTS pipelines are speaker-agnostic, failing to reflect the prosodic timing characteristics of target speakers during zero-shot synthesis. Prior zero-shot approaches such as YourTTS and x-vector/d-vector conditioning showed some generalisation but did not directly use speaker reference audio to inform phoneme timing.
Method
A2TTS is built on Grad-TTS, a DDPM-based architecture that produces mel-spectrograms via a reverse diffusion process. The system retains Grad-TTS’s text encoder, which produces phoneme-level embeddings from input text, and replaces the standard duration predictor with a cross-attention module. In this module, text embeddings serve as queries while a 2-second reference mel-spectrogram from the same speaker (but unrelated content) provides the keys and values. This design forces the model to capture speaker-specific timing and intonation rather than memorising phoneme durations from the text, since the reference clip does not share linguistic content with the input. The resulting phoneme durations are then predicted from the attention output.
For speaker conditioning throughout the diffusion decoder, A2TTS adopts the speaker encoder from UnitSpeech, pre-trained on VoxCeleb2 (6,112 speakers), to extract speaker embeddings from short reference audio. These embeddings condition the score function used in the reverse diffusion process. During inference, classifier-free guidance is applied using the dataset-wide mean mel-spectrogram as the unconditional condition, amplifying the influence of the speaker and text conditions without changing the training procedure.
Language-specific models are trained separately on IndicSUPERB for seven languages (Bengali, Gujarati, Hindi, Marathi, Malayalam, Punjabi, and Tamil), with audio resampled from 16 kHz to 22 kHz. The models share the same architecture as Grad-TTS but with double the channel count to accommodate multispeaker modelling. HiFi-GAN is used as the vocoder. Training runs for 1,500 epochs and fine-tuning for an additional 1,000 epochs on 8 NVIDIA H100 GPUs.
Key Results
Speaker similarity (SIM-O, cosine similarity of speaker embeddings) is the primary metric. A2TTS consistently outperforms a Grad-TTS baseline conditioned on speaker embeddings across all seven languages: Hindi 0.729 vs. 0.725, Marathi 0.734 vs. 0.716, Bengali 0.708 vs. 0.683, Tamil 0.726 vs. 0.675, with similar margins for the remaining languages (Table 2). All systems remain below the test-set average SIM-O of 0.725-0.772, indicating a gap between synthesised and natural speech speaker similarity.
Intelligibility, measured by CER on ASR conformer outputs, is reported across multiple evaluation sets (Table 1). On the primary IndicSUPERB evaluation, CER ranges from 6.16% (Hindi) to 17.93% (Malayalam) for the synthesised speech. WER results in the appendix show TTS error rates roughly 1.5 to 4 times higher than the corresponding ASR baseline on the same test sets, indicating that synthesised speech intelligibility degrades substantially relative to ground-truth audio on out-of-domain evaluation sets such as CommonVoice and FLEURS.
No MOS or UTMOS naturalness scores are reported.
Novelty Assessment
The primary architectural modification is the cross-attention duration predictor conditioned on reference mel-spectrograms. This mechanism is genuinely new relative to Grad-TTS’s standard duration predictor, and the deliberate use of content-unrelated reference audio to prevent duration memorisation is a thoughtful design choice. However, the surrounding system is a direct engineering integration: the diffusion backbone is Grad-TTS unchanged (except channel count), and the speaker encoder and CFG implementation are taken directly from UnitSpeech. The application to Indian languages and IndicSUPERB is the primary scope extension.
The evaluation is limited in several ways. Comparisons are made only against a single baseline (Grad-TTS with speaker conditioning), without including UnitSpeech, YourTTS, or other multilingual diffusion TTS systems. Naturalness is not evaluated with MOS or automated predictors. WER results are relegated to the appendix because the baseline ASR systems themselves have high error rates, which limits their diagnostic value for TTS quality. The paper does not ablate the cross-attention mechanism independently from other modifications.
Field Significance
Low — A2TTS provides a practical demonstration that Grad-TTS can be extended with speaker-reference-conditioned duration prediction for low-resource Indian languages. The cross-attention duration modification is a useful architectural variant, but the contribution is narrowly scoped and the evaluation does not include comparisons against other zero-shot TTS baselines or naturalness metrics. The work primarily adds evidence for the viability of diffusion TTS in South Asian language settings rather than establishing a new direction.
Claims
-
supports: Conditioning duration prediction on reference audio from the same speaker can improve speaker similarity in zero-shot diffusion TTS without modifying the training objective.
Evidence: Cross-attention over an unrelated reference mel-spectrogram (same speaker, different content) improves SIM-O over a speaker-embedding-conditioned Grad-TTS baseline across all seven Indian languages tested. (§2.2, Table 2)
-
supports: Classifier-free guidance with a dataset-mean unconditional condition improves speaker consistency at inference time in diffusion-based TTS for unseen speakers.
Evidence: A2TTS applies CFG at inference, following UnitSpeech, to amplify speaker conditioning without changing the training procedure; speaker similarity gains are observed across all languages evaluated. (§2.3, Table 2)
-
complicates: Diffusion-based zero-shot TTS can achieve reasonable speaker similarity, but intelligibility on out-of-domain benchmarks degrades substantially relative to ground-truth speech.
Evidence: A2TTS TTS WER on FLEURS, CommonVoice, and IndicTTS is 1.5-4x higher than the corresponding ASR baseline on the same test sets across Hindi, Marathi, Punjabi, Bengali, Gujarati, Tamil, and Malayalam. (§Appendix A, Tables 3-9)
-
supports: Language-specific diffusion TTS models can be trained on standard benchmark speech corpora to achieve multi-speaker zero-shot synthesis in low-resource settings.
Evidence: A2TTS trains separate models for seven Indian languages using only the IndicSUPERB dataset (except Malayalam and Tamil which also use IndicTTS), achieving SIM-O above 0.70 for all languages on the IndicSUPERB evaluation set. (§3.1, Table 2)
Limitations and Open Questions
Warning
No naturalness evaluation (MOS or automated predictors) is reported. Speaker similarity (SIM-O) and CER are the only metrics, so claims about speech quality and naturalness cannot be verified from the paper’s own experiments.
The model is only compared to a single baseline (Grad-TTS with speaker embeddings), without evaluating against UnitSpeech (from which the speaker encoder is borrowed), YourTTS, or other zero-shot multilingual systems. This makes it difficult to assess how much of the improvement comes from the cross-attention duration predictor versus the overall system design.
The paper trains language-specific models rather than a single multilingual model, which limits scalability to new languages. The authors acknowledge that out-of-domain speakers and languages not covered by IndicSUPERB would require fine-tuning. High training cost (2,000 epochs) is also noted as a practical constraint.
The elevated TTS WER relative to ASR baselines on CommonVoice and FLEURS suggests intelligibility issues on speakers and recording conditions not well represented in IndicSUPERB training data.
Wiki Connections
- Diffusion TTS — A2TTS builds directly on Grad-TTS, a DDPM-based TTS model, extending it with cross-attention duration prediction and speaker conditioning.
- Multilingual TTS — the paper demonstrates language-specific diffusion TTS across seven Indian languages using IndicSUPERB as the training corpus.
- Zero-Shot TTS — A2TTS uses a reference audio clip and classifier-free guidance to synthesise speech for speakers not seen during training.
- Speaker Adaptation — speaker embeddings from a VoxCeleb2-pretrained encoder condition both the duration predictor and diffusion decoder to preserve target speaker identity.