arXiv · 2025 · Preprint
Pandey et al. (Indian Institute of Technology Bombay) · → Paper · Demo: ? · Code: ?
A comparative study of infill-based versus speaker-prompted duration prediction strategies within a Voicebox-inspired flow-matching TTS system trained on Indian language data, revealing language-dependent trade-offs between intelligibility and speaker similarity.
Problem
Standard duration prediction methods used in flow-matching TTS systems rely on forced alignment to extract character-level timing targets during training. In low-resource settings like Indian languages, forced aligners are often unreliable due to the high prosodic variability, regional dialectal diversity, and agglutinative morphology that differ substantially from high-resource languages. Prior work on Indian language TTS had not examined how different duration prediction strategies interact with these linguistic properties at the zero-shot, speaker-specific generation level.
Method
The system adapts Voicebox (Le et al., 2023) to five Indian languages: Hindi, Tamil, Telugu, Bengali, and Marathi. The audio backbone is a 103M parameter Transformer-based continuous normalising flow (CNF) trained with the flow-matching objective over 80-dimensional log-mel spectrograms. The model performs speech infilling, learning to predict missing segments conditioned on surrounding audio context and character-level transcripts. A probabilistic masking strategy during training applies segment masking with varying rates from 30% to 100% of the sequence.
Two duration predictors are compared. The first follows the Voicebox design (infill-style): a 51M parameter flow-matching model that predicts masked character durations conditioned on unmasked duration context and the phonetic transcript, trained with MSE loss over log-scaled durations. A convolutional pre-network is added before the Transformer compared to the original design. The second is a speaker-prompted predictor (PFlow-style): an 84M parameter model inspired by PFlow (Kim et al., 2023) that replaces duration context with a randomly sampled 3-second mel spectrogram segment from the context audio. Cross-attention from text tokens to speech frames allows the model to extract speaker-specific prosodic cues directly, learning durations without depending on forced alignments at inference. Training data aggregates multiple publicly available Indian language corpora, filtered by WER thresholds using IndicWhisper transcription, yielding between 932 hours (Telugu) and 2,913 hours (Hindi). All models train in FP32 using AdamW with linear warmup and decay.
Key Results
Evaluation covers continuous sentence completion (50% masking) across five languages using the Vistaar test set. For intelligibility (WER via IndicConformer), the speaker-prompted predictor is best for Tamil (WER 0.295 vs. 0.312 for infill overall) but substantially worse for Marathi (WER 0.365 vs. 0.224 for infill). For Bengali and Hindi, differences are small or mixed across test subsets. Speaker similarity (Sim-o cosine scores) favours the PFlow predictor for Tamil (0.693 vs. 0.683), Telugu (0.633 vs. 0.629), and Bengali (0.671 vs. 0.657).
Human evaluations on Hindi, Tamil, and Bengali confirm this pattern at scale. For Tamil, the speaker-prompted predictor exceeds the infill approach on both QMOS naturality (4.49 vs. 4.48), intelligibility (4.61 vs. 4.47), and SMOS similarity (4.56 vs. 4.41). For Hindi, the speaker-prompted predictor achieves notably higher SMOS similarity (4.14 vs. 3.57) with competitive QMOS naturality (4.33 vs. 3.97), though intelligibility MOS is slightly lower (4.13 vs. 4.24). Bengali shows the reverse: infill leads on both QMOS and SMOS dimensions.
Novelty Assessment
The contribution is engineering integration with an empirical focus. Both the audio model (Voicebox) and the speaker-prompted duration predictor design (PFlow) are existing architectures; the paper adapts and compares them in the Indian language low-resource context. Modifications to the audio training objective (weighted loss mixing masked and unmasked frames at 0.9/0.1 ratio) and the duration model (convolutional pre-network, log-duration MSE loss) are incremental refinements rather than structural innovations. The paper’s value lies in the comparative evaluation itself: it provides the first systematic assessment of these two duration strategies across multiple Indian languages with both objective and human metrics, surfacing language-specific patterns that are not predictable from prior work in higher-resource settings.
Field Significance
Low — this technical report provides useful empirical evidence that duration prediction strategy choice has language-dependent consequences in low-resource multilingual TTS, a practical concern for systems targeting Indian languages. The evaluation is honest and methodical, but the scope is narrow, the models are adaptations of established architectures, and the findings do not generalise beyond the five languages studied.
Claims
-
complicates: Duration prediction strategy choice in zero-shot TTS presents language-dependent trade-offs between intelligibility and speaker similarity that are not consistent across multilingual settings.
Evidence: The speaker-prompted (PFlow-style) predictor improves Tamil intelligibility (WER 0.295 vs. 0.312) and speaker similarity (SMOS 4.56 vs. 4.41) but degrades Marathi intelligibility substantially (WER 0.365 vs. 0.224 for infill), while Bengali human evaluation favours infill on both dimensions. (§5, Table 3, Table 5)
-
supports: Speaker-conditioned audio prompting enables duration prediction without forced alignment, yielding competitive or superior speaker similarity in low-resource multilingual TTS.
Evidence: The 84M-parameter speaker-prompted predictor, conditioning on a 3-second mel spectrogram via cross-attention, achieves higher Sim-o scores than the infill predictor for Tamil, Telugu, and Bengali, and substantially higher human SMOS for Hindi (4.14 vs. 3.57), without using explicit duration context at inference. (§3.4.2, §5.1, Table 4, Table 5)
-
complicates: Pooled low-resource training corpora for multilingual TTS contain significant quality variance that requires aggressive filtering before flow-matching models can be reliably trained.
Evidence: WER-based filtering using IndicWhisper reduced the Hindi training set from 1.3M to 1M utterances; a partial recovery via CTC alignment scores added 30K back, indicating that roughly 23% of the raw data was too noisy or misaligned to be usable. (§4.2.2)
-
refines: The relative benefit of speaker-prompted duration conditioning over text-and-alignment conditioning is not uniform and depends on the phonological and prosodic characteristics of the target language.
Evidence: Tamil, with complex phoneme structures and high prosodic variability, benefits consistently from speaker prompting across both objective WER and human SMOS; Marathi shows a clear regression; Hindi and Bengali exhibit mixed patterns depending on test subset. (§5.1, §5.2, Table 3, Table 4, Table 5)
Limitations and Open Questions
Warning
All experiments are conducted on speech infilling (50% sentence masking), not full-sentence TTS generation. Intelligibility and speaker similarity results may not transfer directly to the more common single-utterance TTS evaluation setting.
The cross-sentence completion task is described but no quantitative results are reported in the paper text. Comparisons are against ground-truth durations and the two proposed predictors only; no external TTS baselines from the Indian language literature are included. Training data for some languages is below 1,000 hours, and the five languages evaluated represent a small fraction of India’s linguistic diversity. All models train in FP32, which is resource-intensive and may limit reproducibility for lower-compute settings. No code or model checkpoints are publicly released.
Wiki Connections
- Multilingual TTS — this paper directly targets the multilingual low-resource setting, providing duration strategy comparisons across five typologically distinct Indian languages.
- Flow Matching — the audio model is a CNF trained with the flow-matching objective, following the Voicebox framework adapted for Indian language data.
- Zero-Shot TTS — zero-shot speaker-specific generation is the primary task, using a 3-second audio prompt to condition both duration and audio generation without speaker fine-tuning.
- Speaker Adaptation — the speaker-prompted duration predictor extracts speaker-specific prosodic patterns from a reference mel segment, enabling speaker adaptation without explicit speaker embeddings.
- Evaluation Metrics — the paper uses WER, Sim-o cosine similarity, QMOS, and SMOS, combining objective and human evaluation for a multilingual low-resource setup.
- Subjective Evaluation — human evaluations with 30 annotators per language and 10 annotations per sample provide SMOS and QMOS scores across Hindi, Tamil, and Bengali.
- VALL-E (VALL-E) — cited in related work as an autoregressive codec-based approach to zero-shot TTS; contrasted with the non-autoregressive flow-matching approach taken here.