arXiv · 2026 · Preprint
Prathosh A P (Indian Institute of Science, Bengaluru) · → Paper · Demo: ✓ · Code: ✓
An experience report on building a Sanskrit chant (parayana) synthesizer by fine-tuning an Indic flow-matching TTS backbone on a single voice, with a Sanskrit-specific text frontend, meter-matched reference selection, and a negative result showing that a text-side prosody conditioner has no effect in a self-infilling flow-matching model.
Problem
Sanskrit recitation is a chanted register, not read speech. A faithful synthesizer must hold long vowels, sustain terminal visarga, keep retroflex, aspirated and sibilant contrasts, render dense consonant conjuncts, and follow the verse meter (vrutta). General-purpose and Indic TTS systems do not handle these, and almost no chant-domain training data exists. Indic backbones also apply Hindi-style schwa deletion to Devanagari input, which is wrong for Sanskrit.
Method
The system has four parts built around off-the-shelf components.
Text frontend. All input in any Brahmic script is transliterated to Kannada script before synthesis, because Kannada orthography keeps the inherent vowel and avoids schwa deletion in the Indic backbone. The mapping round-trips, so display text stays in Devanagari. A rule-based frontend then applies Sanskrit phonology: optional internal visarga sandhi, danda-final visarga handling by vowel length, homorganic (parasavarna) anusvara, explicit long vocalic r, a metathesis rule for two conjuncts the backbone renders poorly, and stripping of editorial parentheticals (§3, Appendix B). Visarga allophones (jihvamuliya, upadhmaniya) are learned acoustically, not substituted.
Meter scanning and reference selection. Each verse is scanned into a light/heavy (laghu/guru) syllable sequence and matched per quarter-verse (pada) to a meter signature calibrated from a reference bank. The backbone has no duration or pitch head, so the reference clip carries voice, melodic contour and pace. The system picks a reference with an exact per-quarter-verse weight match, the cleanest harmonics-to-noise ratio and a clean danda ending. Meter also sets the per-syllable duration budget and the caesura position. The “half-reference rule” prefers a clean half-verse reference of about seven seconds, repeated three times to about twenty seconds, over a full fifteen-second verse, which tends to swallow the first synthesized syllable (§5.2).
Backbone and vocoder. The production acoustic model is IndicF5, a multilingual Indic instance of the F5-TTS diffusion transformer (dimension 1024, depth 22, 16 heads, about 337M parameters) trained with conditional flow matching on masked mel infilling. It is fine-tuned on about five hours of the author’s own chant recordings (learning rate 1e-5, bfloat16, two GPUs, 100-band mel at 24 kHz). A short voice-steering retrain on 179 paired clips makes the voice more responsive to the reference, and this is the shipped voice. The BigVGAN vocoder is fine-tuned on the backbone’s mels and averaged into an exponential-moving-average soup (§6). Inference uses an Euler solver, sway sampling, speed factor 0.90 and classifier-free guidance around 3.0 for batch rendering (Appendix A).
Production QC. A post-render gate trims vocoder onset transients and tail ring. Forced-alignment QC was replaced by a duration gate, an internal-silence gate, best-of-N reseeding of flagged clips, recognition-based error detection, and source-text validation (§9).
The corpus was designed for meter and phoneme balance, with explicit coverage targets for retroflexes, long vowels, repha and hard conjuncts. Two meters (rucira, malini) were planned as held-out meters (§8.2). The paper does not report a synthesis result on these held-out meters.
Key Results
All quality figures are expert single-listener MOS estimates. The paper states that the Table 1 values are not directly comparable across rows. It treats them as a trajectory, not a controlled benchmark.
- Architecture lineage (Table 1). Four families were trained on the same single-speaker data. StyleTTS2 (70M) reached MOS 3.0 to 4.2 with muffled conjuncts. VITS2 (39.9M) is reported only as ”> Era 1”, with conjuncts still muffled. Matcha-TTS (18.2M) reached 4.2 to 4.3 with better conjuncts but no meter prosody. The IndicF5 flow-matching backbone (337M), fine-tuned on five hours, reached about 4.6 with all conjuncts rendered correctly, including retroflex aspirates. On the same informal scale, expert estimates put strong commercial systems at 4.0 to 4.6 and human recitation at about 4.6 to 4.7 (§12).
- Script routing. A Kannada-versus-transliteration ablation found that Kannada input did better (Appendix C, E30 and E32). No numbers are given.
- Prosody controllability (§7). A text-side conditioner encoding the syllable-weight pattern or a melodic (swara) template had a gradient norm near 0.02. Outputs with it on and off could not be told apart, and at high guidance its effect was about zero semitones. An implicit meter-conditioning on/off ablation changed results by +0.00 percent (Appendix C, E41). Pitch correlation was about 0.13 for the same reference with different verses, and about 0.40 to 0.54 for different references with the same verse. The paper reads this as the contour being driven by the text. Classifier-free guidance trades expressiveness against tremor.
- Vocoder. The Fourier-domain Vocos vocoder produced a long-vowel phase artifact that BigVGAN does not, so the vocoder choice matters for this domain. Residual wobble and jitter were traced to the acoustic model (§5.1). These findings come from listening, with no metrics.
- QC. Forced-alignment QC raised about 1,648 flags on about 10,220 clips. A manual sample found essentially none to be real defects. Real defects came mostly from errors in the source text (§9).
- Efficiency. The real-time factor is about 1.24 at 64 function evaluations and 0.63 at 32 (the serving setting). Peak inference memory is about 2.5 GB (§6).
- Deployments. The system rendered 5,183 verses (about 17.5 hours) of the Mahabharata Tatparya Nirnaya as 32 videos. It also rendered about 18,000 verses of the Srimad Bhagavatam for a mobile audio app (§10, §11, Table 3).
No baseline is evaluated under the same protocol, and no objective metrics (CER, conjunct accuracy, duration fidelity) are reported. The subjective-evaluation tag is not applied: the only listening evidence is a single expert listener, and the paper itself calls this informal and defers a multi-listener study.
Novelty Assessment
The paper calls itself an experience report rather than a new architecture, and that is accurate. The backbone, vocoder and flow-matching objective are used as published. The contributions are domain engineering: Kannada script routing to avoid schwa deletion, a Sanskrit phonology frontend, meter-keyed reference selection, a balanced chant corpus, and production QC practice. The most transferable finding is the negative result on text-side prosody conditioning in self-infilling flow-matching models, together with its mechanism: the context mel already reveals pitch, so the conditioner gets almost no gradient. The evidence is a set of internal probes on one voice and one backbone, not a controlled study. The released single-speaker chant dataset (about 5.3 hours) is a secondary contribution.
Field Significance
Low — The paper documents a single-voice, single-language chant system that is evaluated only by one expert listener, so its quality claims cannot be compared with other work. Its broader value is a mechanistic negative result: in self-infilling flow-matching TTS, text-side prosody embeddings can be inert, and the reference prompt is the only working prosody lever. It also shows that script routing and meter-matched prompts can adapt a multilingual Indic backbone to a low-resource liturgical register.
Claims
- contradicts: Adding a text-side prosody embedding to a self-infilling flow-matching TTS model gives explicit control over the melodic contour.
Evidence: A swara or syllable-weight conditioner on the IndicF5 backbone had a gradient norm near 0.02. Outputs with it on and off could not be told apart, and the effect at high guidance was about zero semitones. An implicit meter-conditioning ablation changed results by +0.00 percent. The paper attributes this to the context mel already revealing pitch during infilling. (§7, Appendix C)
- supports: In prompt-based flow-matching TTS without explicit duration or pitch predictors, the choice and construction of the audio reference is the main lever over pace and melodic contour.
Evidence: Meter-matched reference selection sets the prosodic template. A clean half-verse reference repeated to about 20 seconds improves the contour more than a full verse, which swallows the first syllable. The reference alone caused a 1.6 times swing in duration (E35). Different references with the same verse gave pitch correlations of about 0.40 to 0.54. (§5.2, §7, Appendix C)
- complicates: In multilingual Indic TTS backbones, the input script can change pronunciation rules, so a language written in a shared script may need to be routed through another orthography.
Evidence: Devanagari input made the Indic backbone apply Hindi-style schwa deletion to Sanskrit. Transliterating to Kannada avoided this, and a Kannada-versus-transliteration fine-tune ablation favored Kannada. (§3.1, §4, Appendix C)
- supports: A short single-speaker fine-tune of a large pretrained flow-matching backbone can overcome phonetic failures that smaller TTS models trained on the same low-resource data could not fix.
Evidence: StyleTTS2, VITS2 and Matcha-TTS trained on the same Sanskrit data all left conjuncts muffled or prosody missing. A five-hour fine-tune of the 337M IndicF5 backbone rendered all conjuncts, including retroflex aspirates, at an expert single-listener MOS of about 4.6. The paper notes that these MOS values are not comparable across rows. (§4, Table 1)
- complicates: Classifier-free guidance strength in flow-matching TTS trades prosodic expressiveness against pitch tremor.
Evidence: A low guidance value gave expressive but tremulous output, and a high value gave clean but flatter output, because both ride the same stochastic pitch and jitter. Batch rendering used guidance of about 3.0 to suppress tremor. (§7, Appendix A)
Limitations and Open Questions
The main limitation is evaluation. Every quality figure is an expert single-listener estimate with no blind baselines, confidence intervals or multi-listener study, and the Table 1 lineage values are not comparable across rows. The paper names conjunct accuracy, recognition-based CER, duration fidelity and a pitch-correlation table as missing objective measures.
- Verse-independent, designable prosody is out of reach on this backbone. It would need an explicit duration predictor and length regulator, which was designed but not built.
- The released weights are single-speaker. Chanting zero-shot in an arbitrary voice would need a multi-reciter chant fine-tune.
- Synthesis on the held-out meters (rucira, malini) is planned in the corpus design but not reported.
- Repeated syllables beyond a depth of about four cannot be recovered, and some rare artifacts are fixed per clip.
- Prose (gadya) passages needed a separate chunking heuristic because the model was not built for them.
- Earlier eras used a different (pilot) reciter from the production voice, which further limits the lineage comparison.
Wiki Connections
- Flow Matching — The production system fine-tunes a diffusion-transformer flow-matching backbone and reports that text-side prosody conditioning gets no gradient under self-infilling.
- Prosody Control — Meter-matched reference selection is the working prosody lever, and probes show a text-side prosody conditioner is inert.
- Speaker Adaptation — A five-hour single-speaker fine-tune plus a 179-clip voice-steering retrain adapts a multilingual Indic backbone to the author’s chant voice.
- GAN Vocoder — BigVGAN is fine-tuned on the backbone’s mels and checkpoint-averaged, and is preferred over a Fourier-domain vocoder that caused long-vowel phase artifacts.
- F5-TTS — The production backbone is IndicF5, an Indic instance of the F5-TTS architecture, fine-tuned for Sanskrit chant.
- BigVGAN — Used as the production vocoder after fine-tuning on the backbone’s 100-band mels.
- Vocos — Compared as an alternative vocoder and rejected for a long-vowel phase artifact.
- CosyVoice — Probed as a semantic-token zero-shot synthesizer for chant control and dropped for overfitting on the small chant corpus.
- Parler-TTS (Natural Language Guidance) — Its description-conditioned approach (via Indic Parler-TTS) was probed for prosody steering and dropped for content drift.
- Flow Matching for Generative Modeling — Provides the conditional flow-matching objective that the backbone is trained with.