arXiv · 2026 · Preprint

Sivateja Trikutam (Independent Researcher) · → Paper · Demo: ✓ · Code: ✓

Compresses a 337M-parameter Hindi flow-matching TTS teacher (IndicF5, an F5-TTS adaptation) into 131M-249M students by keeping the teacher’s width, removing evenly spaced transformer blocks in small stages, and re-fine-tuning on about 17.6 hours of teacher-generated audio, with each stage gated by an ASR word-error-rate check.

Problem

Open Hindi TTS is split between large, high-quality models (IndicF5 at about 350M parameters, Indic Parler-TTS at about 880M) and small models of noticeably lower quality (MMS-TTS at about 36M, Kokoro at about 82M with weak Hindi coverage). The paper targets the middle range of roughly 100-190M parameters for on-device and low-cost deployment. Training a small model from scratch is not an option under an in-house data budget: the author’s earlier 79M flow-matching model, trained from random initialization on the same 17.6 hours of audio, produced unintelligible speech (§1; no metrics are reported for this attempt).

Method

The teacher is IndicF5, a DiT flow-matching model (dimension 1024, 22 blocks, 16 heads, text dimension 512, 4 convolutional text layers) that predicts a 100-channel mel spectrogram at 24 kHz, with a 13.6M Vocos vocoder; the flow transformer has 337.1M parameters (§3.1). A narrower student could not reuse the teacher’s tensors, so the student keeps every dimension fixed and only reduces the number of transformer blocks. All non-block tensors (input embedding, text encoder, time embedding, rotary embedding, output norm and projection) therefore copy one-to-one, and the retained blocks are an evenly spaced subset of the teacher’s 22 with both endpoints kept (§3.2). Each block costs about 14.7M parameters on top of a fixed cost of about 13.8M, giving 102.0M, 131.4M, 190.2M and 248.9M students at depths 6, 8, 12 and 16 (Table 1).

Before any training, the author runs pruned but untrained students and uses output RMS amplitude as a proxy for whether the remaining velocity field still integrates to speech (Table 2). The descent then proceeds in stages (22 → 16 → 12 → 8 → 6 blocks), each pruned from the previous fine-tuned student and re-fine-tuned with the standard conditional flow-matching objective: learning rate 5×10⁻⁵, 500-step warmup, dynamic batches of 24,000 mel frames, an Euler ODE solver and EMA weights (§3.4). The distillation signal is sequence-level: the training corpus is 9,999 utterances (about 17.6 h) synthesized by the teacher from real Hindi text plus templated sentences covering numbers, dates and domain phrases, tokenized with IndicF5’s full 2,545-token character vocabulary (§3.5). There is no logit or feature matching against the teacher. A stage is only pruned further once it recovers to near the previous stage’s WER, measured by IndicWav2Vec-Hindi on one known and one unseen sentence with two stochastic generations each (§3.6). Inference conditions on a reference speaker mel, so the system clones a single reference voice.

Key Results

On the gated two-sentence check, the 249M (depth 16) student transcribes the unseen sentence at WER 0.00, the 190M student at 0.00-0.06 and the 131M student at 0.06-0.12 (Table 3). At 102M (depth 6) the student garbles and repeats words on long sentences (fresh WER 0.41-0.59; about 0.11 on short ones) even after 150 epochs and 42.9k steps, more than the 28.6k steps given to the 131M model; the author attributes this capacity cliff to the data budget (§4). Known-sentence WER is higher than fresh-sentence WER at every depth (0.18 for 249M up to 0.29-0.41 for 102M); the paper does not explain this pattern. The teacher is not included in Table 3.

Pruning tolerance without retraining is sharp: removing 6 of 22 blocks leaves untrained output RMS at 0.073 (“near-teacher; speaks”), while keeping 11 or 6 blocks drops it to 0.005 and 0.0016 (Table 2). RMS measures loudness, not intelligibility, so this table shows the field collapsing rather than how well the 16-block model speaks.

On the independent benchmark of 50 unseen FLEURS hi_in sentences (10-22 words), the teacher scores best of the three systems on WER (0.098), CER (0.03), SIM (0.784) and UTMOS (3.79) (Table 4). The 190M student reaches WER 0.170, CER 0.077, SIM 0.750 and UTMOS 3.65, so its WER is 0.072 higher (worse) than the teacher’s, while it keeps about 96% of the teacher’s UTMOS and SIM at 56% of the parameters. Its median RTF on an Apple M4 (fp32, MPS) is 3.13 against the teacher’s 5.56, about 1.8× faster. Against MMS-TTS-hin (36M), the student has lower WER (0.170 vs. 0.195) and higher UTMOS (3.65 vs. 3.54), but a higher CER (0.077 vs. 0.066). MMS-TTS-hin is far faster (RTF 0.31 on CPU, measured on different hardware), and its SIM of 0.278 is not comparable because it is a fixed-voice model without cloning (Table 4 footnote).

Novelty Assessment

The contribution is engineering integration. Layer dropping with a re-fine-tune (LayerDrop, Poor Man’s BERT, ShortGPT) and sequence-level distillation on teacher outputs are established techniques. What the paper adds is applying them to a flow-matching DiT TTS model, running the prune in small steps with an ASR gate, and measuring how much depth the untrained velocity field survives. The training objective and architecture are those of the teacher, unchanged. The paper argues that gradual descent is necessary but does not show it directly: it has no trained one-shot-prune control at the same depth, and no from-scratch control at a matching size (the from-scratch attempt is a different 79M model with no reported metrics). The deployment lessons in §5.2 (mel filterbank drift across torchaudio versions, rotary-embedding differences across x-transformers versions, and a silent non-strict load of torch.compile checkpoints) are useful practical notes rather than research findings. Releasing the weights, per-sentence benchmark outputs and scripts on Hugging Face is a practical strength.

Field Significance

Low. This single-author, single-language paper is evaluated with automatic metrics only. It gives a concrete, reproducible recipe and a parameter-versus-intelligibility curve for compressing a flow-matching TTS model by depth pruning when data is scarce, which can serve as a reference point for compact low-resource TTS. It does not introduce a new method.

Claims

  • supports: Initializing a compact generative TTS student from a subset of a larger teacher’s transformer blocks makes compression feasible under data budgets where training a small model from random initialization fails.

    Evidence: A 79M flow-matching model trained from scratch on about 17.6 h produced unintelligible speech, whereas the depth-pruned warm start begins stage A at loss near 1.0 (vs. 2-4 from random initialization), is intelligible by step 2,000, and reaches fresh-sentence WER 0.00 at 249M; the from-scratch run is a different size and has no reported metrics. (§1, §3.4, Table 3)

  • refines: Flow-matching TTS transformers tolerate removal of roughly a quarter of their blocks with the velocity field intact, but larger single-step depth reductions collapse the output, which favours staged pruning with re-fine-tuning.

    Evidence: Untrained pruned students give output RMS 0.073 at 16/22 blocks (“near-teacher; speaks”), 0.005 at 11/22 and 0.0016 at 6/22; RMS is an amplitude proxy only, and no trained one-shot-prune control is reported. (§3.3, Table 2)

  • complicates: Under a fixed distillation corpus, compressed TTS students reach a capacity limit below which extra training no longer restores intelligibility on long utterances.

    Evidence: The 102M (depth 6) student still garbles long sentences (fresh WER 0.41-0.59) after 42.9k steps, more than the 28.6k steps used for the robust 131M student (fresh WER 0.06-0.12); attributing the limit to the data budget rather than the recipe is not tested with a larger corpus. (§4, Table 3, §7)

  • complicates: Distilled compact TTS students lose content accuracy faster than predicted naturalness or speaker similarity, and small gated test sets can hide the loss.

    Evidence: On 50 FLEURS sentences the 190M student keeps UTMOS 3.65 vs. 3.79 and SIM 0.750 vs. 0.784 relative to the teacher, but its WER rises from 0.098 to 0.170 and CER from 0.03 to 0.077, a gap much larger than on the two-sentence gated check (fresh WER 0.00-0.06). (§4.1, Table 3, Table 4)

  • complicates: Mismatches between training-time and inference-time feature extraction or model-math library versions can silently degrade neural TTS output without any load-time error.

    Evidence: A different torchaudio mel filterbank muffled output, a different x-transformers rotary-embedding implementation degraded all output, and a non-strict load of torch.compile checkpoints left the model at random initialization while it still produced audio; freezing the training filterbank (matching to about 10⁻⁷) and pinning library versions fixed the first two. (§5.2)

Limitations and Open Questions

Automatic-only, single-voice evaluation

There is no human listening test: quality rests on ASR-WER and predicted MOS (UTMOS), and the stage gates use only two sentences with two generations each. Training data is entirely teacher-generated from a single reference voice, and evaluation covers Hindi only, so the reported “high-quality voice” and the robustness of each operating point are not established beyond these automatic measures (§3.6, §6).

  • The necessity of gradual descent is argued from untrained-student RMS (Table 2) rather than from a trained comparison against a one-shot prune at equal depth.
  • The real-time claim for a 6 GB laptop GPU is not backed by a reported measurement; the only reported RTF for the student is 3.13 on an Apple M4 in fp32 (synthesis time ÷ audio duration, so slower than real time in that setting) (Table 4).
  • Attributing the 102M capacity cliff to the 17.6 h budget is a hypothesis; a larger teacher corpus is proposed as future work but not tested (§7).
  • The system clones only the single reference voice used in training data generation, so zero-shot cloning of unseen speakers is not evaluated.
  • A residual trailing artifact at utterance ends is reduced with an energy-based trim but not removed, and utterance-initial tokens remain difficult (§6).
  • Why known-sentence WER exceeds fresh-sentence WER at every depth is not discussed (Table 3).

Wiki Connections

  • Flow Matching — compresses a DiT flow-matching TTS model by depth pruning plus re-fine-tuning with the conditional flow-matching objective, and measures how much depth the velocity field tolerates.
  • F5-TTS — the base architecture of the IndicF5 teacher; the student inherits its DiT design, mel I/O and training objective.
  • Flow Matching for Generative Modeling — the conditional flow-matching objective used to re-fine-tune each pruned student.
  • Vocos — the vocoder that reconstructs waveforms from the student’s mel output, inherited unchanged from the teacher.
  • UTMOS — the predicted-MOS metric used in the FLEURS benchmark in place of a human listening test.
  • ECAPA-TDNN — the speaker-embedding model used to compute SIM against the reference voice.
  • Parler-TTS — cited as the basis of Indic Parler-TTS, one of the large open Hindi models motivating the compact-model target; not evaluated as a baseline.