DravidianLangTech 2026 · 2026 · Workshop

Bhavana Nali et al. (IIT Bhubaneswar) · → Paper · Demo: ✗ · Code: ✓

A controlled comparison of a cascaded ASR, machine translation and TTS pipeline against SeamlessM4T-v2 for speech-to-speech translation into and between Tamil, Telugu and Kannada, with both systems LoRA-adapted on identical splits; the cascade scores above the end-to-end model in every reported cell, though several margins are very small.

Problem

Speech-to-speech translation progress concentrates on high-resource languages. Tamil, Telugu and Kannada have limited parallel speech data, and no prior study compared the cascaded and end-to-end paradigms for these three languages under matched zero-shot and fine-tuned conditions. Mann-ki-Baat also lacks explicit Dravidian-to-Dravidian pairs, so a direct DL→DL evaluation set has to be built first.

Method

The study compares two pipelines on two directions: English to Dravidian (EN→DL) and Dravidian to Dravidian (DL→DL) (§2, Figure 1).

  • Cascaded Modular. ASR is Whisper Large-v3 for English input and IndicConformer for Dravidian input. IndicTrans2 translates the transcript (English-to-Indic or Indic-to-Indic configuration). Indic Parler-TTS synthesises the target speech from the translated text and a speaker prompt.
  • End-to-End. SeamlessM4T-v2 maps source speech to target speech without exposing intermediate text.

Both systems receive LoRA adapters (rank 16, scaling 32, dropout 0.1, under 1% trainable parameters). In the cascade, LoRA goes into Whisper, IndicConformer and IndicTrans2’s attention layers. In SeamlessM4T-v2 it goes into the cross-attention projections of the speech encoder and text decoder, trained with a speech-to-text cross-entropy objective. The TTS stage is frozen in all settings: Indic Parler-TTS is never fine-tuned because the datasets lack paired target-language audio at the needed scale (§2). No module on the speech-generation side of either system is trained, so fine-tuning changes only what text (or text-decoder state) reaches synthesis. Training runs 4,000 steps at batch size 8 and learning rate 1e-4, with checkpoint selection on validation BLEU (§3).

Data come from FLEURS (predefined splits, about 3,000 training utterances per language) and the Mann-ki-Baat subset of BhasaAnuvaad. For DL→DL pairs, Mann-ki-Baat sentences are aligned with LaBSE embeddings, cosine similarity and Hungarian one-to-one matching at a 0.6 threshold. This gives about 15,284 pairs, split 80/10/10 (§3, Appendix B).

Evaluation uses BLEU, chrF++ and COMET (WMT22 COMET-DA) for translation. For SeamlessM4T-v2, chrF++ and COMET come from its speech-to-text (S2TT) output head. Speech output is scored by back-ASR: IndicConformer re-transcribes the generated audio and WER and CER are computed. The paper does not state which text the back-ASR transcripts are scored against. A component diagnostic reports standalone ASR WER and MT BLEU on gold versus ASR transcripts for the cascade (§3, Table 3).

Key Results

Relative changes below are (new − old) / old, recomputed from the tables.

  • Cascade above E2E in every cell. Across Tables 1 and 2 (3 EN→DL languages and 6 DL→DL pairs, zero-shot and fine-tuned, FLEURS and Mann-ki-Baat, all ten metric columns), the cascade has higher BLEU, chrF++ and COMET and lower back-ASR WER and CER than SeamlessM4T-v2 in the same setting. The paper’s own wording (“across most settings”) is more cautious than its tables.
  • Headline examples. On FLEURS EN→Telugu, cascade BLEU is 18.96 vs. 15.72 zero-shot (+20.6%) and 19.18 vs. 16.84 fine-tuned (+13.9%) (Table 1). On FLEURS Tamil→Telugu it is 8.13 vs. 6.84 zero-shot (+18.9%) and 10.12 vs. 8.97 fine-tuned (+12.8%) (Table 2). Larger relative gaps occur elsewhere, e.g. zero-shot EN→Kannada on Mann-ki-Baat (8.70 vs. 6.24, +39.4%) (Table 1).
  • Near-ties. Several BLEU margins are within the paper’s own “below about 1 BLEU” caution zone. EN→Kannada on Mann-ki-Baat after fine-tuning is 7.03 vs. 7.01 (0.02 BLEU). EN→Tamil on Mann-ki-Baat zero-shot is 3.17 vs. 3.02. EN→Kannada on FLEURS after fine-tuning is 14.33 vs. 13.67. Telugu→Tamil on FLEURS after fine-tuning is 9.11 vs. 8.54 (Tables 1-2). No significance tests are reported.
  • Fine-tuning gains are uneven. For the cascade’s DL→DL pairs, BLEU rises by 24.5% to 39.3% on FLEURS and 22.0% to 37.2% on Mann-ki-Baat (Table 2). For EN→DL the gains are small on FLEURS: Tamil 13.75 to 14.43, Telugu 18.96 to 19.18 and Kannada 14.32 to 14.33. Cascade EN→Kannada on Mann-ki-Baat drops from 8.70 to 7.03 BLEU (−19.2%) after fine-tuning, while its chrF++ and COMET rise (Table 1). The diagnostic shows the same direction: EN→Kannada Mann-ki-Baat MT BLEU falls from 11.23 to 10.14 on gold transcripts and 9.72 to 8.29 on ASR transcripts (Table 3). The paper reports the overall effect as “approximately 20-30%” for Dravidian pairs and does not discuss the Kannada drop.
  • Back-ASR error rates are very high. Back-ASR WER ranges from 0.7687 to 1.0687 across the two tables. Values above 1.0 occur for all three E2E zero-shot EN→DL rows on Mann-ki-Baat, for E2E fine-tuned EN→Tamil on Mann-ki-Baat (1.0341), for E2E zero-shot EN→Tamil on FLEURS (1.0234) and for cascade zero-shot EN→Tamil on Mann-ki-Baat (1.0214) (Table 1). The paper attributes values above 1.0 to insertion-dominated errors (§3).
  • Error propagation. MT BLEU on ASR transcripts is below MT BLEU on gold transcripts in every Table 3 row; for EN→Tamil on FLEURS zero-shot it falls from 17.42 to 14.81. ASR fine-tuning lowers ASR WER in every row, e.g. English FLEURS (Tamil target) from 0.124 to 0.089 (Table 3).

The comparison controls training data and splits, but it is not fully symmetric. IndicConformer is both the cascade’s DL→DL ASR and the back-ASR judge, which the authors note may favour the cascade (§3, §6). E2E translation metrics come from SeamlessM4T-v2’s text head rather than its speech output.

Novelty Assessment

The contribution is empirical. Every component is an existing pretrained model, LoRA is applied with a single fixed configuration, and the evaluation reuses standard scorers. The new elements are the matched cascaded-versus-end-to-end comparison for three Dravidian languages and a LaBSE-aligned pseudo-parallel DL→DL test set from Mann-ki-Baat, which the authors note has not been human-validated. Because the TTS stage is frozen and speech output is judged only through back-ASR, the study says little about speech generation itself. Its findings concern recognition and translation quality upstream of synthesis.

Field Significance

low — This is a short single-family study with automatic-only evaluation, no significance tests and a frozen synthesiser. It adds a consistent data point that a cascade of strong language-specific modules can stay ahead of a general multilingual end-to-end model for low-resource Dravidian speech-to-speech translation. Its small margins and the shared back-ASR judge limit how far that point generalises.

Claims

  • supports: For low-resource language pairs, a cascade of language-specialised ASR, machine translation and TTS modules can outperform a general multilingual end-to-end speech-to-speech translation model under matched adaptation data.

    Evidence: With LoRA on identical FLEURS and Mann-ki-Baat splits, the Whisper/IndicConformer + IndicTrans2 + Indic Parler-TTS cascade scores above SeamlessM4T-v2 on BLEU, chrF++, COMET and back-ASR WER/CER in every EN→DL and DL→DL cell, e.g. FLEURS EN→Telugu BLEU 19.18 vs. 16.84 after fine-tuning. (§4, Tables 1-2)

  • complicates: The advantage of cascaded over end-to-end speech-to-speech translation can shrink to within noise for some language pairs and domains, so rankings need significance testing.

    Evidence: Fine-tuned EN→Kannada BLEU differs by 0.02 on Mann-ki-Baat (7.03 vs. 7.01) and by 0.66 on FLEURS (14.33 vs. 13.67); the authors performed no significance tests. (§4, §6, Table 1)

  • complicates: Back-ASR intelligibility scores for translated speech conflate recognition, translation and synthesis errors and can favour systems that share the judging ASR model.

    Evidence: Back-ASR WER ranges from 0.7687 to 1.0687, exceeding 1.0 for several EN→DL settings; IndicConformer scores both systems while also serving as the cascade’s DL→DL ASR, which the authors flag as a possible bias. (§3, §6, Tables 1-2)

  • supports: ASR errors propagate measurably into downstream translation in cascaded speech-to-speech systems, and adapting the ASR stage narrows the loss.

    Evidence: MT BLEU on ASR transcripts is below MT BLEU on gold transcripts in every diagnostic row (EN→Tamil FLEURS zero-shot 17.42 vs. 14.81); LoRA lowers ASR WER in every row (English FLEURS 0.124 to 0.089). (§4, Table 3)

  • complicates: Parameter-efficient domain adaptation of translation components does not uniformly improve speech-to-speech translation quality.

    Evidence: Cascade DL→DL BLEU rises 22.0% to 39.3% after LoRA, but EN→Kannada on Mann-ki-Baat drops from 8.70 to 7.03 BLEU, with MT BLEU on gold transcripts also falling from 11.23 to 10.14. (§4, Tables 1-3)

Limitations and Open Questions

Warning

Speech generation is not studied directly. Indic Parler-TTS is frozen, no human listening test is run, and speech output is judged only by back-ASR WER/CER using IndicConformer, which is also part of the cascade. Naturalness, prosody and speaker quality of the translated speech are unmeasured.

  • No significance testing; the authors ask readers to treat differences below about 1 BLEU with caution, which covers several of the cascaded-versus-E2E margins (§6).
  • The DL→DL Mann-ki-Baat pairs come from automatic LaBSE alignment and are not human-validated (§6).
  • GPU limits restricted training to a subset of Mann-ki-Baat, and FLEURS (read speech) and Mann-ki-Baat (broadcast speech) differ in domain (§6).
  • The paper does not state the reference text for back-ASR WER/CER, or whether BLEU is computed on translated text or back-ASR transcripts; for the E2E system, chrF++ and COMET come from the S2TT text head rather than its generated speech (§3).
  • Only one LoRA configuration is tried, and the cascade EN→Kannada Mann-ki-Baat BLEU drop after fine-tuning is not analysed.
  • Whether adapting the TTS stage would change the comparison remains open (§6).

Wiki Connections

  • Speech-to-Speech — Provides a matched cascaded-versus-end-to-end comparison for speech-to-speech translation in three low-resource Dravidian languages, with the cascade ahead in every reported cell.
  • SeamlessM4T — The cited source for the end-to-end system; its v2 model is the LoRA-adapted E2E baseline that the cascade outperforms.
  • Seamless — Introduces SeamlessM4T-v2, the exact end-to-end model evaluated here.
  • Whisper — Whisper Large-v3 is the English ASR front end of the EN→DL cascade and receives LoRA adaptation.