arXiv · 2026 · Preprint
Maltais et al. (Mila - Quebec AI Institute) · → Paper · Demo: ? · Code: ?
Introduces NaijaS2ST, a parallel speech corpus pairing English (Nigerian and British accents) with Hausa, Igbo, Yorùbá, and Nigerian Pidgin, and uses it to benchmark cascaded, end-to-end, and audio-LLM approaches to speech-to-text and speech-to-speech translation in both directions.
Problem
Existing Nigerian-language speech resources mostly target ASR and TTS, and speech translation evaluations for African languages are largely restricted to one direction (XX to English) and a single accent. Without parallel, accent-diverse speech in both directions, the field cannot compare cascaded, end-to-end, and LLM-based translation paradigms on authentic low-resource data, and S2ST in particular (which also requires a speech generation stage) has almost no coverage.
Method
The dataset covers Hausa, Igbo, Yorùbá, Nigerian Pidgin (Naijá), and English in two accents. Text was assembled from NTREX, SSA-MT, and MAFAND plus newly collected VOA sentences (500 Nigerian-context and 500 UK-context English sentences) to limit contamination, with Igbo and Naijá evaluation data supplemented by translation. Volunteers recorded the sentences via the Telegram app at 48 kHz (target SNR of at least 30 dB). Each volunteer recorded 250 utterances; 72 volunteers covered Hausa, Igbo, and Yorùbá, and 32 plus 8 more volunteers recorded Nigerian-accented English. Quality control sampled 3-5 utterances per reader; problematic dev and test recordings were fully re-recorded, whereas the train split was only filtered (Table 2). Train splits are roughly 55-61 hours per language after QC.
The benchmark compares three paradigms. Cascaded systems use Omnilingual-ASR 1B followed by NLLB-200 3.3B or a few-shot-prompted Tiny-Aya-Global 3B; for S2ST they add Gemini 2.5 Flash TTS. End-to-end systems use SeamlessM4T (V2 for S2TT, the older large model for S2ST fine-tuning) in zero-shot, per-language fine-tuned (Mono-FT), and multilingual fine-tuned (Multi-FT) settings; Hausa is mapped to an Arabic proxy language token, and the English-to-LRL S2ST direction is omitted because the model only produces speech for high-resource languages. Audio LLMs (Gemini 2.5, Gemini 3.1, GPT-Audio 1.5) are prompted zero-shot or with five few-shot examples; since Gemini lacks native speech output, its S2ST is a cascade of its own S2TT output plus Gemini 2.5 TTS. S2TT is scored with SSA-COMET and ChrF; S2ST outputs are transcribed with Omnilingual-ASR and scored with the same text metrics (ASR-COMET, ASR-ChrF, ASR-SpBLEU). The authors also fine-tune Omnilingual-ASR on the NaijaS2ST train and dev sets (Naija-Omni) to reduce ASR bias against Nigerian-accented speech during evaluation.
Key Results
For S2TT, Gemini 3.1 with few-shot prompting is best overall (XX to English SSA-COMET average 63.8, versus 55.8 for SeamlessM4T Mono-FT and 49.2 for the Omnilingual-ASR plus NLLB cascade) (Tables 3, 5). Zero-shot SeamlessM4T is highly uneven (Hausa 14.6, Igbo 20.6, Yorùbá 57.0 into English). In the English-to-LRL direction its SSA-COMET looks moderate for Igbo and Yorùbá but ChrF is much lower, and manual inspection showed many outputs remained in the source language. Fine-tuning fixes much of this: Mono-FT is better into English, while Multi-FT is better from English (62.2 versus 45.2 average SSA-COMET). GPT-Audio 1.5 is weaker and less stable than Gemini.
For S2ST, translation quality dominates. Into English, Gemini 2.5 few-shot plus TTS gives the best Hausa result (58.5 ASR-COMET versus 51.5 for the cascade) but the cascade is close for Igbo and Yorùbá; English-to-LRL favours the audio-LLM pipeline (e.g. Hausa 44.2 versus 37.1) (Table 6). The multilingual fine-tuned SeamlessM4T is far behind (Hausa to English 23.5). Output accent has little effect into English, but British-accent TTS scored slightly higher than Nigerian-accent TTS even with the accent-tuned ASR, and Naija-Omni improves scores over base Omnilingual-ASR (Table 7). The comparison is not fully like-for-like: audio LLMs are proprietary and few-shot prompted, while Seamless is fine-tuned on the in-domain data.
Novelty Assessment
The contribution is data and benchmarking, not modelling. The corpus is new in providing genuinely recorded, multi-accent, bidirectional parallel speech for Hausa, Igbo, Yorùbá, and Pidgin with English, with fully re-recorded clean dev and test sets. The benchmark uses off-the-shelf systems and standard fine-tuning recipes; the accent-tuned ASR evaluator and the observation about metric disagreement under wrong-language output are useful by-products rather than methodological advances.
Field Significance
Low. The dataset and baseline table are a useful reference point for low-resource speech translation, where S2ST coverage is thin. Its bearing on speech generation is indirect: the synthesis stage is off-the-shelf Gemini TTS, the generated speech is scored only through ASR transcripts and text metrics, and no human listening evaluation is reported. The accent-bias and metric-disagreement findings are useful by-products rather than advances in generation.
Claims
- supports: Translation quality, rather than the speech synthesis stage, is the main bottleneck of speech-to-speech translation for low-resource languages.
Evidence: Cascaded and audio-LLM pipelines share the same TTS yet differ mostly according to the translation component, and the weaker translation of the fine-tuned end-to-end model propagates into its generated speech (far lower ASR-COMET than both). (§5.2, Table 6)
- complicates: Embedding-based translation metrics can mask wrong-language outputs, so low-resource translation evaluation needs several complementary metrics.
Evidence: Zero-shot SeamlessM4T English-to-Igbo and English-to-Yorùbá obtain moderate SSA-COMET (53.1, 55.4) but much lower ChrF (24.5, 14.2) because many outputs remained in the source language. (§5.1, Tables 3, 4)
- complicates: Automatic scoring of generated speech through ASR transcripts is biased by the ASR system’s accent robustness and by the accent of the synthesized speech.
Evidence: Replacing base Omnilingual-ASR with an accent-fine-tuned version raised ASR-COMET for both Nigerian- and British-accented TTS output, and British-accent output still scored slightly higher than Nigerian-accent output. (§5.2, Table 7)
- refines: Whether monolingual or multilingual fine-tuning is better for adapting a multilingual translation model to low-resource languages depends on translation direction.
Evidence: Mono-FT gave the best SSA-COMET into English (e.g. Yorùbá 60.3), whereas Multi-FT was better from English for Hausa and Igbo and roughly tied for Yorùbá (62.2 versus 45.2 average, driven by the Hausa collapse of Mono-FT at 12.1). (§5.1, Table 3)
Limitations and Open Questions
The evaluation is offline and ignores latency, streaming, and inference cost, which matter for audio LLMs. Audio-LLM prompting was limited to a single few-shot setup. S2ST results rely on text-level metrics computed from ASR transcripts, and no human listening evaluation of the generated speech is reported. The English-to-LRL end-to-end S2ST direction was not evaluated, and the S2ST audio for Hausa, Igbo, and Yorùbá targets comes only from Gemini TTS in the cascades. Train-split recordings were not re-recorded after QC. One minor reporting slip: the GPT-Audio 1.5 few-shot XX to English average in Table 5 (38.5) does not match the mean of its three per-language scores (about 33.5).
Wiki Connections
- Speech-to-Speech — Provides a bidirectional S2ST benchmark and compares cascade, end-to-end, and audio-LLM paradigms for low-resource African languages.
- Evaluation Metrics — Shows how SSA-COMET, ChrF, and ASR-based scoring of generated speech disagree or carry accent bias.
- SeamlessM4T (SeamlessM4T) — Serves as the end-to-end baseline that is evaluated zero-shot and fine-tuned (Mono-FT, Multi-FT).
- Omnilingual ASR (Omnilingual ASR) — Used as the ASR front end of the cascades and as the evaluator of S2ST outputs, with an accent-tuned variant.
- AudioPaLM (AudioPaLM) — Cited as an LLM-based speech translation precursor motivating the audio-LLM paradigm.
- LLaMA-Omni (LLaMA-Omni) — Cited as an example of LLM-based speech-to-speech modelling in related work.
- Seamless (Seamless) — Cited for expressive and streaming translation as context for the benchmark’s scope.
- CoVoST 2 (CoVoST 2) — Cited as an established speech translation benchmark that NaijaS2ST complements for African languages.