AfricaNLP · 2026 · Workshop
Kausar Yetunde Moshood et al. (LyngualLabs / Oregon State University / Carnegie Mellon University) · → Paper · Demo: ✓ · Code: ✓
Evaluates four existing Yoruba TTS systems and tests the two most intelligible against Braille and human narration for delivering standardized exam questions to five visually impaired secondary-school students.
Problem
Computer-based testing has become the default exam delivery format in many educational systems, but it creates new accessibility barriers for visually impaired learners, especially where the local language is underrepresented in assistive technology. Yoruba is spoken by over 40 million people and has an official WAEC exam paper, yet almost no published work has examined whether Yoruba TTS systems are actually usable for delivering high-stakes exam content to blind students. Prior TTS research for Yoruba and other tonal, low-resource languages has focused on model-level intelligibility rather than on whether the resulting speech is fit for real assessment use, leaving a gap between reported TTS capability and demonstrated usability for the population that would benefit most.
Method
The study uses a three-phase within-subject design. Ten multiple-choice questions from the 2024 WAEC Yoruba exam paper were delivered to five visually impaired students (ages 15-18) through three modalities: human voice narration, Braille, and a custom TTS web interface, with each student experiencing all three formats and responding verbally.
Before the live exam phase, four publicly available Yoruba TTS models (Facebook’s MMS-TTS-Yor, YarnGPT, Tacotron 2, and YorubaTTS) were screened using Mean Opinion Score (MOS): four native Yoruba speakers rated each system on a 5-point Likert scale following the ITU-T P.800 protocol, and models scoring at or above 3.0 were considered acceptable for the live exam. Only MMS-TTS-Yor and YarnGPT met this threshold and were carried forward. For the TTS phase, exam questions were served through a Streamlit web application; each question’s text was passed to the Facebook mms-tts-yor model via Hugging Face’s AutoTokenizer/AutoModelForTextToWaveform classes, and the resulting waveform was played back in-browser. The interface is deployed publicly and the source code is released on GitHub. After all three phases, each student reported which delivery method they preferred and why, and this preference and accompanying qualitative feedback form the paper’s main evidence.
Key Results
MOS screening (Table 1) ranked the four systems as MMS-TTS-Yor (3.7, “best overall, moderately natural”), YarnGPT (3.2, “moderate”), YorubaTTS (2.5, “partially intelligible”), and Tacotron 2 (2.1, “poor clarity and hard to follow”); only the first two cleared the 3.0 acceptability bar set by the authors.
In the live exam comparison (Table 2), preference was lopsided: all five students (5/5) preferred Braille, citing independence, quiet operation, and no distractions; three of five found human narration clear but reported feeling dependent on another person; only one of five preferred the TTS system, citing interest in the technology but flagging robotic delivery, mispronunciation, and tonal inaccuracies as barriers to comprehension. The best-MOS system in screening (MMS-TTS-Yor) was also the system deployed in the live test, so the live-test result reflects a best-case TTS condition rather than an average one.
Novelty Assessment
The paper proposes no new TTS model, training method, or architecture; all evaluated systems are pre-existing (MMS-TTS-Yor, YarnGPT, Tacotron 2, YorubaTTS). Its contribution is empirical: a small but genuine human-subject comparison of Braille, human narration, and TTS for exam delivery to visually impaired Yoruba-speaking students, a use case and population that the authors state has not previously been studied for this language. The engineering component (a Streamlit-based TTS delivery interface with public demo and code) is an integration of existing tools rather than a novel system design. The main empirical finding, that a favorable MOS score for a tonal low-resource-language TTS system does not translate into student preference or usability in a high-stakes exam setting, is a useful, if narrowly scoped, data point given the small sample (5 students, 10 questions, one exam paper).
Field Significance
low — This is a small-scale (5-participant) applied user study rather than a benchmark or architectural contribution, so its evidentiary weight is limited. Its value lies in documenting, with real visually impaired end users rather than automated or expert-only ratings, that current Yoruba TTS systems are not yet adequate substitutes for Braille in exam delivery, and in surfacing MOS-preference divergence as a concrete measurement caveat for accessibility-oriented TTS evaluation in tonal, low-resource languages.
Claims
- complicates: A subjective quality score (MOS) that clears a conventional acceptability threshold does not guarantee that a text-to-speech system is usable in a real, high-stakes deployment context for its target population.
Evidence: MMS-TTS-Yor scored 3.7/5 MOS in listener screening, above the paper’s own 3.0 acceptability cutoff, yet in the live exam-delivery test only 1 of 5 visually impaired students preferred TTS overall, citing robotic and unclear output. (§3.6.1, §4.1, Table 1, Table 2)
- complicates: Text-to-speech systems for tonal, low-resource languages remain prone to flat prosody and mispronunciation that materially affect comprehension in accuracy-critical tasks such as multiple-choice exam delivery.
Evidence: All four evaluated Yoruba TTS systems were reported to struggle with tone handling; the two selected for live testing were still described by students as robotic and hard to follow, with tonal inaccuracies raised as a specific complaint. (§2.2, §5.1)
- supports: For visually impaired users completing accuracy-sensitive tasks, an established non-speech assistive modality can be strongly preferred over synthetic speech even when the speech system has cleared a minimum intelligibility bar.
Evidence: All five participants ranked Braille as their top-preference delivery method over both TTS and human narration, citing independence and freedom from distraction. (§4.1, Table 2)
- refines: Automated or small-panel MOS evaluation of TTS systems for underrepresented languages should be interpreted as a screening signal rather than as a proxy for end-user acceptability in the deployment context that matters.
Evidence: The authors explicitly note that “many TTS models today are built and tested using automated metrics, but this study shows that actual student experience tells a very different story,” based on the mismatch between MOS ranking and live-test preference. (§6.1)
Limitations and Open Questions
Warning
The live-exam evidence comes from five students answering ten multiple-choice questions with a single TTS system (MMS-TTS-Yor) drawn from one exam paper; the study is not designed to support quantitative generalization, only a qualitative signal about current system adequacy.
The MOS screening used only four native-speaker raters per system and did not report inter-rater agreement. The study does not report objective intelligibility or comprehension-accuracy metrics (e.g., answer correctness by modality), relying instead on stated preference and qualitative feedback. The authors themselves frame the core gap as a lack of tone-aware, user-centered Yoruba TTS training data and call for larger, higher-quality open-source Yoruba speech corpora and closer co-design with educators, accessibility centers, and Yoruba speakers as future work.
Wiki Connections
- Evaluation Metrics — Uses MOS under the ITU-T P.800 protocol as a screening metric for selecting Yoruba TTS systems, then documents a divergence between that automated-style listener score and downstream user preference.
- Subjective Evaluation — Combines a small-panel MOS listening test with a separate human-subject preference study (five visually impaired students comparing TTS, Braille, and human narration), grounding TTS quality assessment in real end-user experience rather than only expert ratings.