ComputEL · 2025 · Workshop
Chan and Hammerly (University of British Columbia) · → Paper · Demo: ? · Code: ?
A participatory design workshop with five Ojibwe teacher-learners surfaces community priorities for Indigenous language TTS deployment, revealing a gap between phonetic accuracy and contextual naturalness, and identifying representation, accessibility, and cultural alignment as the primary barriers to classroom adoption.
Problem
Existing low-resource TTS systems for Indigenous languages have been developed primarily from a technical standpoint, with limited investigation of whether and how language communities actually use these tools. The Border Lakes Ojibwe TTS feature on the Anishinaabemodaa platform had been available to teachers but was largely unused, and community consultation had not identified why. Standard quantitative evaluation is also difficult for Indigenous language TTS: the small L1 speaker population makes large-scale perceptual studies impractical, calling for alternative evaluation methods that centre community expertise.
Method
The study uses a participatory design (PD) methodology adapted from human-computer interaction research and Indigenous co-design practices. Two workshop sessions were conducted with a total of five Ojibwe teacher-learners, each recruited through the CultureFoundry mailing list for the Anishinaabemodaa platform. Participants first completed a pre-workshop Qualtrics questionnaire including a guided trial of the TTS feature: they entered six types of text input (single word, sentence, paragraph, question, emotion-conveying sentences, free choice) and rated the output on four-point Likert scales for accuracy, naturalness, and contextual appropriateness. The workshop itself (approximately two hours, conducted on Zoom) involved a structured brainstorm and lesson-plan co-design task using Zoom Whiteboard. Participants collaboratively designed lesson activities that incorporated the TTS feature, surfacing their practical priorities and perceived barriers through the design process itself. The TTS system under evaluation is the one described in Hammerly et al. (2023), built for Border Lakes Ojibwe and deployed as a web-based feature with a single male synthetic voice.
Key Results
The pre-workshop questionnaire (n=5 participants, 20 Likert responses across six task types) showed strong agreement that phonetic accuracy is a strength of the system: 11 “Strongly Agree” and 8 “Agree” responses for word accuracy across conditions, with only 1 “Disagree.” Contextual appropriateness was rated more negatively, particularly for emotion-conveying sentences (3 “Disagree” out of 4 responses for that sub-task). Participants consistently reported that the speech lacks natural rhythm and tone modulation, making it better suited for pronunciation reference than for conversational practice.
Four teacher priorities emerged from the workshop: (1) representation (diverse voice options beyond a single middle-aged male voice, including female and younger voices); (2) accessibility (feature discoverability, interface clarity, and internet access in rural areas); (3) encouraging language usage (independent practice, family involvement); and (4) inclusion (multimodal resources aligned with Universal Design for Learning). Both groups independently designed lesson activities (phrasebook and flashcard creation) that incorporate the TTS feature as a pronunciation reference.
Novelty Assessment
The primary contribution is methodological and community-facing: the paper demonstrates that participatory design is a viable evaluation approach for Indigenous language TTS when quantitative perceptual studies are not feasible at scale. The workshop protocol, combining a structured guided trial with a co-design lesson-planning task, offers a transferable template for similar low-resource Indigenous language communities. The paper does not propose technical improvements to the TTS system itself, nor does it introduce new evaluation metrics. The qualitative findings, while specific to Ojibwe, generalise as a set of community priorities (representation, accessibility, cultural alignment) that apply broadly to Indigenous language TTS deployment.
Field Significance
Moderate — This paper establishes that phonetic accuracy alone is insufficient for community adoption of Indigenous language TTS: naturalness, representation, and accessibility are distinct barriers that standard technical evaluation does not surface. It provides a replicable participatory design protocol for contexts where large-scale perceptual evaluation is not feasible, contributing a methodological model alongside its empirical findings.
Claims
-
supports: Quantitative accuracy ratings and qualitative community feedback reveal distinct dimensions of TTS quality that do not co-vary for low-resource Indigenous language systems.
Evidence: Participants rated phonetic accuracy highly (11 “Strongly Agree” / 8 “Agree” aggregated across tasks) while rating contextual appropriateness and naturalness much lower, and identified natural rhythm and tone modulation as absent from the system. (§6, Table 1)
-
complicates: Small L1 speaker populations make standard large-scale perceptual evaluation of Indigenous language TTS impractical, limiting how generalisable accuracy findings can be.
Evidence: Five participants constituted the entire available pool; the paper explicitly flags sample size as a limitation and uses qualitative PD methods as an alternative to quantitative MOS-style evaluation. (§Limitations)
-
supports: Community adoption of TTS tools for language education depends on factors beyond output quality, including interface discoverability, voice representation, and alignment with cultural pedagogy.
Evidence: The Ojibwe TTS feature had been available for teacher use but was unknown to participants before this workshop; teacher priorities surfaced in the PD sessions included voice diversity, accessible UI labelling, and offline download capability for rural use. (§7.1, §7.2)
-
refines: Participatory design workshops can serve as an evaluation method for Indigenous language TTS when quantitative listener studies are not feasible, eliciting both usability feedback and pedagogical priorities in a single session.
Evidence: The combined pre-workshop guided trial and co-design session produced both Likert-scale quality ratings and lesson-plan artefacts, with unprompted participant insights addressing all four research questions without formal prompting. (§4, §8.2)
Limitations and Open Questions
Warning
The study involves five participants from a single community and a single TTS system. The workshop findings, while rich, cannot be generalised quantitatively and are specific to Border Lakes Ojibwe and the Anishinaabemodaa platform context.
The online format (Zoom) was a logistical necessity given participant geography but was identified as a limitation by the researchers: in-person sessions with physical materials would better support rapport-building and co-design activities. The single synthetic voice (male, middle-aged) was consistently flagged as a representation gap; multi-speaker and voice-cloning extensions are noted as future directions but not evaluated. The evaluation does not cover speech intelligibility in real classroom settings or longitudinal usage data post-workshop.
Wiki Connections
- Evaluation Metrics — this paper foregrounds the limits of standard accuracy metrics for low-resource Indigenous TTS, proposing participatory design as a complementary evaluation mode.
- Subjective Evaluation — the pre-workshop guided trial applies four-point Likert scales for accuracy and naturalness judgements, offering a lightweight perceptual evaluation protocol adapted to a small-population context.