AmericasNLP 2025 · 2025 · Workshop
Daniel Menendez et al. (Pontificia Universidad Católica del Perú) · → Paper · Demo: ? · Code: ?
The first TTS model and speech dataset for Shipibo-Konibo, a low-resource Panoan indigenous language of Peru, using Tacotron 2 fine-tuned from a Latin American Spanish checkpoint and HiFi-GAN, achieving MOS 4.01 and IR 88.56% with only 4h 37m of training data.
Problem
Shipibo-Konibo is an indigenous Amazonian language spoken by ~40,000 people in Peru, with at least two orthographic traditions and very limited digital or NLP resources. No TTS model or structured speech dataset existed for this language. The challenges include: no existing dataset to draw from, the need to build a corpus from scratch under a limited budget, selection of appropriate model architectures for low-resource settings, and determining which evaluation metrics apply when high-quality reference audio is unavailable (ruling out objective metrics like PESQ and POLQA).
Method
Dataset creation: 3,025 sentences were compiled from the Shipibo-Konibo translation of The Little Prince and Peruvian Ministry of Education bilingual educational materials (texts post-2015 alphabet normalization). Audio was recorded by a single native speaker with prior voice documentation experience over three months (sessions up to 2 hours each). Audio was sampled at 22.05 kHz, trimmed of leading/trailing silences, volume/speed/text normalized per Tacotron 2 requirements. Final corpus: 3,025 sentences, 4h 37m total duration, 5.1s average sentence length.
Model: Tacotron 2 (encoder-attention-decoder architecture) was selected based on prior success in similar low-resource scenarios with datasets under 3 hours. A pretrained Latin American Spanish Tacotron 2 checkpoint (from HuggingFace, via Cedillo 2023) was fine-tuned using transfer learning on the Shipibo-Konibo corpus. Training: Google Colab with Nvidia A100 GPU, 225 epochs, batch size 16, Adam optimizer (lr=3×10⁻⁴, β₁=0.9, β₂=0.999, weight decay=1×10⁻⁵), trained until validation loss of 0.1434.
Vocoder: HiFi-GAN (GAN-based), trained on Tacotron 2-generated spectrograms. Initialized from a pretrained universal female voice model for faster convergence. Training converged after 34 epochs.
Evaluation: 400 additional sentences from Koshi Shinanya Ainbo (a pre-2015 Shipibo-Konibo book) served as the test set. 26 native Shipibo-Konibo speakers (11 men, 15 women, all under 30, university-educated) evaluated both natural and synthetic speech. Intelligibility Rate (IR) was measured as the percentage of correctly transcribed words. MOS was rated 1–5 on clarity, naturalness, and fluency. 25% of the 20-audio evaluation set were natural speech samples (included as a blind reference).
Key Results
- Intelligibility Rate: Natural voice 83.45% vs. Synthetic 88.56% (synthetic exceeds natural)
- MOS: Natural voice 3.75 ± 1.2 vs. Synthetic 4.01 ± 1.09 (synthetic exceeds natural on both metrics)
- Encoder-decoder alignment was stable from the first epoch, indicating the transfer learning initialization was effective
Qualitative analysis identified recurring synthesis issues: ambiguous intonation on certain suffixes (titai, tiai, tian, wai) due to insufficient training examples, and pronunciation inconsistencies for specific phonemes (iki, nai, non, ani, ja, ea, baon, kon, xe, noa).
The unexpected result that synthetic speech outperformed natural speech on both IR and MOS is attributed to the native speaker’s years of residence in Lima, which introduced pronunciation variations and inconsistencies into the reference recordings.
Novelty Assessment
The primary contribution is applied rather than methodological: this is the first TTS system and speech corpus for Shipibo-Konibo. The architecture (Tacotron 2 + HiFi-GAN) is well-established. The novelty lies in demonstrating the viability of transfer learning from a related language (Latin American Spanish → Shipibo-Konibo) with minimal data, and in the systematic evaluation by native speaker judges. The work serves as a template for other Panoan and Amazonian indigenous languages. FastSpeech 2 and Transformer-TTS were considered but discarded due to inferior low-resource performance.
Field Significance
Moderate — This paper contributes the first speech corpus and TTS system for Shipibo-Konibo, serving as a documented instance of the cross-lingual fine-tuning recipe (related-language checkpoint → low-resource target) applied to an Amazonian indigenous language. The evaluation design, using 26 native-speaker judges with both IR and MOS, provides a transferable template for similarly resource-scarce settings where objective metrics are inapplicable.
Claims
- Transfer learning from a closely related language can enable intelligible TTS for an indigenous language with fewer than 5 hours of training data. (§4.1, §5.1)
- Subjective evaluation by native speakers is necessary when objective metrics such as PESQ and POLQA cannot be applied due to absent high-quality reference recordings. (§5.3)
- Pronunciation drift in a native speaker living outside the language community can cause natural speech to score lower on intelligibility than synthetic speech on standardised evaluation. (§5.4)
- GAN-based vocoders initialized from a universal pretrained model converge rapidly when the target speaker is similar to the pretraining distribution. (§4.2, §5.2)
Limitations and Open Questions
- Corpus is only 4h 37m from a single female speaker, limiting prosodic and phonetic diversity and voice coverage.
- Transfer learning from Spanish may introduce Spanish phonotactic biases that are not present in Shipibo-Konibo.
- Evaluation relied entirely on subjective metrics (no PESQ/POLQA due to lack of reference data); subjectivity and evaluator bias are acknowledged.
- Two writing conventions for Shipibo-Konibo exist; the 2015 standard was used, limiting applicability in communities using older orthographies.
- The speaker’s extended residence in Lima introduced pronunciation drift that complicates the natural vs. synthetic comparison.
- The model has not been tested on adaptation to other Panoan or Amazonian languages yet, though the framework is intended to generalize.
Wiki Connections
- multilingual-tts — specifically the low-resource indigenous language subproblem
- gan-vocoder — HiFi-GAN vocoder training and convergence
- transformer-enc-dec-tts — Tacotron 2 encoder-attention-decoder architecture
- speaker-adaptation — cross-lingual fine-tuning from Spanish as a form of language adaptation
- subjective-evaluation — MOS and IR evaluation by 26 native speakers
- evaluation-metrics — metric selection in the absence of high-quality reference recordings
- Scaling Under-Resourced TTS (Thai) — Thai low-resource TTS with GAN-based acoustic modeling (comparable low-resource setting)
- INTP — zero-shot multilingual intelligibility (related cross-lingual transfer work)