ABJAD-NLP 2026 · 2026 · Workshop
Ahmed Khamis et al. (Georgia Institute of Technology, Nile University) · → Paper · Demo: ? · Code: ✓
Introduces NileTTS, a 38-hour Egyptian Arabic speech dataset built through a fully synthetic pipeline (LLM-generated text, AI-narrated audio, automatic transcription and diarization), and fine-tunes XTTS v2 on it to substantially improve intelligibility and speaker similarity for this under-resourced dialect.
Problem
Neural TTS has reached near-human naturalness for high-resource languages, and multilingual zero-shot systems like XTTS and VALL-E now cover many languages from a single model. But this coverage is uneven within language families: Arabic TTS resources concentrate on Modern Standard Arabic (MSA) and Gulf dialects, leaving Egyptian Arabic, spoken natively by over 100 million people and the most broadly understood Arabic variety due to Egypt’s media presence, severely under-resourced. Prior Egyptian Arabic efforts exist (a 20.5-hour single-speaker news corpus; roughly one hour of Egyptian Arabic within a larger multi-dialect benchmark) but are limited in scale, speaker diversity, domain coverage, or public availability. Building larger dialectal TTS resources through manual recording is costly, motivating a synthetic alternative.
Method
The core contribution is a fully synthetic data-construction pipeline that requires no human recording session. First, large language models (Gemini and Claude variants) are prompted to write PDF-style reports in authentic Egyptian Arabic across three domains (medical, sales and customer service, and general conversation), with instructions to avoid Modern Standard Arabic; prompts were kept deliberately simple, with dialect authenticity checked only by manual inspection. Second, each report is converted to speech using NotebookLM’s audio-generation feature, which renders a podcast-style two-speaker (one male, one female) discussion of the report content in natural Egyptian Arabic, each file roughly 10–15 minutes long. Third, the audio is transcribed and segmented into utterance-level chunks with Whisper Large, using its word-level timestamps to cut at natural speech boundaries and discarding segments under one second. Fourth, speaker identity is assigned via unsupervised diarization: ECAPA-TDNN speaker embeddings (from SpeechBrain) are extracted per segment, clustered with K-means (k=2) into per-speaker centroids, and each segment is labeled by cosine similarity to the nearest centroid. Manual quality control then reviewed the full Sales and Customer Service category plus samples of the other two domains for transcription accuracy, speaker-label consistency, and audio artifacts, correcting or discarding flawed segments. The result, NileTTS, contains 9,521 utterances (38.1 hours) split 90/10 into training and evaluation sets with no report-level overlap and unseen evaluation topics.
The dataset is used to fine-tune XTTS v2 (Coqui), a GPT-style autoregressive TTS model that generates discrete audio tokens later decoded to waveform by a DVAE, and which supports zero-shot voice cloning from a short reference clip. Only the GPT component is fine-tuned; the DVAE tokenizer is kept frozen. Training largely follows Coqui’s default fine-tuning recipe (30 epochs planned, batch size 2 with gradient accumulation 8, learning rate 5e-6, AdamW), with added Weights & Biases tracking and periodic WER/CER/speaker-similarity evaluation for checkpoint selection. Training was stopped early at 13 epochs (55,719 steps) once metrics plateaued, and the checkpoint at step 34,289 (epoch 8) was selected based on a balance across metrics, confirmed by informal listening review of 50 sampled synthesized utterances.
Key Results
The fine-tuned NileTTS model is compared against the pretrained XTTS v2 baseline (trained on other Arabic dialectal data, primarily MSA and Gulf) on the held-out Egyptian Arabic evaluation set. NileTTS reduces Word Error Rate from 26.8% to 18.8% (29.9% relative reduction) and Character Error Rate from 8.1% to 4.1% (49.4% relative reduction), and raises speaker similarity (cosine similarity of ECAPA-TDNN embeddings) from 0.713 to 0.755 (+5.9%). All three metrics are computed on the same held-out split using the tools embedded in the data pipeline itself (Whisper Large for re-transcription, ECAPA-TDNN for embeddings), which the authors flag as a possible source of self-consistency bias rather than an independent check. No formal MOS or listening-test study was conducted; the 50-sample listening pass served only to validate checkpoint selection.
Novelty Assessment
The contribution is a new dialectal speech dataset and a reproducible synthetic construction pipeline, not a new model architecture. XTTS v2 is used unmodified except for standard supervised fine-tuning of its GPT component; every pipeline stage (LLM text generation, NotebookLM audio rendering, Whisper transcription, ECAPA-TDNN diarization) reuses existing off-the-shelf tools. The genuine novelty is combining these components into an end-to-end recording-free path from topic prompt to labeled (speaker, audio, text) TTS training data for a dialect that previously lacked resources at this scale, and demonstrating with a real fine-tuning run that the resulting synthetic corpus transfers to measurable intelligibility and voice-similarity gains.
Field Significance
moderate — the paper adds a concrete, publicly released resource (dataset, pipeline, and fine-tuned model) for a widely spoken but under-resourced Arabic dialect, and demonstrates that a fully synthetic, recording-free data pipeline can produce data sufficient to meaningfully improve an existing multilingual zero-shot TTS model on that dialect. Its scope is narrow: two synthetic voices, three domains, a single workshop-scale evaluation, and metrics computed with tools that overlap the data-generation pipeline itself.
Claims
- supports: A fully synthetic construction pipeline, LLM-generated text converted to speech by a generative audio tool with no human recording, can produce TTS training data sufficient to measurably improve an existing multilingual TTS model’s intelligibility and speaker similarity for an under-resourced dialect.
Evidence: Fine-tuning XTTS v2 on NileTTS (built entirely from LLM-generated Egyptian Arabic text rendered by NotebookLM’s audio generation, then auto-transcribed and diarized) reduces WER from 26.8% to 18.8% and CER from 8.1% to 4.1%, and raises speaker similarity from 0.713 to 0.755, relative to the pretrained baseline. (§5.2, Table 3)
- refines: Dialect-specific fine-tuning can substantially improve target-dialect TTS quality even when the base multilingual model already nominally supports the target language family.
Evidence: XTTS v2’s pretraining already includes Arabic (MSA and Gulf dialects), yet fine-tuning on 38 hours of Egyptian-Arabic-only data yields a 29.9% relative WER reduction and 49.4% relative CER reduction on Egyptian Arabic evaluation data versus the untuned baseline. (§5.2)
- complicates: Automatic intelligibility and speaker-similarity metrics can be inflated by self-consistency bias when the same tools used to construct the training data are reused to score the resulting model.
Evidence: WER/CER are scored by re-transcribing synthesized speech with Whisper Large, the same model used to transcribe the training corpus, and speaker similarity uses ECAPA-TDNN embeddings, the same model used for diarization during dataset construction; the authors explicitly flag this as a “form of ASR self-consistency bias” and model-specific bias, with no independent ASR model or human-verified transcript subset used to cross-check. (§6.2)
- complicates: Synthetic dialectal speech data generated through a commercial AI audio tool inherits that tool’s limited voice inventory, constraining the speaker diversity available for downstream TTS training.
Evidence: The entire 38-hour NileTTS corpus derives from exactly two NotebookLM-generated podcast voices (one male, one female); the authors note this “limits speaker diversity” and that models trained on it “may not generalize well to synthesizing voices with different characteristics.” (§6.1)
Limitations and Open Questions
Evaluation relies entirely on automatic metrics computed with tools that overlap the data-construction pipeline (Whisper for transcription-based WER/CER, ECAPA-TDNN for speaker similarity), with no independent ASR model, human-verified transcription subset, or formal MOS/listening-test study to corroborate the gains.
Beyond the evaluation-tooling overlap, the dataset itself has narrow scope: only two speakers (one male, one female), which the authors acknowledge limits voice diversity for downstream generalization; the underlying speech is entirely synthetically generated (via NotebookLM) rather than human-recorded, though the authors argue the text content and rendered prosody preserve authentic dialectal properties; and domain coverage is limited to medical, sales/customer-service, and general conversation, excluding specialized registers such as news broadcasting or poetry. The authors propose speaker expansion, extension to other under-resourced Arabic dialects, formal MOS studies, and cross-validation with independent ASR and embedding models as future work.
Wiki Connections
- Autoregressive Codec TTS — fine-tunes XTTS v2, a GPT-style autoregressive model that generates discrete audio tokens decoded by a DVAE, applying it to a new dialect rather than modifying its architecture.
- Zero-Shot TTS — builds on XTTS v2’s zero-shot voice-cloning capability, adapting it to a dialect (Egyptian Arabic) underrepresented in its original multilingual pretraining data.
- Multilingual TTS — targets a gap in a multilingual TTS system’s dialectal coverage, showing that dialect-specific fine-tuning improves quality within a language family the base model already nominally supports.
- Evaluation Metrics — surfaces a concrete self-consistency risk in automatic TTS evaluation when the transcription and embedding models used for evaluation overlap with those used to construct the training data.
- XTTS (XTTS v2) — the base multilingual zero-shot TTS model this paper fine-tunes on its new Egyptian Arabic dataset.
- Whisper (Whisper) — used both to transcribe and segment the synthetic audio during dataset construction and to compute WER/CER at evaluation time, motivating the paper’s own self-consistency-bias caveat.
- SpeechBrain (SpeechBrain) — supplies the ECAPA-TDNN speaker-embedding model used for unsupervised speaker diarization during dataset construction and for speaker-similarity evaluation.