Habibi: Laying the Open-Source Foundation of Unified-Dialectal Arabic Speech Synthesis

arXiv · 2026 · Preprint

Yushen Chen et al. (Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✓

Introduces Habibi, the first open-source unified-dialectal Arabic TTS framework, together with the first standardized multi-dialect Arabic zero-shot TTS benchmark, and shows a single model trained across 12+ regional dialects can match dedicated per-dialect systems and a leading commercial baseline.

Problem

Arabic spans over 30 spoken varieties, but Modern Standard Arabic (MSA), the formal written standard, is rarely spoken in daily life, while the dialects that dominate everyday communication differ substantially in lexicon and phonology. TTS research on Arabic has stayed largely MSA-centric or covered only a handful of dialects with limited support, and even those systems lag behind zero-shot TTS quality achieved for languages such as English and Chinese. The paper identifies three concrete barriers: existing Arabic TTS corpora (e.g. ArVoice) are small, MSA-only, and partly synthetic; large-scale dialectal speech data exists mainly as noisy ASR corpora unsuited for TTS training; and no standardized benchmark exists for evaluating multi-dialect Arabic TTS. Diacritization, which normally disambiguates Arabic pronunciation, is largely absent from both training transcripts and real-world text, ruling out approaches that depend on it.

Method

Habibi repurposes existing Arabic ASR corpora (MASC, SADA, MGB-2/3/5, FLEURS, Omnilingual ASR Corpus, and several public and in-house sources) into TTS training data through a multi-step curation pipeline: character-per-second (CPS) filtering to remove missing-text or corrupted-audio samples, source-separation-based denoising for low-SNR datasets, and channel-specific text-pattern filtering plus segment merging applied to the MASC subset to raise average utterance duration into a TTS-appropriate range. The resulting corpus covers over 20 ISO 639-3 languoids grouped into 12 regional identifiers (MSA, SAU, UAE, ALG, IRQ, EGY, MAR, OMN, TUN, LEV, SDN, LBY) totaling roughly 1,857 hours after data expansion (§2.2, Table 1, Table 2).

The backbone is the open-source F5-TTS framework, a flow-matching, non-autoregressive model that operates directly on mel spectrograms and raw text character sequences rather than discrete codec tokens, chosen to avoid cascading errors from separate text/audio encoder modules. Training follows a two-stage, linguistically-informed curriculum: the model is first initialized from an F5-TTS checkpoint pre-trained on Chinese and English data, then supervised fine-tuned on MSA-only data, using UTMOS convergence (rather than continued WER improvement) as the criterion for ending this stage, since further MSA-only training narrows the output distribution and degrades speaker similarity and naturalness. In the second stage, the model is fine-tuned either on a single dialect (specialized models) or jointly on all dialects (unified model). A dialect-aware supervised fine-tuning variant augments the text sequence with special regional-identifier tokens wrapping the dialectal text, intended to help the model disambiguate dialect-specific phonological patterns during generation, including at zero-shot inference where a reference audio-text pair supplies additional in-context dialectal cues (§2.4, §2.5).

Alongside the model, the paper releases the first standardized multi-dialect Arabic zero-shot TTS benchmark: 7 dialect subsets (MSA, SAU, UAE, ALG, IRQ, EGY, MAR) totaling over 11,000 utterances, each filtered for 3-12 second duration, Arabic-script-only transcription, and at least two utterances per speaker (needed to supply distinct reference and target speech for zero-shot evaluation), with strict train/benchmark separation (§2.3).

Key Results

On the released benchmark, the unified model (Uni.D2-I, trained on the full expanded dataset with regional identifiers) is competitive with or exceeds ElevenLabs’ Eleven v3 (alpha), a leading commercial system, on the majority of evaluated metrics. In subjective evaluation, Uni.D2-I achieves higher dialect-pronunciation MOS (DMOS) than 11Labs-3a on 5 of 7 dialects (e.g. MSA: 4.24 vs. 4.12, IRQ: 4.21 vs. 3.97) and higher speaker-similarity MOS (SMOS) on all 7 dialects, sometimes by a wide margin (ALG: 4.10 vs. 2.48). Objectively, Uni.D2-I obtains lower WER-O on 6 of 7 dialects and consistently higher SIM across all dialects (e.g. ALG SIM: 0.731 vs. 0.306), while 11Labs-3a leads on UTMOS and NMOS on most dialects, a pattern the authors attribute to ElevenLabs favoring perceptual naturalness over strict speaker fidelity (§3.3, Tables 4-5).

Internally, the unified model (Uni.D2-I) achieves performance close to, and on several dialects (MSA, MAR) better than, per-dialect specialized models, while specialized models retain a WER edge on SAU, UAE, ALG, IRQ, and EGY (§3.4, Table 7). Ablations show that the MSA-first curriculum outperforms both training from scratch and direct dialectal fine-tuning without the MSA stage, even when the no-MSA-stage variant receives double the dialectal training updates (§3.5, Table 8). Removing reference-context audio at inference substantially degrades WER across all 7 dialects (e.g. UAE WER-S: 4.88 to 10.68), confirming reliance on in-context learning from the reference speech-text pair (§3.6, Table 9). Explicit regional-identifier tokens improve WER with limited impact on SIM or UTMOS, and the model remains robust across different inference-time identifier templates (§3.8, Table 11).

Novelty Assessment

The architecture itself is not new: Habibi adapts the existing F5-TTS flow-matching backbone without structural modification. The genuine contributions are a data-engineering pipeline that converts noisy, ASR-oriented Arabic corpora into usable TTS training data at scale, a linguistically motivated MSA-first curriculum learning recipe validated by controlled ablations against both from-scratch training and direct dialectal fine-tuning, and the first open, standardized multi-dialect Arabic TTS benchmark with manually verified transcripts. Together these establish reusable open-source infrastructure (data curation pipeline, benchmark, and checkpoints) for a language family that previously lacked any unified, open multi-dialect TTS system, rather than advancing TTS modeling technique per se.

Field Significance

High field significance

high — this paper fills a concrete infrastructure gap (open-source unified-dialectal Arabic TTS, plus the first standardized cross-dialect benchmark) for a widely spoken but historically under-served language family, and demonstrates via ~8,000 H100 GPU hours of ablations that a training-recipe intervention (curriculum ordering) can be more important than raw quantity of dialectal training data.

The paper provides reusable open-source assets, training data curation pipeline, checkpoints, and a manually verified benchmark, for future dialectal and low-resource Arabic TTS work, and its curriculum-ordering ablation (MSA-first fine-tuning beats direct dialectal fine-tuning even under double the compute) offers a transferable lesson for other pluricentric or dialect-rich languages facing similar data scarcity. It does not introduce a new TTS architecture; its significance is primarily as a dataset, benchmark, and training-methodology contribution rather than a modeling one.

Claims

• supports: For low-resource dialectal speech synthesis, staging fine-tuning from a high-resource, linguistically related variant before adapting to low-resource dialectal data outperforms both training from scratch on dialectal data and direct dialectal fine-tuning without an intermediate stage.

  Evidence: An MSA-first two-stage curriculum consistently beats both from-scratch dialectal training and direct fine-tuning across WER-O, WER-S, SIM, and UTMOS in both specialized and unified settings, and still outperforms a variant trained with twice as many dialectal updates but no MSA stage. (§3.5, Table 8)

• supports: A single unified multi-dialect zero-shot TTS model can match or exceed the performance of dialect-specialized models trained separately on each dialect.

  Evidence: The unified model (Uni.D2-I) matches specialized models on WER for MSA and MAR and equals or exceeds them on speaker similarity across all seven dialects, with the largest gain on MAR (SIM 0.705 vs. 0.607). (§3.4, Table 7)

• supports: Zero-shot TTS models that condition on paired reference speech and transcript actively exploit that context to reproduce target-language or target-dialect phonological patterns, rather than relying solely on parameters learned during training.

  Evidence: Zeroing out the reference audio at inference degrades WER across all seven benchmarked dialects, e.g. IRQ WER-O rises from 17.12 to 20.15 and UAE WER-S from 4.88 to 10.68. (§3.6, Table 9)

• complicates: Automatic ASR-based word error rate is an unreliable absolute quality signal for dialectal speech synthesis when the available ASR systems themselves perform poorly on the target dialect.

  Evidence: Ground-truth recordings for ALG and MAR yield WER-O of 41.19 and 54.42 respectively under the paper’s ASR evaluation setup, higher than several synthesized outputs, which the authors attribute to noise in the underlying ASR-sourced ground truth data rather than genuine synthesis quality differences. (§3.2, §3.4, Table 6)

• refines: Providing explicit categorical conditioning tokens for sub-language variety (dialect/region) during fine-tuning improves recognition-based metrics at inference even when those tokens are omitted at test time, refining the general in-context-learning capability of zero-shot TTS models.

  Evidence: Training a unified model with regional-identifier tokens improves WER relative to a no-identifier variant while leaving SIM and UTMOS largely unchanged, and the identifier-aware model remains robust across plain-text, dialect-agnostic, and dialect-aware inference templates. (§3.8, Table 11)

Limitations and Open Questions

WER-based evaluation is unreliable for several dialects in this benchmark

The ASR models used to compute WER-O and WER-S remain weak for Arabic dialects, particularly ALG and MAR, where even ground-truth audio yields WER above 40%. This limits how much confidence can be placed in absolute or cross-dialect WER comparisons, and the paper compensates with (but does not fully replace this weakness with) subjective DMOS/SMOS/NMOS evaluation.

The comparison against ElevenLabs’ Eleven v3 (alpha) uses a single representative reference voice per dialect subset (three for SAU), chosen due to a strict monthly quota on custom voice uploads in the commercial API; this narrows the speaker-timbre coverage of the head-to-head comparison relative to the full benchmark. A portion of the training data (“in-house” sources for ALG, IRQ, and part of UAE) is not from a named public corpus, which limits third-party reproducibility of the exact training set even though the benchmark and code are released. The paper’s own regional-dialect terminology is explicitly stated to reflect a practical convention rather than an official linguistic classification, and some dialect subsets (e.g. OMN, TUN, SDN, LBY) rely entirely on the Omnilingual ASR Corpus with comparatively few hours, leaving open how well the unified model generalizes to dialects with the least training data.

Wiki Connections

• Zero-Shot TTS — extends zero-shot TTS to a multi-dialect, low-resource-language setting via a curriculum-trained F5-TTS backbone conditioned on reference speech-text pairs.
• Multilingual TTS — treats Arabic dialects as near-language-level variants and demonstrates that a single model, cross-dialect training strategy competitive with per-dialect specialization, directly informing multi-variety TTS design.
• Flow Matching — builds on the flow-matching F5-TTS architecture as its backbone without structural modification, applying it to a new low-resource multi-dialect domain.
• Evaluation Metrics — introduces the first standardized multi-dialect Arabic TTS benchmark and documents where WER-based automatic metrics become unreliable due to weak dialectal ASR.
• Subjective Evaluation — runs controlled DMOS/SMOS/NMOS human listening studies across seven dialects to validate results that automatic metrics alone could not reliably support.
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching — Habibi initializes from and fine-tunes this flow-matching backbone rather than proposing a new architecture.
• ArVoice: A Multi-Speaker Dataset for Arabic Speech Synthesis — cited as the largest prior open-source Arabic TTS corpus, which Habibi’s much larger multi-dialect data curation pipeline explicitly surpasses in scale and dialect coverage.
• Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages — used both as a source of dialectal training data (Omnilingual ASR Corpus) and as the multilingual ASR model behind the WER-O evaluation metric.
• Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis — a concurrent dialectal Arabic TTS effort that Habibi’s unified curriculum-learning approach can be directly compared against.
• Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis — another F5-TTS adaptation to a new language, providing a point of comparison for how the same backbone transfers to low-resource linguistic settings.
• F5-TTS-RO: Extending F5-TTS to Romanian TTS via Lightweight Input Adaptation — a parallel single-language F5-TTS extension against which Habibi’s multi-dialect, curriculum-based adaptation strategy can be contrasted.
• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers — representative of the autoregressive zero-shot TTS paradigm that Habibi’s authors cite as an alternative to the mel-spectrogram-based approach they adopt.
• E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-shot TTS — a related non-autoregressive zero-shot TTS approach cited among the mainstream paradigms Habibi’s F5-TTS-based design belongs to.