arXiv · 2025 · Preprint
Yakov Kolani et al. (Independent Researcher; Reichman University; Tel Aviv University) · → Paper · Demo: ✓ · Code: ✓
Adapts an existing Hebrew diacritization model with lightweight adaptors to output fully-specified IPA transcriptions, resolving phonetic ambiguities (stress, shva) that standard vowel diacritics leave unresolved, and uses this to train small, real-time-capable Hebrew TTS models.
Problem
Modern Hebrew orthography omits vowels, and even when optional vowel diacritics (nikud) are added, features such as lexical stress and shva pronunciation remain ambiguous. Prior Hebrew TTS approaches split into two unsatisfactory camps: end-to-end systems that consume raw unvocalized text directly, which require large models to resolve orthographic ambiguity and therefore incur latency unsuitable for real-time, edge-device applications; and two-stage systems that predict standard vowel diacritics before synthesis, which still leave stress and shva underspecified and produce mispronunciations. There is also no open, phonetically-annotated Hebrew speech corpus, and existing ASR-based TTS evaluation methods for Hebrew operate on unvocalized transcriptions, so they cannot detect stress or vowel-quality errors in generated speech.
Method
Phonikud is a grapheme-to-phoneme (G2P) pipeline with two stages. First, an enhanced diacritization module augments an existing pretrained Hebrew diacritizer (DictaBERT, an encoder-only ~300M-parameter BERT-style model) with a small two-layer MLP prediction head (hidden dimension 256) that predicts three additional symbols per character: a superscript mark for non-final lexical stress, a subscript mark for a shva realized as /e/, and a vertical bar marking cliticized prefix boundaries. The base encoder and its existing diacritization heads are frozen; only the new adaptor head is trained, so inference adds negligible latency and standard diacritization accuracy is unaffected. Second, a deterministic rule-based module, implemented primarily as a finite-state transducer, converts the fully-disambiguated vocalized text into standard IPA, handling many-to-one grapheme-to-phoneme mappings, non-monotonic character sequences, dual-function letters, and irregular/loanword pronunciations via dictionary lookup.
Because ground-truth annotations for Hebrew stress and shva realization do not exist at scale, the authors use a human-in-the-loop distillation procedure: they automatically annotate 5M lines of Hebrew text (IsraParlTweet corpus) with pseudo-ground-truth stress, prefix, and shva labels using a morphophonological analysis API and linguistic rules, then manually correct the most frequent word types before fine-tuning the adaptor on this distilled data. Downstream, the IPA output from Phonikud is used to fine-tune two existing English TTS checkpoints, lightweight VITS-based Piper and StyleTTS2, on Hebrew speech (SASPEECH and the paper’s own ILSpeech dataset) using IPA rather than raw text as the training target; at inference, Phonikud converts input Hebrew text to IPA for synthesis, or users may supply IPA directly for explicit phonetic control. The paper also contributes ILSpeech, roughly two hours of studio-quality Hebrew speech from two speakers with parallel unvocalized-text and expert-annotated IPA transcriptions, and trains an audio-to-IPA ASR model on ILSpeech (first pretrained on SASPEECH with Phonikud-inferred IPA, then fine-tuned on ILSpeech’s manual IPA) to automatically score generated TTS audio for phonetic accuracy, including stress, which standard unvocalized-text ASR evaluation cannot detect. No neural audio codec is used; the TTS backbones operate on IPA phoneme sequences directly.
Key Results
On G2P evaluation against ILSpeech ground-truth IPA (Table 1), Phonikud achieves WER 0.19 and CER 0.04, outperforming the existing SOTA Hebrew diacritizers DictaBERT (WER 0.38) and Nakdimon (WER 0.40) when their vowel output is passed through the same IPA conversion, and far outperforming general-purpose multilingual G2P tools (eSpeak NG, Goruut, CharsiuG2P), all of which score WER ≈ 1.0 on Hebrew due to poor dictionary coverage and hallucination.
For downstream TTS (Table 2), Phonikud-based Piper (Medium, 20M params) reaches WER 0.231 at RTF 0.09 on standard consumer hardware (no GPU), and Phonikud-based StyleTTS2 (90M params) reaches WER 0.188 at RTF 0.50, both real-time-capable. These are compared against open-source two-stage baselines (Robo-Shaul, SASPEECH, MMS, all with higher WER and/or higher RTF), proprietary cloud systems (Gemini, OpenAI), and large end-to-end Hebrew TTS models that consume unvocalized text directly (LoTHM, HebTTS), which the authors report at RTF 84.75 and 25.44 respectively on the same hardware, far outside real-time budgets. A targeted manual stress-accuracy evaluation on 250 challenging sentences (Table 3) shows the full Phonikud-trained model reaches stress word-error-rate 0.03 and exact-match rate 0.76, compared to 0.07/0.57 for a leading baseline (Robo-Shaul) and 0.09/0.46 for an ablated variant trained without stress marking. An ablation (Table 4) shows removing enhanced diacritization increases downstream WER (0.30 to 0.33), removing the IPA conversion step alone has minimal effect (0.30 to 0.32), but removing vowel diacritics entirely and training on raw unvocalized text degrades WER sharply to 0.58 with frequently unintelligible output. Comparisons across proprietary/open-source baselines are not run on identical evaluation protocols in all cases (e.g. proprietary models use cloud APIs while open models run locally), which the paper flags in its methodology.
Novelty Assessment
The core contribution is architectural: extending a frozen, pretrained Hebrew diacritization model with a lightweight adaptor head to predict phonetic features (stress, shva realization, prefix boundaries) that standard diacritization omits, combined with a deterministic IPA conversion module. This is a genuinely new disambiguation scheme for a specific low-resource, phonologically ambiguous language, not a repackaging of existing G2P techniques, though the underlying adaptor-on-frozen-encoder pattern is a well-established fine-tuning technique borrowed from elsewhere in NLP. The downstream TTS component is engineering integration: Piper and StyleTTS2 are existing architectures, fine-tuned from existing English checkpoints, with no architectural modification. The ILSpeech dataset and the audio-to-IPA ASR evaluation methodology are independently valuable contributions, filling a genuine gap (no prior open Hebrew corpus has full IPA annotation, and prior ASR-based TTS evaluation for Hebrew could not detect stress or vowel-quality errors), though ILSpeech itself is small (two hours, two speakers) and explicitly framed by the authors as a proof of concept.
Field Significance
Moderate — this paper demonstrates that lightweight adaptor-based fine-tuning of an existing pretrained model can add fine-grained phonetic disambiguation to a low-resource language’s G2P pipeline without retraining or enlarging the base model, and that resolving this ambiguity in a text-to-phoneme preprocessing stage lets small TTS models match or exceed the speed-accuracy trade-off of much larger end-to-end systems. Its scope is narrow (Hebrew G2P and TTS specifically), and its methodological pattern (frozen encoder plus adaptor head for phonetic disambiguation) is a specific instance of a general fine-tuning strategy rather than a new paradigm.
Claims
- supports: Lightweight adaptor fine-tuning on top of a frozen, pretrained text-disambiguation model can add fine-grained phonetic prediction capability without degrading the base model’s original task performance or adding meaningful inference latency.
Evidence: The enhanced diacritization module freezes the base DictaBERT encoder and its existing diacritization heads, training only a new two-layer MLP head; the authors report negligible additional runtime overhead and unchanged standard-diacritization accuracy. (§3.1)
- supports: Resolving phonetic ambiguity explicitly in a text-preprocessing stage before TTS training allows small models to reach a better speed-accuracy trade-off than large end-to-end models that must learn to resolve the same ambiguity implicitly from raw orthographic text.
Evidence: Phonikud-trained Piper (20M params) reaches WER 0.231 at RTF 0.09, while large end-to-end Hebrew TTS models consuming unvocalized text directly (LoTHM, HebTTS) are measured at RTF 84.75 and 25.44 on the same hardware, far outside real-time budgets. (§5.2, Table 2)
- supports: General-purpose multilingual G2P and phonemization tools trained primarily on high-resource languages transfer poorly to languages with substantially different orthographic and phonological structure, even when they nominally claim support for that language.
Evidence: Multilingual G2P libraries eSpeak NG, Goruut, and CharsiuG2P score WER ≈ 1.0 on Hebrew due to limited dictionary coverage and extensive hallucination, versus WER 0.19 for the proposed Hebrew-specific system, on the same ILSpeech test set. (§5.1, Table 1)
- complicates: Automatic ASR-based evaluation of TTS pronunciation accuracy is insufficient when the ASR system’s transcription target itself omits the phonetic features being evaluated, since errors in those features become invisible to the metric.
Evidence: Standard Hebrew ASR-based TTS evaluation compares unvocalized-text transcriptions and therefore cannot detect vowel-quality or stress errors; the authors build a dedicated audio-to-IPA ASR model trained on phonetically-annotated data to make these errors measurable. (§4.3)
- complicates: Removing an intermediate phoneme-level (or vowel-disambiguated) representation and training a small TTS model directly on raw orthographic text degrades output intelligibility sharply for languages with high orthographic ambiguity, even though removing only a later-stage phoneme conversion step (while retaining vowel disambiguation) has little effect.
Evidence: Ablating vowel diacritics entirely (training on unvocalized text) raises WER from 0.30 to 0.58 with reportedly unintelligible output on uncommon words, while ablating only the IPA conversion step (training on vocalized Hebrew text instead) changes WER by just 0.02 (0.30 to 0.32). (§5.3, Table 4)
Limitations and Open Questions
Warning
ILSpeech, the paper’s own phonetically-annotated corpus and the basis for its TTS fine-tuning and audio-to-IPA evaluation model, consists of only about two hours of speech from two speakers. The authors describe it explicitly as a proof of concept and note in their ethics statement that it lacks coverage of sociolinguistic variation such as the Mizrahi Hebrew accent, which limits how far the reported TTS quality and evaluation results generalize beyond the recorded speakers’ style.
The method inherits any pronunciation errors from the underlying frozen Hebrew diacritization model and does not support user selection between multiple valid pronunciations when genuine ambiguity exists. The enhanced vocalization scheme follows formal written Hebrew conventions, which can diverge from informal spoken pronunciation norms. Prosodic quality of the downstream TTS output is also constrained by the limited capacity of the small, real-time-oriented backbones used (Piper, StyleTTS2), independent of G2P accuracy.
Wiki Connections
- Prosody Control — introduces an explicit stress-marking mechanism in the phonetic representation that both disambiguates pronunciation during training and allows optional user control over stress placement at inference.
- Evaluation Metrics — contributes a new audio-to-IPA ASR evaluation methodology specifically designed to make stress and vowel-quality errors measurable, addressing a gap in standard ASR-based TTS evaluation for orthographically ambiguous languages.
- GAN Vocoder — fine-tunes Piper, a lightweight VITS-based end-to-end GAN TTS system, as one of two backbones for downstream Hebrew speech synthesis.
- Diffusion TTS — fine-tunes StyleTTS2, which uses style diffusion, as the higher-quality of the two TTS backbones evaluated downstream.