arXiv · 2026 · Preprint

Hongkun Yang et al. (Ocean University of China / The Hong Kong Polytechnic University / Cornell University / Nanyang Technological University / Shanghai Jiao Tong University / University of Michigan-Ann Arbor / University of Science and Technology of China / Harbin Institute of Technology) · → Paper · Demo: ? · Code: ✓

Builds a 98.8-minute sentence-level TTS dataset for the endangered Nüshu script by concatenating archival syllable recordings, then fine-tunes an English VITS model with an added frame-level F0 predictor supervised from the waveform.

Problem

Nüshu is a phonetic syllabic script historically used by women in Jiangyong County, Hunan. Each character encodes one syllable of the local Jiangyong dialect, including its tone. Earlier computational work on Nüshu covers text encoding, LLM processing and OCR, but not its spoken form. The only acoustic resource the authors found is a CD-ROM pronunciation dictionary of 873 isolated syllable recordings, with no natural sentence-level speech. The authors argue that this is far below the “low-resource” settings in prior TTS work (for example 4 to 12 hours of clean target speech), and that standard architectures become unstable at this scale (§1, §2.2).

Method

Dataset (NüshuVoice). Text comes from the compilation The Mystery of Jiangyong Nüshu, which pairs Nüshu sentences with phonetic transcriptions in five-level tone notation and standard Chinese translations. Annotators mapped each glyph to its standard Unicode form by hand. A sentence was discarded if any character had no Unicode equivalent, a stroke or component mismatch, or an ambiguous mapping (Table 4). Sentence audio was then built by concatenating the dictionary’s isolated syllable WAV files in Audacity, following each sentence’s transcription. The result has 1,252 utterances (98.8 min), split into 1,002 train, 125 validation and 125 test utterances (§3, Table 2). Two Nüshu experts checked a 10% sample (125 entries) and agreed on 90.00% of entries (Cohen’s κ 0.7887). After adjudication, 117 of the 125 entries (93.6%) were judged correct (§3.4).

Model (Nüshu-PitchVITS). The input is the IPA-style phonetic transcription with a tone digit on each syllable. The model is VITS (a conditional VAE trained with adversarial losses, monotonic alignment search (MAS) and a HiFi-GAN-style waveform decoder) with one addition. After MAS, the frame-expanded text-encoder states feed a two-layer 1D-convolutional pitch predictor that outputs frame-level log-F0. The predicted log-F0 is linearly projected to the hidden dimension and added to the latent before the waveform decoder. The pitch branch is kept out of MAS so that it cannot disturb alignment. The pitch target is log-F0 extracted with WORLD’s DIO algorithm, trained with an L2 loss (§4.3, Figure 3). At inference the predicted F0 is used directly. There is no interface for user-specified pitch (§4.3, Algorithm 1).

Training. The model starts from a pre-trained English VITS checkpoint. Stage 1 freezes the posterior encoder and waveform decoder and trains the text encoder, duration predictor, flow and pitch predictor with the KL, duration and F0 losses. Stage 2 unfreezes everything and trains with the full VITS generator loss plus the F0 term. Training uses AdamW with learning rate 1e-4 in stage 1 and 5e-5 in stage 2, FP16, on one RTX 5090 (§4.4, §5.1). Model size is not reported.

Baselines. Tacotron 2, FastSpeech 2, Glow-TTS, F5-TTS and standard VITS are all trained and evaluated on the same NüshuVoice splits. The paper does not say whether the baselines started from pretrained checkpoints, or whether the VITS baseline used the same English initialization and two-stage schedule (§5.2).

Key Results

All results are on the 125-utterance NüshuVoice test split (Table 3):

ModelMCD ↓F0 RMSE ↓F0 Corr. ↑Naturalness MOS ↑Intelligibility MOS ↑
Tacotron 244.2647.150.321.661.03
FastSpeech 237.5329.510.661.901.10
F5-TTS9.9637.030.473.103.27
Glow-TTS39.9628.440.722.611.03
VITS5.1923.420.733.033.90
Nüshu-PitchVITS3.1113.230.923.314.97
Ground truth---3.34not reported
  • Nüshu-PitchVITS has the lowest MCD, the lowest F0 RMSE and the highest F0 correlation among the six systems. It also has the highest naturalness and intelligibility MOS. Compared with VITS, MCD falls by 40.1% (5.19 to 3.11), F0 RMSE falls by 43.5% (23.42 to 13.23), and intelligibility MOS rises by 27.4% (3.90 to 4.97).
  • The strongest baseline depends on the metric. VITS leads the baselines on MCD, F0 RMSE, F0 correlation and intelligibility. F5-TTS leads them on naturalness (3.10 vs 3.03 for VITS), and Glow-TTS (0.72) nearly ties VITS (0.73) on F0 correlation.
  • Naturalness of the proposed system (3.31 ± 0.53) is within 0.03 of the ground truth (3.34 ± 0.60). The “ground truth” here is itself concatenated syllable audio. The proposed system’s naturalness lead over F5-TTS (3.10 ± 0.92) is 0.21, with no significance test reported.
  • Tacotron 2, FastSpeech 2 and Glow-TTS get MCD above 37 and intelligibility MOS near 1, which the authors describe as unintelligible output. These look like training failures at this data scale rather than close comparisons.
  • The intelligibility MOS of 4.97 ± 0.18 is near the top of the scale. No ASR exists for Nüshu, so there is no WER or CER. The paper does not report how many raters took part or whether they could read or understand Nüshu or the Jiangyong dialect (§5.3, Appendix B.2).
  • Mel-spectrograms of two utterances show sharper harmonics and fewer over-smoothed syllable transitions for Nüshu-PitchVITS than for VITS. This is a qualitative comparison (§6.3, Figure 4).

Novelty Assessment

The main contribution is the dataset, which the authors present as the first sentence-level text-audio resource for Nüshu, with careful orthographic filtering and expert validation. The model change is modest. It adds a FastSpeech-2-style frame-level pitch predictor to VITS and injects its output into the latent before decoding, with the pitch branch kept out of MAS. The paper runs no ablation of the F0 branch beyond the comparison with standard VITS, and that comparison may not share the English initialization or the two-stage schedule. So the gain cannot be attributed cleanly to pitch conditioning rather than to the training recipe. Overall this is an engineering adaptation of an existing architecture to a new single-language, extremely low-resource setting, not a new mechanism.

Field Significance

Low — The paper provides a dataset and a working VITS-based recipe for an endangered tonal script with under 80 minutes of training audio built from isolated syllables. It adds one more data point that explicit F0 supervision can stabilize end-to-end VITS training at very small scale. The evaluation is a single-language test drawn from the same syllable inventory as training, with a human MOS of unreported size and no ablation, which limits how far the result generalizes.

Claims

  • supports: Explicit frame-level F0 supervision added to an end-to-end variational TTS model can improve spectral and pitch fidelity when training data is extremely scarce and tone annotations are available.

    Evidence: Nüshu-PitchVITS lowers MCD from 5.19 to 3.11 and F0 RMSE from 23.42 to 13.23, and raises F0 correlation from 0.73 to 0.92, relative to standard VITS on the NüshuVoice test split. Whether the VITS baseline used the same initialization and two-stage schedule is not stated. (§6.1, Table 3)

  • supports: End-to-end models with learned monotonic alignment hold up better than attention-based autoregressive or duration-supervised non-autoregressive TTS when trained on around an hour of target-language speech.

    Evidence: Trained on 79 minutes, Tacotron 2, FastSpeech 2 and Glow-TTS get MCD of 37.53 to 44.26 and intelligibility MOS of 1.03 to 1.10. VITS reaches MCD 5.19 and intelligibility MOS 3.90. (§6.1, §6.2, Table 3)

  • complicates: Concatenating isolated archival syllable recordings into sentence-level training audio makes extremely low-resource TTS possible, but it leaves out coarticulation and phrase-level prosody, and it lets train and test share acoustic units, so objective scores may overstate generalization.

    Evidence: The authors state that NüshuVoice audio is assembled from 873 isolated syllable WAVs, that it cannot capture coarticulation, pauses, rhythm or discourse prosody, and that train and test splits may share underlying acoustic units. (§3.3, §Limitations)

  • complicates: Without a recognizer for the target language, intelligibility has to be judged by human listeners, and near-ceiling scores are hard to interpret when the panel size and the raters’ language competence are not reported.

    Evidence: WER is replaced by a 5-point intelligibility MOS because no Nüshu ASR exists. The proposed system scores 4.97 ± 0.18, and the paper does not report rater count, rater background, or a ground-truth intelligibility score. (§5.3, Appendix B.2, Table 3)

Limitations and Open Questions

Train-test overlap at the acoustic-unit level

All sentence audio, including the test split and the “ground truth” used for MCD and F0 metrics, is concatenated from one finite set of 873 isolated syllable recordings. The authors acknowledge that train and test may share the same underlying units (§Limitations). The test therefore measures how well the model reproduces known syllable recordings in new orders, not how well it produces natural connected Nüshu speech.

  • The training audio has no natural coarticulation, sandhi, pauses or sentence-level intonation, so the model cannot learn them. Its output can only be as natural as concatenated dictionary speech. That may explain why the ground-truth naturalness MOS is only 3.34.
  • There is no ablation that isolates the F0 branch, the English initialization, or the two-stage schedule. The pretraining data of the English VITS checkpoint is not reported.
  • The MOS test does not report the number of raters, their familiarity with Nüshu or the Jiangyong dialect, the number of samples rated, or any significance test.
  • The paper does not report the number or identity of speakers in the source dictionary, so it is unclear whether the output is single-speaker.
  • The strict Unicode filter discards every sentence containing a historical glyph variant. This biases the corpus toward standardized characters (§Limitations).
  • The code and data are released through an anonymous review link, so long-term availability is uncertain.

Wiki Connections

  • GAN Vocoder — Fine-tunes the full VITS model, including its adversarially trained waveform decoder, with an added F0 branch on under 80 minutes of Nüshu audio.
  • Subjective Evaluation — Replaces ASR-based intelligibility with a blind 5-point human intelligibility MOS because no Nüshu recognizer exists, though the panel size is not reported.
  • F5-TTS — Used as a flow-matching baseline trained on NüshuVoice. It has the best baseline naturalness MOS but trails VITS on MCD, pitch and intelligibility.
  • FastSpeech 2 — Used as a duration-supervised non-autoregressive baseline that fails at this data scale.
  • Tacotron 2 — Used as an autoregressive baseline whose attention-based alignment collapses on the 79-minute training set.
  • Parameter-Efficient Fine-Tuning for Low-Resource TTS — Cited as a less extreme low-resource setting (a 12-hour Korean corpus with cross-lingual continual learning).
  • BnTTS — Cited as a low-resource adaptation setting that still relies on hours of clean target speech and large-scale continued pretraining.
  • South Sámi TTS — Cited as prior endangered-language TTS work motivating speech technology for revitalization.
  • Ojibwe TTS for Education — Cited as evidence that TTS for Indigenous languages can support language education.