arXiv · 2025 · Preprint
Lars Nippert · → Paper · Demo: ? · Code: ✓
Presents reproducible FastPitch-based Arabic TTS baselines, introduces a family of cepstral-domain metrics for quantifying oversmoothing in mel-spectrogram prediction, and shows that a lightweight adversarial spectrogram loss substantially reduces this oversmoothing while extending the system to multiple speakers via synthetic augmentation.
Problem
Arabic TTS remains comparatively underexplored relative to English and Chinese: publicly available Arabic corpora are scarce, and the field lacks reproducible, openly released baselines built on standard architectures such as FastPitch. Separately, neural TTS acoustic models trained with standard L1/L2 reconstruction losses on mel-spectrograms are known to suffer from oversmoothing (over-averaged, muffled spectra with suppressed fine spectral detail), but prior work lacks established objective metrics to quantify this effect during training, and the use of adversarial objectives to counteract it has not been systematically studied for Arabic TTS.
Method
The system follows the standard two-stage mel-spectrogram TTS pipeline: Arabic text is transliterated and phonemized (a simplified version of Nawar Halabi’s Arabic Phonetiser via Buckwalter transliteration, with geminated consonants marked by a dedicated doubling token), tokenized, passed through a FastPitch acoustic model to predict a log-mel spectrogram, and converted to a waveform by a HiFi-GAN vocoder fine-tuned on the target corpus for a few hundred iterations.
FastPitch (a non-autoregressive text-to-mel model derived from FastSpeech) encodes the token sequence with a stack of feed-forward Transformer blocks, predicts per-token duration, pitch, and energy, expands the encoded sequence according to predicted durations, and decodes the expanded, pitch- and energy-augmented sequence into a mel-spectrogram in a single parallel pass. Duration targets are supplied by a RAD-TTS-style alignment network trained with a beta-binomial monotonic alignment prior. The default training objective is a weighted sum of mel L2 reconstruction, duration, pitch, energy, alignment, and binarization losses.
To analyze oversmoothing, the paper introduces a “mel-cepstrogram”: a real FFT applied along the mel-bin axis of each frame (after mean subtraction and Hann windowing), which separates the smooth, low-quefrency spectral envelope from fine, high-quefrency spectral detail. From the resulting quefrency power distribution the paper defines four complementary scores: High-Quefrency Energy Ratio (HQER, the fraction of cepstral energy above a cutoff), Cepstral Slope (log-power decay rate with quefrency), Cepstral Centroid (energy-weighted mean quefrency), and Cepstral Rolloff at 95% (the quefrency index containing 95% of cepstral energy). Lower HQER/Centroid/Rolloff values and more negative slope indicate stronger oversmoothing; three of the four metrics are differentiable and could in principle be used as training losses, though this paper uses them only for evaluation.
To directly counteract oversmoothing, the model is augmented with a lightweight adversarial spectrogram loss. The discriminator follows the convolutional design used in the DeepFillv2 image-inpainting model, adapted to operate on randomly cropped 128-frame log-mel segments: five 5x5 stride-2 convolutions with LeakyReLU activations and spectral normalization on all weights. Training uses the least-squares GAN (LS-GAN) formulation plus a feature-matching loss (MAE between discriminator feature maps for real and generated spectrograms), with the generator’s adversarial term added to the default FastPitch loss at a weight of alpha=4, chosen from limited preliminary tuning. Setting the discriminator/generator Adam momentum beta_1=0 (rather than a positive value) was found necessary for stable, oscillation-free training.
For the multi-speaker extension, the discriminator is additionally speaker-conditioned: a speaker embedding is projected through two spectrally-normalized fully connected layers to a vector of length equal to the mel-channel count, tiled across time, and concatenated to the spectrogram as an extra input channel. Three synthetic voices (one male, two female, based on distinct speaker embeddings) are generated by resynthesizing the full ASC train and test transcripts with XTTSv2, so that all four speakers (the real ASC speaker plus three synthetic ones) share identical linguistic content.
Key Results
For the single-speaker baseline trained with the default L2-based loss, reconstruction metrics (L1, L2, spectral convergence) converge within the first ~50 epochs, while pitch correlation (best Pearson r = 0.953 at epoch 310) and pitch RMSE (best 6.90 Hz at epoch 256, converging to 7.36 +/- 0.078 Hz by epoch 970-1000) improve more gradually. Notably, the four cepstral oversmoothing metrics continue to improve well past epoch 300, after reconstruction losses have already plateaued, indicating that fine spectral detail keeps transferring from training to the held-out test set long after pointwise reconstruction error stops improving (§3.1, Table 3).
Adding the adversarial spectrogram loss produces a small increase in pointwise reconstruction distances (L1/L2/SConv) relative to the default-loss baseline, but the cepstral oversmoothing metrics (HQER, CSlope, CCentroid, CRoll95) are consistently and substantially lower with the adversarial loss throughout training, while pitch RMSE, pitch correlation, and V/UV error rate remain comparable between the two configurations (§3.2, §4.2, Figures 8-9). No exact numeric table comparing the two configurations is given in the main text; the comparison is reported through training curves.
In the multi-speaker setting, adding three XTTSv2-synthesized voices to the real ASC speaker does not degrade performance on the original speaker: mel L2, pitch RMSE, pitch correlation, V/UV error, and the cepstral oversmoothing metrics for the real speaker (S0) remain within the same range as the single-speaker adversarial baseline (S0*) (§3.3, Figure 10). Speakers with lower pitch variance (the synthetic male voice) achieve lower f0-RMSE and faster convergence than the higher-pitch-variance synthetic female voices.
Novelty Assessment
The underlying architecture (FastPitch acoustic model, HiFi-GAN vocoder) and training data (the existing Arabic Speech Corpus) are not new. The paper’s contribution is primarily methodological and empirical: (1) a first openly reproducible FastPitch/HiFi-GAN Arabic TTS baseline with released code, pretrained checkpoints, and training recipes; (2) a new differentiable family of cepstral-domain oversmoothing metrics (HQER, CSlope, CCentroid, CRoll95) derived by applying an FFT across the mel-bin axis, giving a way to track and compare spectral detail loss during training that goes beyond standard L1/L2/SConv reconstruction distances; (3) an engineering adaptation of a DeepFillv2-style image-inpainting discriminator to mel-spectrogram oversmoothing reduction, shown to train stably under an LS-GAN formulation with spectral normalization; and (4) a demonstration that a zero-shot multilingual TTS system (XTTSv2) can be used to synthesize additional training speakers for a scarce single-speaker corpus without harming the original speaker’s quality. None of these individually constitute a new model architecture; the contribution is best read as a reproducibility and measurement paper for a low-resource TTS setting.
Field Significance
moderate - this paper provides a transparent, reproducible baseline and a new measurement tool (the cepstral oversmoothing metric family) for a comparatively underserved language and problem (Arabic TTS oversmoothing), together with a lightweight, architecture-agnostic recipe for reducing it. Its scope is narrow: a single real speaker, one acoustic architecture, and no human perceptual validation, so its significance rests on the reusability of the metrics and training recipe rather than on state-of-the-art results.
Claims
-
supports: A lightweight adversarial spectrogram loss with a spectrally-normalized convolutional discriminator can substantially reduce oversmoothing in mel-spectrogram-based TTS acoustic models without degrading pitch or duration accuracy, at negligible inference-time cost.
Evidence: Cepstral oversmoothing metrics (HQER, CSlope, CCentroid, CRoll95) are consistently and substantially lower for the adversarially-trained FastPitch model than for the default L2-loss baseline throughout training, while pitch RMSE, pitch correlation, and V/UV error rate remain comparable between the two configurations; the discriminator only participates at training time and does not add inference cost. (§3.2, §4.2, Figures 8-9)
-
supports: Synthetic speech from a zero-shot multilingual TTS system can be used to augment a scarce single-speaker corpus into a multi-speaker training set without degrading synthesis quality for the original real speaker.
Evidence: Adding three XTTSv2-synthesized voices (one male, two female, resynthesized from the same ASC transcripts) to the real Arabic Speech Corpus speaker, the multi-speaker FastPitch model’s performance on the original speaker (mel L2, pitch RMSE, correlation, V/UV error, and cepstral oversmoothing metrics) stayed within the same range as the single-speaker adversarial baseline. (§3.3, §4.3, Figure 10)
-
complicates: Standard Lp (L1/L2) regression losses on mel-spectrograms induce a persistent, systematic reduction in predicted pitch and duration variance relative to the ground truth, even after long training.
Evidence: With the default L2 loss, the learned duration predictor shows a lower mean and reduced variance versus the alignment-network-derived targets (mu=6.75 vs. 7.31, sigma=2.33 vs. 3.44), and the pitch predictor similarly shows a lower mean and reduced variance versus ground-truth pitch (mu=125.7 Hz vs. 127.3 Hz, sigma=12.4 vs. 17.2), both persisting after 1000 epochs of training. (§3.1, §4.1, Figures 5-6)
-
refines: Cepstral-domain metrics computed via an FFT across mel-frequency bins provide a more sensitive proxy for spectral detail loss during TTS training than pointwise Lp reconstruction distances, remaining informative after reconstruction losses have plateaued.
Evidence: In the default-loss baseline, L1/L2/spectral-convergence distances plateau within roughly the first 50 training epochs, while the four cepstral oversmoothing metrics (HQER, CSlope, CCentroid, CRoll95) continue to decrease for several hundred additional epochs, reaching their best values between epochs 620 and 936. (§3.1, Table 3)
Limitations and Open Questions
The paper reports no subjective or perceptual evaluation. All results (reconstruction distances, pitch metrics, and the proposed cepstral oversmoothing metrics) are objective proxy measures computed against reference audio; no listening tests or MOS-style human ratings are used to confirm that the observed reduction in oversmoothing translates into perceived quality improvements.
- The system is trained and evaluated on a single real speaker (the Arabic Speech Corpus, ~3.5 hours of Modern Standard Arabic from one speaker); generalization to other Arabic dialects, larger corpora, or unvowelized (undiacritized) text is not evaluated.
- The train and official test splits of the Arabic Speech Corpus differ significantly in duration, phonemes per utterance, speaking rate, and pitch statistics (Mann-Whitney U test, all p < 0.05), which the authors note should be kept in mind when interpreting absolute metric values, since some of the reported differences may partly reflect corpus design rather than model behavior.
- The adversarial loss weight (alpha=4) was selected from limited preliminary experiments; the authors explicitly note that a more extensive hyperparameter study remains for future work.
- Three of the four cepstral oversmoothing metrics are differentiable and could in principle be incorporated directly into the training objective, but this paper uses them only for post-hoc evaluation and leaves training-time use for future work.
Wiki Connections
- Transformer Encoder-Decoder TTS — the acoustic model is a FastPitch feed-forward Transformer encoder-decoder with an explicit duration/pitch/energy predictor pipeline, applied here to Arabic for the first time as a reproducible open baseline.
- Evaluation Metrics — introduces a new family of differentiable, cepstral-domain metrics (HQER, Cepstral Slope, Cepstral Centroid, Cepstral Rolloff) for quantifying oversmoothing in mel-spectrogram TTS, distinct from standard Lp reconstruction distances.
- Speaker Adaptation — extends a single-speaker Arabic FastPitch model to a four-speaker setting using speaker embeddings and a speaker-conditioned discriminator, with three of the four voices generated synthetically.
- XTTS: a Massively Multilingual Zero-Shot Text-to-Speech — used as an external, off-the-shelf zero-shot TTS system to synthesize three additional training speakers by resynthesizing the ASC transcripts, rather than being extended or modified itself.
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis — used as the vocoder, fine-tuned on the target corpus for a few hundred iterations to convert predicted log-mel spectrograms into waveforms.