arXiv · 2025 · Preprint
Falai et al. (Amazon) · → Paper · Demo: ? · Code: ?
Adapter-based fine-tuning — using only 10% additional parameters — enables a 2M-parameter GAN-based TTS model to adapt to unseen speakers and new languages while avoiding catastrophic forgetting, outperforming full fine-tuning on most objective metrics.
Problem
Lightweight on-device TTS models face a dilemma when adapting to new speakers or languages: full fine-tuning risks catastrophic forgetting of previously learned representations, while retraining from scratch is expensive. Cross-lingual voice cloning — synthesising speech in a target language using a source speaker’s voice, with no target-language recordings from that speaker — amplifies this challenge, as it requires decoupling speaker and language representations in a model too small to rely on scale. Prior approaches using large multi-speaker multilingual models or voice conversion pipelines are not viable on-device.
Method
The backbone model (named LE2E, Lightweight End-to-End TTS) is a GAN system comprising a LightSpeech-inspired acoustic model and a MultiBand MelGAN vocoder. The acoustic model takes phonemes and positional embeddings as input, predicts duration and pitch via a variance adaptor, and produces acoustic latents that the vocoder converts to waveforms. Adversarial training uses Multi-Period Discriminators and Multi-Resolution Discriminators with least-squares GAN losses, supplemented by feature matching, multi-resolution STFT, and mel-spectrogram losses. The backbone totals approximately 2M parameters.
Two adapter types are inserted into the frozen backbone. Bottleneck adapters (down-projection → ReLU → up-projection, bottleneck dimension 16) are placed after each convolutional layer in the acoustic model’s encoder, decoder, and variance predictors, adding 150K parameters. Convolutional adapters — three 1D convolutional layers with kernel sizes [3, 5, 3], layer normalisation, and a Squeeze-and-Excitation module — are placed after each upsampling and residual block in the vocoder (15 positions), adding 50K parameters. Together, 200K adapter parameters represent 10% of the original model.
During fine-tuning, all backbone parameters are frozen; only adapter blocks, the phoneme embedding matrix, and speaker/language lookup tables are optimised. Language adaptation fine-tunes the model on target-language data from multiple speakers to transfer to a new language while preserving the source speaker’s voice. Speaker adaptation inverts this: the backbone is pre-trained on a multi-speaker target language, and adapters are fine-tuned on a single source-speaker to transplant that identity.
The paper also introduces a Phoneme Substitution Rate (PSR) metric for accent nativeness, derived from a multilingual wav2vec2 model fine-tuned with CTC loss for phoneme recognition. PSR measures the percentage of phonemes substituted relative to ground truth, as a proxy for non-native accent. Validation against historical MUSHRA accent tests across five language variants achieves over 90% pairwise ranking accuracy.
Key Results
For language adaptation (English → Spanish), adapter fine-tuning outperforms full fine-tuning on naturalness (MOS 4.36 vs. 3.99), signal quality (PESQ 3.29 vs. 3.15), and speaker similarity (SECS 0.51 vs. 0.43). Full fine-tuning achieves slightly better accent nativeness (PSR 0.56 vs. 1.42), but informal listening attributes this to overfitting to a single Spanish speaker, not to genuinely better accent transfer.
For speaker adaptation (Spanish → English speaker identity), the best overall outcome comes from placing adapters in the vocoder only: SECS 0.62, MOS 4.19, PSR 0.75. Adding adapters to the acoustic model as well raises speaker similarity substantially (SECS 0.78) at the cost of accent nativeness (PSR 6.27), indicating that the acoustic model encodes most of the language-agnostic speaker information. Reducing target-language training data to a single speaker in the pre-training phase causes severe degradation across all metrics, with PSR rising to 43.68.
Ablations show that reducing vocoder adapters from 15 to 3 blocks degrades both speaker similarity and accent nativeness (Table 1, ⋆3), confirming that adapter density in the vocoder matters for adaptation quality.
Novelty Assessment
The use of adapters for TTS is not new — prior work (Morioka et al. 2022, Hsieh et al. INTERSPEECH 2023) applied residual and bottleneck adapters for few-shot speaker adaptation. The primary novel claim here is applying adapters to language adaptation in a lightweight GAN-based TTS system, which has not been explored directly. The convolutional adapter formulation for the vocoder is borrowed from prior PEFT work in speech understanding. The PSR accent metric is a genuine methodological contribution: it operationalises mispronunciation detection for TTS evaluation and is validated against human MUSHRA ratings. The overall contribution is engineering integration with a useful new evaluation tool; no architectural invention.
Field Significance
Moderate — This paper provides practical evidence that adapter-based PEFT transfers effectively to cross-lingual TTS in the lightweight regime, with a strong ablation characterising where adapters should be placed (vocoder vs. acoustic model) and how many speakers are needed in pre-training. The PSR accent metric is reproducible and externally validated, filling a gap in objective cross-lingual TTS evaluation. The contribution is applied rather than foundational.
Claims
- Adapter-based fine-tuning prevents catastrophic forgetting in TTS models while achieving better naturalness and speaker similarity than full fine-tuning. (§4.3, Table 1; §4.4, Table 2)
- In GAN-based TTS, language-agnostic speaker representations are encoded primarily in the vocoder rather than the acoustic model, as evidenced by the outsized impact of vocoder-only adapters on accent nativeness. (§4.4, Table 2)
- The number and diversity of speakers in target-language pre-training data is a critical prerequisite for high-quality cross-lingual speaker adaptation; single-speaker pre-training leads to severe degradation. (§4.4, Table 2)
- Phoneme substitution rate, derived from a mispronunciation detection model, provides a reliable and reproducible objective proxy for accent nativeness in cross-lingual TTS, correlating well with human MUSHRA accent judgements. (§3.3)
Limitations and Open Questions
Warning
All evaluations are purely objective — no subjective listening test (MOS, MUSHRA) is reported for the proposed systems. The NORESQA-MOS (TorchAudio-SQUIM) values used as “MOS” are automatic predictions, not human ratings; the paper’s decision to discard subjective evaluation limits the strength of quality claims.
The study is restricted to two languages (British English and Castilian Spanish) and a single source speaker per adaptation direction. It is unclear whether the findings generalise to phonologically more distant language pairs, accents, or speakers with more unusual voice characteristics. The backbone model is proprietary and not publicly released, limiting reproducibility. Adapter hyperparameters (bottleneck dimension 16, kernel sizes [3, 5, 3]) were chosen via early experiments but no systematic search is reported.
Wiki Connections
Concept pages most directly informed: speaker-adaptation, multilingual-tts, fine-tuning, gan-vocoder, evaluation-metrics.