arXiv · 2025 · Preprint
Chivereanu & Boros (RACAI) · → Paper · Demo: ✓ · Code: ✓
Extends F5-TTS to Romanian by training a lightweight input-level adapter (a new character embedding matrix plus a ConvNeXt module) while keeping the entire pretrained F5-TTS backbone frozen, preserving voice-cloning capability and enabling limited Romanian-English code-switching.
Problem
Open-source zero-shot TTS systems like F5-TTS offer strong voice cloning but lack explicit Romanian support, and existing Romanian TTS work (e.g., FastPitch adaptations) requires separate models rather than extending a strong multilingual zero-shot backbone. Fully fine-tuning a pretrained model like F5-TTS on a new, lower-resource language risks catastrophic forgetting of its existing capabilities (voice cloning, English and Chinese support) and, on a limited-speaker corpus, risks overfitting to the specific training speakers. The paper asks whether a new language can be added to a strong pretrained TTS model without touching its internal weights at all.
Method
The approach keeps every F5-TTS parameter frozen and adds only an input-level adapter: a new trainable character-embedding matrix for Romanian, followed by the ConvNeXt module already implemented in F5-TTS’s own codebase (repurposed here as the adapter’s context-modeling layer) to capture dependencies between the new character embeddings. This adapter acts as a “soft” letter-to-sound layer, mapping raw Romanian text into the same continuous input space F5-TTS’s frozen text encoder was originally trained to consume, without requiring hand-crafted phonetic rules. For code-switched Romanian-English utterances, the method masks the input into Romanian and English character spans; Romanian spans are routed through the new trainable embedding and ConvNeXt adapter, while English spans use F5-TTS’s original frozen embedding layer directly, and the two resulting embedding sequences are summed before being passed to the frozen backbone. The adapter is trained with F5-TTS’s original flow-matching generative objective on the SWARA Speech Corpus (21+ hours, 17 speakers), for 40,500 steps on a single A100 GPU (~12 hours).
Key Results
In a 16-20 native-speaker subjective evaluation across three tasks, listeners frequently could not reliably distinguish the adapter’s synthesized speech from the original reference speaker in a similarity test, suggesting the frozen backbone’s speaker-cloning ability transfers well to the new language. On pronunciation and naturalness, a dedicated Romanian-specific baseline (MMS-TTS-RON, a VITS model trained directly on Romanian) scored higher on most test sentences, which listeners partly attributed to a perceptible residual English accent in the adapter’s output. Code-switching evaluation showed the merged-embedding approach produces perceptibly natural Romanian-English transitions in some cases, though the authors describe the underlying tests as limited in scale. Objectively, the adapter-based system (RO-F5TTS) achieves a lower WER than MMS-TTS-RON (5.27% vs. 5.77%) on 1,000 generated Common Voice Romanian samples, but a fully fine-tuned F5-TTS variant trained on the same data achieves an even lower WER (3.62%). The relationship reverses for speaker similarity: RO-F5TTS’s frozen-backbone adapter achieves substantially higher cosine similarity to the reference speaker (mean 0.9013) than the fully fine-tuned variant (mean 0.7946), indicating full fine-tuning better captures Romanian pronunciation at some cost to voice-identity preservation.
Novelty Assessment
The adapter mechanism itself (a new embedding matrix plus a lightweight context-modeling module, reusing an architectural component already present in the base model) is a standard, well-established technique for parameter-efficient language extension rather than a novel architectural contribution; the paper’s value is in applying and honestly evaluating this recipe on a genuinely under-served language (Romanian) with a strong modern zero-shot TTS backbone, including a direct, transparent comparison against the alternative of full fine-tuning that surfaces a real accuracy-versus-identity-preservation trade-off.
Field Significance
moderate — This is a practically useful, honestly-evaluated extension of a strong open-source TTS system to a low-resource language, with real comparisons against both a dedicated single-language system and a full-fine-tuning alternative, and a genuine (if limited) demonstration of code-switching without explicit bilingual training. Its scope is narrow (one language, one base model, a well-established adapter technique) rather than introducing new methodology, but the trade-off it surfaces between adapter-based and fully-fine-tuned language extension is a useful, transferable data point for similar future work.
Claims
- supports: A pretrained zero-shot TTS model’s language coverage can be extended via a lightweight, frozen-backbone input adapter alone, better preserving the backbone’s original voice-cloning fidelity than full fine-tuning does.
Evidence: The frozen-backbone adapter achieves substantially higher speaker-similarity cosine scores (mean 0.9013) than a fully fine-tuned F5-TTS variant trained on the same Romanian data (mean 0.7946). (§4.2.2, Table 5)
- complicates: Extending a TTS model to a new language via a lightweight, frozen-backbone adapter trades transcription accuracy for voice-identity preservation relative to full fine-tuning, rather than improving both simultaneously.
Evidence: The fully fine-tuned F5-TTS model achieves lower WER, MER, and WIL than the adapter-based approach on the same Romanian objective evaluation, while the adapter-based approach achieves substantially higher speaker similarity. (§4.2.1-4.2.2, Tables 4-5)
- supports: Combining a new language’s trainable embeddings with a frozen backbone’s existing embeddings for a previously-learned language can produce limited but perceptible code-switching ability, without any explicit joint multilingual training.
Evidence: Merging Romanian adapter embeddings with the frozen model’s original English character embeddings within the same utterance produced code-switched speech rated as showing promising naturalness in a dedicated listener evaluation, despite no explicit training to represent both languages simultaneously. (§3.2.1, §4.1.3)
- complicates: A frozen-backbone language-extension adapter can introduce a persistent source-language accent into the new language’s pronunciation, a limitation not resolved by the adapter’s capacity alone.
Evidence: Listeners rated a dedicated Romanian-specific baseline (MMS-TTS-RON) higher than the adapter-based approach on most pronunciation-and-naturalness test sentences, attributed in part to a perceptible residual English accent in the adapter-generated Romanian speech. (§4.1.2)
Limitations and Open Questions
The authors state it remains unclear whether the residual English-accent limitation stems from the adapter’s own capacity, from the frozen base model never having been trained on Romanian, or from the interplay between the two. Code-switching evaluation used only 9 examples and is explicitly described as requiring further qualitative and mechanistic analysis. The current code-switching approach relies on manual language-span annotation rather than automated detection, which the authors identify as future work, alongside explicit bilingual training to improve code-switching quality and extending the adapter framework to additional languages and base models.
Wiki Connections
- Multilingual TTS — extends F5-TTS’s own zero-shot voice cloning to a new language (Romanian) via a frozen-backbone adapter, with dedicated Romanian-English code-switching evaluation.
- Zero-Shot TTS — preserves and evaluates F5-TTS’s original zero-shot voice-cloning capability in the newly added Romanian language.
- Subjective Evaluation — reports a 16-20 native-speaker listening study across speaker-similarity, pronunciation/naturalness, and code-switching tasks.
- F5-TTS — the frozen backbone model this paper directly extends via an input-level adapter, with no modification to its internal weights.