arXiv · 2026 · Preprint
Jose Giraldo et al. (Barcelona Supercomputing Center, Universitat de Barcelona, DFKI GmbH) · → Paper · Demo: ? · Code: ?
Fine-tunes two existing non-autoregressive zero-shot TTS architectures (F5-TTS and StyleTTS2) on in-the-wild data cleaned with a speech-enhancement pipeline, and systematically measures how reference-audio-prompt quality and length affect zero-shot synthesis quality, intelligibility, and speaker similarity.
Problem
Zero-shot TTS models are typically trained on clean, studio-quality audiobook corpora (LibriTTS, MLS), but there is growing interest in synthesizing spontaneous, “in-the-wild” speech that better reflects real conversational use. In-the-wild training data introduces two specific obstacles: environmental noise degrades automatic transcription quality and can prevent convergence to intelligible speech, and the diversity of prosody, hesitations, fillers, and pacing in spontaneous speech makes duration modeling substantially harder than in controlled recordings. This paper is a system-description submission to the TTS track of the 2026 WildSpoof challenge, which evaluates zero-shot TTS systems on the TTS In the Wild (TITW) dataset.
Method
The system combines three ingredients: speech enhancement of noisy training and prompt audio, non-autoregressive TTS architectures with flexible duration modeling, and a systematic sweep of inference-time reference-prompt conditions. Two backbone architectures are evaluated: F5-TTS, a flow-matching text-to-speech model, and StyleTTS2, a style-diffusion and adversarially trained model. Both are fine-tuned rather than trained from scratch: F5-TTS is fine-tuned from the pretrained F5-TTS v1 Base checkpoint for 75,000 steps (learning rate 1e-5, 5,000-step warm-up, batch size 76,800 audio frames), keeping the original tokenizer vocabulary to preserve the pretrained text representation space; StyleTTS2 is fine-tuned from a LibriTTS-pretrained checkpoint for 12,000 steps (5 epochs, learning rate 1e-4, batch size 16), with diffusion training enabled after the first iteration and joint training starting at the second epoch. A from-scratch alternative, a smaller 12-layer/16-head F5-TTS variant (“F5-TTS tiny”) trained for 1M steps directly on TITW, is included as a comparison point.
Both models are fine-tuned on the TITW Easy partition after an additional enhancement pass with the Sidon speech-restoration model, applied on top of the Demucs-based enhancement TITW-Easy already ships with; the paper reports that Sidon improves signal quality by a large margin over Demucs alone. At inference time, both architectures require a reference audio prompt to perform zero-shot synthesis (used as an acoustic-encoder input for StyleTTS2, and as a masked-region-filling prompt for F5-TTS), and the paper varies both the length (shortest vs. longest available prompt per speaker) and the enhancement state (raw vs. Sidon-enhanced) of that prompt to measure the resulting effect on output quality. Evaluation uses the VERSA toolkit to compute UTMOS and DNSMOS Pro for audio quality, WER (via a NeMo Conformer-large ASR model) for intelligibility, and F0 RMSE and Speaker Encoder Cosine Similarity (SECS, using ESPnet’s VoxCeleb-trained RawNet3 embeddings) for speaker similarity, on the TITW-KSKT (Known Speaker, Known Text) and TITW-KSUT evaluation subsets.
Key Results
Prompt length has a clear effect on speaker similarity: on TITW-KSKT, shortening the reference prompt by only about two seconds on average drops SECS from 0.35 to 0.24 for F5-TTS and from 0.19 to 0.14 for StyleTTS2, while WER is essentially unaffected for F5-TTS but rises sharply for StyleTTS2 (0.21 to 0.49). Enhancing the reference prompt consistently improves audio-quality scores for both models on both evaluation subsets (e.g., F5-TTS UTMOS rises from 3.68 to 4.02 and DNSMOS from 3.22 to 3.47 on TITW-KSUT) and also improves WER, more markedly for StyleTTS2 than F5-TTS. However, prompt enhancement degrades speaker similarity on TITW-KSKT for both models (F5-TTS SECS drops from 0.35 to 0.28; StyleTTS2 from 0.19 to 0.18). The from-scratch “F5-TTS tiny” baseline trained on TITW alone reaches only 3.27 UTMOS and 0.10 SECS, well below the fine-tuned checkpoints, which the authors present as confirming the value of transfer learning from large-scale pretraining over training on limited in-the-wild data. The abstract’s headline numbers (4.21 UTMOS, 3.47 DNSMOS) come from two different configurations in Table 2 (StyleTTS2+enhanced-prompt for UTMOS, F5-TTS+enhanced-prompt for DNSMOS), not a single system; the system actually selected for the WildSpoof challenge submission was F5-TTS with the enhanced reference prompt, chosen for its combined WER, DNSMOS, and speaker-similarity performance.
Novelty Assessment
This is a challenge system-description paper, not an architectural contribution: both backbone TTS models (F5-TTS, StyleTTS2) and the enhancement model (Sidon) are used off the shelf, fine-tuned rather than redesigned. The genuine contribution is the systematic empirical analysis of how reference-audio-prompt length and enhancement state trade off against each other in zero-shot TTS on spontaneous, in-the-wild data, particularly the finding that prompt enhancement improves perceptual audio quality and intelligibility but can simultaneously degrade speaker similarity. This is an engineering-integration and empirical-benchmark contribution, appropriate in scope and ambition for a challenge submission report.
Field Significance
low — This paper documents a challenge submission combining existing pretrained TTS architectures and an existing speech-enhancement model, with no new architecture or training method. Its value lies in a small, well-controlled ablation of reference-prompt length and enhancement quality for zero-shot TTS on spontaneous speech, providing a concrete data point on the quality/speaker-similarity trade-off introduced by prompt enhancement.
Claims
- supports: Reference-audio-prompt length affects zero-shot speaker similarity, with shorter prompts producing measurably lower speaker-similarity scores even when the duration difference is small.
Evidence: Shortening the reference prompt by roughly two seconds on average dropped SECS from 0.35 to 0.24 for F5-TTS and 0.19 to 0.14 for StyleTTS2 on the TITW-KSKT set. (§3, Table 1)
- supports: Denoising or enhancing the reference audio prompt at inference time improves perceptual audio-quality and intelligibility metrics for zero-shot TTS, even when the target voice itself originates from noisy, in-the-wild recordings.
Evidence: UTMOS and DNSMOS Pro improved for both F5-TTS and StyleTTS2 on both TITW-KSKT and TITW-KSUT after Sidon enhancement of the prompt, and WER improved for both models, more markedly for StyleTTS2. (§3, Table 2)
- complicates: Enhancing the reference audio prompt can degrade speaker-similarity scores even as it improves audio quality and intelligibility, indicating a trade-off rather than a uniform benefit.
Evidence: On TITW-KSKT, SECS decreased from 0.35 to 0.28 for F5-TTS and from 0.19 to 0.18 for StyleTTS2 after prompt enhancement, despite gains in UTMOS, DNSMOS, and WER. (§3, Table 2)
- refines: Whether enhancing a reference speech prompt degrades intelligibility depends on the specific enhancement model’s content-preservation properties, not enhancement as a general operation.
Evidence: The paper contrasts its Sidon-based enhancement, which did not degrade intelligibility, with a prior report of intelligibility degradation from MP-SENet-based prompt denoising, attributing the difference to MP-SENet’s known content-preservation issues. (§3)
- supports: Fine-tuning a large pretrained zero-shot TTS model on a modest amount of in-the-wild data yields substantially better output quality and speaker similarity than training a smaller model of the same architecture from scratch on that same data.
Evidence: A 12-layer/16-head F5-TTS variant trained from scratch for 1M steps on TITW reached only 3.27 UTMOS and 0.10 SECS, versus 3.51 UTMOS and 0.35 SECS for the fine-tuned pretrained F5-TTS checkpoint on the same evaluation condition. (§2, §3, Table 2)
Limitations and Open Questions
The evaluation relies entirely on automated proxy metrics (UTMOS, DNSMOS Pro, WER, SECS, F0 RMSE) computed via the VERSA toolkit; no human listening test or MOS study is reported, so perceptual quality and speaker-identity judgments are not independently verified by human raters. The prompt-length experiment compares only two conditions (shortest vs. longest available prompt per speaker) with an average duration gap of about two seconds, which is a narrow operating range for generalizing about prompt-length effects. The paper does not report overall model parameter counts, nor does it identify which audio codec or vocoder the fine-tuned systems use. As a challenge submission report, the analysis is scoped to the WildSpoof TTS track evaluation protocol and TITW dataset; it does not test whether the prompt-length or enhancement trade-offs generalize to other in-the-wild corpora.
Wiki Connections
- Zero-Shot TTS — evaluates how reference-audio-prompt length and enhancement quality affect zero-shot voice cloning performance in two established zero-shot TTS architectures.
- Flow Matching — fine-tunes the flow-matching-based F5-TTS model as one of its two backbone architectures.
- Diffusion TTS — fine-tunes StyleTTS2, whose acoustic decoder relies on style diffusion, as one of its two backbone architectures.
- Evaluation Metrics — systematically measures how automated audio-quality, intelligibility, and speaker-similarity metrics respond to controlled changes in reference-prompt length and enhancement.
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching — fine-tunes the pretrained F5-TTS v1 Base checkpoint as one of its two backbone zero-shot TTS systems.
- Emilia — cited as an example in-the-wild speech dataset motivating the shift toward spontaneous-speech TTS training data, alongside TITW.
- The Text-to-speech in the Wild (TITW) Database — uses the TITW dataset’s Easy partition for fine-tuning and its KSKT/KSUT subsets for evaluation, the dataset this paper’s challenge track is built around.