arXiv · 2026 · Preprint
Youngwon Choi et al. (Maum AI Inc., Humelo Inc.) · → Paper · Demo: ? · Code: ?
Proposes ZeSTA, a domain-conditioned fine-tuning framework that recovers the speaker-similarity loss caused by naively augmenting scarce target-speaker recordings with zero-shot TTS speech.
Problem
Personalized TTS adapts a model to a specific target speaker, but real recordings from that speaker are often extremely scarce. Large-scale zero-shot TTS (ZS-TTS) models can generalize to unseen speakers without additional training, but they are computationally demanding for practical deployment; lightweight acoustic-model-based TTS systems deploy more easily but achieve lower speaker similarity. Fine-tuning a lightweight model on real target-speaker data is a natural alternative, but its quality is highly sensitive to data scarcity. Using ZS-TTS output as synthetic augmentation data for fine-tuning seems like a promising fix, but the paper observes that naively mixing large amounts of ZS-TTS speech with a handful of real recordings improves intelligibility while degrading speaker similarity, and notes that principled strategies for incorporating this kind of synthetic speech into low-resource personalization remain largely unexplored.
Method
ZeSTA targets a lightweight multi-speaker VITS model, pretrained on VCTK and then fine-tuned per target speaker. Given a small set of real reference utterances from a target speaker, two publicly available ZS-TTS systems (Fish-Speech and CosyVoice 2) are used as external data generators to synthesize additional speech in that speaker’s style, using the longest available real utterance as the reference prompt to maximize speaker-style coverage. For each target speaker, only 10% of the original training utterances are retained as real data (Real 10%), and the remaining 90% are replaced with ZS-TTS-synthesized speech (Synth 90%), simulating an extreme low-resource personalization scenario.
The core contribution is domain-conditioned training (DC): each training sample is tagged with a domain label d ∈ {real, synthetic}, reframing TTS adaptation as learning p(y | x, d) rather than p(y | x). Concretely, the domain label is injected through the existing speaker-embedding matrix of the multi-speaker VITS implementation (reduced from a hidden size of 256 to 64), so no new architectural component is added; the text encoder still produces a speaker-agnostic linguistic representation, and the acoustic generation module is additionally conditioned on the domain label, letting linguistic augmentation benefits flow through while quarantining synthetic-domain acoustic artifacts. At inference, synthesis is always conditioned on d = real. A second, complementary technique, real-data oversampling (OS), repeats each real target-speaker utterance by a fixed factor (3x in the experiments) during fine-tuning to further emphasize the scarce real data without touching the model architecture or inference procedure. Both DC and OS are applied on top of standard VITS fine-tuning; the base architecture is unchanged.
Key Results
On LibriTTS and an in-house six-speaker dataset (YoBind), fine-tuning on Real 10% alone yields speaker embedding cosine similarity (SECS) of 0.818 (LibriTTS) but a comparatively high CER/WER of 5.93/12.52. Naively mixing Real 10% with 90% ZS-TTS-synthesized speech (Fish-Speech) substantially improves CER/WER to 4.74/10.35 but drops SECS to 0.765, confirming the intelligibility-versus-speaker-similarity trade-off the paper sets out to address. Adding DC restores most of the lost speaker similarity (SECS 0.807) while mostly preserving the intelligibility gains, at the cost of a small CER/WER increase; adding OS on top of DC pushes SECS further to 0.815 while also improving intelligibility slightly, and the same pattern holds for CosyVoice 2 as the synthetic source. Subjective evaluation with 18 listeners shows MOS naturalness is not degraded by DC+OS relative to naive mixing or full-data fine-tuning (3.92 vs. 3.86 vs. 3.67 on LibriTTS), while an ABX preference test shows listeners prefer the DC+OS system’s speaker similarity over the naive-mixing baseline 70.8% and 61.8% of the time for Fish-Speech and CosyVoice 2 sources respectively (statistically significant, p < 0.05). Ablations show a domain-embedding size of 64 gives the best SECS/CER/WER trade-off (versus 16 and 256), and that speaker-matched synthetic augmentation is necessary for the speaker-similarity gains: speaker-mismatched synthetic data yields only minor intelligibility improvement and lower SECS than speaker-matched augmentation.
Novelty Assessment
The contribution is a lightweight training recipe, not a new architecture: the domain label reuses the existing VITS speaker-embedding mechanism, and the authors explicitly frame the method as leaving the base architecture and inference procedure untouched. The core ideas (domain-adversarial-style conditioning to separate real/synthetic distributions, and oversampling a minority class) are both established techniques from adjacent literature (e.g., domain-conditioned generation, class-imbalance oversampling); the novelty here is in applying them specifically to reconcile the ZS-TTS-as-data-augmentation trade-off for personalized TTS under extreme data scarcity, and in characterizing that trade-off empirically. The evaluation is reasonably careful (two independent ZS-TTS sources, an in-house dataset in addition to a public one, and a speaker-matched vs. speaker-mismatched ablation that isolates the mechanism), but it is confined to a single target architecture (VITS) and modest data scale (six to eight target speakers).
Field Significance
Moderate — the paper documents and partially resolves a specific failure mode (synthetic-augmentation-induced speaker-similarity degradation) in low-resource personalized TTS, providing a simple, architecture-preserving mitigation and useful empirical evidence about the trade-off. It is a targeted, practical engineering contribution to a data-efficient adaptation sub-area rather than a new modeling paradigm.
Claims
- supports: Explicitly conditioning a fine-tuned TTS model on whether each training sample is real or synthetically generated allows the model to retain the intelligibility benefits of synthetic data augmentation while recovering speaker similarity lost to naive mixing.
Evidence: Adding domain conditioning to a VITS model fine-tuned on Real 10% + Synth 90% raises SECS from 0.765/0.789 (naive mixing, Fish-Speech/CosyVoice 2) to 0.807/0.808 on LibriTTS, while CER/WER stay close to the naive-mixing numbers. (§4.2, Table 3)
- complicates: Naively augmenting scarce real target-speaker recordings with large amounts of zero-shot-TTS-synthesized speech improves intelligibility but degrades speaker similarity, producing a direct trade-off rather than a free improvement.
Evidence: On LibriTTS, Real 10% alone gives SECS 0.818 / CER 5.932 / WER 12.52, while Real 10% + Synth 90% (Fish-Speech, no DC/OS) improves CER/WER to 4.738/10.348 but drops SECS to 0.765. (§4.2, Table 3)
- supports: Oversampling the scarce real portion of a mixed real/synthetic fine-tuning set provides additional speaker-similarity gains when combined with domain conditioning, but is unreliable as a standalone technique.
Evidence: DC + OS together reach SECS 0.815 (Fish-Speech, LibriTTS) versus 0.807 for DC alone, whereas OS applied without DC shows limited and unstable improvement across configurations. (§4.2, Table 3)
- refines: The benefit of synthetic-speech augmentation for personalized TTS depends specifically on speaker consistency between the synthetic and real target-speaker data, not merely on the lower acoustic variability typical of TTS-generated speech.
Evidence: Speaker-mismatched synthetic augmentation (same-gender but different speaker) yields only minor CER/WER gains over the real-only baseline and substantially lower SECS (0.792) than speaker-matched augmentation (0.807) under otherwise identical training conditions. (§4.4, Table 6)
Limitations and Open Questions
The method is evaluated on a single target architecture, a lightweight non-autoregressive VITS model, and the authors state that extending it to other TTS architectures and architecture-specific conditioning strategies is left to future work. Objective evaluation relies on ASR-based CER/WER from a Whisper medium model and a single speaker-verification embedding model (ECAPA-TDNN) for SECS, so results inherit any biases of those specific off-the-shelf models. The in-house YoBind dataset is not publicly available, which limits independent reproduction of the results reported on it, and the subjective evaluation uses a modest pool of 18 listeners.
Wiki Connections
- Zero-Shot TTS — treats existing zero-shot TTS systems (Fish-Speech, CosyVoice 2) purely as external data generators, studying how their synthetic output degrades speaker fidelity when mixed into fine-tuning data for a separate lightweight target model.
- Speaker Adaptation — the paper’s central problem is data-efficient personalization of a TTS model to a target speaker under extreme data scarcity, proposing domain-conditioned fine-tuning as the adaptation mechanism.
- Subjective Evaluation — validates the objective speaker-similarity gains from domain conditioning and oversampling with an 18-listener MOS naturalness test and an ABX speaker-similarity preference test.
- CosyVoice 2 — used as one of two independent zero-shot TTS source models for generating synthetic augmentation data, demonstrating the proposed method generalizes across ZS-TTS generators.
- 2602.05770 — cited as prior evidence that longer reference prompts improve speaker similarity in zero-shot TTS, motivating the choice to use the longest available real utterance as the ZS-TTS reference prompt.