Abstract
Zero-shot TTS synthesizes an unseen speaker from reference audio without speaker-specific fine-tuning. It is established across codec-language-model, flow, diffusion, and hybrid systems, but cloning quality remains bounded by the prompt, training diversity, language match, and the tension between intelligibility, identity, prosody, and speed.
Current State
Reference conditioning works, but the reference is part of the system. Multiple independent lineages preserve unseen-speaker identity from short prompts, from VALL-E and F5-TTS to real-time conversion systems such as RT-VC. Performance changes materially with prompt duration, noise, channel, speaking style, and selection; there is no corpus-supported universal minimum prompt length.
Representation design determines which qualities survive. Discrete semantic tokens make linguistic planning and language-model training convenient, while continuous acoustic targets can retain fine speaker detail. Hybrid systems explicitly separate semantic planning from acoustic reconstruction. No representation wins every dimension: stronger semantic structure can improve intelligibility while removing acoustic detail, and high-fidelity targets can make generation more expensive or difficult to control (AudioLM, Vevo, Autoregressive Speech Synthesis without Vector Quantization).
Generalization comes from data as well as architecture. Broader speaker, language, and acoustic coverage consistently improves unseen-speaker robustness, but many frontier systems use proprietary corpora. Architecture, scale, filtering, and evaluation therefore remain confounded. Low-resource or parameter-efficient adaptation can help, but it is distinct from strictly zero-shot inference and should be reported separately.
Cross-lingual cloning is still conditional. Most studies find language, accent, or prosody leakage when prompt and synthesis languages differ, although one contradictory result shows the failure is not universal. Language-pair difficulty and multilingual training coverage matter, so the evidence supports vulnerability rather than inevitable failure (ControlSpeech, 2025.acl-long.598, Prosody Diversity Benchmark).
Evaluation and deployment expose unresolved trade-offs. WER or CER and speaker-embedding similarity frequently move in different directions and do not fully capture perceived identity or naturalness. Acceleration can preserve usable quality, but hardware and latency definitions vary. Long-form generation additionally needs alignment, chunking, or memory controls to prevent repetition and omission.
Method Landscape
- Autoregressive codec-language models predict speech tokens sequentially and support in-context voice conditioning. They offer flexible language modeling but inherit long sequences, exposure errors, and codec bottlenecks.
- Hybrid semantic–acoustic systems use autoregressive planning followed by non-autoregressive flow or diffusion reconstruction. They separate intelligibility from acoustic fidelity but add interfaces whose errors are hard to attribute.
- Non-autoregressive flow and diffusion models generate acoustic targets in parallel or through iterative refinement. They reduce sequential failure modes, with quality and latency controlled by solver steps and conditioning design.
- Encoder–decoder, GAN, and continuous-latent systems remain viable for efficient or specialized cloning. Their smaller share in recent work does not imply inferiority under matched data.
- Modular hybrids combine text front ends, speaker encoders, semantic tokens, acoustic models, and vocoders. They make components replaceable but distribute identity and prosody across more interfaces.
Key Trade-offs
- Intelligibility versus identity: improving linguistic predictability can suppress speaker or prosodic detail.
- Prompt convenience versus robustness: short clean references are easy to benchmark but do not represent noisy, far-field, or stylistically mismatched use.
- Discrete modelability versus acoustic fidelity: tokens simplify language modeling while continuous targets retain detail at higher generation cost.
- Speed versus hard-case quality: fewer steps and lower token rates reduce latency but can expose omissions, repetition, or identity drift.
- Capability versus safeguards: personalization and accessibility benefits coexist with unauthorized-cloning risk, which quality benchmarks rarely measure.
Open Questions
- Which reference properties predict success across languages, channels, and speaker demographics?
- How can evaluation separate perceived identity, acoustic similarity, intelligibility, and style without relying on one speaker encoder?
- Which representation split best supports streamable, long-form, expressive cloning?
- How much recent progress comes from architecture rather than proprietary data and filtering?
- Which safeguards deter unauthorized cloning without blocking legitimate personalization?
Go Deeper
Read Zero-Shot Text-to-Speech: In Depth for the evidence on prompts, representation families, cross-lingual transfer, robustness, evaluation, long-form generation, and deployment implications.
Scope
This assessment covers 203 papers from 2017 through September 2025, with 164 published in 2025. The graph spans TTS, voice conversion, spoken systems, codecs, evaluation, and singing, but is weighted toward recent TTS systems and often proprietary large-scale training. Sixteen clusters are strongly supported and two are contested; independence is strongest where findings recur across organizations and architectures, not where successor systems share a lineage.
This page is generated from wiki/_claims/zero-shot-tts.yaml (digest date: 2026-07-26).
For complete structured claims and paper-level provenance, use the source YAML.