Zero-shot Voice Conversion with Diffusion Transformers

arXiv · 2024 · Preprint

Songting Liu (Nanyang Technological University) · → Paper · Demo: ? · Code: ✓

Seed-VC combines a diffusion transformer with a novel external timbre-shifter training strategy to achieve zero-shot voice conversion that simultaneously improves speaker similarity and reduces word error rate compared to strong baselines.

Problem

Zero-shot voice conversion faces three compounding failures in prior work. First, self-supervised and ASR-derived content features retain residual source speaker information (timbre leakage), causing converted speech to imperfectly mimic the target timbre. Aggressive bottlenecking solutions (k-means quantisation, vector quantisation) that filter out this leakage also discard fine linguistic detail, inflating word error rates. Second, most systems represent target timbre with a single d-vector or x-vector extracted from the reference utterance, which is inadequate for zero-shot generalisation to diverse unseen speakers. Third, VC models are typically trained to reconstruct speech from its own content and timbre, creating a mismatch with inference where content and timbre originate from different speakers.

Method

Seed-VC addresses all three issues through two design choices: an external timbre-shifter module active only during training, and a diffusion transformer that conditions on the entire reference utterance rather than a single timbre vector.

The timbre-shifter (OpenVoice V2, a YourTTS-based model) transforms each source utterance into a version with a randomly altered timbre before semantic feature extraction. Semantic features are then extracted from the shifted utterance using the Whisper-small encoder, which is well-suited to capturing linguistic content while discarding speaker characteristics. Because the encoder now operates on speech that no longer carries the original source timbre, residual leakage is suppressed without requiring discrete quantisation. This strategy also closes the training-inference gap: during training the content and timbre inputs already come from different identities, matching the inference scenario.

The acoustic backbone is a diffusion transformer with a flow-matching objective (L1 loss on the predicted vector field). The transformer follows a U-ViT design with U-Net-style skip connections across layers, rotary positional embeddings, and timestep conditioning as both a prefix token and adaptive layer normalisation. At each denoising step the model receives three inputs: a global timbre vector from a pretrained CAM++ speaker verifier, a length-regulated sequence of semantic features from Whisper, and a noisy segment of the target mel-spectrogram. A randomly selected audio segment from the reference is included as an acoustic prompt during training, enabling the transformer to exploit in-context learning and capture fine-grained timbre nuances beyond what the single speaker vector provides. At inference, the full reference utterance serves as this acoustic context.

Mel-spectrograms are generated at 22.05 kHz (80 bins, 1024 FFT / 256 hop) and converted to waveform with a pretrained BigVGAN-v2 vocoder. The base model has 13 transformer layers with 8 attention heads and 512/2048 embedding/FFN dimensions, trained on Emilia-101k (approximately 101,000 hours) for 400k steps.

For singing voice conversion, a larger variant (17 layers, 12 heads, 768/3072 dimensions, 44.1 kHz) adds quantised F0 conditioning concatenated to the semantic features, with gender-based semitone shifts applied at inference to handle cross-gender pitch ranges.

Key Results

On zero-shot voice conversion against LibriTTS test-clean sources and seed-tts-eval targets (Table 1), Seed-VC achieves speaker cosine similarity (SECS) of 0.8676, outperforming CosyVoice (0.844) and OpenVoice (0.755). It also records the lowest WER (11.99%) and CER (2.92%), compared to OpenVoice (15.46% WER) and CosyVoice (18.98% WER). The ablation comparing Seed-VC with and without full reference enrollment shows a SECS drop from 0.8676 to 0.7948 when only the timbre vector is used, quantifying the value of in-context learning.

DNSMOS scores (SIG/BAK/OVRL) are slightly lower for Seed-VC than for OpenVoice, indicating a small naturalness trade-off associated with higher timbre fidelity.

For zero-shot singing voice conversion on M4Singer (Table 2), Seed-VC outperforms RVCv2 on SECS (0.7405 vs. 0.7264) and CER (19.70% vs. 28.46%), while achieving comparable F0 correlation (0.9375 vs. 0.9404 for RVCv2). RVCv2 has a slight edge on F0 RMSE (30.43 vs. 33.35) and DNSMOS, but Seed-VC achieves this in a zero-shot setting while RVCv2 requires per-character training.

Novelty Assessment

The timbre-shifter training strategy is the paper’s most distinctive contribution. Using a pre-existing imperfect VC model as a data augmentation device during training is a pragmatic and non-obvious way to simultaneously suppress timbre leakage and close the training-inference gap, avoiding both the information bottleneck and discrete quantisation approaches that trade intelligibility for disentanglement. The diffusion transformer conditioned on full reference audio is consistent with the in-context learning trend from zero-shot TTS (VALL-E, VoiceCraft, E2 TTS), but applying it to VC rather than TTS is novel in context. The combination of these two ideas into a unified framework is architecturally interesting. The evaluation is objective-only; no MOS listening tests are reported, which limits conclusions about naturalness. Baselines are strong open-source systems but the training set (Emilia-101k at ~100k hours) is substantially larger than what OpenVoice and CosyVoice were trained on, and a fully controlled data-matched comparison is deferred to future work.

Field Significance

Moderate — Seed-VC introduces an effective training-time data augmentation strategy for zero-shot VC that avoids the intelligibility-timbre trade-off endemic to bottleneck and quantisation approaches. The in-context learning mechanism for timbre representation is a useful transferable idea, already established in zero-shot TTS and here validated for VC. The work does not include subjective evaluation and uses substantially more training data than its baselines, so the magnitude of the claimed gains is not fully characterised.

Claims

• Using an imperfect external VC model as a timbre-shifting augmentation device during training reduces timbre leakage in content representations without requiring discrete bottlenecking, preserving intelligibility. (§3.3, Table 1)
• Conditioning a generative VC model on the full reference utterance via in-context learning yields higher speaker similarity than conditioning on a single speaker embedding alone. (§3.4, Table 1)
• The speaker similarity versus intelligibility trade-off that limits bottleneck-based zero-shot VC approaches can be mitigated by separating timbre perturbation from content feature extraction. (§1, §3.3.1)
• F0 conditioning enables diffusion transformer VC models to extend to zero-shot singing voice conversion with competitive pitch preservation and improved lyric intelligibility over per-speaker trained baselines. (§3.5, Table 2)

Limitations and Open Questions

Warning

No subjective listening tests (MOS or MUSHRA) are reported. All quality comparisons rely on DNSMOS P.835, an automatic metric; naturalness conclusions should be treated with caution.

The training data advantage is substantial (Emilia-101k vs. unspecified but likely smaller datasets for OpenVoice and CosyVoice), and the authors acknowledge that a dataset-controlled comparison is future work. The timbre-shifter used during training is OpenVoice V2, a proprietary-trained model, which may limit reproducibility for researchers without access to a comparable off-the-shelf VC system. The DNSMOS scores indicate that Seed-VC’s naturalness is slightly below OpenVoice despite its higher speaker similarity, suggesting the model may benefit from vocoder-level refinements. Real-time deployment is identified as an open direction; the current system’s latency is not characterised.

Wiki Connections

• voice-conversion — Seed-VC directly addresses the core zero-shot VC problem, introducing a timbre-shifter training strategy and in-context diffusion transformer to improve speaker similarity while maintaining intelligibility.
• zero-shot-tts — Seed-VC adapts the in-context learning mechanism from zero-shot TTS systems (VALL-E lineage) to the VC task, validating that reference-audio conditioning generalises across synthesis paradigms.
• flow-matching — The denoising backbone uses a flow-matching objective (L1 loss on the predicted vector field), demonstrating flow matching’s viability for mel-spectrogram generation in VC.
• diffusion-tts — Seed-VC builds on diffusion transformer architectures (U-ViT with skip connections) originally developed for image and speech diffusion, extending them to voice conversion with timbre conditioning.
• disentanglement — The timbre-shifter training strategy is a pragmatic approach to content-timbre disentanglement that avoids explicit bottlenecks, sidestepping the intelligibility loss associated with VQ or k-means methods.
• evaluation-metrics — Seed-VC evaluates zero-shot VC using speaker cosine similarity (SECS), WER, CER, and DNSMOS P.835, relying entirely on objective metrics without subjective listening tests, which limits naturalness conclusions.
• 2104.00355 (Speech Resynthesis from Discrete Disentangled SSL Representations) - Seed-VC’s approach contrasts with discrete SSL resynthesis, which uses quantisation to disentangle content from speaker at the cost of intelligibility.
• 2301.02111 (VALL-E) - The in-context learning paradigm from VALL-E for zero-shot TTS directly inspires Seed-VC’s use of full reference audio as a transformer context for timbre capture.
• 2406.18009 (E2 TTS) - A zero-shot TTS system using a similar flow-matching diffusion transformer design, cited as related architectural work.
• 2410.06885 (F5-TTS) - Zero-shot TTS with flow matching; provides context for in-context conditioning approaches that Seed-VC adapts for VC.
• 2312.01479 (OpenVoice) - Used as both a baseline and as the timbre-shifter component during Seed-VC’s training, making it a dependency as well as a comparison system.
• 2407.05407 (CosyVoice) - A zero-shot TTS/VC baseline that Seed-VC outperforms on both speaker similarity and intelligibility metrics.
• 2304.09116 (NaturalSpeech 2) - Latent diffusion approach to zero-shot TTS and singing, contextualising Seed-VC within diffusion-based generative approaches.
• 2403.16973 (VoiceCraft) - Zero-shot speech editing and TTS using in-context learning; related work on reference-conditioned generation.
• 2407.05361 (Emilia) - The Emilia-101k dataset used to train Seed-VC.
• 2206.04658 (BigVGAN) - The BigVGAN-v2 vocoder used to convert Seed-VC’s mel-spectrogram outputs to waveform.
• 2407.04051 (FunAudioLLM) - Related Alibaba work on voice understanding and generation.