EMNLP · 2025 · Conference
Advait Joglekar et al. (Indian Institute of Technology Madras) · → Paper · Demo: ✓ · Code: ✓
Proposes EZ-VC, an any-to-any zero-shot voice conversion system that pairs a single off-the-shelf self-supervised speech encoder with a flow-matching speech decoder, avoiding the multiple encoders or adapters typically used to disentangle speaker and content in voice conversion.
Problem
Zero-shot voice conversion systems that generalize to unseen speakers, accents, and especially unseen languages remain difficult to build. Prior textless VC architectures typically stack several specialized encoders or adapters (a content encoder, a timbre or speaker encoder, sometimes a style extractor) to disentangle linguistic content from speaker identity before resynthesis. The paper argues this multi-encoder machinery adds architectural and training complexity without a clear payoff in cross-lingual robustness, and that systems built this way still tend to struggle when converting into or out of languages and accents not seen during training.
Method
EZ-VC uses exactly one pretrained self-supervised speech encoder (Xeus, trained on a multilingual corpus spanning roughly 4,000 languages) and one trainable flow-matching speech decoder. No supervised or labeled data (transcripts, speaker labels) is used at training time. For each training utterance, a mel-spectrogram is computed and the corresponding Xeus features (taken from the 14th of the encoder’s layers, at 75% of model depth) are extracted and quantized with a 500-cluster k-means model trained on a subset of the training data; adjacent duplicate discrete units are removed. The decoder, adopted from the F5-TTS architecture (a non-autoregressive Diffusion-Transformer conditional flow matching model built on E2-TTS and Voicebox), is trained on an infilling task: given the discrete units of an utterance and an unmasked portion of its own mel-spectrogram, the model learns to reconstruct the full mel-spectrogram.
At inference, source and target speech are both passed through the same Xeus + k-means pipeline to obtain discrete units. The target speech’s mel-spectrogram and discrete units serve as the CFM model’s reference (supplying speaker identity), while the source speech’s discrete units supply the content and prosody to be resynthesized; the two unit sequences are concatenated as decoder input and the target mel is discarded once inference completes. A BigVGAN vocoder, trained separately on LibriTTS, converts the generated mel-spectrogram to a waveform. Because the discrete units are speaker-agnostic (derived from a shared k-means codebook trained across languages and speakers) while the target mel prompt carries speaker identity, the architecture achieves content/speaker separation implicitly through this input design rather than through dedicated disentanglement modules.
The decoder is trained from scratch at the F5-TTS base configuration (~300M parameters, 22 layers, 16 attention heads) for 1.35 million updates on 4 NVIDIA RTX 6000 ADA GPUs, using a 501-token character-level vocabulary (500 discrete units plus an unknown token). Training data totals 12,840 hours: 3,060 hours of English (LibriSpeech for American accents, VoxPopuli for European accents, NPTEL lectures for Indian-accented English) and 9,780 hours across five Indian languages (Bengali, Hindi, Tamil, Telugu, Kannada) drawn from Vaani, Common Voice, Shrutilipi, and institutional (IIIT-H, IIT-M) sources.
Key Results
On a 10-sample any-to-any evaluation set spanning multiple languages and accents, judged by 20 volunteers, EZ-VC outperforms Seed-VC, kNN-VC, Vec2Wav 2.0, and Diff-HierVC on every reported metric: speaker similarity (SSIM 0.71), naturalness MOS (3.91), similarity MOS (3.90), and UTMOS (3.56) (Table 1). Vec2Wav 2.0 is competitive on naturalness (3.67 NMOS) but trails substantially on speaker similarity, which the authors attribute to the benefit of a CFM-based decoder for capturing speaker style.
In a targeted comparison against Seed-VC (also CFM-based) on one seen language (English) and two unseen languages (German, Spanish), EZ-VC reports higher UTMOS naturalness in all three settings (e.g. German: 3.71 vs. 2.83; Spanish: 3.49 vs. 3.24) with comparable or better speaker similarity (Tables 4–5), supporting the claim that the approach generalizes to languages absent from training.
Novelty Assessment
The architectural components here, a self-supervised discrete-unit encoder, k-means quantization, and an F5-TTS-style flow-matching decoder, are all pre-existing. EZ-VC’s contribution is showing that this minimal combination, without additional disentanglement encoders, adapters, or timbre shifters used by contemporaries such as AdaptVC, StableVC, and Seed-VC, is sufficient to match or exceed those systems on zero-shot any-to-any and cross-lingual VC. This is primarily an engineering-integration contribution (simplifying an existing recipe rather than introducing a new mechanism), with the multilingual generalization result being the most notable empirical finding. The reported gains rely on a small, custom 10-sample evaluation set rather than an established VC benchmark, which limits how much can be concluded about the true scale of improvement over baselines.
Field Significance
moderate — EZ-VC demonstrates that a single off-the-shelf multilingual self-supervised encoder combined with an existing flow-matching TTS decoder architecture, adapted to voice conversion, can match or exceed voice conversion systems that use multiple specialized encoders. It provides an empirical data point that architectural simplification does not necessarily cost performance in cross-lingual zero-shot voice conversion, though the evaluation is conducted on a small, self-constructed test set rather than a standard benchmark.
Claims
- supports: Combining a single off-the-shelf self-supervised speech encoder (discretized via clustering) with a flow-matching speech decoder is sufficient for competitive any-to-any zero-shot voice conversion, without requiring separate encoders for speaker/content disentanglement.
Evidence: EZ-VC uses only one pretrained SSL encoder (Xeus) with k-means-quantized units for content and an unmasked mel-spectrogram as the speaker reference, outperforming Seed-VC, kNN-VC, Vec2Wav 2.0, and Diff-HierVC on speaker similarity (SSIM 0.71), naturalness MOS (3.91), similarity MOS (3.90), and UTMOS (3.56). (§3, Table 1)
- supports: Flow-matching-based voice conversion decoders can generalize to languages entirely unseen during training with naturalness comparable to or better than on seen languages.
Evidence: Against Seed-VC (also CFM-based) on unseen German and Spanish, EZ-VC achieves higher UTMOS naturalness (3.71 vs. 2.83 German; 3.49 vs. 3.24 Spanish) with comparable or better speaker similarity, while its own seen-language (English) UTMOS (3.76) is close to its unseen-language scores. (§5, Table 4, Table 5)
- complicates: Removing dedicated disentanglement encoders in a voice conversion architecture does not reduce the pretrained encoder’s role as a bottleneck on cross-lingual generalization.
Evidence: The authors note the architecture’s generalization is reliant on the quality of the pretrained speech encoder, and that an encoder trained on only one language would likely not achieve the same level of cross-lingual generalization Xeus (trained on ~4,000 languages) provides. (§Limitations)
- complicates: Simplifying a voice conversion architecture to fewer encoder modules does not necessarily reduce its overall computational requirements.
Evidence: Despite dropping the multiple encoders/adapters used by systems like AdaptVC and StableVC, the authors state their computational requirements remain comparable or higher, consistent with training the ~300M-parameter F5-TTS-based decoder from scratch for 1.35 million updates on 12,840 hours of speech. (§Limitations, §4.2)
Limitations and Open Questions
The paper’s own stated limitations are narrow: generalization depends on the linguistic coverage of the pretrained SSL encoder, and despite architectural simplification, training compute remains comparable to or higher than prior systems. Beyond what the authors state, the subjective evaluation is small in scale (10 audio samples judged by 20 volunteers), which limits the statistical strength of the naturalness and similarity MOS comparisons, and the objective cross-lingual comparison (Table 5) is restricted to a single baseline (Seed-VC) on two unseen languages. The authors also flag a dual-use risk: the same cross-lingual, high-naturalness voice conversion capability that enables the paper’s contribution could be used to produce deepfakes.
Wiki Connections
- Voice Conversion — proposes a minimal any-to-any VC architecture that avoids the multiple disentanglement encoders common in prior VC systems.
- Flow Matching — adopts an F5-TTS-derived conditional flow matching decoder, repurposed from TTS to speech-unit-conditioned voice conversion.
- Multilingual TTS — trains across English and five Indian languages and demonstrates generalization to entirely unseen languages (German, Spanish) at inference time.
- Self-Supervised Speech — relies on a multilingual self-supervised encoder (Xeus), quantized via k-means, as the sole source of content representations, with no transcripts or labels used in training.
- F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching — EZ-VC directly adopts the F5-TTS architecture as its units-to-speech decoder, repurposing its masked-infilling training for voice conversion.
- E2 TTS — F5-TTS (and by extension EZ-VC’s decoder) builds on E2-TTS’s non-autoregressive infilling formulation, which EZ-VC cites as foundational to its decoder design.
- Zero-shot Voice Conversion with Diffusion Transformers — used as the primary comparison baseline (Seed-VC), a CFM-based VC system that EZ-VC outperforms on naturalness and speaker similarity, including on unseen languages.
- StableVC — cited as a contemporary CFM-based VC system requiring three separate feature extractors, contrasted with EZ-VC’s single-encoder design.
- BigVGAN — used as the vocoder converting EZ-VC’s generated mel-spectrograms to waveform audio.
- SeamlessM4T — cited as prior work establishing the convention of extracting SSL features at 75% of encoder depth, which EZ-VC follows for its Xeus feature extraction.