arXiv · 2026 · Preprint
Qixi Zheng et al. (Shanghai Jiao Tong University) · → Paper · Demo: ✓ · Code: ✓
X-VC performs zero-shot voice conversion as a single non-iterative transformation inside the latent space of a pretrained semantic-acoustic codec. A small MMDiT-style converter jointly attends over source codec latents and a target mel reference and is trained on generated parallel pairs, which yields streamable conversion at 240 ms model latency and an offline RTF of 0.014.
Problem
Zero-shot VC for interactive use has to deliver three things at once: content preservation, faithful transfer of an unseen target speaker’s timbre, and low-latency streaming. Strong recent systems get their timbre fidelity from rich, sequence-level target conditioning (full-reference attention, timbre-dependent features) and, in the case of Seed-VC, from training on generated pseudo-parallel pairs. They mostly rely on iterative diffusion or flow sampling, though, and that makes streaming expensive. Lightweight streaming systems (StreamVC, Conan, MeanVC) run with low latency but give up some speaker similarity and content fidelity on unseen speakers. Codec latents are an attractive space to work in because a pretrained decoder already handles waveform reconstruction. Even so, the authors argue that codec-space VC still has to answer how heterogeneous conditions should be combined under streaming constraints: codec-space source, mel-space frame-level target cues, and an utterance-level speaker vector.
Method
X-VC wraps a trainable “acoustic converter” around a pretrained SAC codec (16 kHz, 62.5 Hz configuration). The source segment passes through SAC’s frontend (semantic encoder, VQ and adapter, acoustic encoder and VQ, and prenet), and conversion happens on the unified 1024-dimensional latent after the prenet. The authors deliberately avoid converting a single acoustic-code stream: they argue that SAC’s acoustic branch is optimised for reconstruction and still carries content, so SAC is better treated as a high-fidelity frontend than as a disentangled tokenizer. The codec’s encoder-side components and quantizers stay frozen. The converted latent goes to the SAC decoder for waveform reconstruction.
The converter takes three inputs. The first is the source codec latents z. The second is a 128-bin mel-spectrogram c from the target reference (frame-level condition). The third is a 192-dimensional ERes2Net speaker embedding g (utterance-level condition). Following MMDiT, source and condition get separate input projections. They are then concatenated for joint multi-head attention in each of 6 transformer blocks (hidden size 512, 8 heads), and both streams are updated layer by layer, so the condition is not a static context. The speaker embedding passes through an MLP and modulates the source branch through adaptive normalisation (scale, shift and gating). The converter has 44M parameters and produces the converted latent in a single forward pass. The total system, including the frozen codec and speaker encoder, is 539M parameters.

Training uses generated pairs. For each random utterance pair (x0, x1), a pretrained Seed-VC small model produces cross-converted versions in both directions, which adds about 20,000 h of generated speech to about 10,000 h of real English and Chinese speech (DNSMOS-filtered Emilia plus LibriTTS). Each training example is randomly assigned one of three roles: standard (generated source, real target), reversed (real source, generated target) or self-reconstruction (same real utterance), with probabilities (0.4, 0.4, 0.2). Segments are cropped to 2.4 s to match SAC training. The mel condition comes from the target utterance with the target segment removed, which prevents trivial copying. The losses are SAC’s own (semantic MSE, mel reconstruction, speaker-similarity MSE and adversarial discriminator losses), without the VQ loss. Training ran for 446k steps on 8 H200 GPUs.
For streaming, the reference mel and speaker embedding are computed once up front. The source is processed in chunks with a fixed 2.4 s window made up of history, current (120 ms), overlap (20 ms) and future (100 ms) regions. Only the current region is emitted, and a cosine cross-fade with the previous step’s overlap smooths chunk boundaries. The codec is not strictly causal, so this chunking approximates streaming rather than making the model causal.
Key Results
Streaming (Table 1, Seed-TTS-Eval). X-VC has the lowest WER of the streaming systems on both test-en (3.14 vs 3.31 for Seed-VC tiny) and test-zh (2.65 vs 3.36 for Seed-VC tiny and 4.89 for MeanVC). Speaker similarity is clearly ahead on English (0.62 vs 0.40 for Seed-VC tiny) and tied with MeanVC on Chinese (0.72). UTMOS is best on English (3.07) but below Seed-VC tiny on Chinese (2.35 vs 2.46). Model latency is 240 ms, against 250 ms for MeanVC and 380 ms for Seed-VC tiny. Compute per chunk on an RTX 3090 is 58.17 ms, slower than MeanVC (32.12 ms) and faster than Seed-VC tiny (120.75 ms). X-VC’s processing window is much larger than either baseline’s, so the compute comparison is not like for like.
Subjective speaker similarity (Table 2). With 15 listeners, SMOS is 3.98 ± 0.10 (test-en) and 3.89 ± 0.13 (test-zh). The baselines score 3.56 and 3.38 (Seed-VC) and 3.44 (MeanVC, Chinese only). The table does not say which Seed-VC variant was rated. Naturalness was not assessed by listeners, only by UTMOS.
Offline (Table 3). The quality gap narrows. X-VC still leads or ties on SIM (0.63 en, 0.73 zh), but on English WER (2.83) it trails both Seed-VC small (2.57) and Seed-VC tiny (2.24). On test-zh its WER (1.99) is second to Seed-VC tiny (1.79), and its UTMOS is below Seed-VC tiny on both languages. The clear offline advantage is speed: RTF 0.014 against 0.069 to 0.161 for the baselines, which fits the one-step design.
Cross-lingual (Table 4). X-VC ties Seed-VC small on English-to-Chinese SIM (0.52) and leads on Chinese-to-English SIM (0.49 vs 0.42). Its WER is mid-pack in both directions, behind Seed-VC tiny.
Ablations (Table 5, test-zh, 300k steps). Freezing the condition stream (not updating c across layers) drops SIM from 0.72 to 0.66. Removing the speaker embedding g drops it to 0.61. Both ablations also raise WER. The data-construction ablations (standard-only, reversed-only, no reconstruction) leave SIM essentially unchanged at 0.71 to 0.72 and only change WER (2.14 to 2.31 vs 2.02).
The baselines are limited to Seed-VC and MeanVC, and MeanVC has no English results because its content encoder only supports Chinese. X-VC’s training targets come from Seed-VC small, so the comparison is partly student against teacher.
Novelty Assessment
The ingredients are mostly borrowed. Converting in codec latent space follows recent codec-generation practice. The joint-attention two-stream block is MMDiT, AdaLN speaker modulation is standard, generated-pair supervision comes from Seed-VC, and the training losses are SAC’s unchanged. What is new is how the pieces are combined and specialised for VC: an MMDiT-style converter that keeps updating the reference mel stream alongside the source, which the ablation shows matters for speaker similarity; the three-way role-assignment scheme over generated pairs; and a chunking window aligned to the codec’s training segment length. The authors argue that converting after the prenet rather than on one codec stream is the right choice, but they do not test that against the alternative. Overall the contribution is a well-engineered single-pass converter design rather than a new paradigm. Its main empirical point is that a non-iterative codec-space converter can match or beat diffusion and mean-flow VC baselines on speaker similarity while running several times faster.
Field Significance
Moderate. X-VC is concrete evidence that zero-shot VC can drop iterative sampling entirely: with a strong pretrained semantic-acoustic codec frontend and generated-pair supervision, a 44M-parameter one-step converter is competitive with diffusion and mean-flow systems and is streamable. The gains over the baselines are clearest for speaker similarity and offline speed. Content and naturalness gains are not consistent across settings, and the comparison set is small.
Claims
- supports: One-step conversion in the latent space of a pretrained neural codec can give zero-shot voice conversion speaker similarity competitive with iterative diffusion and flow-based converters at a much lower real-time factor.
Evidence: A 44M-parameter single-pass converter over frozen SAC latents reaches SIM 0.63/0.73 (en/zh) offline, matching or exceeding Seed-VC small/tiny and MeanVC, at RTF 0.014 vs 0.069 to 0.161 for the baselines on the same RTX 3090. (§5.2, Table 3)
- supports: In zero-shot voice conversion, frame-level reference acoustics and an utterance-level speaker embedding carry complementary target information, and losing either one lowers speaker similarity.
Evidence: On test-zh, removing the utterance-level ERes2Net embedding lowers SIM from 0.72 to 0.61, and keeping the mel condition stream fixed instead of updating it with joint attention lowers SIM to 0.66; both ablations also increase WER. (§5.4, Table 5)
- refines: The benefit of training voice conversion on generated parallel pairs shows up mainly in content preservation rather than speaker similarity, and depends on how source and target roles are assigned.
Evidence: Training on standard-only, reversed-only, or no-reconstruction role mixes leaves SIM at 0.71 to 0.72 but raises WER from 2.02 to 2.14 to 2.31 relative to the mixed (0.4/0.2/0.4) assignment. (§5.4, Table 5)
- complicates: Chunkwise streaming of a non-causal speech codec trades extra per-chunk compute for streaming quality, because each step has to reprocess a long context window.
Evidence: X-VC re-encodes a 2.4 s window per 120 ms emitted chunk to match the codec’s training segment, giving 58.17 ms compute per chunk vs 32.12 ms for MeanVC, even though its algorithmic latency (240 ms) is lower. (§3.5, §5.1, Table 1)
- complicates: Gains from a streaming-oriented voice conversion design do not carry over evenly to offline intelligibility and naturalness.
Evidence: X-VC has the best streaming WER on both languages, but offline its English WER (2.83) trails both Seed-VC variants and its UTMOS falls below Seed-VC tiny on both test-en and test-zh. (§5.1, §5.2, Tables 1 and 3)
Limitations and Open Questions
Warning
X-VC is trained on about 20,000 h of pairs generated by Seed-VC small and then evaluated mainly against Seed-VC variants and MeanVC. The comparison set is narrow, partly a teacher-student comparison, and on English it covers only Seed-VC, because MeanVC’s content encoder supports Chinese only.
- Naturalness is assessed only with UTMOS. The single listening test (15 listeners) covers speaker similarity only, and the number of rated samples is not reported.
- The streaming design relies on a 2.4 s window around each 120 ms chunk, so streaming is approximated over a non-causal codec rather than achieved with a causal model. Per-chunk compute is higher than the lightest baseline.
- The total system is 539M parameters, about 11 times the size of MeanVC (47M). Most of this is the frozen codec, but it still matters for deployment.
- The choice to convert after the SAC prenet rather than on a single codec stream is argued but not ablated, and the paper does not test whether other codecs would work as frontends.
- Evaluation covers English and Chinese only, and there are no tests on expressive, noisy or out-of-domain speech.
Wiki Connections
- Voice Conversion — Proposes a zero-shot any-to-any VC system with VC-specific evaluation (target speaker SIM, SMOS, content WER) in same-language and cross-lingual settings.
- Zero-Shot TTS — Conversion to unseen target speakers from reference speech alone is evaluated directly on Seed-TTS-Eval’s non-parallel zero-shot pairs.
- Streaming TTS — Chunkwise inference with history, overlap and future windows and cosine cross-fades is evaluated directly for model latency and per-chunk compute.
- Neural Audio Codec — Treats a pretrained semantic-acoustic codec’s unified latent as the conversion space and argues that its acoustic branch is not disentangled enough to convert on its own.
- Subjective Evaluation — Reports a 15-listener SMOS test of speaker similarity with 95% confidence intervals.
- SAC — The pretrained codec X-VC builds on. Its frozen frontend, decoder and training losses are reused directly.
- Seed-VC — Main baseline and source of the generated training pairs (Seed-VC small). X-VC extends Seed-VC’s generated-pair supervision with three-way role assignment.
- MeanVC — Streaming mean-flow baseline. X-VC matches its Chinese SIM with lower WER and latency but higher per-chunk compute.
- Conan — Cited as a prior chunkwise online zero-shot VC framework that motivates the streaming design.
- Vevo — Cited as a recent zero-shot VC system showing that timbre-dependent target conditioning matters.
- DSA-Tokenizer — Contrasted as a tokenizer with explicit semantic-acoustic disentanglement, which motivates converting on SAC’s fused latent instead.
- Seed-TTS — Supplies the Seed-TTS-Eval test sets used for all same-language and cross-lingual evaluations.
- Emilia — Main source of real training speech (DNSMOS-filtered) for both real and generated pairs.