arXiv · 2024 · Preprint
Yao et al. (Northwestern Polytechnical University / Ximalaya Inc) · → Paper · Demo: ✓ · Code: ?
StableVC introduces conditional flow matching with a dual-attention gating mechanism for zero-shot voice conversion that independently transfers timbre and style from different unseen speakers, achieving faster-than-real-time inference while surpassing diffusion and autoregressive baselines on both quality and speaker similarity.
Problem
Zero-shot voice conversion systems up to this point converge timbre and style into a single reference — typically a short enrolled clip from the target speaker. This entanglement means that if a speaker has a flat, slow delivery, the converter adopts that delivery along with the voice identity. Independently controlling whose timbre the output has and which style it expresses requires simultaneous adaptation to two different unseen speakers, a scenario no prior system addressed. Beyond controllability, autoregressive language-model-based approaches suffer from slow inference due to token-by-token decoding, while diffusion-based approaches require many reverse steps; neither achieves fast, high-quality conversion in a non-autoregressive framework.
Method
StableVC decomposes a source utterance into three disentangled streams: linguistic content, timbre, and style. Linguistic content is extracted using a pre-trained WavLM model with K-means clustering (1024 clusters), followed by deduplication of adjacent tokens to enable duration re-prediction under different target conditions. Style representation comes from a factorized codec that produces disentangled subspace representations of speech style. Timbre is represented implicitly via mel-spectrograms extracted from multiple reference utterances by the target speaker.
The core generative module is a conditional flow matching network built from Diffusion Transformer (DiT) blocks. It is trained to predict the vector field that maps Gaussian noise to the target mel-spectrogram distribution using optimal-transport conditional flow matching (OT-CFM). At inference, 10 Euler steps are sufficient; the non-autoregressive design means latency scales with utterance length rather than sequence token count.
The architectural novelty is the DualAGC (Dual Attention with Adaptive Gate Control) mechanism embedded in each DiT block. Rather than concatenating timbre and style features, DualAGC runs two parallel cross-attention streams: timbre attention uses the content query against mel-spectrogram values from multiple reference speeches, augmented by a global speaker embedding from a pre-trained speaker verification model as an instructive prior; style attention uses compressed quantized style embeddings as key and value. A zero-initialized learnable scalar gate controls how much style information is injected at each block, allowing the style signal to grow gradually during training without disrupting timbre modeling. A gradient reversal layer (GRL) adversarial loss discourages timbre leakage into the style encoder.
The total model size is 166M parameters (acoustic model only, excluding shared SSL and vocoder). Mel-spectrograms are reconstructed to waveforms with a pre-trained HiFi-GAN vocoder. The system trains on 20k hours of LibriLight filtered by DNSMOS quality.
Key Results
On the zero-shot VC task (Table 1, VCTK timbre references, ESD style references), StableVC achieves nMOS 3.96, UTMOS 4.12, WER 2.03%, and SECS 0.67 (speaker cosine similarity), compared to the next-best diffusion baseline DDDM-VC at nMOS 3.67, UTMOS 3.75, WER 3.07%, and SECS 0.51. All baseline speaker similarity scores cluster around 0.5 SECS, substantially below StableVC’s 0.67.
On the style transfer task (Table 2), where timbre and style references come from different unseen speakers, StableVC achieves F0 correlation 0.75 and RMSE 12.87 versus the next-best diffusion baseline at 0.69 and 15.68.
For inference speed, StableVC runs at RTF 0.146 on a single NVIDIA 3090 GPU, offering a 25x speedup over autoregressive LM-VC (RTF 3.891) and 1.65x over the diffusion baseline DDDM-VC (RTF 0.287). The ablation (Table 3) shows quality stabilises at 10 Euler steps; using 5 steps costs only a modest UTMOS drop from 4.12 to 3.82, while the RTF falls to 0.068.
Ablations (Table 4) confirm that removing the timbre prior and multiple references causes severe intelligibility collapse (WER jumps to 22.16%), removing the adaptive gate damages speaker similarity (SECS drops from 0.64 to 0.36), and removing the GRL loss degrades both style and timbre metrics.
Novelty Assessment
The primary architectural novelty is DualAGC: the combination of two parallel cross-attention streams with a zero-initialized adaptive gate to independently model timbre and style in a single DiT-based flow-matching decoder. Individual elements (cross-attention for speaker timbre, flow matching for TTS/VC, GRL for disentanglement) are established, but their integration in this configuration for independent multi-speaker style-and-timbre control is new. The idea of separating style and timbre transfer to different unseen speakers at inference time is a clear practical extension of existing zero-shot VC, and the paper credibly demonstrates it is the first to operationalise this in a zero-shot setting. The flow-matching choice over diffusion for VC is well-motivated and the speed advantages are substantial. The evaluation is thorough, with subjective and objective metrics on both tasks, ablations of every component, and fair speed comparisons on the same hardware. One limitation is that the style extractor is a third-party factorized codec not described in detail in the paper, which may affect reproducibility.
Field Significance
Tip
High — StableVC demonstrates that independently controlling timbre and style from two different unseen speakers is achievable in a single non-autoregressive flow-matching system, resolving a practical limitation of prior zero-shot VC approaches that conflated style with identity. Its DualAGC mechanism provides a concrete template for injecting multiple disentangled conditioning signals into DiT-based flow-matching decoders without mutual interference, a design pattern applicable beyond voice conversion to style-controlled TTS more broadly.
Claims
- Independent timbre and style transfer from distinct unseen speakers can be achieved without degrading either attribute when conditioning streams are separated via parallel cross-attention with adaptive gating. (§DualAGC, Table 2)
- Flow matching enables zero-shot voice conversion that surpasses diffusion baselines in both sample quality and inference speed, with quality stabilising in as few as 10 ODE steps. (§Conditional Flow Matching, Table 1, Table 3)
- Timbre leakage into style representations is a failure mode in jointly trained VC systems, and adversarial disentanglement via gradient reversal measurably reduces this cross-contamination. (§Training Objectives, Table 4)
- Using multiple reference utterances alongside a pre-trained speaker verification prior as the timbre attention key dramatically improves intelligibility in cross-attention-based timbre modeling, as its removal causes WER to collapse from 2% to over 22%. (§DualAGC, Table 4)
- Non-autoregressive VC generation maintains near-constant latency regardless of utterance length, a qualitative advantage over token-by-token autoregressive decoding that is not captured by fixed-length RTF comparisons. (§Experimental Results on Zero-shot VC)
Limitations and Open Questions
Warning
The factorized codec used as the style extractor is referenced as a public tool but not described in detail in the paper, which restricts exact reproducibility of the style extraction stage.
The evaluation covers English speakers only (VCTK for timbre, ESD for style), leaving cross-lingual style transfer untested. The style space is effectively bounded by the factorized codec’s subspace representation, and it is unclear how fine-grained or compositional the style control is in practice beyond the five ESD emotion categories. The system uses HiFi-GAN as vocoder rather than a codec-based decoder, which may limit audio bandwidth compared to neural codec approaches. Results on real-world noisy conditions are not reported; training was filtered by DNSMOS quality, so robustness to in-the-wild speech is assumed but unverified.
Wiki Connections
- voice-conversion — StableVC extends zero-shot VC to independently transfer timbre and style from different unseen speakers, a new capability relative to prior single-reference VC systems.
- flow-matching — StableVC applies optimal-transport conditional flow matching with DiT blocks as the core generative module for mel-spectrogram synthesis, demonstrating flow matching’s viability for VC beyond TTS.
- zero-shot-tts — the paper’s zero-shot capability (generalising to unseen speakers from short reference clips) shares conditioning strategy and benchmark evaluation with zero-shot TTS, and uses VCTK for timbre reference evaluation.
- disentanglement — DualAGC and the GRL-based style encoder are explicit mechanisms for separating style and timbre into distinct representation spaces, with ablations quantifying the cost of removing each.
- self-supervised-speech — WavLM with K-means clustering is used as the linguistic content extractor, making self-supervised speech representations a core dependency of the content pipeline.
- 2210.02747 (Flow Matching for Generative Modeling) — StableVC’s conditional flow matching module derives directly from the flow matching framework introduced here.
- 2302.00482 (Conditional Flow Matching / OT-CFM) — the optimal-transport conditional flow matching training objective used in StableVC is taken from this paper.
- 2304.09116 (NaturalSpeech 2) — NS2VC, a VC variant of NaturalSpeech 2, is one of the primary diffusion-based baselines compared against StableVC.
- 2301.02111 (VALL-E) — VALLE-VC, an adaptation of VALL-E with VC content representations, is used as the primary autoregressive LM baseline; StableVC achieves 25x faster inference at higher quality.
- 2204.02152 (UTMOS) — UTMOS is used as the primary automatic speech quality predictor throughout StableVC’s objective evaluation.