ACL · 2025 · Conference
Yisi Liu et al. (UC Berkeley) · → Paper · Demo: ✓ · Code: ✗
A zero-shot real-time voice conversion system that achieves 61.4 ms CPU latency — 13.3% faster than the prior SOTA StreamVC — by combining the SPARC articulatory coding framework with a causal DDSP vocoder.
Problem
Prior zero-shot voice conversion systems either lack interpretability (relying on abstract latent spaces from SSL models such as HuBERT) or are too computationally expensive for real-time CPU deployment. The Speech Articulatory Coding (SPARC) framework provides grounded, interpretable disentanglement of speaker and content via vocal tract kinematics (EMA features), but its encoding pipeline is not designed for streaming and is too slow for real-time use.
Method
RT-VC decomposes voice conversion into four streamable modules:
- Source extractor — a causal SoundStream-style encoder that extracts pitch (CREPE-supervised), periodicity, and loudness from a mel spectrogram at 200 Hz. Trained with noise augmentation for robustness.
- EMA inverter — a causal dilated convolution network (11 layers) accepting MFCC input, supervised on pseudo-EMA labels generated offline by the SPARC AAI model (from WavLM). Information perturbation augmentation (random EQ, pitch/formant shift) encourages content-only encoding.
- Speaker encoder — frozen WavLM CNN features combined with a trainable dilated conv network, pooled to a 128-d speaker embedding weighted by periodicity.
- DDSP vocoder — a harmonic-plus-noise vocoder (from Liu et al. 2024) conditioned on articulatory features and speaker embedding via FiLM layers. Causal SoundStream encoder with post-causal convolution layer.
At inference, a 15 ms input chunk is processed through all modules sequentially (32 ms lookahead for windowing + 14.4 ms compute = 61.4 ms total). Pitch rescaling to the target speaker’s range uses the ratio of running median pitches. The system is deployed as a web demo on an AWS CPU (C7i/Intel Xeon) with 10 pre-enrolled VCTK target speakers.
Key Results
Compared to StreamVC (the prior real-time SOTA) on LibriTTS test-clean → VCTK zero-shot conversion:
| StreamVC | RT-VC | |
|---|---|---|
| WER | 6.22% | 6.69% |
| CER | 2.17% | 2.12% |
| Resemblyzer SPK-SIM | 77.81% | 76.65% |
| f0 PCC | 0.842 | 0.865 |
| CPU Latency | 70.8 ms | 61.4 ms |
RT-VC adds a subjective MOS of 3.87 and SMOS of 3.59 (StreamVC did not report these). The system is robust to static noise down to ~20 dB SNR (WER stays near 10%); white noise at 10 dB degrades quality substantially.
Novelty Assessment
The primary contribution is systems engineering: adapting SPARC (which was not designed for streaming) into a real-time causal pipeline with a fast DDSP vocoder. The articulatory-space disentanglement is inherited from prior SPARC work (Cho et al. 2024). The 13.3% latency reduction over StreamVC is genuine and practically significant for live voice conversion applications. Architectural novelty is incremental.
Field Significance
Moderate — RT-VC demonstrates that articulatory-feature-based disentanglement, previously limited to offline use, can be made real-time on commodity CPUs without significant quality loss. The contribution is primarily engineering integration: the SPARC framework, DDSP vocoder, and causal streaming strategies are established components combined into a deployable system. It provides useful evidence that interpretable, physics-grounded representations are a viable path for low-latency voice conversion.
Claims
- Articulatory feature spaces enable interpretable content-speaker disentanglement in voice conversion without sacrificing intelligibility relative to SSL-based approaches. (§5.3, Table 1)
- Real-time zero-shot voice conversion on CPU hardware is achievable below 70 ms end-to-end latency while maintaining naturalness MOS above 3.8. (§3.6, Table 1)
- Causal DDSP vocoders conditioned on articulatory features provide competitive synthesis quality compared to GAN-based alternatives at substantially lower computational cost. (§2.3, §3.5)
- Voice conversion systems trained with static noise augmentation degrade gracefully down to approximately 20 dB SNR input but fail at 10 dB, suggesting a practical noise floor for real-time deployment. (§5.4)
Limitations and Open Questions
- EMA representation omits nasal cavity and laryngeal dynamics, limiting modeling of nasal sounds and vocal fry.
- Pseudo-EMA labels come from a WavLM model pretrained on English; cross-lingual performance is limited.
- Sensitivity to input quality below 20 dB SNR, especially for white noise.
- Model weights will not be open-sourced due to misuse concerns.
- Future work: prompt-free conversion by offline target speaker design (gender, age, emotion, accent).
Wiki Connections
- voice-conversion — RT-VC advances real-time zero-shot VC using an interpretable articulatory feature space
- streaming-tts — the causal four-module pipeline demonstrates sub-70 ms CPU latency for streaming VC
- disentanglement — the SPARC-derived articulatory space explicitly separates vocal tract kinematics (content) from speaker embeddings at training time
- self-supervised-speech — WavLM is used both for pseudo-EMA label generation and as the frozen CNN backbone in the speaker encoder
- zero-shot-tts — all source and target speakers are unseen during training; target adaptation requires only a reference utterance