arXiv · 2025 · Preprint
Yu Zhang et al. · → Paper · Demo: ✓ · Code: ?
Conan is a chunkwise online zero-shot voice conversion system that achieves real-time speaker style transfer (including timbre, prosody, and emotion) to unseen speakers with as low as 37 ms latency, by combining an Emformer-based streaming content extractor, a clustering-VQ style encoder, and a causal pixel-shuffle vocoder.
Problem
Real-time voice conversion must simultaneously preserve source content fidelity, match an unseen reference speaker’s representation (timbre and speaking style), and produce natural audio — all under strict latency constraints. Prior streaming VC systems fell short in at least one of these: self-supervised encoders like HuBERT have large receptive fields incompatible with online processing; most zero-shot systems transfer only global timbre but ignore prosody and emotion; and converting non-causal vocoders to causal ones via zero-padding introduces spectral artifacts. Distillation-based causal content encoders were tried but underperformed in accuracy.
Method
Conan’s pipeline processes source speech in fixed-size chunks (80 ms in the full setting, 20 ms in the fast setting). Three core components work end-to-end:
Stream Content Extractor (SCE). An Emformer transformer distilled from offline HuBERT-Large to extract per-frame content labels at 20 ms intervals. Emformer uses a chunk-and-memory-bank architecture: at each chunk it attends to the current chunk, an optional small right-context block, and cached left-context and summary vectors from prior chunks. The output is hard content labels from a softmax over a fixed vocabulary, keeping the representation streaming-compatible and computationally light.
Adaptive Style Encoder (ASE). A convolutional block processes the full reference mel-spectrogram (available upfront), pooling it into 80 ms chunk-level representations. A Clustering Vector Quantization (CVQ) layer — using a contrastive loss plus commitment loss to prevent codebook collapse — filters non-style content and quantizes into a style codebook. Align Attention then fuses the style embedding with the concatenation of current chunk content and a timbre embedding (from a separate convolutional timbre encoder over the reference mel), yielding a per-chunk style representation z_s,i aligned to content.
Causal Shuffle Vocoder (CSV). Rather than zero-padding a non-causal HiFi-GAN, the CSV replaces transposed convolutions with causal convolutions followed by pixel shuffle (channel-to-time rearrangement). This achieves temporal upsampling without accessing future frames and avoids checkerboard artifacts. A causal mel decoder generates the per-chunk mel-spectrogram from content, timbre, style, and F0 (from a causal pitch predictor), which the CSV then converts to waveform.
Training uses LibriTTS (586 hours, 2,456 speakers) at 16 kHz. The SCE trains separately (cross-entropy distillation, 80 k steps). The main model trains for 160 k steps with mel reconstruction (MAE + SSIM), LSGAN adversarial loss, VQ loss, and pitch MSE. The CSV trains for 600 k steps with HiFi-GAN losses (GAN + mel + feature matching).
Inference streams source speech chunk-by-chunk, prepending a context window at each step for smooth boundary transitions; the complete reference is fed once upfront.
Key Results
On LibriTTS (source) → VCTK (6 unseen speakers):
| System | Online | WER ↓ | SIM ↑ | MOS-Q ↑ | MOS-S ↑ |
|---|---|---|---|---|---|
| VQMIVC | No | 56.73% | 62.54% | 3.81 | 3.76 |
| BNE-PPG-VC | No | 12.29% | 79.06% | 3.86 | 3.90 |
| StreamVC | Yes | 6.22% | 77.81% | 4.01 | 3.86 |
| Conan (Full) | Yes | 6.02% | 85.71% | 4.06 | 4.02 |
| Conan (Fast) | Yes | 10.72% | 78.26% | 3.99 | 3.85 |
Conan Full achieves the highest SIM (85.71%) among all systems while remaining the only online system to exceed 85% similarity. MOS-Q is best-in-class at 4.06. WER is lowest overall at 6.02%.
Latency (single A100 GPU, no engineering optimization): 37 ms (Fast) and 140 ms (Full). Real-time factors: 0.74 (Fast) and 0.25 (Full).
Ablation confirms all three components contribute: removing the SCE raises WER to 26.92% (CMOS-Q −0.33); removing ASE drops CMOS-S by −0.19; replacing CSV with zero-padding causal HiFi-GAN drops CMOS-Q by −0.21.
Novelty Assessment
The primary contributions are engineering and integration. Emformer for streaming content extraction is a direct application of an existing architecture; the key insight is using it as a distillation target from HuBERT, which is novel in the online VC context. CVQ with contrastive loss for stylistic feature extraction is an adaptation of online clustered codebook methods to the VC setting. The Causal Shuffle Vocoder’s use of pixel shuffle in place of transposed convolutions to avoid checkerboard artifacts in a strictly causal setting is the most technically novel component. The system is primarily an engineering contribution achieving a new combination of streaming + zero-shot + full-style-transfer that prior work had not managed simultaneously.
Field Significance
Moderate — Conan demonstrates that the combination of streaming content extraction, zero-shot style transfer, and fully causal waveform synthesis can be achieved simultaneously in a single system, a combination prior online VC work had not delivered. The Causal Shuffle Vocoder’s pixel-shuffle upsampling provides a reusable solution to checkerboard artifacts in causal vocoders that could transfer to streaming TTS pipelines. The contribution is primarily a system integration advance; the individual components are adapted from existing work.
Claims
- Streaming voice conversion quality degrades significantly when offline HuBERT representations are naively replaced by causal alternatives, but knowledge distillation into an Emformer backbone can recover content accuracy with acceptable latency. (§III.B, Table III)
- Causal temporal upsampling via pixel shuffle eliminates the checkerboard artifacts introduced by zero-padding non-causal vocoders, without sacrificing subjective quality. (§III.D, Table III)
- Explicit style modeling with clustering-based vector quantization improves zero-shot speaker similarity in streaming VC beyond what timbre embeddings alone provide. (§III.C, Table I)
- Online voice conversion systems can achieve speaker similarity comparable to offline systems when style transfer is modeled at chunk level rather than at the global utterance level. (§IV.B, Table I)
Limitations and Open Questions
- Evaluation is in English only; cross-lingual style transfer is untested.
- The reference speaker must be fully available before streaming begins, limiting applications where reference is also captured in real-time.
- CER is slightly higher than StreamVC because StreamVC reuses source pitch; Conan introduces some pitch variation that ASR penalizes as CER.
- Model size is not reported; the cost of the Emformer + main model + CSV in production deployment is unclear.
- Perceptual evaluation used 15 listeners per pair on Amazon Mechanical Turk style tasks; larger-scale evaluation would be needed to establish statistical robustness.
Wiki Connections
This paper directly advances voice-conversion with a new state-of-the-art streaming zero-shot system. It connects to streaming-tts (shared concern with latency constraints and causal architectures) and self-supervised-speech (HuBERT distillation for content extraction). The CVQ style encoder relates to disentanglement (content/style/timbre separation). The causal vocoder design relates to gan-vocoder (causal HiFi-GAN variant). zero-shot-tts is a related concept given the speaker generalization objective. CosyVoice is an in-corpus citation, used in the style modeling related work discussion.