VLSP 2025 · 2025 · Workshop
Minh Nghia Vu et al. · → Paper · Demo: ? · Code: ?
Adapts the Seed-VC diffusion-transformer voice conversion framework to Vietnamese by swapping in a Vietnamese-adapted Whisper encoder and multilingual training data, taking first place in the VLSP 2025 Vietnamese Voice Conversion shared task.
Problem
Zero-shot voice conversion (VC) lets a system reproduce an unseen target speaker’s voice from a short reference clip, without speaker-specific training data, but three problems persist: timbre leakage (content features retain residual source-speaker identity), coarse single-vector timbre representations that cannot capture fine-grained speaker characteristics, and a training-inference mismatch (models trained to reconstruct source speech must, at inference, combine source content with a different target timbre). For Vietnamese specifically, prior work has focused mainly on low-resource adaptation and cross-lingual transfer, and no system had yet demonstrated high-fidelity zero-shot Vietnamese VC before the VLSP 2025 shared task, which the authors treat as the target application.
Method
Twinkle-VC adapts Seed-VC, a diffusion-transformer (DiT) zero-shot VC framework, to Vietnamese rather than proposing a new architecture. The DiT backbone (13 layers, 8 attention heads, 512-dim embeddings with a 2048-dim feed-forward layer) predicts mel spectrograms conditioned on content and timbre features, using U-Net-style skip connections without downsampling, a prepended time-embedding token used both as a prefix and as an adaptive layer-norm signal, and rotary positional embeddings. A convolutional length regulator aligns semantic (content) and acoustic feature sequences that run at different frame rates. Diffusion uses 50 denoising steps during training and 25 at inference.
The paper’s contribution is three targeted substitutions and refinements within this framework. First, the semantic (content) encoder is replaced with PhoWhisper-large, a 1.55B-parameter Vietnamese-adapted variant of Whisper (4.67% WER on VIVOS), used to extract continuous linguistic embeddings without discretization. Second, an external timbre shifter (OpenVoiceV2, a further-trained YourTTS-based model) perturbs source-speech timbre during training only, aligning the training objective (which otherwise reconstructs source timbre) with the inference-time task of combining source content with a different target timbre, and reducing residual source-identity leakage into content features. Third, timbre/speaker embeddings are extracted with CAM++, a speaker-verification network pretrained on over 200k speakers, replacing a single learned timbre vector with a more generalizable representation. Waveform synthesis from predicted mel-spectrograms uses a pretrained BigVGAN-v2 vocoder at 22 kHz. All modules are aligned to a consistent 80-band mel-spectrogram configuration to avoid feature mismatch across the pipeline.
Training data combines the VLSP 2025 Vietnamese VC corpus with English (VCTK), Japanese (JVS), and Korean (Zeroth-Korean) speech, restricted to 10-15 minutes per speaker to encourage generalization to unseen voices rather than per-speaker overfitting. This multilingual mix is motivated by Vietnamese’s frequent phonetic borrowing from English, Chinese, and French. On top of this, the authors apply Spectrogram Resize (SR) augmentation (from FreeVC): bilinear interpolation along the frequency axis (pitch/formant perturbation) and the time axis (speaking-rate perturbation) of the mel-spectrogram, generating roughly 120 hours of additional offline-augmented Vietnamese data (Table 1) so the content encoder learns speaker-invariant linguistic features. The DiT model is trained for up to 100k steps, batch size 8 per GPU (1×A100), with AdamW at a fixed 1e-4 learning rate.
Key Results
In the official VLSP 2025 Task T1 leaderboard (Table 2, 4 competing teams), Twinkle-VC placed first with a Final Score of 72.66, computed as a weighted combination of speaker-similarity gap (SMOS_target minus SMOS_source), MOS, and (100 - WER). It achieved the highest MOS (4.29 ± 0.16, reported on the challenge’s 0-100 scale as 85.8, i.e. ≈4.29 on a 1-5 scale) and the lowest WER (9.83%) among all teams, which the authors attribute primarily to the PhoWhisper-large semantic encoder. On speaker-similarity metrics, Twinkle-VC ranked second in SMOS_TGT (3.65 ± 0.23, versus 3.66 for the top team, ViettelRoar) and third in SMOS_SRC (1.17 ± 0.09, a residual-similarity-to-source metric where lower is better). The next-best system by Final Score (ViettelRoar, 67.53) had a notably higher WER (12.95%) and lower MOS (3.53), suggesting Twinkle-VC’s advantage comes mainly from content preservation and naturalness rather than target-speaker similarity. All comparisons are within a single shared-task leaderboard of 4 systems evaluated under one protocol, so they are internally fair but represent a narrow benchmark.
Novelty Assessment
The paper explicitly states it does not introduce a new architecture, and this is accurate: the DiT, length regulator, timbre shifter, and vocoder are all reused from Seed-VC, OpenVoice, and BigVGAN respectively. The contribution is an engineering integration and domain adaptation — swapping a Vietnamese-specific ASR encoder into an existing VC framework, curating a multilingual training mix, and applying an existing augmentation technique (Spectrogram Resize from FreeVC) — rather than a methodological advance. The leaderboard win is real within the shared-task comparison, but the authors themselves note they ran no ablations, so it is not established which of the three changes (PhoWhisper, SR augmentation, CAM++) drives the improvement, or how much of the gain is simply because competing teams used weaker semantic encoders.
Field Significance
Low — this is a shared-task system-description paper: it demonstrates that combining a language-specific ASR encoder with an established diffusion-transformer VC framework yields strong content preservation for Vietnamese zero-shot voice conversion, but it does not propose new methods, provides no ablations to isolate the source of its gains, and is evaluated against only three other systems on one shared-task leaderboard.
Claims
- supports: Replacing a general-purpose semantic encoder with a language-specific ASR encoder improves content preservation in zero-shot voice conversion for a low-resource target language.
Evidence: Substituting Seed-VC’s original Whisper-small encoder with PhoWhisper-large (Vietnamese-adapted, 1.55B params, 4.67% WER on VIVOS) is credited with the lowest WER (9.83%) among four VLSP 2025 shared-task submissions. (§4.2, §5.2, Table 2)
- supports: Perturbing source-speech timbre during training (rather than only at inference) helps align the training objective with the inference-time voice-conversion task and can reduce residual source-identity leakage.
Evidence: An external timbre shifter (OpenVoiceV2) perturbs speaker timbre during training only, so the model is not simply trained to reconstruct source timbre; the resulting system achieves the lowest WER and highest MOS on the shared-task leaderboard, though the isolated effect of this component alone is not measured. (§4.2)
- complicates: Mel-spectrogram-level augmentation used to improve speaker-content disentanglement can trade off phonetic fidelity for robustness.
Evidence: The authors note that Spectrogram Resize augmentation (frequency- and time-axis mel perturbation) “can slightly distort phonetic details, potentially increasing WER in extreme cases,” though they argue the strong semantic encoder largely offsets this risk. (§6.1 “Effect of SR augmentation”)
- complicates: Leaderboard rankings for zero-shot voice conversion do not agree across metrics, so a single aggregate score can mask which quality dimension actually improved.
Evidence: Twinkle-VC wins on MOS, WER, and Final Score but ranks second in SMOS_TGT (target-speaker similarity) and third in SMOS_SRC (residual source similarity) among the four VLSP 2025 T1 submissions. (§5.2, Table 2)
Limitations and Open Questions
Warning
No ablation studies were conducted, so the causal contribution of PhoWhisper-large, SR augmentation, and CAM++ cannot be individually verified; the reported gains are attributable only to the system as a whole. (§6.1 “Lack of ablation studies”)
Beyond the missing ablations, the evaluation is limited to aggregate metrics on a single shared-task test set with only four participating systems, with no per-speaker, per-accent, or objective speaker-verification breakdown. The authors also flag that their multilingual training strategy showed no observed Vietnamese-performance degradation, but state this was not systematically verified per language or dialect. The system is evaluated only on Vietnamese speech; extension to singing voice conversion or noisy/low-quality inputs is left as future work by the authors.
Wiki Connections
- Voice Conversion — Twinkle-VC is a zero-shot VC system adaptation targeting a specific low-resource language (Vietnamese) rather than a new general-purpose VC method.
- Disentanglement — the system explicitly trains for content/speaker separation via an external timbre-perturbation shifter during training and Spectrogram Resize augmentation of the content encoder.
- Speaker Adaptation — target-speaker timbre is captured zero-shot from a reference clip using a pretrained CAM++ speaker-verification embedding rather than speaker-specific fine-tuning.
- Multilingual TTS — training combines Vietnamese with English, Japanese, and Korean corpora specifically to generalize to phonetic borrowings common in Vietnamese speech.
- Subjective Evaluation — the shared task’s official ranking is based on human MOS and SMOS ratings rather than automatic proxies alone.
- Seed-VC — Twinkle-VC directly adopts and adapts Seed-VC’s diffusion-transformer architecture and external timbre-shifter training strategy as its base framework.
- OpenVoice — an OpenVoiceV2 model (further trained on proprietary data) is used as the timbre-perturbation shifter during Twinkle-VC’s training.
- BigVGAN — BigVGAN-v2 is used as the pretrained vocoder that synthesizes waveforms from Twinkle-VC’s predicted mel-spectrograms.
- VALL-E — cited as prior evidence that neural codec language models can perform zero-shot TTS and voice conversion via autoregressive token modeling, contrasted with Twinkle-VC’s non-autoregressive diffusion-transformer approach.