VLSP 2025 · 2025 · Workshop
Nguyen Thi Thu Trang et al. (Hanoi University of Science and Technology) · → Paper · Demo: ? · Code: ?
Establishes the first Vietnamese voice conversion shared task, releasing a 26-hour multi-genre, style-annotated speaker corpus and a composite MOS/SMOS/WER scoring protocol, then reports comparative results from the 18 registered teams’ submitted systems.
Problem
Voice conversion research has advanced on English and a handful of other well-resourced languages, but Vietnamese had no standardized dataset or evaluation protocol for the task before this shared task. Existing text-free VC approaches (bottleneck, adversarial, or augmentation-based disentanglement methods) still commonly suffer from speaker information leakage, degraded naturalness, and linguistic information loss, but progress on these failure modes is hard to compare across papers without a common benchmark. Separately, prior large-scale Vietnamese speech corpora tend to be built from single-speaker or turn-respecting structured recordings, which under-represent the spontaneous, overlapping, multi-accent conversational speech that a robust VC system would need to handle. The paper’s goal is to close both gaps: provide a shared Vietnamese VC dataset and establish a common way to score submitted systems.
Method
The organizers built a multi-genre Vietnamese corpus from YouTube, collecting 1,859 audio samples (642 hours) across eight genre categories (Sharing, Talk show, Review, TV show, Game, Lecture, Vlog, News) and two delivery styles (spontaneous vs. prepared speech). A processing pipeline then reduces this raw pool to a labeled training corpus: pyannote-based speaker diarization defines a “speaker” as an individual tied to a specific speaking style (so an intentional style shift, e.g. impersonation, is treated as a different speaker), retaining only single-speaker, cleanly segmented utterances; DeepFilterNet denoises the audio while preserving naturalness (chosen over traditional waveform-domain denoisers, which the authors report harm perceived voice quality); NISQA and WV-MOS are combined to filter out samples below a MOS threshold of 3.2; WhisperX provides ASR transcription and word-level timestamps, with utterances exceeding 10 words/second removed as outliers. Final speaker labeling clusters utterances using ECAPA-TDNN speaker embeddings (pretrained on VoxCeleb): a 0.75 cosine-similarity threshold groups utterances within an episode (validated via t-SNE cluster separation), and a stricter 0.97 threshold merges speaker identities across episodes. The resulting released corpus contains 26.35 hours across 16,551 utterances from the 100 highest-quality speakers (100-200 utterances each). A separate 33-sample test set (internal recordings, student-contributed samples, and YouTube-retrieved audio) spans North/Central/South Vietnamese accents, includes singing samples, and is evaluated under a strict zero-shot protocol in which test speakers never appear in the training data, including cross-regional conversion trials.
Systems are scored with a composite formula, Score = 0.4 x (SMOS(ref,out) - SMOS(src,out)) + 0.3 x MOS + 0.3 x (100 - WER), where SMOS(ref,out) is perceived speaker similarity between the converted output and a reference target-speaker clip, SMOS(src,out) is similarity between the output and the source clip, MOS is naturalness, and WER is computed with the ChunkFormer ASR model. Weighting the reference-minus-source similarity difference at 40% is intended to specifically penalize systems that retain source-speaker identity (speaker leakage) rather than rewarding target similarity alone. MOS and SMOS were each collected from 30 audio pairs, each independently rated by 5 of 13 total human raters (university students). The organizers also distributed a Gradio-based evaluation toolkit to teams to standardize automated WER computation.
Key Results
Of 18 registered teams, the paper details the four top-ranked submissions. Team Twinkle (Twinkle-VC), an end-to-end diffusion-transformer system built on Seed-VC with a PhoWhisper-large semantic encoder, CAM++ speaker embeddings, BigVGAN-v2 vocoding, an OpenVoiceV2 timbre shifter to reduce leakage, and multilingual training data (VCTK, JVS, Zeroth-Korean, PhoAudioBook, plus the VLSP2025 data), achieved the best composite score of 72.66 (MOS 4.29, SMOS_TGT 3.65, SMOS_SRC 1.17, WER 9.83). Team ViettelRoar, a cascade ChunkFormer ASR + F5-TTS pipeline trained sequentially on ViVoice (Vietnamese) and VCTK (English), reached the highest target-speaker similarity (SMOS_TGT 3.66) but a higher WER (12.95), giving a lower composite score of 67.53. Team VCL used MKL, a training-free, optimal-transport-based VC method operating on fine-tuned WavLM features (in place of a KNN-VC-style retrieval step), scoring 64.49 (MOS 3.72, WER 10.98). Team ProfessorAgasa’s cascade ChunkFormer + ZipVoice system scored lowest of the four (62.40; MOS 3.29, WER 12.84). Across all four, end-to-end systems achieved better naturalness and avoided ASR-to-TTS error propagation, while the two cascade systems showed lower SMOS_SRC (better source-identity removal), which the organizers interpret as clearer disentanglement at the cost of content-accuracy sensitivity to upstream ASR errors. These comparisons are all against the same organizer-defined test set and scoring formula, so they are internally consistent, though only the top four of the 18 registered teams are analyzed in detail.
Novelty Assessment
This is a shared-task overview: its own contribution is the dataset, the annotation pipeline (notably operationalizing “speaker” as a style-conditioned identity rather than a purely biometric one), and the composite evaluation protocol, not a new VC architecture. The systems it reports on are themselves built from established components (Seed-VC, F5-TTS, ZipVoice, WavLM-based optimal transport), so the architectural novelty in the write-up belongs to the cited participant systems rather than to the organizers. The value of the paper is as a controlled bake-off: it gives the field a first standardized Vietnamese VC benchmark and a documented comparison of end-to-end versus cascade design choices under matched data and scoring conditions.
Field Significance
Moderate — this paper establishes the first Vietnamese-language voice conversion shared task, contributing a released dataset, annotation pipeline, and composite scoring protocol that other Vietnamese VC work can adopt as a common benchmark. Its scope is narrow (a single language, a single evaluation track, one shared-task edition), and it reports rather than proposes methodology, so its significance is as an evaluation and dataset resource rather than a paradigm-shifting result.
Claims
- supports: Combining multilingual training data with a language-specific semantic encoder improves zero-shot voice conversion generalization to a target low-resource language.
Evidence: The winning submission trains an end-to-end diffusion-transformer system on a mix of VCTK, JVS, Zeroth-Korean, and Vietnamese PhoAudioBook/VLSP2025 data together with a PhoWhisper-large (Vietnamese-tuned) semantic encoder, and achieves the best composite score among four systems evaluated on unseen Vietnamese speakers across three regional accents. (§5.2, Table 6)
- complicates: Cascade ASR-TTS voice conversion pipelines can achieve stronger target-speaker similarity than end-to-end systems, but remain vulnerable to content-accuracy degradation from upstream ASR errors.
Evidence: The cascade ChunkFormer+F5-TTS submission attains the highest SMOS_TGT (3.66) of all four evaluated systems but a higher WER (12.95) than the top-ranked end-to-end system (9.83), with the organizers attributing the naturalness gap to ASR error propagation. (§5.2, Table 6)
- refines: Evaluating voice conversion by the difference between reference- and source-speaker similarity, rather than target similarity alone, better isolates a system’s ability to remove source-speaker identity.
Evidence: The composite scoring formula weights SMOS(ref,out) minus SMOS(src,out) at 40%, and the two cascade submissions’ lower SMOS_SRC scores are interpreted as evidence of stronger source-speaker disentanglement despite one of them scoring lower overall. (§5.1, Figure 5)
- refines: For voice-conversion corpus construction, defining “speaker” identity by speaking style rather than biometric identity alone changes what counts as a distinct training class.
Evidence: The diarization and labeling pipeline explicitly treats intentional style shifts (impersonation, joking, acting) as a different speaker label from the same individual’s default voice, validating resulting clusters with t-SNE separation and embedding-similarity thresholds of 0.75 (within-episode) and 0.97 (cross-episode). (§4.1, Figure 3, Figure 4)
Limitations and Open Questions
Warning
The paper reports preliminary results while the shared task’s own evaluation and verification process was still underway at submission time; camera-ready acceptance and the final conference presentation were scheduled for after this overview was written, so the reported rankings and scores should be treated as provisional.
The held-out test set is small (33 total samples across internal recordings, student contributions, and YouTube-retrieved audio), which limits statistical confidence in the reported system rankings. Subjective ratings come from only 13 raters, all students from a single university, which may not generalize to a broader or more demographically varied listener population. Only the top 4 of 18 registered teams are analyzed in detail, so the full distribution of submitted approaches and failure modes is not characterized. WER is computed with a single ASR model (ChunkFormer), which conflates true content-preservation errors in the converted speech with residual ASR recognition errors on Vietnamese.
Wiki Connections
- Voice Conversion — establishes the first standardized Vietnamese-language benchmark for voice conversion, releasing a labeled multi-genre corpus and composite evaluation protocol for future work in the language.
- Evaluation Metrics — introduces a composite MOS/SMOS/WER scoring formula that explicitly weights the difference between reference- and source-speaker similarity rather than target similarity alone.
- Subjective Evaluation — collects MOS and SMOS ratings from 13 human raters across 30 audio pairs per system to rank the shared task’s participating systems.
- Speaker Adaptation — curates a 100-speaker, style-annotated corpus with 100-200 utterances per speaker explicitly intended to support speaker-dependent voice conversion and adaptation experiments.
- Twinkle-VC — the winning shared-task submission, an end-to-end diffusion-transformer system built on Seed-VC with Vietnamese-specific components, whose results this overview reports and analyzes.
- ViettelRoar — a cascade ChunkFormer+F5-TTS submission whose highest target-speaker similarity but higher WER this overview uses to illustrate the cascade-versus-end-to-end trade-off.
- Seed-VC — the zero-shot diffusion-transformer voice conversion framework that the winning Twinkle-VC submission adapts with Vietnamese-specific encoders and augmentation.
- ZipVoice — the flow-matching TTS module used as the synthesis stage inside the ProfessorAgasa cascade submission.
- F5-TTS — the flow-matching TTS module used as the synthesis stage inside the ViettelRoar cascade submission.
- Training-Free Voice Conversion with Factorized Optimal Transport — the training-free optimal-transport method used by the VCL submission in place of KNN-based retrieval.
- Zero-Shot Text-to-Speech for Vietnamese — the PhoAudioBook Vietnamese dataset from this paper is one of the multilingual training sources used by the winning Twinkle-VC submission.