VLSP 2025 · 2025 · Workshop
Hong Nhat Tran et al. (ViettelAI, Viettel Group) · → Paper · Demo: ? · Code: ?
Builds a two-stage cross-lingual zero-shot voice conversion system for Vietnamese by cascading a ChunkFormer ASR module with an F5-TTS synthesis module, using phoneme-level inputs to stabilize duration modeling for tonal Vietnamese speech, and placing second in the VLSP 2025 Vietnamese Voice Conversion shared task.
Problem
Cross-lingual zero-shot voice conversion requires a system to reproduce an unseen target speaker’s voice while transferring linguistic content across languages, without paired training data or speaker-specific fine-tuning. Two-stage ASR-TTS pipelines address this by separating content extraction from voice synthesis, but this separation introduces its own difficulties: long-form transcription must be both memory-efficient and accurate, the TTS stage must generalize to unseen speakers across languages, and duration modeling must handle language-specific pronunciation rules. For Vietnamese specifically, character-level text representations are ill-suited to duration estimation in diffusion-based TTS, because a single character can map to multiple phonetic realizations and Vietnamese tone and vowel length are directly tied to duration.
Method
The system cascades two off-the-shelf components without architectural modification. ChunkFormer, a transformer-based ASR model, processes the source utterance in fixed-size chunks with a relative right context, using masked batching to eliminate padding overhead; it transcribes both the source utterance (yielding a content transcript) and a short target-speaker reference clip (yielding a reference transcript that captures speaker prosody and phrasing). F5-TTS, a flow-matching diffusion-transformer (DiT) TTS model, then conditions on the content transcript, the reference transcript, and the reference mel-spectrogram to synthesize a mel-spectrogram of the target speaker’s voice; an ODE solver integrates a predicted velocity field from noise to a mel using a sway timestep policy, after which a neural vocoder produces the waveform. The authors state they adopt the original F5-TTS architecture “without structural modifications,” using classifier-free guidance strength 2, 32 function evaluations, and the Euler ODE solver.
The paper’s principal design choice is at the input-representation level rather than the architecture level: Vietnamese text is converted to IPA-based phoneme sequences with explicit tone marking (via vPhon) to give a one-to-one mapping between symbolic units and acoustic segments, removing the character-to-phoneme ambiguity that otherwise destabilizes duration estimation, while English text is left at the grapheme level. The two vocabularies are assigned disjoint index ranges but share a unified embedding space, and a routing mechanism sends Vietnamese phoneme tokens through an extended phoneme encoder and English grapheme tokens through the base encoder, allowing a single F5-TTS model to handle both languages. The model is trained in a single stage on a combined corpus mixing 1,000 hours of Vietnamese speech (ViVoice) with the English multi-speaker VCTK corpus, intended to jointly learn Vietnamese prosodic patterns and English phonetic structure for cross-lingual generalization. Training and inference were run on a single NVIDIA A100 GPU (40GB) with fp16 mixed precision.
Key Results
On the official VLSP 2025 Voice Conversion Task 1 leaderboard (4 competing teams, Table 1), the system achieved a Final Score of 67.53, ranking second overall, behind the top team (3N, 72.66) and ahead of VCL (64.49) and ProfessorAgasa (62.40). It obtained the highest target-speaker similarity score among all teams (SMOS_TGT = 3.66 ± 0.21), matching the paper’s claim of best speaker-timbre preservation, but had the second-worst WER (12.95) and a mid-range MOS (3.53 ± 0.21), notably below the leaderboard-topping team’s MOS of 4.29 and WER of 9.83. All comparisons are drawn from a single shared-task leaderboard of four systems under one evaluation protocol, so rankings are internally fair but represent a narrow, non-standard benchmark rather than a broadly recognized one.
Novelty Assessment
The contribution is an engineering integration rather than an architectural advance: ChunkFormer and F5-TTS are both used as published, with F5-TTS explicitly unmodified at the model level. The one input-level innovation, a phoneme/grapheme dual-vocabulary routing scheme with a shared embedding space, is a targeted adaptation of F5-TTS’s text-conditioning input rather than a new module or training objective, and its effect is not isolated by ablation. The paper’s own discussion attributes gains qualitatively to phoneme-level inputs and joint Vietnamese-English training but provides no controlled comparison against a character-level or Vietnamese-only baseline to substantiate the claim. As in the companion VLSP 2025 submissions, the value of the paper lies in demonstrating that this combination of components performs competitively on a specific low-resource shared task, not in a novel method.
Field Significance
Low — this is a shared-task system-description paper that demonstrates a phoneme-level, cross-lingual ASR-TTS cascade can achieve strong target-speaker similarity for Vietnamese zero-shot voice conversion, but it proposes no new architecture, offers no ablations isolating the effect of its phoneme-routing scheme, and is evaluated against only three other systems on one shared-task leaderboard.
Claims
- supports: Phoneme-level text representations reduce duration-estimation instability in diffusion-based TTS for tonal languages, compared to character-level inputs.
Evidence: The authors describe the character-level duration heuristic in F5-TTS (ratio-based duration estimation with filler-token padding) as unstable for a language where a single character can map to multiple phonetic realizations, and adopt IPA-based phoneme sequences with explicit tone marking for Vietnamese instead. (§2.2 “Phoneme-level representations”)
- supports: A two-stage ASR-TTS pipeline can achieve strong target-speaker similarity in cross-lingual zero-shot voice conversion by explicitly separating content transcription from speech synthesis.
Evidence: The system obtained the best target-speaker similarity score (SMOS_TGT = 3.66 ± 0.21) among all four teams on the VLSP 2025 Voice Conversion Task 1 leaderboard. (§3.2, Table 1)
- complicates: Optimizing a cascaded ASR-TTS voice conversion system for speaker similarity can come at the cost of content accuracy relative to competing designs.
Evidence: The system’s WER (12.95) was the second-worst of four teams despite its best-in-class SMOS_TGT, and its Final Score (67.53) trailed the top team (72.66), which achieved both a lower WER (9.83) and higher MOS (4.29). (§3.2, Table 1)
- complicates: Diffusion-based TTS models retain characteristic failure modes such as repetition and hallucination on long text inputs even when integrated into an otherwise well-performing cascaded voice conversion pipeline.
Evidence: The authors state that “the known weaknesses of diffusion-based TTS models, such as repetition and hallucination on long text inputs, persist in our system.” (§3.3 “Discussion”)
Limitations and Open Questions
The paper does not ablate its own design choices: the relative contribution of phoneme-level inputs, the phoneme/grapheme routing mechanism, and the joint ViVoice+VCTK training mix to the reported results is not isolated, so it is unclear which component drives the observed speaker-similarity gain. The authors acknowledge that Vietnamese tonal and dialectal variation are not explicitly modeled, with the system relying instead on reference audio for prosodic and regional cues, which can cause mismatches when tones are under-realized or dialects diverge. They also note that the asymmetry between Vietnamese phonemes and English graphemes may restrict cross-lingual alignment, and that the shared-task evaluation covers only the VLSP 2025 test set against three other competing systems, not a broader or standard benchmark.
Wiki Connections
- Voice Conversion — the paper builds a two-stage ASR-TTS cascade specifically for cross-lingual zero-shot Vietnamese voice conversion, evaluated in the VLSP 2025 shared task.
- Multilingual TTS — the TTS stage is trained jointly on Vietnamese (ViVoice) and English (VCTK) data with a shared phoneme/grapheme embedding space to support cross-lingual synthesis.
- Zero-Shot TTS — target-speaker timbre and prosody are reproduced from a short reference clip without speaker-specific fine-tuning, using F5-TTS’s zero-shot conditioning.
- Flow Matching — the synthesis stage relies on F5-TTS’s flow-matching diffusion-transformer, used here without architectural modification.
- Subjective Evaluation — the VLSP 2025 shared task’s official ranking and this paper’s headline results are based on human MOS and SMOS ratings.
- F5-TTS — F5-TTS is adopted unmodified as the zero-shot TTS backbone, with only its text-conditioning input adapted via phoneme/grapheme routing for cross-lingual Vietnamese-English synthesis.