arXiv · 2025 · Preprint
Wenyu Wang et al. · → Paper · Demo: ✓ · Code: ?
Proposes FabasedVC, an end-to-end VITS-based voice conversion system that supplements the disentangled self-supervised content representation with textual modality features and phoneme-level duration control to improve content integrity, timbre similarity, and prosody matching.
Problem
Voice conversion systems that rely purely on speaker-independent representations extracted from self-supervised learning (SSL) models face three recurring shortcomings. First, the frame-level features used to carry linguistic content inherit the source speaker’s duration and prosody, so the converted speech cannot fully take on the target speaker’s speaking style. Second, content integrity is often imperfect: information is lost during the disentanglement process, which degrades semantic accuracy. Third, common disentanglement schemes leave residual timbre information in the content representation, causing timbre leakage from the source speaker into the converted output. The paper’s premise is that techniques standard in text-to-speech (TTS), namely explicit textual conditioning and phoneme-level duration prediction, can be imported into a VC pipeline to address these three gaps without abandoning the SSL-based disentanglement approach that VC systems already rely on.
Method
FabasedVC is built on a VITS-style backbone: a conditional variational autoencoder (CVAE) with a posterior encoder, prior encoder, and decoder, trained end to end with adversarial training. The paper’s contributions are concentrated in the front-end feature processing and the prior encoder, leaving the posterior encoder (non-causal WaveNet residual blocks) and decoder (HiFi-GAN generator with multi-period and multi-scale discriminators) close to their VITS/HiFi-GAN origins.
A front-end annotation pipeline derives, for each utterance, character (Words), phoneme, and tone sequences from an ASR model and forced-alignment tool, plus a BERT embedding of the text (WordsBERT) and a disentangled SSL representation (ContentVec) of the audio. Phoneme-level timestamps from forced alignment give per-phoneme frame counts, used to align text and audio features at matched granularity (§3.1, Table 1).
The prior encoder has three new components. A text feature encoder separately encodes words, phonemes, tones, and WordsBERT features and sums the four resulting embeddings into a single text representation (§3.2.1). An SSL feature encoder converts the frame-level ContentVec features into phoneme-level features using two complementary mechanisms: phoneme-level average pooling, which averages frames within each phoneme’s duration window, and an attention-based module that uses the text encoding as the query and the frame-level SSL features as key and value, integrating information across each phoneme’s frames via scaled dot-product attention (§3.2.2). The two resulting features are combined with weights tuned to 1.0 (pooling) and 0.5 (attention). A duration predictor, conditioned on the phoneme-level encoded feature and the target speaker embedding, re-predicts each phoneme’s log-duration; when enabled at inference, it shifts the frame length used to expand phoneme-level features back to frame level toward the target speaker’s characteristics rather than the source’s (§3.2.3-3.2.4). An auxiliary mel-spectrogram prediction module, inspired by VISinger2, is added during training to strengthen the frame-level prior network. The total training objective combines mel reconstruction loss, an auxiliary mel-prediction loss, KL divergence between prior and posterior, the duration-prediction L2 loss, and adversarial generator/discriminator losses (§3.5).
The model is trained on AISHELL-3 (about 85 hours, 218 Mandarin speakers) for 500,000 steps on a single A100 GPU with batch size 16, using 192-dimensional Mandarin BERT and 256-dimensional ContentVec features, 44.1 kHz audio, and 80-band mel spectrograms (§4.1).
Key Results
On the AISHELL-3-to-AISHELL-3 conversion scenario, FabasedVC obtains the best or near-best scores against six baselines (VQMIVC, PPGVC, FreeVC, SOVITS-VC, CosyVoice-VC, and SeedVC): MOS naturalness 4.43 and MOS similarity 4.49, versus 4.09/4.11 for the strongest baseline (SeedVC), along with the lowest CER (6.27% vs. 7.51% for the next best, SOVITS-VC) and PER (2.22% vs. 2.26%), and the highest speaker cosine similarity (0.8088 vs. 0.7836-0.7827 for CosyVoice-VC/SeedVC) (§4.3, Table 2). In the harder DataBaker-to-AISHELL-3 scenario, where source and target differ in speaking style, FabasedVC still leads on similarity (4.27) and objective metrics, though its naturalness (3.85) trails SeedVC (3.90) by a small margin. Ablating either the attention or pooling branch of the SSL encoder substantially degrades all metrics (e.g., CER rises from 6.27% to 19.19% without pooling), and disabling the duration predictor reduces similarity from 4.49 to 4.25 in the matched scenario while leaving speech timed identically to the source (§4.3, Table 2). A separate analysis using a proposed Relative Duration Deviation (RDD) metric shows converted speech stays close to the source speaker’s duration profile (average RDD 3.25%) while shifting substantially toward the target speaker’s duration profile (average RDD 11.34%, versus 20.85% between source and target directly), which the authors read as evidence of partial but meaningful duration adaptation (§4.3, Table 3). Comparisons against CosyVoice and SeedVC use the authors’ official open-source checkpoints, while the other baselines are retrained on the same AISHELL-3 data as FabasedVC, so the comparison basis differs across baselines.
Novelty Assessment
The contribution is primarily engineering integration: a VITS-style CVAE-GAN backbone, BERT text features, ContentVec SSL features, scaled dot-product attention, and a FastSpeech/VITS2-style duration predictor are all established components, recombined here specifically for VC rather than TTS. The one genuinely new element is the dual pooling-and-attention mechanism for converting frame-level SSL features to phoneme-level features conditioned on text, which the ablations show matters substantially for both content accuracy and similarity. The claimed novelty around “text modality fusion” for VC is modest: it amounts to porting a TTS front-end (text, phoneme, tone, BERT features) into a VC system’s conditioning stack, which is a reasonable but incremental idea rather than a new modeling paradigm. The RDD metric introduced for duration analysis is a simple normalized deviation measure rather than a new evaluation methodology with broader validation.
Field Significance
Moderate - the paper offers a concrete, ablated recipe for reducing timbre leakage and improving content integrity in SSL-based VC by moving from frame-level to phoneme-level SSL features and by giving the system explicit control over target-speaker duration, both validated with clear ablations against a fairly strong and diverse baseline set including a diffusion-transformer VC system and a zero-shot TTS system repurposed for VC.
Claims
- supports: Converting frame-level self-supervised content features to phoneme-level features, guided by forced-alignment duration information, improves speaker disentanglement and content preservation relative to using frame-level SSL features directly in voice conversion.
Evidence: Removing either the pooling or attention branch of the phoneme-level SSL encoder raises CER from 6.27% to 19.19% (pooling ablation) or 9.45% (attention ablation) and lowers naturalness/similarity MOS, while the full phoneme-level encoder achieves the lowest CER (6.27%) and PER (2.22%) among all compared systems. (§4.3, Table 2)
- supports: Adding an explicit duration predictor conditioned on the target speaker’s identity, rather than inheriting the source utterance’s duration by default, improves perceived speaker similarity in voice conversion.
Evidence: Disabling the duration predictor lowers similarity MOS from 4.49 to 4.25 in the aishell-to-aishell scenario, and a Relative Duration Deviation analysis shows converted speech shifts from an average 20.85% RDD relative to the target (source duration unchanged) to 11.34% RDD when the predictor is enabled. (§4.3, Table 2 and Table 3)
- supports: Supplementing a voice conversion system’s content representation with explicit textual features (word, phoneme, tone, and pretrained text-embedding features) improves content integrity beyond what a purely acoustic self-supervised representation provides.
Evidence: The full text-and-SSL conditioned model achieves lower CER (6.27%) and PER (2.22%) than all six baseline VC systems, including a diffusion-transformer VC system (SeedVC, CER 8.15%) and a repurposed zero-shot TTS system (CosyVoice-VC, CER 10.80%) trained or evaluated on the same task. (§4.3, Table 2)
- complicates: Duration adaptation toward a target speaker in voice conversion is partial rather than complete when the source and target differ substantially in speaking style, and gains in naturalness can trade off against gains in similarity.
Evidence: In the DataBaker-to-AISHELL-3 scenario, where source and target styles differ more than in the matched AISHELL-3-to-AISHELL-3 scenario, FabasedVC’s naturalness MOS (3.85) falls slightly below SeedVC’s (3.90) even though its similarity MOS (4.27) remains the highest; the RDD toward the target speaker averages 11.34%, indicating a shift toward but not full adoption of the target’s duration profile. (§4.3, Table 2 and Table 3)
Limitations and Open Questions
The evaluation is confined to Mandarin Chinese speech and to a single training corpus (AISHELL-3) supplemented by one additional test-only speaker set (DataBaker); the paper does not report results on other languages, despite noting that non-tonal languages would require different textual features. The front-end annotation pipeline depends on an ASR model and a forced-alignment tool, both of which introduce their own error sources into the phoneme, tone, and duration annotations that the rest of the system is conditioned on; the paper does not quantify how alignment or transcription errors propagate into conversion quality. The subjective evaluation uses only 20 raters over four target speakers, and comparisons against CosyVoice and SeedVC use official pretrained checkpoints while other baselines are retrained on AISHELL-3, so the comparison conditions are not fully matched across the baseline set. Model size and code/data release status are not reported in the paper.
Wiki Connections
- Voice Conversion — proposes a VITS-based VC system that targets the field’s recurring challenges of content loss, timbre leakage, and source-locked prosody via phoneme-level feature processing and explicit duration control.
- Self-Supervised Speech — depends on ContentVec, a pretrained self-supervised representation, as the core disentangled content feature that the system converts from frame level to phoneme level.
- Prosody Control — introduces a duration predictor that explicitly re-estimates each phoneme’s duration toward the target speaker, evaluated with a dedicated duration-deviation analysis.
- Evaluation Metrics — defines a Relative Duration Deviation metric to quantify how far converted speech duration shifts from the source toward the target speaker.
- Subjective Evaluation — reports a 20-listener MOS study on naturalness and similarity across two conversion scenarios in addition to objective metrics.
- CosyVoice — used as a zero-shot TTS baseline repurposed for voice conversion (CosyVoice-VC), run via its official open-source checkpoint.
- Zero-Shot Voice Conversion with Diffusion Transformers (SeedVC) — used as the strongest VC baseline, a diffusion-transformer zero-shot VC system evaluated via its official open-source checkpoint.
- HiFi-GAN — supplies the decoder architecture (generator plus multi-period and multi-scale discriminators) used for waveform generation.
- WaveNet — its non-causal residual block design is used as the posterior encoder architecture, following VITS.
- BERT — provides the pretrained text embeddings (WordsBERT) fused into the text feature encoder.
- Paraformer — used as the ASR model that generates character and phoneme annotations for the front-end feature-processing pipeline.