arXiv · 2025 · Preprint
Meiying Melissa Chen et al. (University of Rochester) · → Paper · Demo: ? · Code: ?
Introduces SpeakerVAE, a hierarchical VAE that samples novel, unseen speaker representations for voice conversion, removing the need for a target-speaker utterance at inference.
Problem
Most voice conversion (VC) systems require a recorded utterance from the target speaker at inference time, whether to derive a disentangled speaker embedding or as a prompt for an LLM-based converter. This blocks two practically important use cases: generating a genuinely novel voice for dubbing or identity protection, and offering users a diverse catalogue of unseen voices to choose from. Prior attempts to synthesize new speakers either interpolate between existing speaker embeddings, which does not guarantee novelty or audio quality, or treat speaker generation as a side capability of a TTS system controlled by coarse labels (sex, age) rather than a first-class, systematically evaluated feature of a VC pipeline. No prior work supplies a systematic methodology to check whether generated speakers are actually distinct from the training population and whether they retain high audio quality.
Method
SpeakerVAE models the distribution of speaker embeddings extracted from a pretrained VC system’s own speaker module, then samples from that learned distribution to produce novel pseudo-speaker embeddings that can be substituted directly into the VC pipeline’s speaker-conditioning slot. The backbone is a Nouveau Variational Autoencoder (NVAE), a deep hierarchical VAE originally designed for image generation, chosen for its ability to capture a multi-scale latent structure and to avoid the posterior collapse and over-regularization that limit standard VAEs.
To adapt NVAE from 2D image data to 1D speaker embeddings, the authors replace all convolutional operations with 1D equivalents, remove the autoregressive normalizing flows from the original design while keeping the hierarchical latent structure, add quantile-based normalization of the input embeddings, and introduce free-bits regularization with KL-coefficient warmup to prevent posterior collapse. The model is trained with the standard hierarchical ELBO objective (reconstruction likelihood minus a per-level KL term against a learned conditional prior), using only speaker embeddings as input, with no text, content, or prosody information involved in training the VAE itself.
At inference, a novel speaker embedding is sampled from SpeakerVAE at temperature 1.0; the source utterance’s content and prosody representations are kept unchanged; and the underlying VC model synthesizes speech using the source content/prosody together with the sampled speaker embedding. Because SpeakerVAE operates entirely in the same latent domain as the host VC system’s speaker embeddings, no additional alignment, co-training, or fine-tuning of the base VC model is required, and the same procedure transfers across backbones by re-training only the (lightweight) VAE on that backbone’s embedding space. The method is evaluated on top of two VC systems with structurally different speaker spaces: FACodec, a factorized neural codec that represents timbre in a 1024-dimensional space learned end-to-end with the codec, and CosyVoice2, a streaming zero-shot TTS/VC model that conditions its text-speech language model on a 192-dimensional CAM++ speaker-verification embedding.
Key Results
The paper’s headline claim is that sampled SpeakerVAE speakers are usable as VC targets without degrading intelligibility or perceptual quality relative to real training speakers. Word error rate, character error rate (via Whisper transcription and JiWER), and UTMOSv2 naturalness scores for speech synthesized with generated speakers are reported as comparable to those for speech resynthesized with real speakers’ embeddings, for both the FACodec and CosyVoice2 backbones (Fig. 2). The one exception is a Gaussian Mixture Model (GMM) baseline, which noticeably degrades FACodec’s intelligibility while remaining comparable to SpeakerVAE on CosyVoice2.
On the cosine-similarity-based speaker generation metrics (Table 1), SpeakerVAE and the GMM baseline perform similarly overall, each with its own trade-offs rather than one method dominating. For FACodec, SpeakerVAE’s generated speakers are slightly less diverse than the resynthesized-original set (Pairwise 0.74 vs. 0.67 original diversity, where lower pairwise similarity indicates higher diversity) but cover the original distribution better than GMM (coverage 0.70 vs. 0.64); the GMM baseline is more stable (0.90 vs. SpeakerVAE’s 0.85) but has lower coverage. For CosyVoice2, the ranking of stability flips: SpeakerVAE is more stable (0.90) than GMM (0.86). Both methods achieve speaker-fidelity scores around 0.92-0.94 when reconstructing or resynthesizing known speakers, indicating both VC backbones faithfully render whatever speaker embedding they are given, generated or real.
Novelty Assessment
The architectural contribution is an adaptation, not an invention: NVAE is an existing 2020 computer-vision architecture, and the modifications applied here (1D convolutions, removed autoregressive flows, quantile normalization, free-bits warmup) are standard engineering adjustments needed to fit a new data modality rather than a new model class or training objective. Prior work has already generated novel speakers for TTS using simpler mechanisms (interpolation, label-conditioned generation); this paper’s contribution is narrower and more specific: applying a VAE-based generative model to the VC setting as a backbone-agnostic plug-in, and proposing a small but systematic evaluation protocol (diversity, coverage, fidelity, stability, computed via cosine similarity between speaker-verification embeddings) to check whether generated speakers are genuinely novel and high quality. The strongest evidence for the method’s value is its backbone-agnostic design: the same training/inference recipe transfers across two structurally very different speaker embedding spaces (FACodec’s disentangled 1024-d codec space and CosyVoice2’s 192-d CAM++ verification embedding) without touching the underlying VC model.
Field Significance
Low — this is a narrow, useful engineering contribution to a specific sub-problem (novel speaker generation as a VC add-on) rather than a new direction for the field. Its value lies in demonstrating that a backbone-agnostic generative module can be swapped into existing VC pipelines without retraining them, and in proposing evaluation metrics for a capability (speaker novelty and diversity) that prior work rarely measured systematically.
Claims
- supports: A generative model of a voice conversion system’s own speaker-embedding space can synthesize novel, unseen speaker identities that the same system renders at audio quality comparable to real training speakers, without needing any target-speaker utterance at inference.
Evidence: SpeakerVAE, trained only on speaker embeddings extracted from a pretrained VC system’s speaker module, produces sampled embeddings that yield WER/CER and UTMOSv2 scores comparable to speech synthesized from real training speakers’ embeddings, for both the FACodec and CosyVoice2 backbones. (§4, Fig. 2)
- supports: A lightweight speaker-generation module can be attached to different voice conversion backbones as a plug-in without co-training or fine-tuning the base system, as long as it operates in the same latent domain as the backbone’s own speaker embedding space.
Evidence: The same SpeakerVAE training and inference procedure, re-trained only on backbone-specific embeddings, is applied unmodified to FACodec’s 1024-d disentangled timbre space and to CosyVoice2’s 192-d CAM++ speaker-verification embedding. (§2.2, §2.3)
- complicates: Cosine-similarity-based diversity and stability metrics for generated speaker embeddings can rank competing generation methods inconsistently across voice conversion backbones, complicating claims that one speaker-generation method is uniformly superior.
Evidence: For FACodec, a GMM baseline achieved higher stability (0.90±0.06) than SpeakerVAE (0.85±0.10), but for CosyVoice2 the ranking reversed (SpeakerVAE 0.90±0.06 vs. GMM 0.86±0.09); SpeakerVAE achieved better distribution coverage than GMM on FACodec (Pairwise(G_syn,S_syn) 0.70 vs. 0.64) but comparable diversity to GMM on CosyVoice2. (§4, Table 1)
- complicates: Evaluating generated-speaker naturalness and diversity using ground-truth and resynthesis sets drawn only from the training corpus, rather than a held-out test split, leaves open whether apparent fidelity generalizes beyond the training distribution.
Evidence: All evaluation sets (GT, GT-SameSpeaker, S_syn, S_recon, G_syn) are constructed by sampling m=1000 utterances “from the train dataset,” with no separately held-out evaluation corpus reported. (§3.4.1)
Limitations and Open Questions
Warning
Speaker generation quality and diversity are evaluated only against ground-truth and resynthesized utterances sampled from the same training corpus used to fit SpeakerVAE, rather than a held-out test set; the paper does not report whether the generated-speaker distribution generalizes to out-of-corpus speakers or acoustic conditions.
The evaluation is confined to a single dataset family (LibriTTS) and two VC backbones from the same lab-adjacent ecosystem (FACodec, CosyVoice2), with a single baseline (a 12-16 component GMM) for comparison; no comparison is made against other published speaker-generation approaches (e.g., embedding interpolation, label-conditioned TTS speaker generation). Stability results are mixed and backbone-dependent, and the paper does not analyze why. The authors note as future work the extension to attribute-controlled (age, gender, accent) speaker generation via guided latent-space sampling, which the current method does not support.
Wiki Connections
- Voice Conversion — proposes a backbone-agnostic module that removes the need for a target-speaker utterance in VC by sampling novel speaker embeddings instead of extracting them from real recordings.
- Speaker Adaptation — models the speaker-embedding space of existing VC systems to generate new speaker identities, complementing the more common problem of adapting synthesis to a fixed, known target speaker from limited data.
- Evaluation Metrics — introduces a systematic cosine-similarity-based protocol (diversity, coverage, fidelity, stability) specifically for assessing whether generated speaker embeddings are novel and high-quality, a gap the authors identify in prior speaker-generation work.
- NaturalSpeech 3 — its FACodec component is used unmodified as one of the two VC backbones into which sampled SpeakerVAE speaker embeddings are substituted, relying on its factorized, disentangled timbre space.
- CosyVoice2 — used unmodified as the second VC backbone, with SpeakerVAE’s sampled output replacing its CAM++ speaker-verification embedding input during the language-model conditioning stage.
- ECAPA-TDNN — cited as an alternative speaker-verification embedding extractor that SpeakerVAE could in principle be trained on, though the paper’s own experiments use FACodec’s and CosyVoice2’s native speaker embeddings instead.