arXiv · 2026 · Preprint

Meiying Melissa Chen et al. (University of Rochester) · → Paper · Demo: ✗ · Code: ?

Trains a hierarchical NVAE on speaker embeddings and samples it to produce novel pseudo-speaker embeddings, which are plugged into two frozen voice conversion backends (FACodec and CosyVoice2) to turn them into speaker anonymization systems, with the number of resampled latent groups acting as an anonymization-strength control.

Problem

Speaker anonymization systems (SAS) must hide the source speaker’s identity while keeping the content and paralinguistics of the speech usable. The paper names three requirements: pseudo-speakers should be dissimilar to the original speaker (privacy), distinct from one another (diversity), and distributed like real speakers (naturalness). DSP-based anonymizers (formant or pitch shifts) are vulnerable to attackers that retrain speaker verification on anonymized data, and the authors argue that DNN-based approaches built on embedding pools, such as the averaging-plus-noise selection in Stream-Voice-Anon, do not explicitly guarantee speaker diversity across the generated identities.

Method

NouveauVoice is a pseudo-speaker generator trained separately from any synthesis model. It adapts NVAE (Nouveau VAE), a deep hierarchical VAE originally proposed for images, to fixed-dimensional speaker embeddings. The encoder maps an embedding to 8 latent groups ordered coarse to fine; each group’s posterior is conditioned on the already sampled coarser groups, and the decoder has a matching conditional prior p(z_l | z_<l). Training maximizes the ELBO (reconstruction likelihood minus per-level KL terms). To fit 1D embeddings rather than images, the authors replace 2D convolutions with 1D ones, apply quantile-based normalization to the embeddings, and add free-bits regularization with KL coefficient warmup to prevent posterior collapse. Encoder cells use two BN-Swish Conv1D layers (kernel size 3, stride 1 or 2); decoder blocks include residual cells, upsampling and self-attention, with residual connections between encoder and decoder levels (Figure 1). Model size is not reported.

At inference, z_1 is drawn from the unconditional prior and each subsequent group is drawn from its conditional prior; the decoder maps the full latent to a pseudo-speaker embedding in the original embedding space.

The generator is integrated into two frozen VC backends, using official pretrained checkpoints and no backend fine-tuning:

  • FACodec-NV: an NVAE is trained on embeddings from the speaker (timbre) encoder of FACodec (the factorized codec from NaturalSpeech 3), extracted from LibriTTS train-clean-100 and train-clean-360 (about 150k embeddings). At inference, FACodec content tokens from the source speech are combined with a sampled pseudo-speaker embedding and decoded.
  • CosyVoice-NV: the CAM++ speaker embedding that conditions CosyVoice 2 is replaced by the NVAE output, with an NVAE trained and sampled the same way on that embedding space.

A second experiment uses the hierarchy as an anonymization-strength control: a real speaker’s embedding is encoded into the 8 groups, the first N (coarsest) groups are replaced by prior samples and the rest kept, giving x̃(N). N = 0 is a reconstruction; N = 8 replaces everything.

The baseline pseudo-speaker generator is a 16-component diagonal-covariance GMM fitted to the same training embeddings, from which 1,000 samples are drawn (FACodec-GMM and CosyVoice-GMM).

Evaluation follows a protocol modelled on the VoicePrivacy Challenge: EER from an ECAPA-TDNN (ECAPA-TDNN) speaker verification attacker pretrained on VoxCeleb, comparing original enrollment against anonymized trial utterances (1,000 from LibriTTS test-clean; higher EER means more privacy, 50% is chance); WER from a wav2vec 2.0 CTC ASR model fine-tuned on LibriSpeech; and emotion preservation as UAR of a wav2vec2 SER model on 1,000 IEMOCAP utterances balanced over four emotions. Diversity is measured as mean top-5 cosine similarity among 1,000 embeddings (lower = more diverse), and distributional divergence as cross-group RBF-kernel MMD between original and generated embeddings (higher = further from the real distribution).

Key Results

Privacy and utility (Table I, full generation at N = 8):

SystemEER ↑ (%)WER ↓ (%)UAR ↑ (%)
Original0.002.2866.74
CosyVoice Recon7.883.9942.37
CosyVoice-GMM36.204.2941.88
CosyVoice-NV36.404.2942.53
FACodec Recon6.264.5036.58
FACodec-GMM42.309.9038.59
FACodec-NV38.267.5640.36
  • On CosyVoice2, NV and GMM are close: EER 36.40 vs. 36.20, identical WER (4.29), and UAR 42.53 vs. 41.88. The differences are small and no significance testing or repeated runs are reported.
  • On FACodec there is a trade-off in both directions: the GMM baseline gives more privacy (EER 42.30 vs. 38.26), while NV gives better intelligibility (WER 7.56 vs. 9.90) and emotion preservation (UAR 40.36 vs. 38.59). The authors judge the NV balance more favorable.
  • Most of the emotion loss comes from the backends themselves: reconstruction alone drops UAR from 66.74 to 42.37 (CosyVoice) and 36.58 (FACodec).

Anonymization-strength control (Figure 2, §V-B; values from the text): For CosyVoice-NV, EER rises monotonically from 7.88% at x̃(0) to 36.40% at x̃(8), with jumps of 13.8 and 10.8 points at x̃(1) and x̃(2) and diminishing gains afterwards, while WER and UAR stay mostly flat. For FACodec-NV, privacy plateaus around x̃(2) (35.60% EER, WER 4.52%), but WER exceeds 7% from x̃(3) onward, so the authors identify x̃(2) as FACodec’s best operating point.

Diversity versus distributional match (Table II):

ConfigurationTop-5 cos ↓ (FACodec)MMD ↓ (FACodec)Top-5 cos ↓ (CosyVoice2)MMD ↓ (CosyVoice2)
Original0.84 ± 0.06not applicable0.75 ± 0.05not applicable
NV, N = 00.80 ± 0.070.00530.77 ± 0.070.0043
NV, N = 20.68 ± 0.070.06420.49 ± 0.030.1318
NV, N = 80.66 ± 0.060.10220.42 ± 0.030.1690
GMM0.80 ± 0.070.01530.74 ± 0.060.0722

(The table also reports N = 1, 3, 4, 6; the lowest CosyVoice2 top-5 similarity is 0.40 at N = 6.) Every NV setting with N ≥ 1 is more diverse than the GMM (lower top-5 cosine similarity) but further from the real embedding distribution (higher MMD); the GMM matches the real distribution better. The authors frame this as an inherent tension: generating new, varied voices pushes embeddings away from the original distribution. The paper’s third stated requirement (distribution similarity) is therefore met better by the baseline in these measurements.

All results use a single ECAPA-TDNN attacker scored against original enrollment; no listening test or naturalness MOS is reported.

Novelty Assessment

The contribution is the application of an existing hierarchical VAE (NVAE) to speaker-embedding generation for anonymization, with the modifications needed for 1D embeddings (1D convolutions, quantile normalization, free bits with KL warmup). The VC backends are used off the shelf. The most distinctive element is the use of the latent hierarchy as an anonymization knob (resample the coarsest N groups, keep the rest), which gives a graded privacy-utility trade-off from a single trained model. The comparison is against only one baseline generator (a GMM); the VoicePrivacy Challenge baselines and other anonymizers cited in the introduction are not run, so the standing of the method relative to prior SAS is not established by the paper. Overall this is an engineering integration with an incremental recipe.

Field Significance

Low — The paper provides a small-scale data point that a hierarchical generative model over speaker embeddings can serve as a backend-agnostic pseudo-speaker source for VC-based anonymization, and that resampling only the coarse latent groups yields a controllable privacy level. Its evidence is limited by a single GMM baseline, one attacker type, and close or mixed comparisons against that baseline.

Claims

  • supports: Pseudo-speaker generation can be decoupled from the synthesis model, so a single embedding generator can turn different pretrained voice conversion systems into speaker anonymizers without retraining them.

    Evidence: The NVAE generator is trained only on speaker embeddings and plugged into frozen FACodec and CosyVoice2 backends, raising EER from 6.26% and 7.88% (reconstruction) to 38.26% and 36.40% while WER goes from 4.50% to 7.56% and from 3.99% to 4.29%. (§III-A, Table I)

  • supports: In a hierarchical latent model of speaker identity, the coarsest latent levels carry most of the identity, so resampling only those levels provides a graded control over anonymization strength.

    Evidence: For CosyVoice-NV, replacing the first one and two of eight groups raises EER by 13.8 and 10.8 points, with diminishing gains thereafter and nearly flat WER and UAR; FACodec-NV reaches 35.60% EER at x̃(2) with 4.52% WER. (§V-B, Figure 2)

  • complicates: Increasing the diversity of generated pseudo-speakers moves them away from the distribution of real speaker embeddings, so diversity and distributional realism trade off against each other.

    Evidence: NV settings with N ≥ 1 have lower top-5 cosine similarity than the GMM baseline (CosyVoice2: 0.42 at N = 8 vs. 0.74) but higher MMD to real embeddings (0.1690 vs. 0.0722); the same pattern holds on FACodec (0.66 vs. 0.80 similarity, 0.1022 vs. 0.0153 MMD). (§V-C, Table II)

  • complicates: The privacy-utility trade-off of a pseudo-speaker generator depends on the voice conversion backend it drives; a simple density model can give stronger privacy at a utility cost on one backend while being indistinguishable on another.

    Evidence: The GMM baseline yields higher EER than NV on FACodec (42.30% vs. 38.26%) but worse WER (9.90% vs. 7.56%) and UAR (38.59% vs. 40.36%), whereas on CosyVoice2 the two are within 0.2 EER points with identical WER. (§V-A, Table I)

  • complicates: In VC-based anonymization, much of the loss of emotional expressiveness comes from the resynthesis backend rather than from the identity change.

    Evidence: Reconstruction without anonymization drops SER UAR from 66.74% to 42.37% (CosyVoice2) and 36.58% (FACodec), larger than any change between reconstruction and anonymized outputs. (§V-A, Table I)

Limitations and Open Questions

Privacy is measured only against an attacker using a pretrained ECAPA-TDNN on original enrollment utterances; no attacker retrained on anonymized data is evaluated, although the introduction cites such attackers as the reason DSP methods fail. The only comparison system is a GMM pseudo-speaker generator, not the VoicePrivacy Challenge baselines or other cited anonymizers.

  • The attacker is described as “an uninformed attacker system” (§IV-A), while the results subsection is titled “Robustness against a semi-informed attacker” (§V-A); the described setup matches the uninformed case.
  • No listening test or naturalness measure is reported, so the perceptual quality and realism of the pseudo-voices are not assessed; realism is approximated only by embedding-level MMD, on which the GMM baseline does better.
  • Comparisons on CosyVoice2 differ by fractions of a point, with no confidence intervals or repeated sampling.
  • Model size, NVAE training hyperparameters and compute are not reported. Code is not mentioned and the demo is stated to be released on acceptance.
  • Diversity is measured in each backend’s own embedding space (FACodec timbre embeddings vs. CAM++), so the two backends’ diversity numbers are not directly comparable.
  • Attribute-controlled generation (age, gender, accent) is left to future work.

Wiki Connections

  • Voice Conversion — Uses frozen zero-shot VC backends driven by generated pseudo-speaker embeddings as speaker anonymization systems, and measures the converted speech for EER, WER and emotion preservation.
  • Generating Novel and Realistic Speakers for Voice Conversion — Earlier speaker-generation work for voice conversion by the same group (same first and last authors), with a VAE-based speaker generator; NouveauVoice’s reference list does not cite it.
  • NaturalSpeech 3 — Source of FACodec, the factorized codec used frozen as one of the two VC backends (FACodec-NV).
  • CosyVoice 2 — Used frozen as the second VC backend, with its CAM++ speaker embedding replaced by NouveauVoice samples (CosyVoice-NV).
  • Stream-Voice-Anon — Cited as a language-model anonymizer whose averaged-pool-plus-noise pseudo-speaker selection motivates the focus on speaker diversity; not run as a baseline.
  • ECAPA-TDNN — Architecture of the speaker verification attacker used to compute EER.
  • LibriTTS — Provides the speaker embeddings for training the generator and the 1,000 test-clean trial utterances for evaluation.
  • Vevo — Cited among standalone voice conversion systems capable of voice cloning, as motivation for anonymization.