arXiv · 2026 · Preprint
Thanapat Trachu et al. (University of Southern California) · → Paper · Demo: ? · Code: ?
Formalizes Speech Generation Speaker Poisoning (SGSP), the task of modifying a trained zero-shot TTS model so it can no longer reproduce specific “forgotten” speaker identities while preserving synthesis quality for all other speakers, and introduces a distribution-aware privacy evaluation framework to measure it.
Problem
Zero-shot voice cloning lets a TTS model reproduce any speaker’s voice from a few seconds of reference audio, which creates a concrete privacy and misuse risk (impersonation, misinformation). Conventional machine unlearning, which tries to make a model behave as if it had never seen certain training data, does not solve this: because zero-shot TTS generalizes from short prompts at inference time, simply approximating a model trained without a speaker’s data does not stop the deployed model from reconstructing that speaker’s voice when handed a reference clip. The paper also argues that inference-time defenses (filtering out forget-set reference prompts before they reach the model) are inherently fragile whenever model weights are public, since an adversary can bypass the filter and query the raw model directly. This motivates methods that modify the model’s internal parameters, plus an evaluation protocol suited to zero-shot generalization rather than closed-set classification accuracy.
Method
The paper defines the SGSP problem over a speaker set S, partitioned into a forget set F (identities the model must no longer synthesize) and a retain set R = S \ F (identities the model must keep synthesizing correctly). Forgotten speakers are deliberately drawn from the training set rather than held-out test speakers, since in-domain speakers are reproduced with higher fidelity and are therefore harder to suppress. Three forget-set sizes are studied: 1, 15, and 100 speakers, using LibriTTS (train-clean-100 + train-clean-360 for training, test-clean for evaluation).
The backbone is StyleTTS2, a pretrained model combining a text/style/speech-decoder path with duration and prosody predictors and a diffusion sampler; only the diffusion module is fine-tuned, keeping the text encoder, decoders, and discriminators frozen so that poisoning targets speaker identity specifically without degrading other capabilities. Two naive inference-time baselines are established first: replacing a forget-set reference prompt with a retain-set sample using a WavLM speaker-similarity threshold (Pretrained + Speaker Filtering), and the same replacement with oracle knowledge of forget/retain membership (Pretrained + Ground Truth Filtering).
For parameter modification, the paper adapts Teacher-Guided Poisoning (TGP), originally proposed for VoiceBox and here ported to StyleTTS2: a student model is fine-tuned so that, when given a forget-set reference (swapped in with probability p_forget), it reproduces the output the frozen teacher model would generate for a randomly sampled retain-set speaker, using an L2 distillation loss. The paper’s own contribution, Encoder-Guided Poisoning (EGP), keeps the same training procedure but replaces the teacher-generated audio target with the style encoder’s own output, reasoning that distillation between same-capacity teacher and student yields a noisier signal than a direct encoder target. Both TGP and EGP can be augmented with an explicit triplet (contrastive) loss applied only during forget-conditioned generation, which pushes the diffusion output away from a forget-set negative sample while anchoring it to the ground-truth retain-set output.
Evaluation covers utility (WER via Whisper-medium, UTMOS as an automated naturalness proxy, and WavLM-based speaker-similarity for identity preservation) and privacy under two conditions: an “easy” condition measuring AUC separability between the retain-set and forget-set prompt-output similarity distributions (0.5 = fully overlapping, 1.0 = fully separated), and a “strong” condition, Forget Set Similarity (FSSIM), which measures each generated sample’s similarity against every speaker in the forget set (aggregated by average and by worst-case maximum).
Key Results
In the single-speaker setting, most methods preserve utility close to the unmodified pretrained model, while speaker-filtering fails to provide real privacy (AUC ≈ 0.47–0.5) because forget- and retain-set similarity distributions remain nearly identical. Ground-truth filtering and EGP+Triplet achieve the strongest privacy separation (AUC ≈ 0.90–0.95). EGP consistently outperforms TGP in AUC (e.g. 0.79 vs. 0.71 without triplet loss), consistent with the paper’s explanation that distillation between equal-capacity teacher/student models is a noisier training signal than a direct encoder target. Adding the triplet loss to EGP raises AUC to 0.95, but at a cost: forget-set WER rises from 3.0 to 7.86 and UTMOS on forget-set utterances drops from 4.29 to 3.38, an explicit privacy-utility trade-off (Table 1).
Scaling the forget set to 15 and 100 speakers degrades every parameter-modifying method. AUC for TGP and EGP variants falls from the 0.7–0.95 range at 1 speaker to roughly 0.5–0.65 at 100 speakers, and the retain/forget similarity gap that is clearly visible at 15 speakers largely collapses at 100. Under the strong (FSSIM) condition, average forget-set similarity stays below the 0.86 verification threshold across settings, but maximum forget-set similarity remains high (0.91–0.95) at both 15 and 100 speakers, showing that averaged metrics understate worst-case identity leakage (Table 2).
Novelty Assessment
The core task, using a distillation-based training procedure to make a zero-shot TTS model stop reproducing specific speaker identities, was already introduced by prior work (Teacher-Guided Poisoning for VoiceBox); this paper does not originate that idea. Its genuine contributions are: (1) porting TGP to a structurally different backbone (StyleTTS2) to test whether the approach generalizes across architectures, (2) Encoder-Guided Poisoning, an incremental but well-motivated variant that swaps the distillation target from teacher-generated audio to the style encoder’s own output, (3) a systematic multi-scale study (1, 15, 100 forget speakers) that surfaces a scalability limit not characterized in prior single-scale evaluations, and (4) a distribution-aware evaluation methodology (AUC separability plus average/worst-case FSSIM) that is more diagnostic than raw average speaker-similarity. The contribution is best read as an evaluation-methodology and generalization study layered on an existing poisoning technique, rather than a new architecture or a new poisoning principle.
Field Significance
moderate — This paper strengthens the empirical and methodological foundation of a narrow but growing subfield (targeted speaker erasure / voice privacy for zero-shot TTS) by showing that an existing poisoning technique transfers across backbones, and by contributing an evaluation protocol (AUC-based distributional separability, worst-case FSSIM) that exposes failure modes average similarity metrics hide. Its main limitation for broader significance is scope: a single backbone, a single English dataset, and automated proxy metrics rather than human listening tests, evaluated on forget sets no larger than 100 speakers where the method itself is shown to already be breaking down.
Claims
-
supports: Preventing a zero-shot TTS model from reproducing a specific speaker’s voice requires modifying the model’s internal parameters, because inference-time reference filtering is bypassable whenever model weights are accessible.
Evidence: Pretrained + Speaker Filtering leaves the retain/forget similarity distributions nearly indistinguishable (AUC ≈ 0.47–0.5), while an oracle version of the same filter (Ground Truth Filtering) reaches AUC ≈ 0.9-0.91, showing the filtering mechanism itself, not the underlying model, is where privacy is enforced (§3.2, §6.1, Table 1).
-
complicates: Speaker-identity erasure methods that succeed at small forget-set sizes degrade substantially as the number of forgotten identities grows, because the forget and retain identity spaces increasingly overlap in the model’s representation space.
Evidence: AUC for parameter-modifying methods (TGP, EGP, and their triplet-loss variants) falls from roughly 0.7–0.95 at 1 forgotten speaker to roughly 0.5–0.65 at 100 forgotten speakers, and the retain/forget similarity separation visible at 15 speakers largely collapses at 100 (§6.2, Table 2, Figure 2).
-
refines: Average speaker-similarity to a forget set is an insufficient privacy metric on its own; worst-case similarity against the full forget set is needed to detect residual identity leakage that an average can mask.
Evidence: Introducing Forget Set Similarity (FSSIM) with both average and maximum aggregation shows Avg-FSSIM staying below the 0.86 verification threshold at 15 and 100 speakers while Max-FSSIM remains high (0.91–0.95) in both settings, indicating persistent worst-case leakage that the average alone would miss (§4.2, §6.2, Table 2).
-
complicates: Explicit contrastive suppression of a forgotten speaker’s identity can trade off intelligibility and naturalness on exactly the inputs it is meant to suppress.
Evidence: Adding a triplet loss to Encoder-Guided Poisoning raises privacy AUC from 0.79 to 0.95 in the single-speaker setting, but forget-set WER rises from 3.0 to 7.86 and forget-set UTMOS drops from 4.29 to 3.38, compared to the non-triplet EGP variant (§6.1, Table 1).
Limitations and Open Questions
The method's central claim, robust targeted speaker erasure, holds only at small scale (up to 15 forgotten speakers) and the paper's own 100-speaker results show this scalability limit is fundamental, not just a matter of more training: increasing identity overlap between retain and forget sets erodes separation regardless of which parameter-modification method is used (§6.2, §7).
Beyond the scalability ceiling, the study is limited to a single TTS backbone (StyleTTS2) and a single English dataset (LibriTTS), so it is unclear whether the reported AUC/FSSIM gaps hold for other zero-shot architectures (e.g. codec-language-model or flow-matching TTS systems) or other languages. Utility and privacy are assessed entirely with automated metrics (Whisper WER, UTMOS, WavLM speaker similarity); no human listening test verifies that the perceptual naturalness or residual identity leakage tracks these automated proxies. The privacy threshold (0.86 cosine similarity) is tied to a specific speaker-verification model (WavLM-base-plus-sv), and the paper itself notes elsewhere that verification thresholds are model-dependent, which limits how directly the reported AUC/FSSIM numbers transfer to a different verification system.
Wiki Connections
- Zero-Shot TTS — studies a targeted failure mode of zero-shot voice cloning (unwanted reproduction of specific speaker identities) and proposes model-level interventions to suppress it while preserving zero-shot synthesis for other speakers.
- Evaluation Metrics — introduces a distribution-aware privacy evaluation framework (AUC separability and average/maximum Forget Set Similarity) as an alternative to raw average speaker-similarity for assessing speaker-erasure methods.
- Speaker Adaptation — operates directly on the speaker-conditioning pathway of a zero-shot TTS model, manipulating how the model responds to forget-set versus retain-set speaker references.
- Do Not Mimic My Voice — this paper adapts that work’s Teacher-Guided Poisoning framework, originally built for VoiceBox, to a structurally different backbone (StyleTTS2) and proposes Encoder-Guided Poisoning as a variant.
- Neural Codec Language Models are Zero-Shot TTS Synthesizers — cited as an example of the zero-shot voice cloning capability that motivates the paper’s privacy concern.
- UTMOS — used as the automated naturalness proxy for utility evaluation across all forget-set settings.