arXiv · 2026 · Preprint
Myungjin Lee et al. (Ewha Womans University) · → Paper · Demo: ✓ · Code: ✓
Introduces TruS, the first training-free, inference-time speaker unlearning framework for zero-shot TTS, which dynamically steers identity-specific hidden activations in a pretrained F5-TTS model to suppress a target opt-out speaker’s voice while preserving prosody and emotion, generalizing to speakers never seen during the base model’s pretraining.
Problem
Zero-shot TTS systems can now synthesize convincing speech in the voice of any reference speaker, including individuals who never consented, creating a genuine privacy and security risk since speech carries strong biometric identity cues. Existing countermeasures fall short: watermarking is purely post hoc (it can trace synthetic speech after generation but cannot prevent misuse), voice anonymization substitutes one identity for another rather than prohibiting a specific identity outright, and machine-unlearning approaches for TTS (SGU, TGU) require costly retraining, cannot handle speakers outside their training set, and need re-training from scratch whenever a new opt-out request arrives. Real-world opt-out requests are most likely to come from individuals never seen during training, which retraining-based unlearning is structurally unable to address, and prohibiting voice generation entirely would undermine TTS’s legitimate accessibility and content-creation uses.
Method
TruS operates entirely at inference time on a frozen, pretrained zero-shot TTS model (F5-TTS, a DiT-based flow-matching architecture), requiring no retraining and generalizing to any DiT-based TTS backbone in principle. The method maintains a pool of retain-speaker reference utterances R and, for each, extracts intermediate feed-forward-network (FFN) activations from the DiT blocks, since these outputs carry strong timbre and identity signal after nonlinear channel mixing. Averaging these activations across N retain speakers at each block ℓ and flow-matching timestep t produces an “ID-prototype” P_Ret, a centroid representing typical (non-target) identity in that region of the model. Given a one-shot reference utterance from an opt-out speaker, the corresponding activation X_Opt is compared against the ID-prototype, and their L2-normalized difference defines an identity-specific steering vector S at each block and timestep, representing the direction in activation space associated with that speaker’s distinctive identity.
Rather than steering every layer and timestep uniformly, the method dynamically selects intervention points per target sample: cosine similarity between each opt-out activation and the corresponding ID-prototype is computed across all layers and flow steps, and layers whose average similarity falls below a dynamic threshold (empirically the global mean plus one standard deviation, μ+σ, found best via ablation) are selected as intervention layers, since lower similarity indicates the target speaker diverges more strongly from typical identity at that point. Within each selected layer, a further step-level filter restricts intervention to flow steps whose similarity is below that layer’s own average, producing a sparse, sample-specific set of layer-step intervention points rather than blanket steering. At each selected (layer, step) pair during the denoising process, the component of the activation aligned with the steering vector is projected out and subtracted (scaled by a steering strength α=1.2), suppressing identity-specific information while leaving the orthogonal, non-identity-aligned components of the representation, carrying linguistic content and prosody, untouched.
Key Results
On seen opt-out speakers (10 Emilia training-set speakers held out as an opt-out set), TruS reduces speaker similarity from the unmodified F5-TTS baseline’s 0.657 to 0.477 while matching or slightly improving word error rate (3.25% vs. 3.36% baseline), outperforming the retraining-based SGU and TGU baselines on content fidelity despite requiring zero training hours versus 48 and 430 GPU-hours respectively; SGU achieves the strongest identity suppression but at the cost of substantially degrading retain-speaker similarity too (generating effectively random voices for everyone), while TGU achieves only modest identity suppression alongside degraded retain-set performance. Critically, TruS generalizes to unseen opt-out speakers from LibriSpeech, entirely absent from F5-TTS’s Emilia pretraining data: speaker similarity drops from 0.668 to 0.488 and voice-randomness (Spk-ZRF-UO) rises from 0.906 to 0.913, at the cost of a moderate WER increase (2.03% to 3.26%), a capability retraining-based methods cannot offer since they can only remove influence of speakers present in their training data. On the CREMA-D emotional speech benchmark, TruS reduces identity similarity from 0.217 to 0.131 (confirming effective unlearning) while emotional similarity (SIM-Emo) remains close to the non-unlearning F5-TTS upper bound (0.723 vs. 0.732), indicating paralinguistic content is largely, though not perfectly, preserved alongside identity suppression. Ablations show the dynamic μ+σ layer-selection threshold provides the best overall balance: stricter thresholds under-suppress identity, while steering all layers uniformly yields only marginal additional identity suppression at a disproportionately larger WER cost; a retain-speaker pool size of N=30 provides the best trade-off between seen and unseen performance, with N=50 favoring unseen speakers at the cost of seen-speaker performance.
Novelty Assessment
The core contribution, treating speaker unlearning as an inference-time activation-steering problem rather than a retraining problem, is a genuine paradigm shift for the specific problem of opt-out speaker suppression in TTS, directly enabling a capability (generalization to unseen opt-out speakers) that prior retraining-based TTS unlearning methods (SGU, TGU) structurally cannot provide. The dynamic, per-sample layer-and-step selection mechanism is a meaningful advance over related activation-steering work in TTS (EmoSteer, which the paper explicitly contrasts itself against): EmoSteer applies a fixed, input-independent rule for selecting top-k activation channels, while TruS’s threshold-based selection adapts to where a given target speaker’s identity signal actually diverges from the retain-speaker prototype, validated by a clear ablation showing this adaptivity outperforms both fixed-threshold and uniform-all-layer alternatives. The projection-based subtraction mechanism for removing only the identity-aligned component of an activation, rather than perturbing it wholesale, is adapted from activation-steering techniques in the LLM literature but is a novel application to the TTS speaker-identity suppression problem specifically.
Field Significance
high — this paper is the first to demonstrate that speaker identity can be suppressed from a pretrained zero-shot TTS model's output entirely through inference-time activation steering, with no retraining, and crucially the first speaker-unlearning method for TTS shown to generalize to opt-out speakers never present in the base model's training data, a capability that is structurally impossible for retraining-based unlearning and directly relevant to real-world opt-out requests, which most plausibly come from individuals outside any training set.
Claims
- supports: A target speaker’s identity can be suppressed from a pretrained zero-shot TTS model’s output entirely at inference time, without any retraining, by steering intermediate diffusion-transformer activations away from a direction computed from the difference between the target speaker’s activations and an averaged retain-speaker prototype.
Evidence: TruS reduces speaker similarity on seen opt-out speakers from 0.657 (F5-TTS baseline) to 0.477 while WER (3.25%) matches or improves on the baseline (3.36%) and outperforms training-based SGU/TGU baselines, with zero training hours versus 48 and 430 GPU-hours respectively. (§3.2, Table 1)
- supports: Training-free, inference-time speaker unlearning generalizes to speakers never seen during the base TTS model’s pretraining, a capability retraining-based unlearning methods cannot structurally provide.
Evidence: Applying the same one-shot steering mechanism to LibriSpeech speakers absent from F5-TTS’s Emilia pretraining data reduces speaker similarity from 0.668 to 0.488 and increases voice-randomness (Spk-ZRF-UO) from 0.906 to 0.913, at the cost of a moderate WER increase (2.03% to 3.26%). (§3.2, Table 2)
- complicates: Projecting out and subtracting an identity-related activation direction at inference time can suppress speaker identity while leaving non-identity paralinguistic attributes like emotional expression largely intact, but this preservation is imperfect and comes with a measurable cost to linguistic fidelity.
Evidence: On CREMA-D, TruS reduces identity similarity from 0.217 to 0.131 while emotional similarity drops only slightly relative to the non-unlearning F5-TTS upper bound (0.723 vs. 0.732), and WER increases modestly under unlearning across all evaluated settings relative to the unmodified baseline. (§3.2, Table 3)
- supports: The layers and generation timesteps within a diffusion-transformer TTS model that carry the strongest identity-specific signal for a given target speaker vary dynamically across both depth and denoising step, so a fixed, input-independent set of intervention points is less effective than dynamically selecting intervention points per target sample.
Evidence: An ablation over layer-selection thresholds shows the dynamic mean-plus-one-standard-deviation criterion achieves the best overall balance, outperforming both stricter fixed thresholds (weaker identity suppression) and uniform all-layer steering (disproportionately larger WER increase for only marginal additional suppression). (§3.3, Table 4)
Limitations and Open Questions
Unlearning quality is asymmetric between seen and unseen opt-out speakers: WER for seen opt-out speakers is essentially unchanged from baseline (3.36%→3.25%), but for unseen opt-out speakers WER rises substantially (2.03%→3.26%, a 60% relative increase), indicating the method's content-fidelity preservation is meaningfully weaker when generalizing beyond speakers the base TTS model was pretrained on.
The steering strength α (fixed at 1.2) and the retain-speaker pool size N (found to trade off seen-versus-unseen performance non-monotonically, with N=30 chosen as a compromise rather than a clear optimum) are set empirically rather than adaptively per speaker or per deployment context. All evaluation is conducted on English speech using a single base TTS architecture (F5-TTS); the authors state the method should generalize to other DiT-based TTS architectures but this is not empirically tested within the paper.
Wiki Connections
- Zero-Shot TTS — operates on a pretrained zero-shot TTS model (F5-TTS), suppressing specific opt-out speaker identities at inference time without disrupting zero-shot cloning behavior for retained speakers.
- Disentanglement — explicitly isolates and removes only the identity-aligned component of intermediate DiT activations via projection-based steering, validated by showing non-identity attributes (emotion, measured via SIM-Emo) are largely preserved after suppression.
- F5-TTS — the pretrained, frozen DiT-based flow-matching TTS backbone this method operates on; all steering interventions target F5-TTS’s internal DiT block activations.
- Do Not Mimic My Voice — supplies the SGU and TGU retraining-based speaker-unlearning baselines this paper directly compares against, both requiring substantial GPU-hours and unable to generalize to unseen opt-out speakers.
- EmoSteer-TTS — a related training-free activation-steering method for TTS (targeting emotion control via fixed top-k channel selection), explicitly contrasted against this paper’s dynamic, per-sample layer-and-step selection mechanism.
- CosyVoice 2 — cited as an example of modern streaming, large-scale TTS systems within the broader landscape of zero-shot speech generation this paper’s unlearning framework is positioned to protect against misuse of.
- ECAPA-TDNN — supplies the pretrained speaker verification model used to compute the SIM metric quantifying identity similarity between generated and reference speech.
- Whisper — used (Whisper large-V3) to transcribe generated speech for computing word error rate as the linguistic fidelity metric.
- emotion2vec — used to compute SIM-Emo, the emotional similarity metric verifying that paralinguistic content is preserved after speaker unlearning.
- Emilia — the large-scale multilingual corpus F5-TTS was pretrained on; its English subset supplies the retain-speaker pool and the seen opt-out speaker set used for evaluation.