arXiv · 2026 · Preprint

Jinju Kim et al. (Sungkyunkwan University / Korea University) · → Paper · Demo: ✓ · Code: ?

Shows that speaker identity unlearning methods for zero-shot TTS revive previously erased voices when removal requests arrive one at a time, and proposes CORTIS, which combines contrastive Fisher-saliency parameter masking with orthogonal projection against prior unlearning directions so that earlier speakers stay forgotten without retaining their data.

Problem

Zero-shot TTS systems can clone a voice from a few seconds of audio, and privacy regulation (GDPR, CCPA, the right to be forgotten) gives individuals grounds to request that a deployed model stop reproducing their voice. Prior speaker identity unlearning work, notably the Teacher-Guided Unlearning (TGU) and Sample-Guided Unlearning (SGU) methods from Do Not Mimic My Voice, assumes the full forget set is known up front and removed in one optimization run. In deployment, requests arrive sequentially, and the provider faces a “data retention paradox”: re-running unlearning on the cumulative forget set requires keeping every previous requester’s voice data, which contradicts the right being exercised, while discarding the data after each request removes the anchor that keeps those speakers forgotten.

The paper formalizes continual speaker identity unlearning under two constraints: sequential request arrival (C1) and forget-data non-retention (C2), where each request’s data is discarded after processing. It then identifies the failure mode it calls catastrophic re-learning: because a zero-shot model generalizes to unseen speakers, the retain-loss regularization that protects utility during a later request carries no signal for previously forgotten speakers, so their identities drift back.

Method

CORTIS (Cumulative ORThogonal Identity Suppression) is applied on top of the TGU unlearning objective (the student is trained to match a text-only, random-voice teacher target when prompted with a forget speaker, plus a retain loss). The base VoiceBox model is fine-tuned, not frozen, but each request updates only a masked subset of its parameters. Two mechanisms constrain each request’s update.

Contrastive parameter localization. For request i, the method computes the diagonal Fisher information of the forget loss on the current speaker’s data and divides it (element-wise) by the maximum of the remain-set Fisher and the Fishers of all previously forgotten speakers. The top-k% of this saliency map (k = 30 in the main results, so 70% of parameters are frozen per request) defines a trainable mask. Parameters important for retain quality or for any earlier forget speaker are thus pushed out of the trainable set. A fixed remain subset suffices for the remain Fisher.

Orthogonal projection on a cumulative forget subspace. Following the gradient-projection recipes of orthogonal gradient descent and gradient projection memory from continual learning, gradient snapshots are collected during each speaker’s unlearning run and reduced by truncated SVD to a rank-R orthonormal basis (R = 40 in the main results), after subtracting the prior subspace so each new basis is orthogonal to earlier ones. Rather than concatenating bases indefinitely, the per-speaker bases are scaled by their singular values, stacked, and re-truncated to a fixed merged rank, keeping per-step projection cost constant regardless of the number of requests. After every optimizer step, the masked weight delta is projected onto the orthogonal complement of the merged basis, so updates for the new speaker cannot move along directions previously used to forget earlier speakers. The merge is computed through a small Gram matrix to stay tractable at the scale of a 24-layer model (Appendix A).

Figure 2: Method overview. (a) At sequence i, the saliency score (2) compares the Fisher information of the current forget set f_i against the maximum across the F_{i-1}. Selecting the top-k% from saliency map yields the mask M_i, which restricts updates to forget-relevant parameters. On (b), without intervention, the optimizer step δ would move from θ_{i-1} to a configuration outside the orange region (dashed), reverting prior unlearning. We project δ onto the orthogonal complement of U_{<i}, so the resulting θ_i remains in the intersection of both regions: f_i is unlearned without disturbing F_{i-1}.

The backbone is a VoiceBox reimplementation (flow-matching, mel-spectrogram output, 24 Transformer layers with U-Net-style skips and ALiBi, a regression duration predictor, and a DiffWave vocoder) pre-trained for 500K steps on LibriHeavy. The first request runs 10K steps; subsequent requests run 1K steps at a lower learning rate (Appendix B).

Key Results

Evaluation uses five LibriHeavy forget speakers (the filtered set from Do Not Mimic My Voice) in a fixed randomized order, with remain-set quality measured on LibriSpeech test-clean. Metrics are WER (HuBERT-Large ASR) on the remain set (W-R) and forget sets (W-F), and WavLM-TDCNN speaker similarity on the remain set (S-R) and per forget speaker (S-f_i). Success thresholds (S-R ≥ 0.46, S-f < 0.32) are calibrated from same-speaker and different-speaker similarity distributions on LibriSpeech test-clean (Appendix E). There is no human listening test.

Across three sequential requests (Table 1), CORTIS is the only method that meets both thresholds at every step. After request 3, its forget similarities are 0.172 / 0.148 / 0.124 for f1 / f2 / f3, with S-R 0.557 (original model: 0.649) and W-R 2.8 (original: 2.1). The baselines fail in distinct ways:

  • TGU (sequential) forgets f1 at request 1 (0.164) but f1 reverts to 0.612 after request 2 and 0.603 after request 3; f2 reverts to 0.546.
  • SGU keeps all forget similarities low (at most 0.233) and reaches lower per-speaker values than CORTIS in several cells (e.g., f2 0.075 vs. 0.122 at request 2; f2 0.101 vs. 0.148 at request 3), but its remain similarity collapses to 0.348 and then 0.315, below the retention threshold.
  • UN (update-norm penalty) and SelFT (top-30% gradient-importance masking), adapted from continual unlearning for text-to-image models and composed with TGU, preserve S-R (0.580 and 0.548 after request 3) but let f1 re-emerge (0.638 and 0.553).

CORTIS does not lead on utility. Its remain WER is the highest of all unlearning methods at requests 1 and 2 (2.9 vs. 2.3 to 2.8), and at request 3 (2.8) it is better than TGU and UN (3.0) but behind SGU and SelFT (2.7). Its S-R is below TGU at request 1 (0.602 vs. 0.624), below SelFT and TGU at request 2 (0.553 vs. 0.585 and 0.563), and below TGU and UN at request 3 (0.557 vs. 0.582 and 0.580), exceeding only SelFT (0.548) and SGU (0.315) there. Reported standard deviations of S-R are around 0.09 to 0.11 (Table 6), so small S-R differences among CORTIS, SelFT, UN, and TGU are not clearly separable.

Extending to five requests (Table 7, Figure 3), the largest forget similarity stays below 0.2 at every step (maximum 0.198 for f1 after request 4; all five at or below 0.178 after request 5), while S-R declines to 0.527 and W-R stays at 2.7 to 2.8.

Ablations: removing projection while keeping the mask (Table 2) lets f1 and f2 drift to 0.334 and 0.397 after request 3, so masking alone is insufficient. Shrinking the mask to k = 20 (Table 3) keeps forget similarities comparable but lowers S-R after request 3 from 0.557 to 0.523. Rank R (Table 8) trades off differently: R = 30 gives the lowest prior-speaker similarities after request 3 (0.140, 0.086) but S-R drops to 0.476, while R = 20 gives the highest S-R (0.568) with f1 at 0.187. A merging baseline (TIES-Merging of per-speaker task vectors), which the authors exclude because it requires keeping the pre-trained model and per-speaker checkpoints, lets f3 stay at 0.563 after request 3 (Table 5).

On cost at request 3 (Table 4, two A100 80GB), CORTIS takes 3.5 hours and 3K steps versus 29 hours for a sequential TGU request and 87.5 hours for re-running TGU on the cumulative forget set, but uses the most peak GPU memory (49.3 GB vs. 30.8 GB for TGU). Note that CORTIS runs 1K steps per later request while the TGU, UN, and SelFT baselines run 10K to 11K steps, so the step budgets differ across methods.

Novelty Assessment

The main conceptual contribution is the problem formulation: framing speaker identity unlearning as a continual task under data non-retention, and showing empirically that the zero-shot generalization of the TTS model makes previously erased voices recover under ordinary retain regularization. Catastrophic re-learning is a clear, well-isolated failure mode, and the comparison with continual-unlearning regularizers from the vision literature (UN, SelFT) shows that limiting parameter drift is not enough.

The method itself assembles established continual-learning ingredients: Fisher-based parameter importance (as in EWC and saliency-based unlearning) and gradient-subspace projection (OGD, GPM). The genuinely new pieces are the contrastive saliency ratio that uses prior forget-speaker Fishers in the denominator, and the energy-weighted fixed-rank merge of per-speaker bases. The ablations support that both components are needed. Evaluation is narrow: one backbone (VoiceBox), one ordering of five speakers, automatic metrics only, and a single seed-averaged run per configuration.

Field Significance

Moderate. The paper adds a deployment-realistic dimension to speaker identity unlearning for zero-shot TTS, demonstrating that one-shot unlearning results do not transfer to sequential requests and providing a working recipe and threshold-based evaluation protocol for the continual case. Its evidence comes from a single model and a short request sequence, so it is best read as a first demonstration of the problem and a baseline method rather than a settled solution.

Claims

  • complicates: Speaker identity unlearning in zero-shot TTS that succeeds for a fixed forget set does not stay effective when removal requests are processed sequentially; erased voices recover once later unlearning runs no longer apply a forget loss to them.

    Evidence: Sequential TGU forgets the first speaker at request 1 (speaker similarity 0.164) but that speaker returns to 0.612 after request 2 and 0.603 after request 3, close to the original 0.721; the second speaker similarly returns to 0.546. (§5.4, Table 1)

  • complicates: Regularizers that limit parameter drift preserve zero-shot cloning quality on retained speakers but do not protect previously unlearned speaker identities.

    Evidence: Update-norm and selective fine-tuning baselines hold remain similarity at 0.580 and 0.548 after request 3, yet the first forgotten speaker rises to 0.638 and 0.553; SGU avoids reversion only by collapsing remain similarity to 0.315. (§5.4, Table 1)

  • supports: Constraining unlearning updates to be orthogonal to the gradient subspace used for earlier removals keeps previously erased speaker identities suppressed without retaining their audio.

    Evidence: With projection, prior-speaker similarities after request 3 are 0.172 and 0.148; with masking alone they revert to 0.334 and 0.397. Over five requests no forgotten speaker exceeds 0.198. (§5.5, Table 2; §5.4, Figure 3, Table 7)

  • complicates: Durable continual speaker unlearning in zero-shot TTS carries a measurable cost in cloning quality and intelligibility for retained speakers.

    Evidence: CORTIS lowers remain-set speaker similarity from 0.649 to 0.557 after three requests and 0.527 after five, and raises remain WER from 2.1 to 2.8; its remain WER is the highest of all unlearning methods at requests 1 and 2. (§5.4, Table 1, Table 7)

  • refines: Restricting speaker unlearning to parameters selected by contrastive Fisher saliency yields largely disjoint per-speaker update regions, but the size of that region trades retained-speaker quality against update locality.

    Evidence: Pairwise Jaccard overlap between per-speaker masks is 0.049 to 0.197; shrinking the mask from 30% to 20% of parameters keeps forget similarity comparable but lowers remain similarity after request 3 from 0.557 to 0.523. (§Appendix G, Figure 6; §5.5, Table 3)

Limitations and Open Questions

All experiments use a single VoiceBox reimplementation and a single ordering of five LibriHeavy speakers; the authors leave autoregressive codec-based and diffusion-based backbones to future work (§6), and longer request sequences beyond five are untested. Evaluation is entirely automatic (ASR WER and embedding similarity with calibrated thresholds); there is no human listening test of whether forget-speaker outputs are perceptually non-identifying or of naturalness on remain speakers.

  • Adversarial robustness is out of scope. The threat model assumes an honest provider; recovery of forgotten identities via fine-tuning or activation attacks on released weights is not studied (§6).
  • Retained per-speaker artifacts. The saliency ratio needs the Fisher diagonals of every previously forgotten speaker, and projection needs their gradient bases. These are derived from forget data and kept across requests; the paper argues that retaining per-speaker checkpoints violates non-retention for the merging baseline, but does not analyze whether these smaller artifacts carry recoverable identity information.
  • Unequal training budgets. Later CORTIS requests run 1K steps at a lower learning rate, while baselines run 10K to 11K steps, which complicates attributing differences purely to the mechanisms.
  • Hyperparameter sensitivity. Rank R and mask size k shift the forgetting/retention trade-off noticeably (R = 30 drops S-R to 0.476 after request 3), and selection was made on the same sequence used for reporting.
  • First-request cost. The first request still requires 10K steps, and CORTIS has the highest peak memory of all methods (49.3 GB).

Wiki Connections

  • Zero-Shot TTS — Unlearning is applied to a zero-shot TTS model and evaluated both as loss of cloning ability for forgotten speakers and as preserved zero-shot cloning on unseen LibriSpeech speakers.
  • Flow Matching — The unlearned backbone is a VoiceBox flow-matching model whose parameters are partially fine-tuned at each request.
  • Speaker Adaptation — Speaker-targeted fine-tuning run in reverse, where per-speaker data drives localized updates that remove, rather than add, a speaker’s reproducibility.
  • Do Not Mimic My Voice — Direct predecessor by overlapping authors that introduced speaker identity unlearning for zero-shot TTS with TGU and SGU in a one-shot setting; this paper reuses its forget speakers, adopts TGU as the base objective and both methods as baselines, and adds the sequential, non-retention setting plus the masking-and-projection method that prevents earlier speakers from reverting.
  • VALL-E — Named as an autoregressive codec-based backbone to which cross-architecture validation of the method is left for future work.