EACL 2026 · Workshop

Shenran Wang et al. (University of British Columbia, National Research Council Canada) · → Paper · Demo: ? · Code: ✓

Compares inference-time, off-the-shelf, and training-time voice conversion strategies for anonymizing speaker identity in low-resource text-to-speech systems for two Canadian Indigenous languages, nêhiyawêwin and SENĆOŦEN.

Problem

Neural TTS systems trained on recordings from identifiable community members raise privacy and reputational risks: if a model makes an error or is misused, the credited speaker’s likeness can lend false credibility to the output. This concern is acute for Indigenous language education projects, where the small number of contributing speakers are typically well known within their community. Speaker anonymization, masking or altering the identity conveyed by synthesized speech while preserving intelligibility and naturalness, offers a partial mitigation, but prior work on voice conversion for anonymization has largely targeted high-resource languages with abundant speakers and standard evaluation tooling. It is unclear which anonymization strategy works best when training data per language is scarce and the target languages differ structurally (and are cross-lingually distant) from the models’ pretraining data.

Method

The paper builds all systems on StyleTTS2, chosen for its demonstrated few-shot cross-lingual speaker adaptation and its self-supervised style encoder, which learns speaker and prosodic style embeddings directly from reference audio rather than from explicit speaker labels. Three anonymization strategies are compared. First, inference-time modification of a monolingual StyleTTS2 model’s style embeddings without any additional training: averaging a speaker’s own embeddings, applying a gender-steering offset (difference between mean male and female embeddings), replacing embeddings with the gender-average, injecting Gaussian noise, or substituting embeddings from ten randomly selected LibriTTS-R speakers (“Custom”). Second, post-hoc voice conversion of the no-conversion baseline output using four pretrained SeedVC configurations (v1-base, v1-small, v1-xlsr, v2), varying similarity and accent-related hyperparameters for v2. Third, training multilingual StyleTTS2 models from scratch on combinations of nêhiyawêwin, SENĆOŦEN, and 53 hours of LibriTTS-R English data, with an architectural addition of language embeddings injected into the style encoders, the prosodic text encoder, and the text encoder to disentangle language identity from the learned style representation. At inference for the trained models, the target language’s speech is generated using style embeddings averaged from ten LibriTTS-R speakers, mirroring the “Custom” inference-time setting. Because standard reference-based objective metrics (PESQ, SI-SDR) cannot compare synthesized anonymized speech to a matching ground truth, the paper reports predicted PESQ and SI-SDR via torchaudio-squim, predicted MOS via UTMOS, speaker-embedding cosine similarity (SECS) via Resemblyzer, and accent confidence via an MMS-LID classifier’s probability that generated nêhiyawêwin audio is still recognized as nêhiyawêwin (CRK-Conf); no human listening test was conducted due to the difficulty of recruiting raters for these low-resource languages.

Key Results

On nêhiyawêwin, most inference-time embedding manipulations (gender steering, gender averaging, noise injection) left speaker similarity nearly unchanged from the no-conversion baseline (SECS 0.81-0.87 vs. 0.87 ground truth-relative baseline), indicating poor anonymization despite the different manipulations. The “Average” and “Custom” inference-time settings were the exceptions, reaching the lowest SECS (0.64 and 0.63) with the highest predicted MOS (4.02 and 4.06) among inference-time methods, but the same “Custom” setting was substantially less effective on SENĆOŦEN (SECS 0.74 vs. 0.87 baseline), suggesting inference-time embedding substitution is not robust across languages or speaker-count regimes. Off-the-shelf SeedVC conversion achieved the lowest similarity scores overall (SECS as low as 0.52-0.58) but at a clear quality cost, with SeedVC-v1-base dropping to PESQ 2.46 and predicted MOS 2.34 versus a ground-truth PESQ of 3.58. Training multilingual models from scratch produced the most consistent trade-off: models with language embeddings reduced similarity (SECS 0.61-0.63 on nêhiyawêwin, 0.69 on SENĆOŦEN) while keeping quality high (MOS up to 4.03) and retaining a reasonable accent-confidence score (CRK-Conf 0.55-0.67); models without language embeddings scored marginally better on quality and similarity but exhibited a strong English accent, quantified by CRK-Conf collapsing to 0.003-0.06 versus 0.53 for ground truth. Combining all three languages (CRK-STR-ENG) did not outperform the two-language combination (CRK-ENG) on nêhiyawêwin, showing that adding more pretraining languages is not uniformly beneficial.

Novelty Assessment

The contribution is primarily an engineering and empirical comparison rather than a new architecture: StyleTTS2, SeedVC, UTMOS, torchaudio-squim, Resemblyzer, and MMS-LID are all existing components, applied and combined for a new application (speaker anonymization for two specific Indigenous languages) and a new operating regime (very low speaker counts, cross-lingual transfer from English). The one architectural change, adding language embeddings to StyleTTS2’s style and text encoders to separate language identity from speaker style, is a targeted adaptation of an existing model rather than a novel structure, though it produces a clear, measurable effect (accent confidence) documented for the first time in this cross-lingual anonymization setting. The paper’s most durable contribution may be methodological: it demonstrates that off-the-shelf reference-based VC quality metrics are unusable for anonymized speech evaluation and that even multiple existing proxy metrics (SECS, CRK-Conf) capture different, sometimes conflicting, failure modes (timbre similarity vs. accent leakage).

Field Significance

low - This is a narrowly scoped applied comparison on two specific low-resource, low-speaker-count Indigenous languages, using entirely existing model components, with no human evaluation and no new architecture released beyond configuration code. It provides a useful empirical data point on the limits of inference-time and off-the-shelf voice conversion for anonymization under extreme data scarcity, and identifies a concrete accent-leakage failure mode using an MMS-LID-based diagnostic, but its findings are demonstrated only on this specific data regime.

Limitations and Open Questions

No human evaluation was conducted; all quality and similarity judgments rely on predicted proxy metrics (UTMOS, torchaudio-squim, Resemblyzer, MMS-LID), which the authors themselves note may carry model-specific bias or error, and this could not be validated against human raters given the small number of available Indigenous-language speakers.

  • All target-language test speakers were seen during training, so results characterize anonymization of known training speakers rather than generalization to unseen speakers.
  • SENĆOŦEN evaluation is limited to two speakers total, making it difficult to distinguish genuine method effects from speaker-specific idiosyncrasy.
  • No existing language-identification classifier is trained on SENĆOŦEN, so accent leakage could only be quantified for nêhiyawêwin (CRK-Conf), leaving the SENĆOŦEN accent-entanglement claims impressionistic rather than measured.
  • The authors explicitly flag the lack of an evaluation metric tailored to cross-lingual voice conversion as an open problem, rather than proposing one themselves.

Claims

  • supports: Training a multilingual model with explicit language-identity embeddings, rather than relying on inference-time embedding manipulation alone, produces a more robust trade-off between speaker anonymization and speech quality when transferring style from a higher-resource language to a low-resource target language.

    Evidence: Trained models with language embeddings reduced SECS from a 0.87 no-conversion baseline to 0.61-0.69 while keeping MOS at 3.76-4.03, whereas inference-time “Custom” embedding substitution generalized well on nêhiyawêwin (SECS 0.63) but degraded sharply on SENĆOŦEN (SECS 0.74, versus the trained STR-ENG model’s 0.69). (§4, Tables 2-3)

  • complicates: Removing an explicit language-identity signal from a multilingual voice-style representation causes language identity to entangle with speaker/style embeddings, leaking the higher-resource language’s accent into synthesized low-resource-language speech.

    Evidence: Multilingual models trained without language embeddings scored slightly better on PESQ, SI-SDR, and SECS than their language-embedding counterparts, but their MMS-LID-predicted confidence that generated audio was nêhiyawêwin collapsed to 0.003-0.06, compared to 0.53 for ground-truth recordings and 0.55-0.67 for the language-embedding variants. (§4, Table 2)

  • complicates: Off-the-shelf voice conversion models trained for general-purpose or singing-voice conversion do not transfer cleanly to anonymizing speech in unseen, structurally distinct low-resource languages without sacrificing substantial output quality.

    Evidence: Applying pretrained SeedVC configurations to nêhiyawêwin no-conversion output achieved the lowest speaker-similarity scores in the study (SECS 0.52-0.58) but at the cost of the lowest quality scores observed, with SeedVC-v1-base reaching PESQ 2.46 and predicted MOS 2.34 against a ground-truth PESQ of 3.58 and MOS of 3.27. (§3.3, §4, Table 2)

  • complicates: Standard reference-based objective speech quality and intelligibility metrics are not applicable to evaluating speaker-anonymized speech, and even purpose-fitted proxy metrics for similarity and accent can disagree about which method anonymizes best.

    Evidence: The authors report they could not use PESQ or SI-SDR in their normal reference-matched form because the anonymized output is deliberately produced in a different voice than the ground truth, and instead relied on predicted (non-reference) variants via torchaudio-squim plus a separate MMS-LID accent-confidence proxy, since high SECS-based similarity alone was judged insufficient evidence for successful anonymization when accent entanglement was possible. (§4, “Metrics”)

Wiki Connections

  • Voice Conversion — directly compares three families of voice conversion strategy (inference-time style-embedding edits, off-the-shelf pretrained VC, and trained multilingual transfer) for the specific goal of speaker anonymization rather than speaker mimicry.
  • Multilingual TTS — trains StyleTTS2 jointly on two Indigenous languages plus English LibriTTS-R data to enable cross-lingual style transfer into low-resource languages with very few native speakers.
  • Speaker Adaptation — relies on StyleTTS2’s few-shot cross-lingual speaker-adaptation capability and evaluates several ways of substituting or averaging style embeddings to change the perceived speaker identity at inference time.
  • Disentanglement — adds language-identity embeddings to StyleTTS2’s style and text encoders specifically to separate language identity from the style/speaker representation, and shows via an accent classifier that omitting this mechanism causes language and speaker style to entangle.
  • Zero-shot Voice Conversion with Diffusion Transformers (SeedVC) — used directly as an off-the-shelf voice conversion baseline (four pretrained SeedVC configurations) applied to StyleTTS2’s no-conversion output for anonymization.
  • UTMOS — used as the predicted-MOS metric for evaluating naturalness of anonymized speech in the absence of human listening tests.