arXiv · 2026 · Preprint
Mu Yang et al. (Center for Robust Speech Systems (CRSS), University of Texas at Dallas) · → Paper · Demo: ? · Code: ?
A post-hoc, training-free activation-steering method that neutralizes the accent of a zero-shot TTS system’s output while preserving the reference speaker’s timbre.
Problem
Zero-shot TTS systems clone both the timbre and the accent of a reference speaker from a short prompt, but these two attributes are not independently controllable: when the reference speech is accented, the synthesized output inherits the accent along with the voice. This is a practical obstacle for accent-free voice cloning, which is useful for generating training targets for accent conversion models and for producing personalized, accent-neutral pronunciation feedback for second-language learners. Prior disentanglement approaches typically require retraining or fine-tuning the TTS model to separate accent from speaker identity, which is costly to apply to a large pretrained zero-shot TTS system.
Method
The approach targets Qwen3-TTS, an LLM-based zero-shot TTS model whose backbone is a 28-layer Transformer that consumes aggregated speech-tokenizer features and predicts a semantic token stream, with a lightweight 5-layer Multi-Token-Prediction module producing the remaining acoustic RVQ streams on top of it. The steering method operates only on the backbone LLM’s internal activations and requires no additional training of the TTS model itself.
Steering vectors are extracted offline using ARCTIC (accent-neutral US English) and L2-ARCTIC (Mandarin-accented English) as a contrastive pair: the same target texts are synthesized once with an accent-neutral reference and once with an accented reference, and for each Transformer layer the steering vector is the difference between the mean generated-token activations of the accented condition and the neutral condition (§3.2, Eq. 1). Because accent is entangled with speaker identity in this setup (each L2-ARCTIC speaker always carries the same accent), the reference waveforms used for vector extraction are randomly perturbed with formant scaling, F0 scaling, and frequency-shaping equalization before roughly 70% of extraction samples, to push the steering vectors toward accent-specific rather than speaker-specific directions.
At inference, when the reference speech is accented, the corresponding layer’s activations at each decoding step are shifted by subtracting a scaled steering vector (strength hyperparameter alpha), then renormalized to the original activation norm to better preserve speaker timbre (§3.3, Eq. 2). Only generated-token activations are modified; prompt-token activations are left untouched. The paper restricts experiments to single-layer steering (one Transformer layer modified per run) rather than steering multiple layers simultaneously.
Key Results
On L2-ARCTIC with Chinese-accented reference prompts, single-layer steering at layer 15 sharply reduces the model’s Accent Match Rate toward the target (Chinese) accent while raising the match rate toward the native (US) accent, for both the 0.6B and 1.7B Qwen3-TTS models, at the cost of a drop in speaker-embedding cosine similarity (e.g., 1.7B model: 0.84 unsteered to 0.76 steered) (§5.1, Table 1). Word error rate and UTMOS naturalness are maintained or improved under steering relative to the unsteered accented baseline, and the effect generalizes to unseen accented speakers in speechocean762 (out-of-domain), where WER drops from 56.41% unsteered to 32.43% steered (§5.1, Table 1). A layer-wise sweep shows that steering middle layers (15, 20) gives the best balance of accent neutralization and timbre preservation, while early and top layers under-steer the accent, and higher steering strength trades further accent reduction against larger speaker-similarity loss and (at the earliest layers) inference instability (§5.2, Figure 2). An ablation on steering-vector extraction shows that the proposed speaker-perturbation data augmentation consistently improves speaker similarity relative to extraction without augmentation, and that as few as 1,000 extraction samples achieve results close to those obtained with 4,000 (§5.3, Table 2).
Novelty Assessment
The paper does not propose a new TTS architecture; it applies activation steering, an inference-time technique already explored for TTS attribute control in several concurrent works cited in the paper (prosody/mispronunciation editing, emotion control, speaker unlearning), to a new target attribute: accent. Its specific technical contributions are the data-augmentation strategy for breaking the accent-speaker entanglement during steering-vector extraction, and a single-pass, single-layer steering procedure that avoids the external classifiers and multiple inference passes used by at least one cited prior method. The contribution is best read as a focused, efficient instantiation of an increasingly established technique family (post-hoc activation steering for zero-shot TTS attribute control) applied to a specific, previously unaddressed attribute, rather than a structurally new mechanism.
Field Significance
moderate — This paper extends a growing family of training-free activation-steering methods for zero-shot TTS attribute control to accent neutralization, contributing a concrete extraction-time disentanglement trick (speaker-perturbing augmentation) and demonstrating generalization to unseen, out-of-domain accented speakers. It is a targeted engineering contribution within an established technique family rather than a foundational shift, evaluated on a single accent pair (Mandarin-accented to native English) with a single base model family.
Claims
- supports: Post-hoc activation steering can decouple accent from speaker timbre in a pretrained zero-shot TTS model without any additional training of the model itself.
Evidence: Subtracting a layer-wise steering vector (computed offline from accented vs. neutral activation differences) at inference time reduces Accent Match Rate toward the accented target and raises it toward the native target for both 0.6B and 1.7B Qwen3-TTS models, while speaker-embedding similarity remains partially but not fully preserved. (§5.1, Table 1)
- complicates: Activation steering for one voice attribute trades off against preservation of a correlated attribute, and this trade-off is layer- and strength-dependent.
Evidence: Steering strength alpha=2.0 achieves larger accent reduction than alpha=1.0 but causes a significantly larger drop in speaker similarity and, at early layers, a marked drop in Inference Success Rate; middle layers (15, 20) give the best accent/timbre balance while early and top layers under-steer the accent. (§5.2, Figure 2)
- supports: Explicitly perturbing the correlated attribute during steering-vector extraction (rather than relying on natural variation in the extraction set) improves the disentanglement quality of the resulting steering direction.
Evidence: Applying formant, F0, and equalizer-based speaker perturbations to reference waveforms during steering-vector extraction improves speaker similarity of the steered output relative to extraction without augmentation, across all tested extraction-set sizes (40, 1000, 4000 samples). (§5.3, Table 2)
- refines: A small number of contrastive extraction samples is sufficient to obtain a usable attribute-steering direction, without requiring the full available extraction set.
Evidence: Steering vectors extracted from 1,000 samples produce accent/timbre trade-offs comparable to those extracted from 4,000 samples, on L2-ARCTIC with the 1.7B model at layer 15. (§5.3, Table 2)
Limitations and Open Questions
The method is validated on a single accent pair (Mandarin-accented to native US English), a single base model family (Qwen3-TTS, 0.6B/1.7B), and a small extraction pool (4 native and 4 non-native speakers), leaving open whether the same steering vectors and augmentation recipe generalize to other accent pairs, languages, or TTS architectures.
The paper reports a consistent trade-off between accent neutralization and speaker-similarity preservation that is not fully resolved: steering strong enough to substantially reduce accent classification always costs some speaker similarity, and the authors note that steered speech sometimes exhibits audible local pitch and prosody shifts. Accent Match Rate and Inference Success Rate rely on an external pretrained accent classifier and the TTS model’s own stability under activation perturbation, respectively, rather than on human listening evaluation of accent-neutrality or naturalness. The single-layer restriction is also a design choice rather than a demonstrated optimum; the paper explicitly leaves multi-layer steering, which the framework in Figure 1 is designed to support, unexplored.
Wiki Connections
- Zero-Shot TTS — targets a core failure mode of zero-shot voice cloning, where accent transfers along with timbre from the reference prompt, and proposes an inference-time fix that leaves the underlying zero-shot TTS model unchanged.
- Disentanglement — separates accent from speaker identity in the activation space of a pretrained TTS model using a steering-vector extraction procedure with an explicit speaker-perturbation augmentation, validated by an ablation showing its effect on speaker similarity.
- Autoregressive Codec TTS — operates entirely within the backbone LLM activations of an autoregressive, codec-token-based zero-shot TTS model (Qwen3-TTS), without modifying its architecture or training.
- Voice Conversion — addresses accent neutralization, a goal closely related to accent conversion, as a byproduct of controllable zero-shot TTS rather than through a dedicated conversion model.
- Qwen3-TTS Technical Report — is the base zero-shot TTS model whose backbone activations are steered; no modifications are made to its architecture or weights.
- EmoSteer-TTS — is a closely related prior activation-steering method for TTS attribute control that this paper explicitly contrasts against, noting that EmoSteer requires an external emotion classifier and multiple inference passes where this method uses a single pass.
- TruS — is cited as adopting a similar steering-vector extraction method for inference-time speaker unlearning in a diffusion-based TTS model, situating this paper’s accent-neutralization use case within the same technique family.
- EmoShift — is cited as a related lightweight activation-steering approach for emotion-aware TTS, part of the same emerging family of post-hoc, training-free TTS attribute-steering methods this paper extends to accent.