arXiv · 2026 · Preprint

Haowei Lou et al. (UNSW Sydney / Macquarie University / University of California San Diego / Adobe Research) · → Paper · Demo: ✓ · Code: ?

AutoSIFT splits a global speaking-style embedding into label-grounded category subspaces (gender, age, emotion, language) plus a quantized residual, so a text prompt can overwrite only the categories it names while every other category and the residual are copied from a reference utterance.

Problem

Style-controllable TTS is usually driven either by a text description of the voice or by a reference recording. Text prompts are underspecified: a description such as “happy” or “old male” cannot capture timbre, breathiness, rhythm or micro-prosody. Speech prompts carry those details but encode style as one holistic vector, so speaker, age, gender, language and emotion are entangled and a single category cannot be edited without disturbing the rest. The paper frames the missing capability as Arbitrary Style Infilling (ASI): given a text prompt that specifies an arbitrary subset of style categories and a reference utterance, produce speech that follows the text for the specified categories and inherits all unspecified categories, plus residual style that no label covers, from the reference. Use cases include dubbing, game voice variants and post-production edits of a recorded line.

Method

AutoSIFT is trained in three stages, each freezing the modules from earlier stages.

Stage 1, style-controllable TTS backbone. Speech is represented as a latent sequence (roughly 16 ms per frame) obtained from an STFT followed by an encoder or vocoder latent space; the paper does not name a codec. A Style Extractor (Transformer encoder plus attentive pooling, L2-normalized) compresses the reference latent into a single global style vector S (192 dimensions), which a linear projector expands into block-wise conditioning for each DiT block. The generator is a flow-matching Diffusion Transformer (12 blocks, hidden size 384, 6 heads, DiT-S scale) conditioned through AdaLN-Zero on the style vector and on content embeddings. The content side reuses the frozen text encoder of ParaStyleTTS, adapted by a trainable Feed-Forward Transformer. Freezing the content prior is meant to push all non-content information needed for reconstruction into S. The objective is the standard flow-matching velocity regression loss. No parameter count is reported.

Stage 2, Style Disentangler. Linear projectors map S into four 48-dimensional category subspaces (gender, age, emotion, language) and one residual subspace. Each category has a set of learnable class prototypes. A Prototypical Alignment Loss (PAL, taken from ParaMETA) pulls each category embedding toward its class prototype under a stop-gradient. Unlabelled samples get a pseudo-label from the nearest prototype by cosine similarity. The residual embedding passes through a vector-quantization layer with its own codebook. The selected prototypes and the quantized residual are concatenated and passed through a residual SiLU network that must reconstruct S (MSE against the stop-gradient of S). The total loss is reconstruction + PAL + VQ.

Stage 3, Arbitrary Style Infiller. A text style description is encoded by a pretrained MPNet sentence encoder (768 dimensions), projected into the 192-dimensional style space and split into the same four category sub-embeddings. A hard gate takes the text-derived embedding for each category the prompt specifies and the reference speech’s selected prototype otherwise. The quantized speech residual is always taken from the reference. A linear fuser maps the concatenation back to a style vector trained to reconstruct S. “Censored learning” randomly drops categorical text labels at a 25% rate during training so that the fuser learns to fall back on speech-derived categories.

Figure 1: Overview of AutoSIFT. 1) A FM-DiT-based SC-TTS extracts a style embedding from speech. 2) A Style Disentangler decomposes style embedding S into category-specific and residual style embeddings. 3) An Arbitrary Style Infiller performs the ASI task by following text-specified categories and preserving text-undescribed and residual styles from S.

Training data is a bilingual (English and Mandarin) pool of about 110k utterances, 138 hours and 6,361 speakers, built from Baker, LJSpeech, ESD, CREMA-D, CommonPhone and Genshin Voice. It is labelled for gender (2 classes), emotion (7), age (5) and language (2). The backbone trains for 800k steps on one RTX 4090 (about 48 hours). The disentangler and infiller train in about 2 and 4 hours respectively on a laptop RTX 4060.

The paper also defines two evaluation metrics for ASI. Described Style Accuracy (DSA) is the fraction of text-specified categories that frozen, independently trained category classifiers recognize in the output. Residual Style Accuracy (RSA) is the fraction of unspecified categories whose predicted label matches the reference speech.

Key Results

Category disentanglement (Table 1, speaker-independent split following ParaMETA). On every category and all three metrics (balanced accuracy, Macro-F1, Weighted-F1), AutoSIFT with PAL scores higher than every baseline in the table: three pretrained audio-text models (CLAP general, CLAP speech and music, ParaCLAP) and same-backbone cross-entropy, InfoNCE (CLAP-style) and ParaMETA training. Balanced accuracy compared with the strongest baseline per category:

CategoryAutoSIFT (PAL)Strongest baselineAutoSIFT without PAL
Emotion67.255.2 (same-backbone InfoNCE)5.7
Gender87.878.4 (ParaMETA)47.1
Age59.429.7 (ParaMETA)0.9
Language98.891.1 (ParaMETA)44.3

Removing PAL makes the category subspaces collapse to near or below chance, even though the reconstruction loss still decreases.

Generation quality (Table 2). The table reports phoneme error rate (PER, lower is better), MOS from 10 native listeners per language, and style similarity (S-SIM, cosine similarity in a style-embedding space). The results split by prompt type:

  • Text prompt. AutoSIFT has the lowest PER among text-prompted systems: 3.59 in English against 3.77 for PromptTTS++, 6.11 for CosyVoice, 6.53 for ParaStyleTTS and 9.00 for ParaMETA; 14.32 in Chinese against 16.48 for CosyVoice. F5-TTS with a speech prompt still has a lower PER in both languages (2.97 and 12.30). AutoSIFT’s text-prompted English MOS (4.00) ties F5-TTS (4.00) and is below CosyVoice speech-prompted (4.06).
  • Speech prompt. AutoSIFT has the highest MOS in both languages: 4.11 in English against 4.06 for CosyVoice and 4.00 for F5-TTS; 3.78 in Chinese against 3.61 for CosyVoice. The gaps are well within the reported standard deviations (about 0.8 to 1.1). Its S-SIM is also the highest: 0.8512 in English against 0.6501 for IndexTTS, and 0.8100 in Chinese against 0.6884. However, its PER is worse than F5-TTS and IndexTTS in English (4.74 against 2.97 and 4.19) and worse than every speech-prompted baseline except ParaMETA in Chinese (18.88 against 12.30 to 16.58).

So speech-prompted AutoSIFT trades intelligibility for style preservation.

Arbitrary style infilling (Figure 2). The ASI results are shown only as radar charts, with no numeric values or variance. The text reports that the ASI configuration reaches close to 100% DSA and RSA on language and gender and about 90% on emotion and age. It also reports that control accuracy is highest when only one category is edited and falls as more categories are text-specified, which the authors attribute to interference between multiple text-specified prototypes.

The baselines are run “following their recommended settings or official implementations”. The paper does not state whether the large zero-shot baselines were retrained on the same 138-hour pool, so the Table 2 comparison mixes in-domain training for AutoSIFT with models trained elsewhere.

Novelty Assessment

The contribution is architectural, mainly in the style pathway. The backbone (flow-matching DiT with AdaLN-Zero style conditioning) and the PAL objective are inherited, PAL from the same group’s ParaMETA. The new parts are:

  • the reconstruction-guided disentangler, which adds a VQ residual branch and requires prototypes plus residual to reconstruct the holistic style vector;
  • the gated infiller with censored label dropout, which puts text-derived and speech-derived category embeddings in one shared subspace;
  • the ASI task formulation and its DSA/RSA metrics.

The gate is a hard, label-aware switch rather than a learned selection, so the “sifting” depends on knowing which categories a prompt specifies. The evidence is strongest for disentanglement: Table 1 includes an ablation, statistical-significance testing and several runs. For the headline ASI claim the evidence is weakest, since there is no numeric table and no ablation of the residual branch or of censored learning. The text prompts are short category descriptions that cover four categories, so the result tells us about category-level editing rather than free-form natural-language instruction following.

Field Significance

Moderate — The paper sets out partial-specification style editing (text overrides named categories, the reference supplies everything else) as a concrete TTS task with its own metrics. It shows that prototype-grounded subspaces plus a quantized residual can support that editing in a small flow-matching model. The evidence base is a single bilingual in-house data pool with four categories, and the infilling results are only shown as figures, so the work is best read as an early method and evaluation design for selective style editing, not a settled result.

Claims

  • supports: Explicit prototype alignment is necessary for decomposed style subspaces to carry interpretable category information; a reconstruction objective alone yields subspaces that reconstruct well but are not semantically organized.

    Evidence: Without the Prototypical Alignment Loss, balanced accuracy collapses to 5.7 (emotion), 47.1 (gender), 0.9 (age) and 44.3 (language), against 67.2, 87.8, 59.4 and 98.8 with it, although the reconstruction loss still decreases. (§5.2, Table 1)

  • supports: Category subspaces learned from a generative TTS style embedding can separate paralinguistic attributes better than general audio-text contrastive embeddings or same-backbone classification and contrastive training.

    Evidence: AutoSIFT (PAL) exceeds CLAP, ParaCLAP, cross-entropy, InfoNCE and ParaMETA on balanced accuracy, Macro-F1 and Weighted-F1 for all four categories under a speaker-independent split, e.g. emotion 67.2 against 22.1 for CLAP (speech and music) and 55.2 for the best same-backbone baseline. (§5.2, Table 1)

  • complicates: Stronger preservation of reference style in prompted TTS can come with worse intelligibility than holistic zero-shot cloning.

    Evidence: Speech-prompted AutoSIFT has the highest S-SIM (0.8512 English, 0.8100 Chinese) but a higher PER than F5-TTS and IndexTTS in English (4.74 against 2.97 and 4.19) and than F5-TTS, IndexTTS and CosyVoice in Chinese (18.88 against 12.30 to 16.58). (§5.3, Table 2)

  • complicates: Category-level style control degrades as more attributes are specified at once, because the specified attributes interfere with each other.

    Evidence: Control accuracy under partial text specification is highest for single-category edits and falls as the text-prompting rate increases toward 100%. (§5.4, Figure 2b)

  • refines: Combining text-described attributes with a reference utterance works better when the reference contributes only the categories the text leaves unspecified plus an unlabelled residual, rather than a holistic style vector that competes with the text.

    Evidence: The gated infiller keeps text embeddings for specified categories and speech prototypes plus a VQ residual for the rest, and the authors report about 100% DSA/RSA on language and gender and about 90% on emotion and age, ahead of text-only and speech-only baselines. (§4.3, §5.4, Figure 2a)

Limitations and Open Questions

Warning

The central ASI result (DSA/RSA) is reported only as radar charts with no numeric values, variance or ablation of the residual branch or censored learning. S-SIM is computed with a style extractor that Appendix B does not distinguish from AutoSIFT’s own Style Extractor, so the large S-SIM lead may partly reflect the shared embedding space.

  • The style taxonomy is fixed at four categories. Accent, speaking rate, pitch range and conversational intent are out of scope (§Appendix A), and prosody can only be preserved through the residual, not controlled.
  • Prototypes depend on category labels, so disentanglement quality is bounded by the coverage and consistency of annotations pooled from six heterogeneous corpora. Appendix A acknowledges this.
  • Text prompts are category descriptions encoded by MPNet, and the infiller’s hard gate needs to know which categories a prompt specifies. Robustness to free-form instructions is not evaluated.
  • MOS uses 10 listeners per language, and the reported differences between the top systems are small relative to their standard deviations. Intelligibility is reported only as PER, not WER/CER, and speaker similarity with a speaker-verification model is not reported.
  • Whether the baselines were trained on the same data is not stated, and code availability is not mentioned.

Wiki Connections

  • Flow Matching — The generator is a flow-matching DiT conditioned on a global style vector through AdaLN-Zero, trained with the standard velocity regression loss.
  • Disentanglement — Prototype alignment plus a reconstruction-guided VQ residual factorizes style into category and residual subspaces, with a PAL ablation showing the mechanism is necessary.
  • Instruction-Conditioned TTS — Text style descriptions encoded by MPNet control the specified categories, evaluated through Described Style Accuracy against prompt-based TTS baselines.
  • Emotion Synthesis — Emotion (seven classes) is one of the editable categories, with emotion DSA/RSA and classification accuracy reported.
  • Multilingual TTS — The system is trained and evaluated on English and Mandarin, and language is itself an editable style category (cross-lingual dubbing use case).
  • Subjective Evaluation — Naturalness MOS from 10 native English and 10 native Chinese listeners, with the rating instructions documented.
  • ParaMETA — Source of the Prototypical Alignment Loss and the speaker-independent evaluation protocol; also compared directly as both a representation learner and a TTS baseline.
  • ParaStyleTTS — Provides the frozen text encoder used as the content prior and is a text-prompted TTS baseline.
  • F5-TTS — Speech-prompted flow-matching baseline with the lowest PER in both languages but lower S-SIM than AutoSIFT.
  • IndexTTS — Speech-prompted zero-shot baseline, the strongest on S-SIM after AutoSIFT.
  • CosyVoice — Evaluated with both text and speech prompts; its speech-prompted MOS is closest to AutoSIFT.
  • FastSpeech 2 — Feed-Forward Transformer design reused to adapt content embeddings, and an uncontrolled TTS baseline.
  • Tacotron 2 — Uncontrolled English TTS baseline in the quality comparison.
  • Flow Matching for Generative Modeling — Training objective of the generative backbone.