Speech Generation Research Wiki

ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesis

9 min read

arXiv · 2026 · Preprint

Haitao Li et al. (Ant Group) · → Paper · Demo: ✓ · Code: ?

Introduces ReStyle-TTS, which enables continuous and reference-relative style control (pitch, energy, and emotion) in zero-shot TTS by first decoupling the model’s dependence on reference style from text fidelity, then applying disentangled style-attribute LoRAs while explicitly preserving speaker timbre.

Problem

Zero-shot TTS models clone a speaker’s timbre from a short reference clip, but they also strongly inherit that reference’s speaking style (prosody, emotion), so producing speech in a desired style requires finding a reference that already has that style, which is impractical when only limited or mismatched references are available (e.g. wanting angry output but only having a happy reference). Existing controllable TTS methods (style-audio prompts as in IndexTTS2 and Vevo, or language-based style prompts as in ControlSpeech, EmoVoice, and CosyVoice) address parts of this problem, but all rely on absolute style targets and none support continuous, reference-relative adjustment, where a user incrementally nudges an attribute up or down relative to whatever the reference already sounds like, which the paper argues is a more intuitive interaction model.

Method

ReStyle-TTS fine-tunes F5-TTS with three components. First, Decoupled Classifier-Free Guidance (DCFG) separates the single guidance term of standard CFG (which entangles text and reference-audio influence) into two independent terms: a text-guidance strength λ_t and a reference-guidance strength λ_a, computed from three predictions (fully unconditional, text-only, and text-plus-reference). Standard CFG is recovered as a special case (λ_t = λ_cfg, λ_a = 1 + λ_cfg); lowering λ_a relative to this default reduces reliance on the reference’s style while λ_t continues to enforce text fidelity independently, which the paper shows algebraically is impossible with entangled CFG.

The overall framework of ReStyle-TTS. The proposed method consists of three logically coordinated components: (1) Decoupled Classifier-Free Guidance (DCFG) reduces the model's dependency on the reference style while maintaining text fidelity; (2) Orthogonal LoRA Fusion (OLoRA) reduces interference among multiple style-specific LoRAs by projecting each LoRA onto the orthogonal complement of the subspace spanned by the others; and (3) Timbre Consistency Optimization (TCO) reinforces speaker identity preservation through a similarity-based reward mechanism.

Second, with reference-style dependence relaxed, style-specific LoRA adapters (rank 32) are fine-tuned on attribute-annotated subsets (high/low pitch, high/low energy, and seven emotions) of a dataset built from LibriTTS plus several emotion-focused corpora. To combine multiple style LoRAs without mutual interference, Orthogonal LoRA Fusion (OLoRA) projects each LoRA’s low-rank update onto the orthogonal complement of the subspace jointly spanned by all the others (via least-squares/SVD projection against the concatenated update vectors), then fuses the decorrelated updates with per-attribute weights that provide continuous intensity control; because the projection is computed jointly rather than sequentially, the fusion is order-independent.

Third, because relaxing reference guidance can let speaker timbre drift, Timbre Consistency Optimization (TCO) adds a lightweight reward-weighting term to the standard flow-matching loss: generated speech is scored for speaker similarity against the reference, an exponential-moving-average baseline is subtracted to form an advantage signal, and the flow-matching loss is reweighted by a bounded function of that advantage (following an advantage-weighted regression recipe) so that samples with better speaker-similarity get relatively stronger gradient emphasis, without backpropagating through generation or the reward computation.

Key Results

Sweeping individual style LoRAs (pitch, energy, six emotions) shows smooth, largely monotonic control curves with WER and speaker similarity remaining nearly constant, and negative scaling of a “high-attribute” LoRA naturally produces the opposite effect. Jointly sweeping two or three LoRAs over 2D/3D grids shows each attribute varies smoothly along its own axis with limited cross-attribute interference, and WER/speaker-similarity remain stable across the joint control space. Regression of generated energy against reference energy across LoRA scales yields slopes between 0.77 and 1.22 with near-zero intercepts, evidence that the LoRA effect is roughly proportional to (relative to) each sample’s own starting style rather than collapsing all samples toward one absolute target. On contradictory-style generation (VccmDataset test set, where the reference’s emotion or prosody deliberately mismatches the target), ReStyle-TTS achieves substantially higher target-style accuracy than CosyVoice, EmoVoice, and IndexTTS2 across most reference-target emotion pairs (e.g. neutral-reference-to-happy-target accuracy of 100.0% vs. 82.9/85.7/90.4% for the three baselines) and for pitch/energy reversal (e.g. low-to-high pitch accuracy of 90.2% vs. 74.9/72.4%). Ablation shows standard CFG cannot simultaneously achieve controllability and text fidelity: high guidance yields only 2.1% relative attribute change (though good WER/speaker-similarity), low guidance produces unusable WER (>1.0), while DCFG achieves 51.2% relative attribute change at comparable WER (2.31%) and speaker similarity (0.79); removing TCO drops speaker similarity from 0.79 to 0.71 while controllability and WER stay similar.

Novelty Assessment

The paper identifies a real, previously unaddressed gap, per its own comparison table, no prior controllable zero-shot TTS method supports continuous or reference-relative style control, only absolute targets via style audio or discrete/text prompts, and each of its three components is targeted at a specific sub-problem with a corresponding ablation showing it matters (DCFG for the controllability/fidelity trade-off, OLoRA for multi-attribute interference, TCO for timbre drift). The application of orthogonal LoRA composition to TTS is adapted from image-generation LoRA-fusion techniques (the paper is explicit about this borrowing and about why naive LoRA composition from that literature doesn’t transfer directly, since TTS style is entangled with reference audio in a way image style is not). The relative-control claim is unusually well substantiated: rather than just reporting average improvements, the per-sample regression analysis (slopes near 1, intercepts near 0) is a direct, falsifiable test of the “relative not absolute” claim, which most controllable-TTS papers do not attempt.

Field Significance

Tip

High, this is the first controllable zero-shot TTS method to combine continuous, multi-attribute, and reference-relative style control while preserving zero-shot voice cloning, addressing a specific practical failure mode (mismatched or limited reference audio) that prior style-audio- and text-prompt-based methods do not solve. The decoupled-guidance idea (DCFG) is a reusable technique in its own right, likely applicable beyond this paper’s specific LoRA-based style system to any zero-shot generation setting where a single CFG term conflates two distinct conditioning signals.

Claims

  • supports: Decoupling classifier-free guidance into separate text-guidance and reference-audio-guidance terms allows reducing a zero-shot TTS model’s dependence on reference style without sacrificing text intelligibility, which coupled (standard) CFG cannot achieve simultaneously.

    Evidence: Standard CFG at high guidance strength yields only 2.1% relative attribute controllability while preserving WER/speaker-similarity, and at low guidance strength produces unusable WER (>1.0); DCFG achieves 51.2% relative attribute controllability while maintaining comparable WER (2.31%) and speaker similarity (0.79). (§4.6, Table 4)

  • supports: Style-specific LoRA adapters, when jointly orthogonalized against each other’s parameter subspaces before fusion, can control multiple speech attributes simultaneously without one attribute’s control interfering with another’s.

    Evidence: Jointly sweeping two or three style LoRAs (pitch, energy, emotion) over a 2D/3D grid shows each attribute’s controlled surface varies smoothly and largely independently along its own axis, while WER and speaker similarity remain stable across the entire multi-attribute control space. (§4.3, Figures 3-4)

  • supports: Style control that operates relative to the reference audio’s own style, rather than toward a fixed absolute target, preserves the relative ordering of style intensity across different reference samples.

    Evidence: Regression of generated energy against reference energy across multiple LoRA scaling strengths produces slopes between 0.77 and 1.22 with near-zero intercepts, indicating a roughly proportional transformation rather than collapse toward a single absolute target value. (§4.4, Figure 5)

  • complicates: Reducing a zero-shot TTS model’s dependence on reference audio to enable style control introduces a competing pressure on speaker-timbre consistency that must be explicitly counteracted.

    Evidence: Removing the Timbre Consistency Optimization module while keeping DCFG and Style-LoRAs active drops speaker similarity from 0.79 to 0.71, despite comparable style controllability and WER. (§4.6, Table 4)

Limitations and Open Questions

  • The authors state the primary limitation is scalability to new attributes: adding control for a new style dimension requires collecting a corresponding annotated dataset and performing additional LoRA fine-tuning, rather than being learned zero-shot from a description.
  • Contradictory-style generation is evaluated with automatic accuracy metrics (emotion classifier / pitch-energy direction accuracy) as the primary quantitative evidence; a subjective MOS-SA evaluation is reported only in an appendix rather than the main results.
  • The ablation study varies λ_a and TCO but the paper’s stated hyperparameters (λ_t = 2, λ_a = 0.5, λ = 0.2, β = 5.0) are not accompanied by a broader sensitivity analysis in the main text (deferred to an appendix on OLoRA and λ_a selection).
  • The method depends on the underlying F5-TTS backbone’s flow-matching representation; transferability of DCFG and OLoRA to other zero-shot TTS architectures (autoregressive codec-LM systems, for instance) is not empirically tested in this paper.

Wiki Connections

  • Flow Matching — fine-tunes F5-TTS’s flow-matching framework with LoRA adapters and a decoupled classifier-free guidance formulation, rather than proposing a new flow-matching objective.
  • Zero-Shot TTS — explicitly targets preserving zero-shot speaker-timbre cloning while adding style controllability, in contrast to controllable-TTS methods that assume a fixed or predefined speaker space.
  • Emotion Synthesis — provides continuous, per-emotion intensity control via style-specific LoRAs, including reliable override of a reference’s own emotion in contradictory-style generation.
  • Prosody Control — introduces explicit, continuously adjustable LoRA-based control over pitch and energy, independent of content and largely independent of speaker timbre.
  • Disentanglement — Orthogonal LoRA Fusion explicitly projects each style attribute’s parameter update onto the orthogonal complement of the others’ subspaces before fusion, with ablation and multi-attribute sweep evidence that this reduces cross-attribute interference.
  • Subjective Evaluation — reports MOS-SA (Mean Opinion Score-Style Accuracy) human evaluations of synthesized style accuracy alongside objective metrics.
  • F5-TTS — the pretrained zero-shot TTS backbone that ReStyle-TTS fine-tunes with LoRA adapters rather than training from scratch.
  • Seed-TTS — the Seed-TTS test set is used as the primary evaluation benchmark for single- and multi-attribute style control experiments.
  • CosyVoice 2 — a text-prompt-based controllable TTS baseline compared against in the contradictory-style generation experiments.
  • EmoVoice — a text-prompt-based controllable TTS baseline compared against in the contradictory-style generation experiments.
  • IndexTTS2 — a style-audio-prompt-based controllable TTS baseline compared against in the emotion-transfer contradictory-style matrix.
  • Vevo — discussed as a prior style-audio-prompt-based approach to controllable zero-shot voice cloning that, like IndexTTS2, still requires selecting a suitable style reference clip.
  • Spark-TTS — discussed in related work as a representative autoregressive zero-shot TTS architecture using discrete decoupled speech tokens.

Backlinks

Graph View