arXiv · 2026 · Preprint

Jaehoon Kang et al. (Sungkyunkwan University) · → Paper · Demo: ? · Code: ?

Two training-free inference-time interventions for an existing natural-language-prompted autoregressive TTS model: interpolation along direction vectors between contrastive style-prompt embeddings for continuous per-utterance control of gender, pitch, and speed, and a KV-cache swap plus sliding-window attention mask that makes mid-utterance style transitions possible by countering a diagnosed “style self-referencing” effect.

Problem

Prompt-based TTS models let users describe a speaking style in natural language (“a male voice speaks quickly”), but they offer only coarse, categorical control: small prompt edits (“fast”, “slightly fast”) do not produce smooth, monotonic acoustic changes, and the style prompt is applied as a single global condition over the whole utterance. Use cases such as audiobook narration or conversational responses that shift tone need both continuous intensity control across utterances and time-varying style within one utterance. The authors also find that the obvious inference-time fix for the second goal, swapping the style embedding partway through generation, does not work, and set out to explain why.

Method

All experiments use the pretrained Parler-TTS-mini model, a decoder-only autoregressive Transformer that generates audio tokens and reads the encoded style description through cross-attention (§2, Figure 2a). No parameters are trained; both techniques act only at inference. The model size and audio tokenizer are not reported in the paper.

Inter-utterance style interpolation (§3). The authors write pairs of style prompts that differ only in one attribute word (“male”/“female”, “high”/“low pitch”, “quickly”/“slowly”). A UMAP projection of the text-encoder outputs at those attribute tokens shows tight, well-separated clusters per attribute value (Figure 2b), which motivates linear interpolation. For the attribute token positions only, a direction vector is computed as half the difference between target and source embeddings, and the source embedding is shifted by alpha times that vector: alpha = 0 reproduces the source style, alpha = 2 reaches the target, intermediate values interpolate, and values outside [0, 2] extrapolate. The modified embedding sequence is fed to the decoder’s cross-attention as usual. Appendix B shows that shifting all token positions (with a separate factor beta for non-attribute tokens) gives comparable results, so the simpler attribute-only variant is kept.

Diagnosing style self-referencing (§4.1). Replacing the style embedding at a transition point t* leaves the output in the original style. Inspecting decoder cross-attention between audio positions and style tokens shows that attention patterns change actively only during the first part of generation and then stay nearly constant, concentrating on relatively uninformative tokens such as “with” and the end-of-sequence token (Figure 4; quantified as per-position attention variance in Appendix C). The authors read this as a “set-and-maintain” strategy: style is established in the early generated audio tokens, and later generation conditions mainly on those tokens through self-attention rather than on the prompt, so the decoder resists new style information.

Intra-utterance style transition (§4.2). Two decoder instances run in parallel. Decoder A generates in the source style until t*. Decoder B, conditioned on the interpolated target-style embedding, generates only the first n tokens to build a target-style KV-cache, where n is the number of transcript tokens plus a buffer k of early audio tokens. At t*, Decoder A’s cached keys and values for positions 1..n are overwritten with Decoder B’s, and its cross-attention style input is switched to the target embedding. Because Decoder A could still attend to the source-style tokens generated between n and t*, a sliding-window self-attention mask restricts each later query to the swapped initial n positions plus the most recent w tokens. The window size w trades transition strength against local coherence. The extra cost is generating n tokens with a second decoder.

Figure 5: Intra-utterance style transition. To convert speaking style mid-generation, we combine multiple techniques to avoid style self-referencing: (a) prepare an initial KV-cache derived from the target style prompt, and (b) swap the original KV-cache and style embeddings, together with (c) sliding-window attention masking.

Key Results

Inter-utterance interpolation (400 LibriTTS-R test sentences, Parler-TTS-mini; Table 1 at alpha = 2.0). Gender conversion success, measured with a pretrained gender classifier, reaches 99.0% (Female to Male) and 100.0% (Male to Female). Average F0 (measured with PENN) moves by -36.1 Hz (High to Low) and +35.8 Hz (Low to High), with speaker similarity of 0.76 and 0.78. Speaking rate moves by -1.4 and +1.6 syllables per second, with speaker similarity of 0.84. Figure 3 shows roughly monotonic changes with alpha for pitch and speed (for example, -0.7 SPS at alpha = 1.0 and -1.4 SPS at alpha = 2.0 for Quick to Slow). Note that the speaker similarity is computed between source and converted audio, so lower values partly reflect the intended attribute change rather than only identity loss.

In a listening test with 15 participants (Table 2), perceived style change on a -2 to +2 scale grows with alpha for pitch and speed, reaching 1.62 to 2.00 at alpha = 2.0 across all directions, while MOS stays between 3.99 (Quick to Slow) and 4.40 (Male to Female). Gender behaves more like a switch than a dial: Female to Male scores only 0.09 at alpha = 0.5 and 0.12 at alpha = 1.0 before jumping to 1.74 at alpha = 2.0.

Intra-utterance transition (400 LibriTTS-R test samples of 50 to 70 text tokens, k = 48; Table 3). Smaller windows give stronger transitions: Low to High pitch changes by +27.4 Hz with w = 256 versus +5.5 Hz with full attention, and Quick to Slow speed changes by -2.29 SPS versus -1.34 SPS. Listener-reported transition detection is 88.0% to 96.2% at w = 256 versus 55.6% to 84.6% with full attention. Speaker similarity between the first and last 3-second segments falls to 0.81 to 0.86 at w = 256 versus 0.90 to 0.91 with full attention. Smoothness ratings (1 to 5, human listeners) range from 3.48 to 4.48 across all configurations. Trends across window sizes are not strictly monotonic (for example, the 384 window has a lower detection rate than 512 for pitch).

Ablations. Swapping only the style embedding without the KV-cache swap yields minimal change (-4.3 Hz and -2.40 Hz pitch differences, the latter in the wrong direction for Low to High; Table 4). Swapping only the transcript-token region (k = 0) produces pitch changes in the wrong direction (+6.8 to +16.2 Hz for High to Low), while k of 32 or more produces the intended -5.3 to -12.4 Hz (Figure 6).

No baseline from another controllable-TTS method is compared; all comparisons are internal (alpha values, window sizes, ablated variants). The listener count and rater details for the intra-utterance test are not reported separately.

Novelty Assessment

The inter-utterance technique is a direct application of embedding arithmetic (direction vectors between contrastive prompts) to a prompt-conditioned TTS text encoder; it is useful and simple but conceptually incremental. The more original contributions are the diagnosis of style self-referencing, supported by cross-attention visualization and the ablations in Table 4 and Figure 6, and the intra-utterance procedure that combines a second decoder’s prefix KV-cache, a mid-generation cache overwrite, and attention-sink-style sliding-window masking (adapted from Longformer and streaming-LLM attention work). The novelty is an inference procedure, not a new trained architecture, and it is demonstrated on a single model.

Field Significance

moderate — The paper shows that a natural-language-prompted autoregressive TTS model already contains continuous, interpolatable attribute directions in its prompt-encoder space, and that the model’s reliance on its own early audio tokens, rather than the prompt, is what blocks mid-utterance style changes. The KV-cache swap with windowed attention provides a training-free route to time-varying style, but evidence comes from one model (Parler-TTS-mini) and three attributes, with no external baseline.

Claims

  • supports: Contrastive natural-language style prompts that differ in a single attribute word occupy separable regions of a prompt-conditioned TTS model’s text-encoder space, so linear interpolation and extrapolation along the difference vector yields graded, roughly monotonic control of continuous attributes such as pitch and speaking rate without retraining.

    Evidence: Shifting only attribute-token embeddings by alpha times half the target-minus-source difference in Parler-TTS-mini changes F0 by up to -36.1/+35.8 Hz and speaking rate by up to -1.4/+1.6 SPS at alpha = 2.0, with listener-rated style change increasing with alpha and MOS of 3.99 to 4.40. (§3.3, Table 1, Table 2, Figure 3)

  • complicates: Embedding interpolation does not yield equally graded control for every attribute; categorical attributes such as perceived gender can behave as a threshold switch rather than a continuum.

    Evidence: For Female to Male, listener-rated style change is 0.09 at alpha = 0.5 and 0.12 at alpha = 1.0 but 1.74 at alpha = 2.0, whereas pitch and speed already show changes of 0.68 to 1.45 at alpha = 1.0. (§3.3, Table 2)

  • supports: In autoregressive TTS decoders conditioned on a style prompt through cross-attention, the style is established in the first generated audio tokens and then maintained through self-attention to those tokens, so changing the style condition mid-generation has little effect.

    Evidence: Cross-attention weights over style tokens fluctuate early and then stay nearly constant (attention variance stabilizes); replacing the style embedding at the transition point without touching the KV-cache yields only -4.3 Hz and -2.40 Hz pitch differences, the latter in the wrong direction. (§4.1, §4.4, Figure 4, Table 4, Appendix C)

  • supports: Mid-utterance style transitions in an autoregressive TTS model can be induced at inference time by overwriting the early KV-cache with a cache generated under the target style and masking attention to intermediate source-style tokens, with the cache region needing to include early audio tokens rather than only the text prefix.

    Evidence: With k = 48 extra audio positions swapped and a sliding window, listeners detected the intended transition in 55.6% to 96.2% of samples with smoothness 3.48 to 4.48; with k = 0 (text region only) the High to Low pitch change goes the wrong way (+6.8 to +16.2 Hz), while k of 32 or more gives -5.3 to -12.4 Hz. (§4.3, §4.4, Table 3, Figure 6)

  • complicates: Limiting the self-attention window to strengthen an inference-time style transition costs speaker consistency across the utterance, giving a trade-off between transition strength and identity preservation.

    Evidence: A 256-token window gives +27.4 Hz Low to High pitch change and 88.0% to 96.2% detection but speaker similarity of 0.81 to 0.86 between the first and last 3 s, versus +5.5 Hz, 55.6% to 84.6% detection, and 0.90 to 0.91 similarity with full attention. (§4.3, Table 3, §Limitations)

Limitations and Open Questions

Single-model, single-dataset evidence

Every experiment uses Parler-TTS-mini on LibriTTS-R text with templated prompts, and no other controllable-TTS method is compared. Whether the self-referencing diagnosis and the KV-cache intervention transfer to other prompt-conditioned autoregressive TTS models, or to non-autoregressive and diffusion models without a KV-cache, is untested (the authors list the latter as open).

  • Only three attributes (gender, pitch, speed) are studied; emotion and intonation are left for future work.
  • The transition-strength versus speaker-similarity trade-off depends on a window-size hyperparameter, and results across window sizes are not monotonic, so a practical setting is not clearly identified.
  • Intra-utterance transition requires a second decoder pass; the authors call the overhead marginal but do not measure latency.
  • The listening tests are small (15 participants for the inter-utterance test; the intra-utterance participant count is not stated) and confidence intervals are not reported.
  • The “success” rate for pitch and speed in Table 1 is not defined in the text, unlike the gender classifier success rate.
  • Speaker similarity in Table 1 compares source and converted audio, which conflates intended attribute change with identity drift.

Wiki Connections

  • Instruction-Conditioned TTS — The paper adds continuous interpolation and mid-utterance transitions to a model controlled by natural-language style descriptions, without retraining.
  • Prosody Control — Embedding interpolation gives graded control of F0 and speaking rate, and the KV-cache swap with windowed attention changes pitch or rate partway through an utterance.
  • Autoregressive Codec TTS — The style self-referencing analysis and the KV-cache intervention target the self-attention behaviour of an autoregressive audio-token decoder.
  • Subjective Evaluation — Human listeners rate style change, MOS, transition detection, and transition smoothness for both techniques.
  • Parler-TTS (Natural Language Guidance) — The method is built entirely on the pretrained Parler-TTS-mini model introduced by this work.
  • EmoSteer-TTS — The closest training-free prior work (activation steering for emotion), which the authors contrast as not addressing smooth transitions over time.
  • WeSCon — Word-level emotional control through multi-stage inference, cited as a fine-grained intra-utterance control alternative.
  • ControlSpeech — Cited as an example of prompt-conditioned TTS whose small prompt edits do not give smooth, predictable acoustic changes.
  • Llasa — Cited as support for the importance of local coherence in autoregressive speech generation, which motivates keeping a recent-token window.
  • Spark-TTS — Source of the syllables-per-second speaking-rate measure used throughout the evaluation.