arXiv · 2026 · Preprint

Pouw et al. (University of Amsterdam, Tilburg University) · → Paper · Demo: ? · Code: ?

A controlled behavioural and mechanistic study of few-shot in-context TTS in SpiritLM, showing that speaking rate (not pitch or intensity) drives both task accuracy and acoustic mimicry, and that ablating prefix-matching (induction) heads abolishes the model’s in-context TTS ability.

Problem

In-context learning (ICL) in text LLMs has been dissected both behaviourally (which demonstration properties matter) and mechanistically (induction heads), but speech LMs that interleave speech and text tokens have not been analysed in the same way. Prior speech ICL studies focused on ASR, where output is text, and they compared matched versus unmatched demonstrations without isolating individual acoustic or linguistic factors. TTS is a better probe because the output is speech: one can measure both whether the model infers the task (correct spoken content) and whether it copies acoustic properties of the demonstration speech.

Method

The studied model is SpiritLM base 7B (Spirit LM), a Llama-2-7B backbone continually pretrained on interleaved text and speech tokens, where speech is tokenized by HuBERT units (25 ms frames) and decoded to waveform by a HiFi-GAN vocoder. Output modality is fixed to speech with greedy decoding and at most 50 new tokens (§A.1). TTS is elicited purely by prompting, following the SpiritLM recipe: alternating [TEXT] sentence 'stop' [SPEECH] <speech> <speech:STOP> blocks for 0 to 5 demonstrations, followed by the target text and an open [SPEECH] slot (§3.3).

Prompts come from the PRIME-LM structural-priming corpus, whose Core condition pairs demonstrations and targets with no lexical, semantic, tense, or function-word overlap (§3.2). Demonstration speech is synthesized with two American English female voices (Kokoro-TTS and SpeechT5). Four factors are varied: number of demonstrations (1 to 5); syntactic congruence (active/passive targets with congruent or incongruent demonstrations); linguistic overlap (lexical overlap in nouns, main verb, function words, or all words; semantic similarity in nouns or verb); and Praat-based acoustic manipulations of the demonstrations (2x fast, 0.5x slow, flattened pitch, intensity reduced from about 67 dB to 30 dB). Each condition uses 100 targets x 10 demonstration sets x 4 syntax combinations, giving 4,000 items per linguistic condition and 20,000 per acoustic condition (§3.2).

Task performance is measured by transcribing outputs with whisper-large-v3-turbo and computing WER plus a new Content Word Recall (lemmatized content words recovered), introduced because SpiritLM often gets content words right while erring on function words and inflections (§3.4.1). Acoustic mimicry is measured as output speaking rate (Whisper word count over duration) and Praat pitch and intensity statistics, compared against outputs after unmanipulated Core demonstrations (§3.4.2).

For the mechanistic part, the authors compute modality-specific prefix-matching scores on the ICL prompts themselves (attention from a repeated token to the successor of its earlier occurrence, against a random-position baseline), validated against the standard random-token-sequence method (§5.1.1, §A.4). They define head groups with k = 50 (top speech-prefix, top text-prefix, unique-to-speech, unique-to-text, top non-prefix, random) and zero-ablate each group (§5.1.2).

Key Results

All results are reported as plots with 95% confidence intervals; the paper gives no numeric tables for WER or Content Word Recall, so no values are reproduced here.

  • One demonstration suffices. One-shot clearly beats zero-shot, two demonstrations help further, and performance then plateaus, even in the Core condition with no linguistic overlap (§4.1.1, Figure 2A).
  • Target syntax matters more than congruence. Passive targets get lower WER regardless of demonstration syntax, but the gap disappears under Content Word Recall, indicating the gain comes from function words such as “by” (§4.1.2, Figure 2B).
  • Lexical overlap helps; semantic similarity does not. All-word and function-word overlap reduce WER, and noun overlap additionally raises Content Word Recall, while semantic similarity gives no benefit, unlike reports from text ICL (§4.1.3, Figure 2C).
  • Speaking rate is the only acoustic factor that matters. Fast (2x) demonstrations substantially degrade ICL; slow demonstrations do not, and slightly help in the one-shot case. Flattened pitch and low intensity have little effect (§4.1.4, Figure 2D).
  • Mimicry is asymmetric. Output speaking rate follows the demonstrations (slow effect after one demonstration, fast effect only after three), pitch range is not copied, and intensity is only weakly affected and only with Kokoro-TTS demonstrations (§4.2, Figure 3).
  • Induction heads are causal. Among the top 50 heads per modality, 25 are unique to speech and 25 unique to text. Ablating top speech- or text-prefix heads causes a large performance drop, ablating only the unique speech-prefix heads is nearly as damaging, unique text-prefix ablation hurts less, random-head ablation hurts little, and ablating top non-prefix heads slightly improves performance (§5.2, Figure 4). The abstract describes the top-k ablation as completely removing ICL ability; the results text describes a “substantial drop”.
  • Backbone comparison. Llama-2-7B shows generally higher prefix-matching scores than SpiritLM on random sequences, which the authors tentatively read as speech pretraining weakening induction behaviour (§A.4, Figure 7).

The two TTS voices give highly similar patterns (§4, §A.3, Figure 6). Whisper size checks show smaller Whisper models degrade on fast speech while turbo and large do not, supporting the choice of the turbo evaluator (§A.2, Figure 5).

Novelty Assessment

The contribution is analytical rather than architectural: no model is proposed or trained. What is new is the transfer of two established text-LLM interpretability tools, controlled demonstration manipulation and induction-head ablation, to in-context TTS in a speech LM, together with a modality-specific prefix-matching score computed on well-formed interleaved prompts. Using a structural-priming corpus with zero linguistic overlap as the baseline is a clean design that separates acoustic from linguistic contributions. The scope is narrow: a single model (SpiritLM base), synthetic single-speaker demonstration voices, short template sentences, and an output length cap of 50 tokens, so the findings are a case study rather than a characterisation of speech LMs in general. The proposed link between speaking rate and token repetition available to induction heads is offered as a hypothesis, with only suggestive attention evidence (§6, §A.5).

Field Significance

Moderate — The paper gives the first mechanistic evidence that in-context TTS in an interleaved speech-text LM relies on prefix-matching attention heads, including heads specific to speech tokens, mirroring text-LLM findings. It also provides a controlled demonstration that temporal properties of the prompt, unlike pitch or loudness, shape both content accuracy and the prosody of generated speech, which is directly relevant to prompt design for LM-based speech generation.

Claims

  • supports: In-context learning of TTS in interleaved speech-text language models can emerge from very few demonstrations even when demonstrations share no lexical or semantic content with the target.

    Evidence: SpiritLM one-shot TTS clearly outperforms zero-shot on the PRIME-LM Core condition (no lexical, semantic, tense, or function-word overlap), improves further at two demonstrations, and then plateaus. (§4.1.1, Figure 2A)

  • refines: The prosodic attributes that a speech language model copies from in-context speech prompts are dominated by temporal properties; pitch range and loudness are transferred weakly or not at all.

    Evidence: Output speaking rate tracks 2x and 0.5x demonstrations, while flattened-pitch demonstrations leave output pitch range unchanged and 30 dB demonstrations shift output intensity only slightly and only for Kokoro-TTS voices. (§4.2, Figure 3)

  • complicates: Fast speech in audio prompts degrades content accuracy of in-context speech generation in token-based speech language models.

    Evidence: Demonstrations at 2x speaking rate substantially worsen WER and Content Word Recall relative to Core, while 0.5x demonstrations do not and slightly help with one demonstration; the authors attribute this to fewer speech tokens per word. (§4.1.4, Figure 2D, §A.5)

  • supports: In-context speech generation in speech language models depends causally on induction-style prefix-matching attention heads, as in text LLMs, with some heads specialised for speech tokens.

    Evidence: Zero-ablating the top 50 speech- or text-prefix-matching heads in SpiritLM sharply reduces ICL TTS performance, ablating the 25 speech-unique prefix heads is nearly as harmful, random-head ablation costs little, and ablating top non-prefix heads slightly improves performance. (§5.2, Figure 4)

  • complicates: Exact-match WER can understate content accuracy for LM-based TTS whose errors concentrate on function words and inflections, so content-word metrics give a different picture.

    Evidence: The passive-target advantage under WER disappears under lemmatized Content Word Recall, and noun overlap improves Content Word Recall but not WER. (§3.4.1, §4.1.2, §4.1.3, Figure 2B, Figure 2C)

Limitations and Open Questions

Warning

All findings come from one model (SpiritLM base 7B) prompted with synthetic speech from two female American English TTS voices on short template sentences, and results are reported only as plots; generality to other speech LMs, natural speech prompts, or longer outputs is untested.

  • Speech quality, naturalness, and speaker similarity of the generated speech are not evaluated; no human listening test is run, and the only qualitative listening is an informal check of pitch behaviour (§4.2).
  • Acoustic manipulations are coarse (2x, 0.5x, flattening, 30 dB) and were chosen to cover time, frequency, and amplitude rather than to be comprehensive (§3.2).
  • Speaking rate is computed from Whisper word counts, so it is coupled to ASR accuracy; the Whisper-size check (§A.2) mitigates but does not remove this dependence.
  • The abstract says top-k ablation completely removes ICL ability, while §5.2 reports a substantial drop; the exact magnitude is only visible in Figure 4.
  • Prefix matching is measured on single tokens, though speech tokens (25 ms frames) and text subwords differ in granularity; multi-token spans are left to future work (§6).
  • The hypothesis that slow speech and lexical overlap help because they create more prefix-matching opportunities, and the observation that Llama-2-7B has stronger induction scores than SpiritLM, are not tested causally (§6, §A.4).

Wiki Connections

  • Spoken Language Model — Analyses how an interleaved speech-text LLM consumes demonstration speech in context, at both the behavioural level and the attention-head level.
  • Self-Supervised Speech — The analysed model represents all speech as HuBERT discrete units, and the paper’s speaking-rate findings are explained through the number of such units per word.
  • Evaluation Metrics — Introduces lemmatized Content Word Recall alongside Whisper-based WER and shows the two metrics diverge for LM-based TTS output.
  • Spirit LM — The sole model under study; its interleaved prompting recipe is used to elicit in-context TTS.
  • Whisper — whisper-large-v3-turbo transcribes all outputs for WER, Content Word Recall, and speaking-rate measurement, with a model-size robustness check in the appendix.
  • HiFi-GAN — The vocoder inside SpiritLM that renders predicted speech tokens to waveform.
  • LLaMA — The Llama family backbone of SpiritLM; Llama-2-7B serves as the text-only reference for prefix-matching scores.
  • GPT-3 — Source of the few-shot in-context learning framing that the paper transfers to speech.
  • Speech Language Models Survey — Cited for the definition of speech LMs as next-token predictors over joint speech and text tokens.