arXiv · 2026 · Preprint

Yuxuan Hu et al. (Microsoft) · → Paper · Demo: ? · Code: ?

PRIME-Speech turns a frozen speech-to-text LLM (Phi-4-MM-7B) into a speech-to-speech model by training only a separate audio branch, a causal post-decoder that reads the backbone’s middle-layer hidden states at every decoding step and emits codec tokens alongside the text. Because the backbone never changes, its speech-to-text scores stay where they were, and the added speech output transcribes with low WER.

Problem

Strong speech-to-text (S2T) LLMs already perceive speech and reason in text, but giving them spoken output creates a tension. Training the backbone end to end on speech tokens (as in unified token-interleaving models) forces one decoder to balance text and audio objectives, which can erode the speech understanding, reasoning, and instruction following that made the backbone useful. Keeping the backbone intact and voicing its finished text with a downstream “talker” (Thinker-Talker designs) instead reintroduces a serial text-to-speech bottleneck: the speech branch follows finalized text, text chunks, or hand-off states rather than the backbone’s evolving computation. Multi-token prediction (MTP) can reduce codec decoding steps, but applied only to a light talker it barely shortens end-to-end latency, and applied to the backbone it risks the S2T degradation the authors want to avoid. The paper frames S2S adaptation as a frozen-backbone conversion problem: add speech output without updating any part of the S2T pathway.

Method

The backbone is Phi-4-MM-7B (pre-trained on 2M hours of speech and 5T text tokens), with its speech encoder, projection adapter, transformer stack, text LM head, and text decoding path all frozen. The joint output factorizes as the frozen backbone’s text distribution times a trainable audio-branch distribution conditioned on the text prefix and a stream of intermediate backbone states. That interface is a fixed middle layer at roughly two-thirds of the backbone’s depth, chosen by a layer-wise centered kernel alignment (CKA) analysis that the authors say identifies it as carrying the most paralinguistic information while staying semantically grounded (§II-B).

Speech targets are single-codebook semantic tokens from the CosyVoice 2 tokenizer at 25 Hz. The paired CosyVoice 2 token-to-waveform decoder is used unchanged and is not trained here.

The audio post-decoder is a causal transformer of 10 decoder layers (about 2B parameters) whose hidden size and attention configuration match the backbone’s. It runs in the same streaming update loop as the text head (timestamp-level synchronization): at update s the backbone’s middle-layer state fans out in parallel to the text head, which emits one text token, and to the post-decoder, which emits a block of codec tokens. The post-decoder sees only history committed before s; the text token produced concurrently becomes available at the next update. Its input is an unweighted sum (all weights 1.0, chosen in held-out ablations) of three signals: the synchronized backbone state, the embedding of the previous text token (lexical anchoring), and the mean embedding of the previous audio block (acoustic continuity). Text and audio keep separate token representations and KV caches, so this is not a single interleaved token stream, and no forced alignment, completed responses, or fixed text chunks are needed. Training uses teacher forcing along the same causal graph.

Model architecture. A frozen speech-text backbone remains responsible for speech perception and text reasoning. PRIME-Speech attaches a trainable audio post-decoder to intermediate backbone states, so each streaming update runs text and audio branches in parallel from the same hidden-state trajectory. MTP lets the audio branch commit multiple codec tokens per update.

Multi-token prediction. Once the single-token audio branch is aligned, MTP heads (a multi-head MLP of about 100M parameters, hidden size 2048) let each post-decoder state predict k future codec tokens, trained with a weighted sum of future-token cross-entropy losses. This lowers the effective autoregressive codec rate from 25 Hz to 25/k Hz without touching the reasoning path.

Multi-turn cache policy. Text KV states accumulate across turns, while the audio KV cache and audio position indices reset at each assistant turn. Because multi-turn S2S data is scarce, training packs unrelated single-turn examples into pseudo-dialogues with the same policy (global text positions, audio positions reset per segment), so no multi-turn S2S supervision is used.

Data and curriculum. The mixture totals about 100k weighted hours: LibriHeavy (46k h) for dense text-audio alignment, about 10k h of synthesized x-to-English S2ST targets plus CVSS/CoVoST-2 X2EN (seven source languages), and VoiceAssistant-400K, TriviaQA, and Natural Questions for answer-like outputs. All target speech is synthesized with Microsoft Azure TTS (several hundred speakers) and tokenized with CosyVoice 2. Stage 1 trains the post-decoder at 25 Hz with next-token prediction for one epoch (AdamW, learning rate 1e-4, linear decay). Stage 2 enables MTP for 20k steps, down-weighting pure alignment data.

Key Results

Evaluation covers S2ST (FLEURS and CoVoST-2 X2EN, BLEU / ASR-BLEU), spoken QA (UltraEval-Audio LLaMA-QA, TriviaQA, WebQ), BigBench-Audio, VocalBench, and an in-house multi-turn set of 28 human-validated conversations (179 turns). S2S outputs are transcribed with Whisper Large-V3; WER is measured against the system’s own text response, so it measures rendering consistency rather than accuracy against a reference. Unless noted, PRIME-Speech uses MTP k = 4. Baselines are Qwen3-Omni-30B, GPT-4o, GLM-4-Voice-9B, Kimi-Audio-7B, Step-Audio-2-Mini-7B, VocalNet-8B (evaluated in its streaming mode), Qwen2.5-Omni-7B, and the frozen Backbone-LLM-7B in S2T mode only (Table II).

Preservation (Table II). PRIME-Speech’s S2T scores stay within about one point of the frozen backbone: FLEURS 31.40 vs. 31.41, CoVoST 41.29 vs. 40.65, LLaMA-QA 79.00 vs. 78.67, TriviaQA 46.98 vs. 47.07, WebQ 42.04 vs. 42.18, multi-turn 80.45 vs. 79.33, BigBench-Audio 66.2 vs. 66.5.

Rendering WER (Table II). PRIME-Speech has the lowest WER of all systems reporting it: UltraEval-Audio 3.33 (vs. 7.68 VocalNet, 8.56 Step-Audio-2-Mini, 14.85 Kimi-Audio, 14.92 Qwen3-Omni, 21.5 Qwen2.5-Omni) and multi-turn 3.34 (vs. 4.23 Qwen2.5-Omni, 6.15 Step-Audio-2-Mini, 7.83 GLM-4-Voice, 8.52 VocalNet, 10.9 Kimi-Audio, 11.28 Qwen3-Omni). GPT-4o reports no WER.

S2S task scores (Table II). PRIME-Speech has the highest S2S score among systems with S2S results on FLEURS ASR-BLEU (33.24 vs. 32.72 Qwen3-Omni), CoVoST ASR-BLEU (40.98 vs. 37.62 Qwen3-Omni), LLaMA-QA (74.42 vs. 71.33 Qwen3-Omni), and multi-turn accuracy (79.33 vs. 70.95 GLM-4-Voice, 70.39 Qwen3-Omni). It trails the much larger Qwen3-Omni-30B on TriviaQA (44.54 vs. 57.52), WebQ (40.18 vs. 52.51), and BigBench-Audio S2S (63.4 vs. 72.0; GPT-4o 67.2). Among the 7B to 9B open systems its BigBench-Audio S2S score is the highest (next: Qwen2.5-Omni 53.6). On VocalBench it is mid-pack: knowledge 68.9 (Qwen2.5-Omni 69.5, Qwen3-Omni 89.4, GPT-4o 91.3), UTMOS fluency 4.36 (Step-Audio-2-Mini 4.52, VocalNet 4.45, Qwen3-Omni 4.38 higher), single-round 4.29 (Qwen3-Omni 4.96, GPT-4o 4.67 higher).

S2T to S2S gap. Computed as S2T minus S2S from Table II, PRIME-Speech’s multi-turn gap is 1.12 points, tied with Step-Audio-2-Mini (70.39 to 69.27) and smaller than the other systems. On BigBench-Audio its gap (66.2 to 63.4, 2.8 points) is larger than Qwen2.5-Omni (0.6), VocalNet (1.0), and GLM-4-Voice (2.1), though at a much higher absolute level. The authors’ statement that VocalNet has a larger modality gap holds for the multi-turn set and LLaMA-QA, but not for BigBench-Audio.

Frozen vs. adapted backbone (Table III). Two LoRA variants that update the backbone on the same data (LoRA + ESI from Towards Efficient Speech-Text Jointly Decoding, and LoRA + Post LM from SLM-S2ST) drop BigBench-Audio S2T to 53.75 and 52.96 (PRIME-Speech: 66.3 to 66.4) and LLaMA-QA S2T to 70.00 and 70.33 (PRIME-Speech: 79.00). Their rendering WER is mixed relative to the PRIME-Speech stage-1 model: lower or comparable on UltraEval-Audio (3.25 and 5.13 vs. 6.12) but higher on FLEURS (2.07 and 2.62 vs. 1.51).

MTP (Tables III, V). Going from the stage-1 model to stage 2 with k = 1 already improves rendering (UltraEval-Audio WER 6.12 to 5.66, BigBench-Audio S2S 59.1 to 63.86). Raising k to 2 and 4 keeps S2S task scores broadly stable (BigBench-Audio S2S 64.16 and 63.38) with UltraEval-Audio WER 3.01 and 3.33. FLEURS WER rises at k = 4 (2.19 vs. 1.45 to 1.52 for smaller horizons). On one H100 (Table V), k = 1 to k = 4 cuts time to first audio (TTFA) from 1.07 s to 0.39 s, RTF from 1.088 to 0.296, and raises throughput from 30.62 to 123.76 tok/s. VocalNet-8B is faster in raw terms (RTF 0.243, up to 225.93 tok/s, TTFA 0.40 s at k = 5, time to first text 38 ms vs. 58 ms), while Qwen2.5-Omni-7B has TTFA 1.01 s and RTF 1.093.

Cache policy (Table IV). With text accumulation plus audio reset, multi-turn S2S accuracy stays between 71.43 and 92.86 and WER below 4% across turn-position buckets. Without audio reset, turn 3 drops to 39.29 accuracy at 65.57 WER, and the 5+ turn bucket collapses to 0.00 accuracy at 143.27 WER.

Novelty Assessment

The central idea, keeping the LLM frozen and driving a separately trained speech decoder from its hidden states, has precedent in Freeze-Omni, which the paper cites but does not compare against. What PRIME-Speech adds is the specific coupling: a middle-layer (rather than final-layer) state selected by CKA, a lockstep per-update loop in which a large causal post-decoder emits codec blocks concurrently with each text token, and a mixed hidden-state, text, and audio-history input. The turn-level audio-cache reset with packed single-turn training is a simple but useful finding, backed by a clean ablation. Applying MTP to the audio branch follows VocalNet and related work; the new point is using it to recover latency lost to a heavy 2B post-decoder. The contribution is architectural at the interface level, built from existing parts (Phi-4-MM, the CosyVoice 2 tokenizer and decoder, MTP heads).

Field Significance

Moderate. The paper provides a controlled demonstration that speech output can be added to a strong S2T LLM with its S2T scores essentially unchanged, and its LoRA comparison quantifies what adapting the backbone costs in reasoning (over 12 points on BigBench-Audio S2T). It also gives a concrete, ablated recipe for multi-turn stability without multi-turn S2S data. Its evidence is limited to English output trained on Azure-synthesized targets and to automatic metrics, so it speaks to content fidelity and latency rather than speech quality or expressiveness.

Claims

  • supports: Keeping a speech-to-text LLM backbone frozen and training only a separate speech-output branch preserves its speech understanding and reasoning, whereas adapting the backbone for speech output degrades them.

    Evidence: PRIME-Speech’s S2T scores match the frozen Phi-4-MM backbone within about one point across translation, spoken QA, and BigBench-Audio, while LoRA variants trained on the same data drop BigBench-Audio S2T from 66.3 to 53.75 and 52.96 and LLaMA-QA S2T from 79.00 to about 70. (§IV-B, §IV-C, Table II, Table III)

  • supports: A speech decoder driven by an LLM’s intermediate hidden states in step-by-step synchronization with text decoding can render the LLM’s responses with low word error, without forced alignment or waiting for completed text.

    Evidence: With its post-decoder fed middle-layer states and the previous text and audio tokens at every update, PRIME-Speech reaches rendering WER 3.33 on UltraEval-Audio and 3.34 on the multi-turn set, the lowest among the compared systems, while its multi-turn S2S accuracy (79.33) stays close to its S2T accuracy (80.45). (§II-C, §IV-B, Table II)

  • refines: In multi-turn spoken dialogue, the text-side context and the acoustic-side context of a speech LM need different memory policies: dialogue semantics should persist across turns while acoustic generation state should be reset per response.

    Evidence: Accumulating text KV cache while resetting audio KV cache and positions per turn keeps WER below 4% in every turn bucket; carrying audio cache across turns raises WER to 65.57 at turn 3 and 143.27 at turn 5+, with S2S accuracy falling to 0.00. (§IV-C, Table IV)

  • supports: Predicting several codec tokens per autoregressive step can cut first-audio latency and real-time factor several-fold with little loss in spoken task quality.

    Evidence: Raising the MTP horizon from 1 to 4 lowers the effective codec rate from 25 Hz to 6.25 Hz, TTFA from 1.07 s to 0.39 s, and RTF from 1.088 to 0.296, with BigBench-Audio S2S 63.86 vs. 63.38 and multi-turn S2S unchanged at 79.33. (§IV-D, Table III, Table V)

  • complicates: Larger multi-token horizons do not reduce rendering errors uniformly across tasks, so the latency gain can come with task-specific intelligibility costs.

    Evidence: At k = 4 FLEURS rendering WER rises to 2.19 from 1.45 (k = 1) and 1.52 (k = 2), and multi-turn WER is 3.34 vs. 2.07 at k = 2, even though UltraEval-Audio WER is lower than at k = 1. (§IV-C, Table III)

Limitations and Open Questions

Synthetic targets and automatic-only evaluation

All target speech is synthesized by Microsoft Azure TTS, output is English only, and no human listening test is run. The authors explicitly decline to claim anything about expressive prosody or speaker consistency; speech quality is represented only by VocalBench’s UTMOS-based fluency score (4.36, below three of the baselines).

  • WER is computed against each system’s own text response, so it measures self-consistency of rendering, not correctness; task scores carry the correctness signal.
  • The heavy 2B post-decoder makes the single-token configuration slower than real time (RTF 1.088 at k = 1); real-time operation depends on MTP, and lightweight-talker designs such as VocalNet remain faster in raw throughput.
  • The comparison with public systems mixes different backbones, data, and sizes (Qwen3-Omni is 30B), which the authors acknowledge; the controlled evidence is the Backbone-LLM and LoRA comparisons.
  • The multi-turn set is small and in-house (28 conversations, 179 turns), and the S2ST synthetic data (about 10k hours) is also in-house.
  • Although the backbone is frozen, Table II lists slightly different S2T numbers for PRIME-Speech and Backbone-LLM (e.g. multi-turn 80.45 vs. 79.33, CoVoST 41.29 vs. 40.65); the paper does not explain these differences.
  • Freeze-Omni, the closest frozen-LLM design, is cited but not evaluated, so the benefit of middle-layer lockstep synchronization over that approach is not measured.
  • No code or demo release is mentioned.

Wiki Connections

  • Spoken Language Model — Converts a frozen Phi-4-MM speech-to-text LLM, which consumes input speech, into a speech-in, speech-out model by training only an audio branch.
  • Speech-to-Speech — Evaluated as a spoken dialogue system and on X-to-English speech-to-speech translation, with a cache policy for multi-turn spoken dialogue.
  • Autoregressive Codec TTS — The trained post-decoder autoregressively generates 25 Hz CosyVoice 2 codec tokens, with MTP heads emitting up to four per step.
  • Streaming TTS — Codec blocks are emitted in lockstep with each text token, and time to first audio and RTF are measured across MTP horizons.
  • Phi-4-Mini Technical Report — Describes the Phi-4-MM family whose 7B multimodal model serves as the frozen S2T backbone.
  • Towards Efficient Speech-Text Jointly Decoding — Same group’s earlier speech LM work; its early-stop interleaved (ESI) pattern, with LoRA on the backbone, is an ablation baseline that loses S2T reasoning accuracy.
  • Freeze-Omni — Closest precedent for adding speech output to a frozen LLM; cited as an S2S architecture but not compared experimentally.
  • CosyVoice 2 — Provides the 25 Hz semantic speech tokenizer used for targets and the decoder that turns tokens into waveforms.
  • VocalNet — Baseline that also applies MTP to speech generation; faster in raw throughput but with a larger multi-turn S2T to S2S gap.
  • Qwen3-Omni — Strongest baseline (30B Thinker-Talker); leads on TriviaQA, WebQ, and BigBench-Audio S2S but has higher rendering WER.
  • Qwen2.5-Omni — Thinker-Talker baseline in both quality and efficiency comparisons, with similar TTFA to PRIME-Speech at k = 1.
  • VocalBench — Benchmark used for knowledge, reasoning, creativity, fluency (UTMOS), and single-round scores.