ICML · 2026 · Conference
Congrui Du et al. (University of California, Santa Barbara / MIT-IBM Computing Research Lab, IBM Research) · → Paper · → Preprint · Demo: ✓ · Code: ✓
SpeechCombine builds an instruction-following speech LM with no instruction tuning at all: it continues pre-training a text LLM base model on about 30k hours of prosody-tokenized speech, then adds the text LLM’s instruct-minus-base weight difference, and the merged model follows QA, prosody-understanding and expressive-generation instructions it never saw in training.
Problem
Most speech LMs copy the text-LLM recipe: large-scale speech pre-training followed by speech instruction tuning (SFT and/or RL) on real or synthesized speech. The paper argues that this recipe has two costs. First, speech token sequences are roughly 20 times longer than the equivalent text, so the effective data scale is far smaller than for text LLMs. Second, heavy training on speech tokens causes catastrophic forgetting, so much of the speech training budget goes to relearning what the text LLM already knew. Freezing the LLM and training only adapters (the Freeze-Omni approach) avoids forgetting but supports only text-style instructions, not speech-specific instructions about emotion or emphasis. The paper asks whether a speech LM can keep the text LLM’s knowledge and still acquire speech-specific skills, without a speech instruction-tuning stage.
Method
The system is a cascade around one trained LLM. An external ASR (whisper-large-v3) transcribes input speech into a [text] section, and the ProsodyLM tokenizer converts the same speech into a [speech] section of word-level prosody tokens. The LLM reads and writes interleaved [text][speech] pairs. A frozen ProsodyLM decoder (modified from a StyleTTS2 synthesizer) renders generated text-plus-prosody tokens back into a waveform. Only the LLM is trained; the encoder and decoder stay frozen (§3.1).
The speech representation encodes content and prosody, not timbre. For each word, the tokenizer emits five values (median log-F0, F0 range, F0 slope, duration and energy), each quantized to 512 levels. Silences carry their own duration tokens, and a speaker median-F0 token opens each [speech] section (§3.5, §Appendix A.3). The paper does not report a frame or token rate.
Training is a single round of continued pre-training with the standard next-token objective. It starts from Qwen3-8B-base, not the instruct model, and uses LoRA (rank 64, alpha 16). The data combine Libri-Light with eight emotional and expressive corpora (BEAT, CREMA-D, ESD, JL Corpus, EmoV-DB, Expresso, MEAD, TESS), about 30k hours in total. Each sentence becomes a [text][speech] pair. A natural-language [cap] caption is placed before the speech with probability 0.3, after it with probability 0.3, or left out (0.4). This placement teaches caption-to-speech (generation), speech-to-caption (understanding) and multi-sentence continuation patterns. Captions are written by GPT-OSS 120B from automatically extracted attributes: pitch level (RMVPE), speaking rate, emphasized words (WhiStress), emotion, emotional arousal and transcript. Special tokens such as <|im_end|> and </think> are randomly inserted into the data so the model does not forget them (§3.3, §3.4, §4.1, §Appendix A.1, A.2).
The merge step is task arithmetic in the style of Chat Vector. With Δθ_speech = θ_speech − θ_base and Δθ_inst = θ_inst − θ_base, the final model is θ_SC = θ_base + λΔθ_speech + Δθ_inst, with λ = 0.85. A smaller λ keeps the speech-adapted model closer to the base, which helps the instruct direction transfer but limits how much speech knowledge is kept. The paper stresses that speech pre-training must start from the base model, unlike Fun-Audio-Chat’s merge recipe (§3.1, §4.1).
At inference, the model uses the Qwen3 chat template with [speech] sections added. Format forcing makes the output reliable: speech tokens are masked inside text and thinking sections, the <speech> delimiter is boosted after each text section, and the speaker-F0 token is forced to come first. Long thinking is turned on by forcing <think> followed by “Okay” and banning </think> for at least 160 tokens. It is enabled by default (§3.6, §3.7, §Appendix A.4, A.5).
Key Results
The paper compares against two groups. Group A uses the same Qwen3-8B models and data: ASR plus the text LLM (the paper calls this the topline), continued pre-training from the instruct model, and continued pre-training plus SFT on about 10k hours of SIFT-50M, InstructS2S-200K and VoiceAssistant-400K. Group B contains released speech LMs of similar size, plus GPT-4o-Audio, which the paper excludes from its rankings because it is much larger (§4.1).
Text-oriented tasks (accuracy, Table 1). On OpenbookQA and MMSU, SpeechCombine scores 86.59 and 73.38, the highest of the Group B comparison (Fun-Audio-Chat 83.52 / 71.08, Kimi-Audio 83.52 / 62.17). It also edges past the ASR + text LLM topline (83.29 / 73.22). On the reasoning sets it is second, not first: GSM8kEval 90.03 vs. Kimi-Audio 95.53, TruthfulEval 60.09 vs. Fun-Audio-Chat 61.27, MLCpro 89.01 vs. Fun-Audio-Chat 93.40. On MLC it ties Fun-Audio-Chat at 93.97. The ASR + text LLM topline stays ahead on GSM8k (94.61), TruthfulEval (71.12) and MLCpro (94.13). Some baseline numbers are copied from other papers (Table 1 footnotes), and the reasoning sets are scored by an LLM judge (§4.2, Table 1, §Appendix B.1).
Speech understanding (Table 2, panel a). Emphasis detection F1 is 60.84, well above every baseline, including GPT-4o-Audio (42.99) and Fun-Audio-Chat (28.76). Emotion understanding is mid-table at 52.70, behind Fun-Audio-Chat (74.74), Kimi-Audio (63.69) and ASR + text LLM (55.42). The paper attributes this gap to having emotion captions for only about 100 of the 30k hours. The emphasis result also depends on a forced “Based on the prosody” prefix in the thinking section, which the paper admits gives it “a mild advantage” over the baselines (§4.3, Table 2).
Speech generation (Table 2, panel b). GenEmo score is 45.42, second to GLM-4-Voice (48.13) and ahead of Fun-Audio-Chat (39.30). Emphasis generation F1 is 31.42, the highest among the comparable-size systems (GLM-4-Voice 26.55, Qwen2.5-Omni 24.24). GPT-4o-Audio scores 65.02, about double, so “best on emphasis generation” holds only with GPT-4o-Audio excluded. Group A systems score lowest on generation; the ASR + text LLM pipeline with Kokoro TTS gets a GenEmo score of 5.06 (§4.4, Table 2). All generation scores are automatic: emotion2vec probability weighted by (1 − WER), and the EmphAssess emphasis detector. There is no human listening test.
Ablations (Table 3, Figure 4). Removing Δθ_inst and using in-context examples instead lowers the scores to 71.68 / 40.38 / 15.34 (OpenbookQA / emphasis F1 / GenEmo), against 86.59 / 60.84 / 45.42. Dropping [cap] from pre-training collapses emphasis detection to F1 0.39, and dropping [text] collapses it to 0.40, while OpenbookQA barely moves. Turning off long thinking costs about 22 points on OpenbookQA (64.83) and 9 on GenEmo (36.37). As λ grows, QA accuracy falls and emphasis detection and GenEmo scores rise, with a broad workable range in between. Emotion understanding is the exception: in Table 5 it falls from 62.04 at λ = 0.8 to 29.87 at λ = 1.0. The recipe also transfers to OLMo-3-7B-Think and Llama-3.1-8B, but with much lower scores (OpenbookQA 72.52 and 62.85) (§4.6, Table 3, Figure 4, §Appendix C.1, C.3, Tables 4, 5, 7).
Novelty Assessment
The components are known. Weight-difference transfer comes from Chat Vector and task arithmetic, the prosody tokenizer and decoder come from ProsodyLM (by overlapping authors), and Fun-Audio-Chat already uses model merging in speech LM training. The contribution is the recipe and the finding. Pre-train from the base model on interleaved text, prosody tokens and captions, then add the instruct delta. The merged model then shows compositional generalization: it follows speech-understanding and speech-generation instructions that neither weight direction saw in training, and it carries over long-thinking behaviour. The cleanest evidence is the controlled Group A comparison on identical data and backbone. The Group B comparison is less controlled: several numbers are copied from other papers, and SpeechCombine gets task-specific format prompts and a prosody-reasoning prefix. The system is also far from a fully end-to-end speech LM. It depends on external ASR, represents only prosody (no timbre, voice quality or accent), and needs format forcing to produce well-formed output.
Field Significance
Moderate — The paper shows that weight-space combination of a speech-adapted base model with a text LLM’s instruct delta can produce instruction-following speech LM behaviour, including prosody-aware generation, from about 30k hours of speech and no instruction data. This challenges the view that speech LMs need large instruction-tuning corpora. The evidence comes from a prosody-only token representation with an ASR front end and automatic scoring alone, so it is unclear how far the result carries over to full-waveform speech LMs.
Claims
- supports: Adding a text LLM’s instruction-tuning weight difference to a speech-adapted base model can give a speech LM instruction-following ability without any speech instruction-tuning data.
Evidence: Qwen3-8B-base continued-pretrained on about 30k hours of prosody-tokenized speech, plus the instruct-minus-base delta (λ = 0.85), reaches 86.59 OpenbookQA and 73.38 MMSU accuracy. This exceeds continued pre-training plus 10k hours of speech SFT (80.21 / 60.80) and matches the ASR + text LLM topline (83.29 / 73.22) (§4.2, Table 1).
- supports: Speech-specific instruction following can emerge from combining a speech-knowledge direction with a text instruction direction, even when neither saw speech-understanding or speech-generation instructions in training.
Evidence: SpeechCombine reaches emphasis detection F1 60.84 and emphasis generation F1 31.42 on adapted EmphAssess. Without the instruct delta (in-context examples instead), emphasis detection drops to 40.38 and GenEmo from 45.42 to 15.34 (§4.3, §4.4, §4.6, Tables 2, 3).
- refines: When speech knowledge and text instruction ability are merged, transfer depends on paired text anchors and natural-language descriptions of prosody in the speech pre-training data.
Evidence: Removing the
[cap]captions from pre-training drops emphasis detection F1 from 60.84 to 0.39 and GenEmo from 45.42 to 27.18. Removing the paired[text]transcript drops them to 0.40 and 35.69, while OpenbookQA accuracy changes only slightly (84.83 and 83.95 vs. 86.59) (§4.6, Table 3). - complicates: Merging weights from a text LLM trades text-task knowledge against speech-task ability, with no single setting best for both.
Evidence: As the speech-direction weight λ rises, OpenbookQA accuracy falls (89.89 at λ = 0.6 to 86.81 at λ = 0.8) while emphasis detection F1 rises (55.79 at 0.8 to 64.57 at 0.9). Emotion understanding does not follow the speech-task trend: it falls from 62.04 at λ = 0.8 to 29.87 at λ = 1.0. The paper recommends λ 0.6 to 0.85 for text-oriented use and 0.8 to 1.0 for speech understanding and generation (§4.6, Figure 4, §Appendix C.1, Tables 4, 5).
- complicates: Speech LMs that inherit text reasoning tend to answer prosody questions from the transcript unless prosody-aware reasoning is triggered explicitly.
Evidence: Without the forced “Based on the prosody” prefix, SpeechCombine infers the emphasized word from the text and answers wrongly; with the prefix, it reasons about the speaker’s actual emphasis. Removing format forcing halves emphasis detection F1 (53.14 to 27.73) in non-thinking mode (§4.3, §4.5, Figure 3, §Appendix C.2, Table 6).
Limitations and Open Questions
Warning
The speech representation encodes only word-level prosody (pitch, duration, energy), and an external Whisper ASR supplies the transcript. Timbre, voice quality and accent are outside the model. Speech output quality is judged only by automatic classifiers (emotion2vec with WER, the EmphAssess detector); there is no MOS or listening test, and the paper sends readers to its demo page for perceptual quality.
- Output formatting is unstable. The model needs format forcing and a forced thinking onset (“Okay” after
<think>). Without forcing, the thinking section degenerates into prosody tokens (§5, §Appendix C.2). - The comparison with released speech LMs is not fully controlled. Some baseline numbers are copied from other papers, SpeechCombine gets task-specific format prompts, and the emphasis detection prompt adds a prosody-reasoning prefix that the authors admit is an advantage (§4.3, Table 1, §Appendix B.1, B.2).
- Emphasis evaluation uses one EmphAssess speaker (ex04), and the emphasis-generation prompts are synthesized with CosyVoice (§Appendix B.2, B.3).
- §4.2 lists SDQA among the VoiceBench QA sets, but Table 1 reports no SDQA results.
- Table 2 (λ = 0.85) and the λ = 0.85 row of Table 5 do not quite agree on UnderEmo accuracy (52.70 vs. 50.75) or emphasis generation (F1 31.42 vs. 31.68), although they report the same configuration. The emphasis detection and GenEmo values match. This page uses the Table 2 values (checked against the PDF).
- Turn-taking is learned from audiobook segments spliced across speakers, not from conversational speech. Whether the recipe works with full-waveform codec tokens, where text and speech overlap more, is untested (§Appendix A.1, §3.5).
Wiki Connections
- Spoken Language Model — Proposes training a speech LM by weight combination instead of speech instruction tuning; the adapted LLM consumes external speech as ASR text plus prosody tokens.
- Speech-to-Speech — A cascaded voice-assistant setup (ASR, LLM over text and prosody tokens, frozen prosody decoder) that takes spoken queries and answers with speech.
- Prosody Control — Generates explicit word-level pitch, duration and energy tokens separately from the transcript, which allows instruction-driven emphasis placement on requested words.
- Emotion Synthesis — Produces emotion-constrained speech on GenEmotion-en (second among comparable systems), learned only from emotion captions in pre-training.
- Instruction-Conditioned TTS — Follows natural-language style instructions (read in a given emotion, emphasize a given word) for speech output, through a speech LM rather than a dedicated TTS model.
- ProsodyLM — Supplies the word-level prosody tokenization and the frozen StyleTTS2-derived decoder that SpeechCombine is built on.
- Fun-Audio-Chat — The strongest released baseline and prior use of model merging in speech LM training; SpeechCombine differs by pre-training from the base model rather than the instruct model.
- Kimi-Audio — Baseline that leads on GSM8kEval and outscores SpeechCombine on emotion understanding but trails SpeechCombine on emphasis detection and generation.
- GLM-4-Voice — Baseline with the best GenEmo score in the comparison, ahead of SpeechCombine.
- Step-Audio 2 — Baseline in both thinking and non-thinking variants; SpeechCombine outperforms both on every reported task.
- Freeze-Omni — Frozen-LLM adapter approach that the paper says keeps text knowledge but cannot follow speech-specific instructions.
- VoiceBench — Source of the OpenbookQA and MMSU spoken-QA evaluations.
- URO-Bench — Source of the reasoning, emotion-understanding and emotion-generation evaluations.