Late Fusion and Multi-Level Fission Amplify Cross-Modal Transfer in Text-Speech LMs

arXiv · 2025 · Preprint

Santiago Cuervo et al. (Université de Toulon (LIS) / University of Cambridge) · → Paper · Demo: ? · Code: ?

Introduces late fusion and multi-level fission for text-speech language models, an architecture that respects the different abstraction granularity of text and speech tokens and achieves state-of-the-art-level cross-modal transfer with an order of magnitude less compute than prior text-speech LMs.

Problem

Text-speech language models (TSLMs) typically adopt early fusion and early fission: text and speech tokens are fed into and predicted from a shared transformer backbone through the same linear input/output layers, with both modalities passing through the same number of layers regardless of how abstract or fine-grained their tokens are. The paper argues this design is structurally mismatched to the task of cross-modal transfer: speech tokens are lower-level than text tokens (a single word corresponds to many phonetic or sub-phonetic speech units), and language models build increasingly compositional features with depth. Under early fusion/fission, non-text modalities are forced to reach the abstraction level needed to exploit text-pretrained functions without the extra layers such composition would require, and the model’s final layer - tuned for next-word prediction - is asked to directly emit low-level, non-textual targets. Prior TSLMs such as SpiritLM, Moshi, and GLM-4-Voice (Spirit LM, Moshi, GLM-4-Voice) close this gap mainly by scaling parameters and training compute, which the authors note is expensive: unimodal speech LMs are known to scale far less efficiently than text LMs, making architectural fixes for cross-modal transfer more attractive than brute-force scaling.

Method

The paper reframes modality fusion and fission around three premises: (i) cross-modal transfer works by reusing text-pretrained functions on “text-like” structure in other modalities, (ii) speech tokens are inherently lower-level than text tokens, and (iii) transformer layers build increasingly compositional features with depth. From these, the authors derive three architectural principles: late fusion (an input adapter composes speech tokens into higher-level features before they reach the text backbone), late fission (an output adapter decomposes backbone representations back into low-level speech-predictive features), and multi-level fission (the fission mechanism should draw on both low- and high-level backbone features, since predicting the next speech token requires fine-grained context within a word but coarser, next-word-like context at word boundaries).

The resulting architecture, SmolTolk, implements all three principles with simple causal transformer adapters rather than pretrained bidirectional encoders (e.g., Whisper- or CLIP-style), preserving a unified discrete input/output token representation suited to autoregressive modeling. Contiguous chunks of speech embeddings pass through a two-layer input adapter before being merged with text embeddings at their sequence positions and processed by the shared backbone. For fission, a learned linear layer selector produces input-dependent softmax weights over all backbone layers, computing a weighted average of per-layer contextual representations; a residual connection from the speech input embedding is added to retain fine-grained token identity. This multi-level representation is then passed through a two-layer output adapter before the final linear projection to speech logits. Text logits continue to be read directly from the last backbone layer. An entropy-maximization term is added to the training loss to prevent the layer selector from collapsing onto a single layer, an issue observed in larger models.

SmolTolk is built on the SmolLM family of text backbones (135M, 360M, 1.7B parameters), with the added adapters contributing an 11-15% parameter overhead. Speech is tokenized as HuBERT units quantized into a 500-token vocabulary at 25 Hz (following prior TSLM tokenization practice), with adjacent repeated tokens collapsed. Training follows a two-stage schedule: in stage 1 (~3% of iterations) the text backbone is frozen and only the new speech modules are trained on interleaved text-speech data to limit forgetting of text capabilities; in stage 2 the full model is fine-tuned on a mixture of unimodal speech, unimodal text, and interleaved text-speech samples (with ASR and TTS deliberately excluded from the main training mixture, unlike prior TSLMs such as AudioPaLM (AudioPaLM)). Training uses AdamW, a constant learning rate, and up to 32B training tokens for the largest model.

Key Results

On zero-shot speech language modeling benchmarks (sBLIMP, sStoryCloze, tStoryCloze, evaluated across same-modality and cross-modal settings), SmolTolk consistently and substantially outperforms early fusion/fission baselines trained with identical backbones and data. SmolTolk-150M surpasses the 360M early-fusion baseline and rivals the 1.7B baseline; cross-modal alignment (T→S, S→T) only emerges in the 1.7B early-fusion baseline, but SmolTolk exhibits it even at 150M parameters. Compute-matched scaling curves for LibriSpeech NLL and tStoryCloze accuracy show SmolTolk dominating the early-fusion baselines across the entire compute range tested.

Against state-of-the-art TSLMs, SmolTolk-2B outperforms GLM-4-Voice-1.5B (GLM-4-Voice) - the closest model in parameter count - while using over 20x less training compute (16-32B tokens vs. 1T), and even SmolTolk-400M surpasses GLM-4-Voice-1.5B on several cross-modal metrics. SmolTolk-2B performs comparably to the much larger Moshi (7.7B, Moshi) and GLM-4-Voice-9B on most tasks and achieves the best sBLIMP syntactic score in the comparison, though it trails on speech-to-text continuation. Against the extended benchmark table that also includes SpiritLM (Spirit LM), SyllableLM (SyllableLM), Align-SLM (Align-SLM), and Slamming (Slamming), SmolTolk is competitive or superior on same-modality speech metrics at a fraction of the training budget.

Ablations (Table 2, medium-size model) show that removing any architectural component degrades performance, with removing all adapters causing the steepest cross-modal transfer drop and the input adapter mattering more than the output adapter for cross-modal transfer specifically. Representation analyses (intrinsic dimensionality and cross-modal subspace overlap) corroborate the mechanism: models without multi-level fission show the lowest compositionality and alignment among the ablated variants, and removing input/output adapters selectively reduces alignment in early/late layers respectively. An auxiliary experiment (Appendix D.4) adding explicit ASR and TTS training shows SmolTolk-2B+ASR+TTS reaches WER 8.0/19.2 and TTS CER 12.0/14.5 (clean/other, zero-shot) on LibriSpeech, competitive with the 3.5x larger, 11x-more-data SpiritLM (WER 6.0/11.0, TTS CER 6.7/7.9), while barely affecting cross-modal StoryCloze performance.

Novelty Assessment

The contribution is genuinely architectural: late fusion, late fission, and especially multi-level fission (an input-dependent, entropy-regularized weighted pooling across backbone layers for output generation) are, by the authors’ own account and to the reviewer’s knowledge, not previously combined in this form for text-speech LMs, though late fusion and decoupled output modules individually appear in prior multimodal LLM work (e.g., Qwen-Audio-style late fusion, Xie and Wu’s modality-specific forward passes). The paper’s strength is that it does not just propose the mechanism but validates it mechanistically: ablations isolate each component’s contribution to both downstream metrics and representation-level properties (intrinsic dimensionality, subspace overlap), and a word-segmentation probe on the learned layer-attention weights gives independent, if imperfect, evidence for the hypothesized alternation between low- and high-level feature use. The efficiency claim (rivaling SOTA at >20x less compute) is credible because comparisons are partly compute-matched (Figure 2) rather than relying solely on final-checkpoint numbers. The main open question the paper itself flags is scale: all experiments are conducted at 135M-2B parameters, and the authors’ own analysis of models at SpiritLM’s 7B scale (Appendix D.1) is limited to representation alignment, not full downstream re-training with their architecture.

Field Significance

Tip

High - the paper demonstrates that respecting the compositional-hierarchy mismatch between text and speech tokens, via late fusion and a novel multi-level fission mechanism, closes much of the cross-modal transfer gap that prior early-fusion text-speech LMs closed mainly through scale, at over an order of magnitude less training compute.

The paper’s significance lies less in the raw benchmark numbers and more in providing a mechanistic account - backed by representation-level analysis, not just downstream metrics - of why early fusion/fission architectures underperform at cross-modal transfer and how a specific structural fix addresses it. This gives future spoken language model designs a concrete, tested alternative to simply scaling parameters and data when cross-modal transfer is the bottleneck.

Claims

• supports: Architectures that let non-text modalities undergo additional feature composition before entering a shared text-pretrained backbone (late fusion), and additional decomposition after leaving it (late fission), improve cross-modal transfer efficiency relative to early fusion/fission at matched model size and training compute.

  Evidence: SmolTolk-150M surpasses the 360M early-fusion baseline and rivals the 1.7B early-fusion baseline on cross-modal StoryCloze metrics, and compute-matched scaling curves show SmolTolk dominating early-fusion baselines across the tested compute range (§5, Table 1, Figure 2).

• supports: Allowing a speech-generation output mechanism to draw on both low-level and high-level backbone representations, rather than only the final layer, improves the model’s ability to abstract compositional features and increases the overlap between text and speech representation subspaces.

  Evidence: Removing multi-level layer pooling (”- Layer pooling”) is the ablation with the largest drop in intrinsic dimensionality and cross-modal subspace overlap among non-baseline variants, and reduces tStoryCloze T→S from 80.9 to 77.8 (§5.1, §5.2, Table 2, Figure 3).

• complicates: Increasing model scale under early fusion/fission architectures narrows, but does not close, the cross-modal transfer gap with architectures explicitly designed around the compositional-hierarchy mismatch between modalities.

  Evidence: Cross-modal alignment in early-fusion baselines only emerges at the 1.7B parameter scale, yet even the 7B-parameter early-fusion SpiritLM shows lower cross-modal representation alignment than the much smaller SmolTolk-2B (§7, Appendix D.1, Figure 5).

• complicates: Layer-attention patterns learned for cross-modal language modeling can reflect linguistically meaningful structure (e.g., word boundaries) without this translating into a competitive unsupervised segmentation method as model scale increases.

  Evidence: A word-boundary segmentation probe derived from the layer selector’s final-layer weights outperforms the SCPC unsupervised baseline at all three model sizes on TIMIT, but segmentation F1 decreases from 48.4 (150M) to 31.2 (2B) as model size grows (§5.2, Table 3).

• complicates: Text-speech language models trained primarily for linguistic/cross-modal competence, without instruction tuning or high-expressivity speech tokenizers, have limited direct applicability as spoken conversational agents.

  Evidence: The authors note SmolTolk is not instruction fine-tuned, limiting its direct use for spoken conversational agent tasks, and that its HuBERT-based tokens constrain expressivity relative to cascaded ASR+TTS pipelines (§7 Limitations).

Limitations and Open Questions

Warning

All experiments are conducted at 135M-2B parameters; the paper’s own scale-up evidence (Appendix D.1) is limited to a representation-alignment comparison against the 7B-parameter SpiritLM, not a full re-training of SmolTolk at that scale, so whether the architecture’s advantages persist under full-scale training remains an extrapolation rather than a demonstrated result.

Beyond scale, the authors note that SmolTolk is not instruction fine-tuned, limiting its direct applicability to spoken conversational agent use cases without further work. The HuBERT-based speech tokenizer used throughout constrains output expressivity (e.g., prosody, non-linguistic vocalizations) relative to cascaded ASR+TTS pipelines or more expressive speech representations, and the authors flag this as a direction for future work rather than something the current architecture addresses. The auxiliary ASR/TTS experiment (Appendix D.4) is a single configuration added on top of the main model rather than a systematically tuned ASR/TTS setup, so its WER/CER numbers should be read as a feasibility check rather than an optimized result.

Wiki Connections

• Spoken Language Model — proposes a late fusion/late fission architecture with a novel multi-level fission mechanism specifically to improve cross-modal transfer in joint text-speech language models.
• Self-Supervised Speech — speech is represented throughout as HuBERT-derived discrete units, making a self-supervised speech tokenizer a core input/output representation for the model.
• Speech-to-Speech — the trained model performs speech-context-to-speech-continuation generation as one of its four evaluated cross-modal settings, alongside text-to-speech and speech-to-text.
• Spirit LM — the closest prior TSLM in training/evaluation methodology; SmolTolk is compared directly against it and, per the paper’s own scale analysis, shows higher cross-modal alignment despite being far smaller.
• Moshi — used as a state-of-the-art early fusion/fission baseline; SmolTolk-2B performs comparably to Moshi’s 7.7B-parameter model on most cross-modal metrics.
• GLM-4-Voice — used as the closest-sized state-of-the-art baseline; SmolTolk-2B outperforms its 1.5B variant while using over 20x less training compute.
• AudioPaLM — represents the mixed speech-text task training recipe (including ASR/TTS) that SmolTolk deliberately departs from in its main training mixture.
• Align-SLM — included as an extended-benchmark comparison point among recent textless/text-speech speech LMs evaluated on the same StoryCloze-style benchmarks.
• Slamming — included as an extended-benchmark comparison point illustrating efficient speech LM training at comparable parameter scale.
• SyllableLM — included as an extended-benchmark comparison point among speech-token-based language models evaluated on sBLIMP/StoryCloze.