arXiv · 2025 · Preprint
Adel Moumen et al. (University of Cambridge) · → Paper · Demo: ? · Code: ?
Introduces a cross-lingual interleaving scheme that mixes speech tokens from different languages within a single training sequence, without any text supervision, to build textless spoken language models that transfer semantic competence across languages.
Problem
Spoken language models (SLMs) learn linguistic competence directly from discrete speech units, without text, which in principle widens access to NLP for languages with limited written resources. In practice, progress has stayed largely English-centric because cross-lingual evaluation benchmarks and training data are scarce. Prior attempts to encourage cross-lingual sharing in speech-capable language models generally depend on text, for example speech-text interleaving in text-speech LMs (TSLMs) such as VoxtLM/SUTLM and Spirit-LM (Spirit LM), which switch between text and speech tokens within a language. That mechanism is not compatible with textless SLMs, which have no text tokens to interleave with, and existing multilingual speech-LM efforts either report only English spoken evaluations (Spirit-LM) or train on multiple languages but evaluate semantics only in English (GLM-4-Voice-style systems). The paper targets this gap: building a textless, cross-lingual SLM together with the bilingual benchmarks needed to actually measure cross-lingual semantic competence.
Method
The system follows the standard three-stage SLM pipeline: quantise a waveform into discrete speech units, train an autoregressive language model with a next-token objective over the unit sequence, and resynthesise waveforms with a neural vocoder. Speech is tokenised with Mimi, a convolutional RVQ audio codec (32 codebooks, 12.5 Hz, semantic distillation from WavLM); following prior evidence that early RVQ codebooks capture semantic content, only the first codebook is modelled by the SLM. SLMs are initialised from pretrained text LLM checkpoints (a 1B-parameter Llama 3.2 checkpoint and a 360M-parameter Qwen2 checkpoint, following TWIST), with the text embedding table replaced by embeddings for the new discrete audio token vocabulary; both models use a 2048-token context window (about 2.73 minutes of speech).
The core contribution is a cross-lingual interleaving scheme that operates purely on speech tokens. Given sentence-level segmentation and cross-lingual sentence alignment between two languages, the method samples a language label per segment index and concatenates the corresponding speech-token segments at sentence boundaries into a single training sequence, so each next-token prediction is conditioned on a context that may span multiple languages. No text tokens are introduced at any point, keeping the method compatible with textless pipelines. Training proceeds in three stages to isolate the effect of interleaving: (1) English-only pre-training for 50k steps to simulate a high-resource SLM; (2) cross-lingual interleaving for 20k steps with a 0.5 language-sampling probability, to transfer from the high-resource language (English) into the lower-resource one (French); and (3) an alternating monolingual fine-tuning stage in French and English for 15k steps (“stabilisation”) to restore monolingual generation quality in both languages. All comparisons use a matched training-token budget across systems, so that any effect is attributable to interleaving rather than to data volume.
To make cross-lingual interleaving possible without text, the authors construct sentence-aligned bilingual speech resources: a spoken, sentence-aligned English-French corpus derived from TinyStories (approximately 42k hours total; GPT-4 translates each sentence with full story context, and a multi-speaker delayed-streams TTS system from the Moshi lineage synthesises the audio, reusing a similarity-matched voice across the paired languages to promote cross-lingual speaker consistency). The same construction pipeline produces two new bilingual evaluation benchmarks, spoken StoryCloze (sSC) and spoken TopicCloze (sTC), each comprising four-sentence story prefixes paired with a true and an adversarial ending (semantically incompatible for sSC, off-topic for sTC), used to probe whether a model prefers the correct continuation via log-likelihood.
Key Results
On monolingual semantic tasks (sSC, sTC), a French-only baseline already reaches non-trivial competence (56.44% sSC, 69.85% sTC on the 360M model), and interleaving without stabilisation trades a little English polish for cross-lingual gains: English sSC drops modestly (e.g. 56.06% to 54.40% for the 1B model) while English sBLiMP/sWUGGY drop more sharply (61.96%/69.92% to 52.73%/56.74%), consistent with rapid language alternation perturbing low-level syntactic/lexical regularities more than semantics. French sSC still improves during interleaving despite far fewer French tokens than the monolingual French baseline (360M: 57.08% with interleaving vs. 56.44% with 3x more French data), indicating the gain is transfer from English rather than data volume. A brief bilingual stabilisation stage after interleaving restores English performance close to the English-only baseline (1B sBLiMP/sWUGGY recover to 61.75%/69.15%) while consolidating the French transfer: at a matched French token budget, stabilised interleaving beats the French-only baseline on both sSC (58.31% vs. 55.31%) and sTC (70.39% vs. 67.07%) for the 1B model.
On cross-lingual continuation (prompt in one language, continuation scored in the other), a mixed EN+FR baseline trained without interleaving performs only modestly above chance (360M: 50.56%/51.25% sSC, 55.58%/57.34% sTC for EN→FR/FR→EN). With interleaving, cross-lingual performance rises substantially and approaches monolingual accuracy: the 360M interleaved model reaches 56.44%/55.37% on cross-lingual sSC and 65.20%/65.84% on cross-lingual sTC, comparable to its own monolingual scores (55.90% EN / 57.08% FR sSC; 64.00% EN / 68.67% FR sTC). The stabilisation stage preserves most of this cross-lingual ability with only a small degradation (average relative drops of 1.39 points on sSC and 4.45 points on sTC), while gaining monolingual quality. Layer-wise cosine similarity between hidden states on 1,000 aligned EN-FR sentence pairs (1B models) increases from 0.73 (EN+FR baseline, no interleaving) to 0.75 (interleaving) to 0.76 (interleaving + stabilisation), indicating progressively stronger cross-lingual representational alignment. These trends hold at both 360M and 1B scale.
Novelty Assessment
The core idea, interleaving segments across languages within a single training sequence, is a direct textless adaptation of an idea already established for text-speech interleaving (VoxtLM/SUTLM, Spirit-LM); the novelty is in applying it purely to speech tokens across languages rather than across modalities, which removes the dependence on text and had not been demonstrated before. The training-recipe contribution (three-stage schedule: monolingual pretraining, cross-lingual interleaving, monolingual stabilisation) is a reasonable but incremental engineering choice on top of that idea. The more durable contribution may be the released resources: a sentence-aligned, ~42k-hour bilingual spoken corpus and the first bilingual spoken StoryCloze/TopicCloze benchmarks, which fill a real gap since almost all prior SLM semantic evaluation is English-only. The experimental scope is narrow (one language pair, 360M/1B scale, short-paper format), so the generality of the interleaving recipe across more distant language pairs or larger models is not established here.
Field Significance
moderate — the paper demonstrates a simple, textless mechanism for building cross-lingual spoken language models and provides genuinely new sentence-aligned bilingual data and semantic benchmarks that address a real evaluation gap in SLM research. Its scope is confined to a single language pair and modest model scales, so it is best read as a proof of concept and a resource release rather than a demonstration of general multilingual scalability.
Claims
- supports: Interleaving segments from different languages within a single training sequence, applied directly to discrete speech tokens without any text supervision, induces cross-lingual transfer that plain multilingual mixing without interleaving does not achieve.
Evidence: At matched training-token budgets, the EN+FR baseline without interleaving reaches only 50.56%/51.25% cross-lingual sSC accuracy (360M, EN→FR/FR→EN), barely above chance, while the interleaved model reaches 56.44%/55.37% and approaches its own monolingual accuracy of 55.90%/57.08% (§6.2, Table 3).
- supports: Cross-lingual interleaving strengthens shared cross-lingual structure in a speech LM’s internal representations, not just its output-level accuracy.
Evidence: Layer-wise cosine similarity between hidden states on 1,000 aligned EN-FR sentence pairs rises from 0.73 (mixed baseline, no interleaving) to 0.75 (interleaving) to 0.76 (interleaving plus monolingual stabilisation) for the 1B model (§6.2).
- complicates: Cross-lingual interleaving without a subsequent monolingual stabilisation stage degrades low-level syntactic and lexical competence in the high-resource language more than it degrades semantic competence.
Evidence: For the 1B model, English sBLiMP and sWUGGY drop from 61.96%/69.92% (English-only baseline) to 52.73%/56.74% after interleaving, a much larger relative drop than the corresponding sSC decrease (56.06% to 54.40%) (§6.1, Table 3).
- supports: A brief monolingual fine-tuning stage after cross-lingual interleaving can recover most of the lost monolingual quality while retaining the cross-lingual gains from interleaving.
Evidence: After the stabilisation stage, the 1B model’s English sBLiMP/sWUGGY recover to 61.75%/69.15% (within a hair of the English-only baseline), while French sSC/sTC (58.31%/70.39%) still exceed the French-only baseline (55.31%/67.07%) at the same French token budget (§6.1, Table 3).
Limitations and Open Questions
The evaluation covers a single, related language pair (English-French), both of which are Latin-script, high-resource languages for existing MT and TTS tooling; the paper does not test whether the interleaving scheme transfers to typologically distant or genuinely low-resource language pairs, which is the scenario the introduction motivates the work with. Model scale is limited to 360M and 1B parameters, and all training and evaluation data is synthetic speech generated by a TTS system rather than naturally recorded bilingual speech, so results may not transfer directly to found (non-synthetic) cross-lingual audio. The interleaving scheme also requires sentence-level cross-lingual alignment during training, which is itself a resource that had to be constructed via machine translation and cannot be assumed to exist for arbitrary language pairs.
Wiki Connections
- Spoken Language Model — extends the textless GSLM lineage of speech-only autoregressive language models to a cross-lingual training setting.
- Self-Supervised Speech — relies on Mimi’s semantic codebook, distilled from WavLM, as the discrete input representation the SLM is trained on.
- Neural Audio Codec — adopts the Mimi RVQ codec and models only its first (semantic) codebook, following prior evidence that early codebooks capture semantic content.
- Evaluation Metrics — introduces the first bilingual spoken StoryCloze and TopicCloze benchmarks, filling a gap left by predominantly English-only SLM semantic evaluation.
- Fission TSLM — a closely related text-speech LM from an overlapping author group, contrasted as a text-dependent route to cross-modal transfer versus this paper’s textless, speech-only interleaving.
- Spirit LM — the key text-speech interleaving precedent this paper adapts to a purely speech-token, cross-lingual setting without text.
- Moshi — source of both the Mimi codec used as the speech tokeniser and the delayed-streams TTS system used to synthesise the paper’s bilingual training and evaluation corpora.
- Scaling Speech-Text Pre-training with Synthetic Interleaved Data — cited as a text-dependent multilingual speech-LM approach that trains on multiple languages but evaluates semantics only in English.
- AudioLM — foundational cascaded semantic/acoustic token SLM referenced as part of the textless SLM lineage this work builds on.
- Slamming — prior work establishing that spoken TinyStories is an effective, compute-efficient SLM training resource, motivating this paper’s TinyStories-based cross-lingual corpus.
- Align-SLM — cited as a recent textless SLM fine-tuned with preference optimisation, part of the comparison set of SLM training recipes this paper positions against.
- Scheduled Interleaved Speech-Text Training for S2S Translation — directly contrasted as an interleaving scheme that mixes monolingually with no cross-language interaction during continuation, unlike this paper’s genuinely cross-lingual interleaving.