arXiv · 2025 · Preprint
Nouriborji et al. (Nlpie Research; University of Zurich) · → Paper · Demo: ? · Code: ✓
Compresses a 7B interleaved speech-text language model into a 2B student via layer-aligned knowledge distillation, retaining 93-97% of the teacher’s accuracy on spoken commonsense and acoustic-consistency benchmarks.
Problem
Interleaved speech-text language models that generate expressive, prosody-aware speech end-to-end (e.g. SpiritLM) exceed 7B parameters, putting them outside the latency and memory budgets of edge and commodity-hardware deployment. The common workaround, an ASR-frontend + text-LLM + TTS-backend cascade, is more tractable but discards the paralinguistic signal carried in raw speech tokens, producing less natural output. Meanwhile, model providers typically train a separate checkpoint from scratch for each target size, which is compute- and data-inefficient when multiple deployment points are needed. No prior work had shown that a large, interleaved speech-text foundation model could be compressed to a fraction of its size while preserving both its semantic reasoning and its expressive (prosody, speaker, style) behavior.
Method
TinyWave is a family of three 2B-parameter models distilled from 7B-parameter SpiritLM teachers, which extend LLaMA-2 with speech capability by training on continuous streams of HuBERT-based audio tokens interleaved with text. Two speech-only variants, BASE-SPEECH and EXPRESSIVE-SPEECH, are initialized from SpiritLM-Base-7B and SpiritLM-Expressive-7B respectively by pruning every third transformer block, reducing depth to 10 layers (~2B parameters) while retaining the original embeddings, RoPE settings, and LM head. The base variant consumes a 100-cluster HuBERT phonetic tokenizer; the expressive variant uses a tokenizer that adds pitch and style tokens for richer prosodic and speaker modeling. A third variant, EXPRESSIVE-INTERLEAVED, handles mixed audio-text sequences and is initialized from the already-distilled expressive-speech student rather than the 7B teacher, to reuse its learned speech priors and speed up training.
Before distillation, the teacher is “corrected”: SpiritLM-Base-7B is fine-tuned with LoRA adapters on a Libri-Light subset (a 20% subset for the base-speech teacher; a 10,000-hour subset for base-speech teacher correction) to reduce the distribution mismatch between the teacher’s original training data and the target domain; the expressive teacher is skipped since it already encodes prosodic cues. For the interleaved variant, interleaved training data is constructed by combining Libri-Light speech with pseudo-transcriptions from Whisper-v3 Large, inserting text spans into the audio token stream at word boundaries; training samples from five interleaving patterns ([Speech], [Text], [Speech][Text], [Text][Speech], [Speech][Text][Speech]) so the model learns to switch modality mid-sequence. The 7B expressive teacher is briefly fine-tuned (10k steps) on this interleaved corpus before its knowledge is distilled into the student.
The distillation objective combines three terms. A layer-alignment loss maps each student layer l to a teacher layer via g(l) = 3l + 4, applying a cosine-similarity loss between hidden states and a KL divergence between attention maps at each mapped pair. An output-alignment loss applies KL divergence between softened (temperature-scaled) teacher and student logits, encouraging the student to match the teacher’s full output distribution rather than only its argmax predictions. A standard autoregressive cross-entropy loss over ground-truth tokens is added as a third term. The three losses are combined in a weighted sum for the final training objective. All models train on 50,000 hours of publicly available Libri-Light audio, a fraction of the private data used to train the SpiritLM teacher.
Key Results
On Libri-Light continuations, distillation closes 85-93% of the Normalized Perplexity Score (NPS) gap to the teacher while halving (or further reducing) parameter count; a from-scratch 2B baseline trained without distillation trails both the teacher and the distilled student by a wide margin (Table 1, §3.1). On spoken commonsense reasoning, TinyWave-Base reaches 55.3% on Spoken StoryCloze and 74.8% on topic StoryCloze, versus 60.3%/82.1% for the 7B teacher and only 53.6%/53.6% for the matched-size from-scratch baseline; TinyWave also outperforms prior published models GSLM, VoxtLM, and AudioLM on sWUGGY/sBLIMP surface-level linguistic tests, and closely tracks the teacher (70.26% vs 70.98% sWUGGY; 58.08% vs 58.37% sBLIMP) (Table 2, §3.2-3.3). On the SALMon acoustic-consistency suite, TinyWave-Expressive matches or exceeds the teacher on background-domain tests (57.0% vs 55.0%) but drops 8-10 percentage points on speaker- and gender-consistency metrics relative to the teacher (71.0% vs 81.0% speaker consistency; 76.5% vs 85.0% gender consistency) (Table 3, §3.4). Overall, distilled models retain 93-97% of teacher accuracy across the evaluated benchmark suite while running at roughly one-third the parameter count.
Novelty Assessment
The contribution is primarily engineering integration rather than architectural innovation: TinyWave keeps the SpiritLM/LLaMA-2 transformer structure unchanged (only pruning layers) and combines established knowledge-distillation techniques, softened-logit matching (Hinton et al., 2015), hidden-state and attention-map alignment (as in DistilBERT- and Minitron-style compression), and standard cross-entropy supervision, into a single training recipe. The one procedural addition is “teacher correction,” a brief domain-adaptive fine-tuning pass on the teacher before distillation begins, which the paper shows improves supervision quality but which is itself a straightforward application of LoRA fine-tuning. What is new is the application of this recipe to a large, interleaved speech-text foundation model in the expressive speech domain, a setting where most prior distillation work has focused on ASR encoders rather than generative speech-text LMs. The evaluation is reasonably broad (four benchmark families, comparison against five prior published systems and a matched-size from-scratch baseline), which supports the compression claim, but no new architecture, training objective, or evaluation methodology is introduced.
Field Significance
Moderate — this paper provides a concrete engineering demonstration that layer-aligned distillation, rather than additional model scale, can produce compact interleaved speech-text language models suitable for commodity hardware while retaining most of a larger teacher’s semantic and acoustic behavior. It extends distillation practice, previously concentrated on ASR encoders, to generative expressive speech modeling, and the released checkpoints, training code, and evaluation scripts make the recipe directly reproducible. The contribution is incremental relative to the underlying SpiritLM architecture rather than a new direction for the field.
Claims
- supports: Layer-aligned distillation (combining hidden-state, attention-map, and softened-logit alignment) can compress a large interleaved speech-text language model to a fraction of its parameter count while retaining most of its linguistic competence, outperforming an equally-sized model trained from scratch on the same data.
Evidence: The 2B TinyWave-Base student closes 85-93% of the teacher’s Normalized Perplexity Score gap and scores 55.3%/74.8% on spoken/topic StoryCloze versus 53.6%/53.6% for a matched-size baseline trained without distillation. (§3.1-3.2, Table 1, Table 2)
- complicates: Distillation of speech language models preserves global semantic and acoustic content more reliably than fine-grained speaker-identity information.
Evidence: TinyWave-Expressive matches or exceeds the 7B teacher on SALMon background-domain accuracy but drops 8-10 percentage points relative to the teacher on gender- and speaker-consistency metrics. (§3.4, Table 3)
- refines: Domain-adapting a teacher model with lightweight fine-tuning before distillation reduces distribution mismatch and improves the quality of the supervisory signal passed to the student.
Evidence: The base-speech teacher is fine-tuned with LoRA adapters on a Libri-Light subset before distillation begins (“teacher correction”), a step the authors report as necessary to align supervision with the target domain; it is skipped for the expressive teacher, which already encodes the relevant cues. (§2.1-2.2)
- complicates: Compact distilled speech language models remain limited by the naturalness ceiling of their underlying discrete tokenizer and vocoder, independent of the distillation method itself.
Evidence: The authors report that the HuBERT-based tokenizer and vocoder components struggle to capture conversational stylistic nuance, producing flat or unnatural synthetic speech, and propose residual vector quantization tokenizers as a future direction. (§Limitations)
Limitations and Open Questions
Warning
The paper’s own Limitations section reports constrained semantic understanding, the models prioritize phonetic accuracy over deeper linguistic comprehension, and naturalness bottlenecks in the inherited HuBERT-based tokenizer and vocoder, which produce flat or unexpressive synthetic speech independent of distillation quality.
Training and evaluation use only the public Libri-Light corpus, a small fraction of the private data used to train the SpiritLM teacher, so it is unclear how the distillation recipe would behave with access to the teacher’s full training distribution. The paper does not evaluate the models in an interactive dialogue setting; all benchmarks (NPS, StoryCloze, sWUGGY/sBLIMP, SALMon) measure continuation quality or linguistic/acoustic consistency on single-speaker prompts rather than multi-turn conversational behavior. The speaker- and gender-consistency drop observed for the expressive model under SALMon is flagged by the authors as an open trade-off for future optimization rather than a solved problem.
Wiki Connections
- Spoken Language Model — shows that a 7B interleaved speech-text spoken language model (SpiritLM) can be distilled to 2B parameters while retaining most of its semantic reasoning and expressive behavior.
- Speech-to-Speech — the speech-only TinyWave variants perform audio-to-audio continuation directly over discrete tokens, without a text intermediate.
- Autoregressive Codec TTS — both teacher and student are autoregressive transformers generating discrete speech tokens sequentially, inheriting this modeling paradigm rather than replacing it.
- Self-Supervised Speech — both teacher and student operate over HuBERT-derived discrete phonetic and expressive tokens as their core speech representation.
- SpiritLM — TinyWave’s direct teacher; its base and expressive HuBERT tokenizers, interleaving scheme, and 7B checkpoints are pruned and distilled into the 2B student models evaluated here.
- AudioLM — used as a published baseline in the spoken commonsense and morphosyntax comparison table.
- VALL-E — cited as a related codec-based language-modeling approach that separates phonetic and acoustic information to improve fidelity.
- AudioPaLM — cited as a related large multimodal speech-text model that unifies ASR, TTS, and spoken continuation but at substantially larger parameter counts than TinyWave targets.