arXiv · 2026 · Preprint
Dehua Tao et al. · → Paper · Demo: ? · Code: ?
Extends a fully frozen vision-language backbone with speech understanding and generation by attaching lightweight, trainable speech-input and speech-output modules, trained on manufactured spoken-QA supervision derived cheaply from ASR corpora.
Problem
Omni-modal models that unify text, vision, and speech typically reach strong performance only by fine-tuning the full backbone on massive multimodal corpora, which is expensive to collect and compute, and risks catastrophic forgetting of the backbone’s original vision-language competence. Existing speech extensions to language models either discretize speech into tokens for joint next-token training (requiring large-scale speech-text alignment data) or use a Thinker-Talker design that still requires at least partial fine-tuning of the LLM backbone, coupling the learned speech modules tightly to one specific fine-tuned checkpoint and making them hard to transfer to other backbones. Separately, training a speech-output module for spoken question answering normally depends on large curated or synthesized spoken-QA corpora, which are costly to build at scale.
Method
Speech-Omni-Lite keeps a pre-trained vision-language (VL) backbone (Qwen3-VL-8B-Instruct) fully frozen and adds two lightweight trainable modules on either side of it: a speech projector for input and a speech token generator for output, alongside a pre-trained discrete speech tokenizer and a pre-trained speech de-tokenizer.
On the input side, a streaming discrete speech tokenizer converts speech into tokens at 12.5 Hz using a HuBERT LARGE encoder, convolutional downsampling, and a Finite Scalar Quantization (FSQ) module; convolution and attention layers are refactored to operate causally so the tokenizer can run in a streaming, chunk-by-chunk fashion (640 ms chunks). During tokenizer training, an auxiliary CTC head predicts phonemes from the FSQ codes to encourage the tokens to carry phonetic/lexical content; this head is discarded at inference. The resulting tokens are mapped into the VL backbone’s input embedding space by a speech projector (an MLP followed by several LLaMA-style decoder layers and a final linear projection), trained in two stages: first on ASR speech-text pairs, then on spoken-QA data, with the tokenizer and VL backbone frozen throughout.
On the output side, a speech token generator (an encoder-decoder design following VocalNet and Freeze-Omni, with multi-token prediction to speed up decoding) converts the VL backbone’s hidden states into discrete speech tokens, which a de-tokenizer based on F5-TTS then converts into a waveform via flow matching. The de-tokenizer replaces F5-TTS’s DiT block with a cross-attention-augmented variant (CA-DiT), where noisy/masked mel features are queries and the (temporally upsampled) speech tokens serve as keys/values, rather than being concatenated as in the original F5-TTS.
The central data problem the paper addresses is that a speech token generator trained only on ASR-style conditioning (speech in, matching tokens out) generalizes poorly to QA-style conditioning, because the VL backbone’s hidden states differ between the two settings even for similar text content. To supply QA-style supervision without collecting spoken-QA recordings, the authors construct Question-Text Answer-Text-Speech (QTATS) data: for each existing ASR pair, an LLM (Qwen3-30B-A3B-Instruct-2507) generates a plausible question given the transcript as the answer, yielding roughly 1.45 million question-answer-speech triplets from about 4,128 hours of ASR data (Table 1). To make this text-only supervision usable for training a module that must eventually consume VL hidden states from spoken questions, an auxiliary text projector (mirroring the speech projector but for text input) is trained first, then discarded after the speech token generator is trained against its QA-style hidden states.
Key Results
On speech-to-text ASR and QA benchmarks, Speech-Omni-Lite (8B) trails large-scale omni-models trained on far more speech data on raw ASR accuracy (5.77% CER on AISHELL-1 and 4.39% WER on LibriSpeech test-clean, versus 1.12–2.54% for Qwen2.5-Omni-7B and Baichuan-Omni-1.5), but reaches competitive or better spoken-QA accuracy on some sets, including the best LLaMA Questions S→T accuracy (85.0%) among all compared systems (Table 2). On speech-to-speech generation, content-consistency WER (18.46% average) and UTMOS (3.97 average) sit in the middle of the comparison group rather than at the top (Table 3), reflecting that the speech token generator is trained entirely on manufactured QTATS supervision rather than genuine spoken QA data. A transferability experiment shows that a speech projector trained against the 8B backbone can be adapted to Qwen3-VL-4B-Instruct and Qwen3-VL-32B-Instruct by retraining only the final linear layer, with QA accuracy improving as backbone size grows (e.g., LLaMA Questions S→T accuracy: 78.33% at 4B, 85.0% at 8B, 86.0% at 32B). An end-to-end latency breakdown (Table 4) shows the flow-matching de-tokenizer dominates response latency (1,139 ms of 1,346 ms total for a 1-second response), while the streaming tokenizer contributes a fixed 54.3 ms per chunk regardless of input length.
Novelty Assessment
The individual components are largely adapted from prior systems: the speech token generator follows VocalNet’s encoder-decoder-with-multi-token-prediction design, the de-tokenizer builds on F5-TTS, and the frozen-backbone Thinker-Talker paradigm follows Freeze-Omni. The paper’s own genuine contribution is the QTATS data construction strategy, which the authors state is the first to convert ASR speech-text pairs into spoken QA data via LLM-driven reverse question generation, paired with an auxiliary text projector that makes this text-only supervision usable for training the speech-output path. The CA-DiT cross-attention conditioning is a modest structural change to F5-TTS’s DiT rather than a new generative paradigm. The transferability result (reusing a trained speech projector across VL backbones of different sizes) is a useful empirical demonstration of the frozen-backbone design’s practical value rather than a new architectural idea.
Field Significance
moderate — This paper contributes a low-cost recipe (frozen backbone plus lightweight adapters, trained on manufactured QA data) for adding speech I/O to vision-language models, showing that thousands of hours of speech data can achieve spoken-QA performance comparable to systems trained on millions of hours. Its main transferable insight is the QTATS construction strategy for sidestepping the need for dedicated spoken-QA recordings, and its transferability experiment provides empirical evidence that adapters trained against one backbone size can be cheaply reused at other backbone sizes.
Claims
- supports: Keeping a large pretrained multimodal backbone fully frozen while training only lightweight input and output adapter modules can reach spoken question-answering performance competitive with systems trained end-to-end on far larger speech corpora.
Evidence: With roughly 4,128 hours of ASR data and 1.45 million QTATS pairs, the 8B system attains spoken-QA accuracy comparable to or exceeding Qwen2.5-Omni-7B and Baichuan-Omni-1.5, both trained on millions of hours of speech, including the best LLaMA Questions S→T accuracy (85.0%) in the comparison. (§5.1, Table 2)
- supports: Spoken question-answering training data for a speech-generation module can be manufactured at scale from ordinary ASR corpora by having an LLM generate a plausible question for each transcript, removing the need for dedicated spoken-QA recording or TTS-synthesized QA corpora.
Evidence: QTATS converts ASR speech-text pairs into ~1.45M question-answer-speech triplets via LLM-driven reverse question generation, and the speech token generator trained solely on this synthetic supervision produces “robust speech generation behavior” without any genuine spoken QA corpus. (§3.2.1, §4.1)
- complicates: A speech-generation module conditioned on a frozen backbone’s hidden states does not generalize across conditioning regimes: supervision collected under one input style (e.g., ASR) does not transfer to another (e.g., QA), even when the underlying text content is similar.
Evidence: A generator trained solely under ASR-style conditioning “generalizes poorly to QA settings,” which the authors attribute to VL hidden states differing between ASR-style and QA-style conditioning despite similar text content, motivating the QTATS construction and an auxiliary text projector. (§3.2.1)
- supports: An adapter trained to interface a frozen speech tokenizer with one backbone size can be reused on backbones of different capacities by retraining only a small projection layer, with downstream performance improving as backbone size grows.
Evidence: A speech projector trained against Qwen3-VL-8B-Instruct is adapted to Qwen3-VL-4B-Instruct and Qwen3-VL-32B-Instruct by updating only the final linear layer, with LLaMA Questions S→T accuracy rising from 78.33% (4B) to 85.0% (8B) to 86.0% (32B). (§5.3, Table 2)
Limitations and Open Questions
The speech token generator is trained entirely on LLM-manufactured QA supervision rather than genuine spoken QA data, and the authors acknowledge this "manufactured supervision lacks genuine QA reasoning structure," producing an unavoidable training-inference mismatch in the VL hidden states that condition speech generation (§5.2).
Reduced ASR training data relative to large-scale omni-models leaves raw ASR accuracy noticeably behind systems like Qwen2.5-Omni and Baichuan-Omni-1.5 (§5.1, Table 2). The de-tokenizer dominates end-to-end latency (over 1 second of a roughly 1.35-second total for a 1-second response), which the paper reports but does not attempt to reduce further (§5.4, Table 4, Appendix C). The transferability experiments reuse the same VL model family (Qwen3-VL) at different sizes rather than testing transfer across distinct backbone architectures, so cross-family portability of the frozen-backbone design is untested.
Wiki Connections
- Spoken Language Model — extends a frozen vision-language backbone with speech input and output via lightweight adapters, an alternative to fine-tuning the whole backbone for speech modality integration.
- Speech-to-Speech — implements a full speech-in, speech-out dialogue pipeline following the Thinker-Talker paradigm, evaluated on speech-to-speech spoken QA.
- Self-Supervised Speech — its streaming discrete speech tokenizer is built on a HuBERT LARGE encoder pre-trained via self-supervised masked prediction.
- Neural Audio Codec — introduces a custom streaming FSQ-based discrete speech tokenizer and de-tokenizer pair operating at 12.5 Hz token rate.
- Flow Matching — its speech de-tokenizer is trained with a flow-matching objective, adapting F5-TTS’s DiT backbone into a cross-attention-conditioned CA-DiT variant.
- Multilingual TTS — trains and evaluates the system on a balanced mix of Chinese and English ASR corpora.
- MOSS-Speech — cited as related work on speech-to-speech models, contrasted as a text-guidance-free approach versus Speech-Omni-Lite’s Thinker-Talker design that still relies on text-mediated hidden states.
- Qwen3-Omni Technical Report — used as a large-scale omni-model comparison point demonstrating full multimodal input/output coverage that Speech-Omni-Lite’s frozen-backbone approach aims to approximate at much lower training cost.
- MiMo-Audio — cited among prior omni-model approaches to speech-language modeling that Speech-Omni-Lite contrasts against for training cost.
- X-Talk — cited as related work on modular speech-to-speech dialogue systems.
- DSA-Tokenizer — cited as a related discrete speech tokenization approach; Speech-Omni-Lite’s own tokenizer follows the same general training methodology lineage (ToneUnit/DSA-Tokenizer family).