ACL 2026 (Findings) · 2026 · Conference
Yan Zhou et al. (Institute of Computing Technology, Chinese Academy of Sciences / University of Chinese Academy of Sciences) · → Paper · → Preprint · Demo: ? · Code: ✓
CSLM builds a Chinese-English speech LLM from Llama-3.1-8B-Instruct with about 77k hours of paired speech-text data, using continual pre-training on ASR, TTS and machine-translation tasks plus a chunk-level interleaved text-speech chain-of-modality for fine-tuning, which enables cross-lingual spoken dialogue and cuts response latency by ×1.92 to ×4.62 versus generating the full text answer first.
Problem
Speech LLMs that jointly model discrete speech tokens and text, such as GLM-4-Voice and Moshi, rely on millions of hours of paired speech (Table 4), and the paper notes that speech data is scarce for many languages. In the paper’s framing, modular encoder-plus-LLM systems lack intrinsic speech-text alignment and are weaker at generation. Existing token-based speech LLMs are also mostly evaluated monolingually, and the authors argue their cross-lingual alignment is limited, so tasks such as answering a Chinese spoken question in English speech are hard. The paper asks whether cross-modal and cross-lingual alignment can be learned together from limited paired data.
Method
CSLM has three parts. The speech tokenizer is the supervised semantic tokenizer of CosyVoice (CosyVoice-300M-25hz: Conformer encoder plus vector quantization, 4,096 tokens at 25 Hz). Consecutive repeated tokens are merged before they enter the LLM. The LLM is Llama-3.1-8B-Instruct with its vocabulary extended by the 4,096 speech tokens, following SpeechGPT. The speech decoder reuses the CosyVoice conditional flow-matching model and HiFi-GAN vocoder, with one new component: a two-layer convolutional duration predictor that re-expands the merged token sequence. Only the duration predictor is trained on the decoder side (on LJSpeech-1.1 and the Baker Mandarin corpus); the flow-matching model and vocoder are taken from CosyVoice.
Alignment is designed as a hub: within each language, speech aligns to text; across languages, alignment passes through text only (§3.2). Training has two full fine-tuning stages. Continual pre-training mixes ASR and TTS instruction data in English (MLS English, GigaSpeech) and Chinese (WenetSpeech for ASR, WenetSpeech4TTS for TTS), Chinese-English WMT17 translation data, and Infinity-Instruct text instructions to limit text-ability loss. This totals 6.1B speech tokens and 3.9B text tokens (Table 3). Supervised fine-tuning then trains on mono-lingual and cross-lingual speech-to-speech instruction data, replaying text instructions and translation data. All SFT speech is synthesized with CosyVoice-300M-SFT, with random input timbres from Fish-Speech 1.5 and a fixed response timbre (Appendix C).
The fine-tuning output format is a speech-text interleaved chain-of-modality. The original chain-of-modality from SpeechGPT generates text question, full text answer, then full speech answer. CSLM instead emits the text question, then alternates short text chunks with their matching speech tokens. Chunk boundaries come from CTC forced alignment between the synthesized response audio and its text (wav2vec 2.0 for English, SenseVoice Small for Chinese), cutting at punctuation unless a segment is under 7 words (§3.3.2). At inference, each speech chunk can be decoded and played while the LLM generates the next chunk, which is how the method reduces response latency (§3.4, Appendix B).
Key Results
Data scale. CSLM uses 77k hours of paired speech-text data, against 11k+ (SpeechGPT), 57k (AnyGPT), 7,000k+ (Moshi) and 10,000k+ (GLM-4-Voice) hours (Table 4).
Base-model ASR and TTS (Tables 1, 2). On English TTS, CSLM-base reaches WER 3.2 on LibriSpeech test-clean, 3.2 on LibriTTS and 2.8 on VCTK. This is lower than SpeechGPT (24.6 / 29.1 / 5.7), AnyGPT (27.1 / 27.9 / 8.5), Moshi on LibriSpeech (4.7) and GLM-4-Voice on LibriTTS (5.6). Against the specialized CosyVoice-300M-SFT it is lower on LibriSpeech (3.2 vs 3.4) and VCTK (2.8 vs 3.7) but higher on LibriTTS (3.2 vs 2.9). On Chinese TTS, CosyVoice is better on all three AISHELL sets (CER 3.3 / 5.0 / 3.8 vs CSLM-base 3.8 / 5.3 / 4.9). ASR is the weaker side: on LibriSpeech CSLM-base gets WER 6.7 versus 2.8 for GLM-4-Voice, 5.7 for Moshi and 2.5 for Whisper-large-v3, and on AISHELL-1 CER 8.6 versus 2.5 for GLM-4-Voice. SFT raises English ASR WER from 6.7 to 9.8 and LibriTTS TTS WER from 3.2 to 4.5. The paper describes the base model as comparable to Moshi and GLM-4-Voice; on ASR the gap is substantial.
Monolingual spoken dialogue (Table 5). On InstructS2S-Eval (En), CSLM has the highest UTMOS among the compared systems (4.4 vs GLM-4-Voice 4.0, SpeechGPT 3.9, AnyGPT 3.3), the lowest speech-text ASR-WER (9.0 vs 12.8 and 45.0), and 0.0 off-target ratio. Content quality, scored by GPT-4o, is below GLM-4-Voice (text 3.50 vs 4.10; speech 3.27 vs 4.02) but above SpeechGPT and AnyGPT. On BELLE-eval-S2S (Zh), GLM-4-Voice leads on GPT score (4.80 / 4.70 vs 3.78 / 3.37), ASR-CER (4.6 vs 6.9) and off-target ratio (text 0.4 vs 0.8; speech 4.8 vs 5.2).
Cross-lingual spoken dialogue (Table 6). For Chinese questions answered in English, GLM-4-Voice mostly answers in the wrong language (off-target 97.2% text, 96.8% speech; GPT score 1.20 / 1.16), while CSLM stays on target (1.6% / 0.8%) with GPT scores 3.53 / 3.20 and ASR-WER 7.4 vs 42.6. For English questions answered in Chinese, GLM-4-Voice is better on content (4.29 / 4.23 vs 3.31 / 2.95) and ASR-CER (5.5 vs 17.5), though CSLM again has lower off-target ratios (1.0 / 0.5 vs 8.5 / 8.5).
Human evaluation (Table 7). Human ratings of content (C-MOS) favour GLM-4-Voice for En→En (4.17 vs 3.25), En→Zh (3.50 vs 3.42) and Zh→Zh (4.42 vs 4.00), and favour CSLM for Zh→En (4.33 vs 3.08). Acoustic quality (A-MOS) is 4.11 for GLM-4-Voice, 4.00 for CSLM and 3.22 for SpeechGPT. The A-MOS ordering of CSLM and GLM-4-Voice is the reverse of their UTMOS ordering in Table 5, although the paper states the human ratings align with UTMOS. The number of raters and rated samples is not reported.
Ablations. Hidden-state similarity between parallel speech and text on LibriSpeech is 72.5% for CSLM vs 39.4% for GLM-4-Voice and 1.2% for SpeechGPT, but random speech-text pairs also score 56.7% for CSLM (Table 8). Removing MT data from continual pre-training lowers cross-lingual GPT scores (En→Zh text 3.31 to 3.00; Zh→En 3.53 to 3.08) and raises ASR-ER (17.5 to 25.4; 7.4 to 9.2), while the Zh→En off-target ratios are slightly lower without MT (1.2 / 0.4 vs 1.6 / 0.8) (Table 9). In Table 10, the interleaved format cuts latency against full chain-of-modality by ×2.87 (En→En), ×1.92 (Zh→Zh), ×4.62 (En→Zh) and ×2.30 (Zh→En), an average of ×2.93 (latency values are labelled in seconds, e.g. 466.46 vs 1338.68 for En→En). Its quality gain over full chain-of-modality is not uniform: full chain-of-modality has lower ASR-ER in three of four directions (En→En 8.5 vs 9.0, Zh→Zh 6.1 vs 6.9, En→Zh 11.8 vs 17.5) and a higher Zh→Zh speech GPT score (3.42 vs 3.37). Chunk size 4 degrades all content scores, and dropping the text question collapses GPT scores (e.g. Zh→En 1.22 vs 3.53).
The Chinese dialogue test inputs and all SFT speech are synthesized with the same CosyVoice-300M-SFT model, so the dialogue evaluation does not cover natural recorded questions in Chinese.
Novelty Assessment
The contribution is a training recipe assembled from known parts. Vocabulary expansion with discrete speech tokens and the chain-of-modality format come from SpeechGPT; the tokenizer and decoder come from CosyVoice; interleaved text-speech output is already used by GLM-4-Voice. The new elements are the use of machine-translation data in continual pre-training to route cross-lingual alignment through text, CTC-aligned chunk-level interleaving (rather than word-level or fixed-ratio interleaving), and the duration predictor that lets the LLM model deduplicated tokens. The most distinctive result is cross-lingual: on Zh→En spoken dialogue CSLM answers in the requested language where GLM-4-Voice almost never does. The “language scalability” claim (§3.5) is argued, not tested beyond Chinese and English.
Field Significance
Moderate — The paper shows that a speech LLM trained on roughly 77k hours of paired data, about one percent of what GLM-4-Voice reports, can reach competitive TTS intelligibility and controllable cross-lingual spoken responses when translation data anchors alignment through text. It provides an ablated recipe (MT data, chunk granularity, text question, interleaving vs full chain-of-modality) but remains well behind a large-data model on content quality and ASR in most directions.
Claims
- supports: Including text machine-translation data in continual pre-training of a speech-text LLM improves cross-lingual spoken response quality and the intelligibility of generated speech.
Evidence: Removing WMT17 zh-en data lowers GPT scores on En→Zh (3.31 to 3.00 text) and Zh→En (3.53 to 3.08 text) and raises ASR-ER (17.5 to 25.4; 7.4 to 9.2), though Zh→En off-target ratios are slightly lower without it. (§6.2, Table 9)
- supports: Interleaving short text chunks with their speech tokens, instead of generating the complete text answer before any speech, sharply reduces spoken-response latency in text-guided speech LLMs.
Evidence: Chunk-level interleaving gives ×1.92 to ×4.62 speedups (average ×2.93) over full chain-of-modality across four language directions, with comparable GPT scores but higher ASR-ER in three of four directions. (§6.3, Table 10)
- complicates: Interleaved text-speech generation is sensitive to alignment granularity; chunks that are too short damage both content and speech-text consistency.
Evidence: Chunk size 4 instead of a minimum of 7 words drops En→Zh GPT score from 3.31 to 1.62 and raises ASR-ER from 17.5 to 34.7, with almost no extra latency saving (×4.65 vs ×4.62). (§6.3, Table 10)
- supports: Speech LLMs trained with much less paired speech can match or exceed large-data speech LLMs on TTS intelligibility while still trailing them on speech recognition and dialogue content quality.
Evidence: With 77k hours of paired data, CSLM-base reaches TTS WER 3.2 on LibriTTS vs 5.6 for GLM-4-Voice, but ASR WER 6.7 on LibriSpeech vs 2.8, and lower GPT and human content scores than GLM-4-Voice in three of four dialogue directions. (§5.1, §5.2, Tables 1, 4, 5, 7)
- complicates: Automatic naturalness predictors and human acoustic ratings can rank speech LLM outputs differently.
Evidence: UTMOS ranks CSLM above GLM-4-Voice on English responses (4.4 vs 4.0), whereas human A-MOS ranks GLM-4-Voice higher (4.11 vs 4.00). (§5.2.1, §5.2.2, Tables 5, 7)
Limitations and Open Questions
Warning
Only Chinese and English are trained and evaluated. The claim that the recipe scales to further languages (§3.5) rests on argument, not experiment, and the paper’s own Limitations note that more languages are still needed (§Limitations).
- The Chinese dialogue test set (BELLE-eval-S2S) and all SFT speech are synthesized with CosyVoice-300M-SFT, and responses use a single fixed timbre, so robustness to natural, noisy or accented spoken input is not measured.
- The human evaluation reports neither the number of raters nor the number of rated samples, and its acoustic ranking disagrees with UTMOS.
- The “latency” in Table 10 is not defined further than its column label, so the absolute values are hard to interpret; the speedup ratios are the usable result.
- UTMOS is reported only for English, so Chinese speech naturalness is not measured automatically.
- SFT degrades the base model’s ASR and English TTS intelligibility (Table 1), and the paper does not study how to limit this.
- The paper notes that data and compute limits prevented training at larger scale (§Limitations).
Wiki Connections
- Spoken Language Model — CSLM extends a text LLM’s vocabulary with discrete speech tokens so it consumes and produces speech directly.
- Speech-to-Speech — Evaluates end-to-end monolingual and cross-lingual spoken dialogue, answering in the language requested rather than the input language.
- Multilingual TTS — The system is trained on Chinese and English and its generated speech is evaluated in both (TTS WER/CER on LibriSpeech, LibriTTS, VCTK and AISHELL; ASR-ER of dialogue responses in both languages).
- Streaming TTS — Chunk-level interleaved text-speech generation lets audio playback start before the full response is generated, with latency measured against full chain-of-modality.
- Subjective Evaluation — Reports double-blind human content and acoustic MOS against SpeechGPT and GLM-4-Voice, which disagree with UTMOS on acoustic ranking.
- SpeechGPT — Source of the vocabulary-expansion approach and the original chain-of-modality that CSLM makes interleaved; also a baseline.
- GLM-4-Voice — Main baseline; stronger on content, ASR and most human ratings, but fails on Zh→En spoken dialogue by answering in Chinese.
- CosyVoice — Provides the 25 Hz speech tokenizer, the flow-matching decoder, the TTS baseline, and the TTS model used to synthesize training and test speech.
- Llama 3 — Llama-3.1-8B-Instruct is the backbone LLM that CSLM continually pre-trains.
- HiFi-GAN — The inherited CosyVoice decoder uses a HiFi-GAN vocoder for waveform synthesis.
- Moshi — Baseline on English ASR and TTS and a reference point for large-data training scale.
- LLaMA-Omni — Source of the InstructS2S-200K instruction text, the InstructS2S-Eval set and the GPT-4o scoring protocol.
- UTMOS — Automatic naturalness predictor used for English speech responses.