ESPnet-SpeechLM: An Open Speech Language Model Toolkit

NAACL · 2025 · Conference

Jinchuan Tian et al. (Carnegie Mellon University) · → Paper · Demo: ✓ · Code: ✓

An open-source toolkit that unifies ASR, TTS, and general speech-language-model development under a single sequential-modeling workflow, with a fully transparent 1.7B-parameter multi-task model as its flagship demonstration.

Problem

Speech language models (speech LMs) have demonstrated rapid progress in zero-shot generalization, multi-task learning, instruction following, and real-time interaction, but building one remains a fragmented, resource-intensive effort. Existing open-source speech LM codebases (VoxtLM, UniAudio) cover only narrow subsets of tasks, while widely used systems such as Moshi, Mini-Omni, and GLM-4-Voice release inference code and weights but not training data, training code, or evaluation scripts. Text-LLM training frameworks (Megatron-LM, LLaMA-Factory) offer scalability but lack native support for speech features such as multi-stream audio tokenization. No existing toolkit offers an end-to-end, fully open pipeline spanning data preprocessing, multi-task pre-training, inference, and standardized evaluation for speech LMs.

Method

ESPnet-SpeechLM frames all speech-related tasks (ASR, TTS, machine translation, speech-to-speech translation, singing voice synthesis, speech enhancement, target speaker extraction, and others) as a single sequential modeling problem: conditions and targets are tokenized into discrete sequences and spliced into one stream that a decoder-only Transformer learns to predict autoregressively under a standard cross-entropy objective. Each task is declared via a “task template” that specifies the composition of the training sequence, which data items it draws from, and which tokenizer applies to each item; the toolkit then automates preprocessing, training, inference, and evaluation from this template without further code changes.

The toolkit natively supports multi-stream language models (sequences with an extra token-stream axis, needed because audio codecs encode each frame into multiple codebook tokens), with four interchangeable multi-stream architectures including VALL-E-style and delay-interleave (MusicGen-style) patterns. Either an ESPnet built-in Transformer or any HuggingFace AutoModelForCausalLM can serve as the backbone, allowing initialization from pre-trained text LLMs to preserve text capability. Audio is tokenized via configurable codec models (ESPnet-Codec, DAC, EnCodec, UniAudio codec) and/or self-supervised tokens (XEUS, S3PRL front-ends); the authors report that concatenating codec and SSL tokens frame-by-frame (“Codec_SSL”) works well for both understanding and generation tasks and use this scheme for their headline experiments. Training efficiency relies on DeepSpeed, FlashAttention, and Liger-Kernel, reaching up to 30% model FLOPs utilization on multi-node H100 training; the toolkit also supports RLHF fine-tuning for speech LMs. Inference supports greedy search, beam search, and top-k/top-p sampling, with a modality-tracking heuristic that filters out cross-modality token predictions. Evaluation is standardized through VERSA, a bundled collection of over 60 speech and audio metrics.

The flagship use case is a 1.7B-parameter multi-task model initialized from SmolLM2, pre-trained jointly on ASR, TTS, TextLM, and AudioLM objectives over roughly 200k hours of open-source English speech (Emilia, Yodas, OWSM English subsets, LibriSpeech, MLS) and 115B text tokens, totaling 240B effective text tokens or audio frames across two data passes on 24 H100 GPUs.

Key Results

On English ASR, a 442M-parameter single-task ESPnet-SpeechLM model achieves an average WER of 5.4% across LibriSpeech-clean/other, MLS, TEDLIUM3, WSJ, and FLEURS, matching OWSM v3.1-medium (1.02B params, 5.4%) and outperforming Whisper-medium (769M, 5.7%) and Whisper-small (244M, 6.4%) despite using fewer parameters than the larger comparators (Table 3). On zero-shot TTS over full LibriSpeech test-clean, the system reaches WER 3.1%, SPK-SIM 0.55, and Proxy MOS 4.03, outperforming CosyVoice (WER 5.0, SPK-SIM 0.51, Proxy MOS 4.15), Parler-TTS, ChatTTS, WhisperSpeech, and third-party VALL-E/VALL-E-2 implementations on WER and SPK-SIM, though CosyVoice still leads on Proxy MOS (Table 4).

The 1.7B multi-task model achieves ASR WER 2.8/5.9% (clean/other), TTS WER 6.0%, SPK-SIM 0.701, and Proxy MOS 3.99, while preserving TextLM benchmark performance (MMLU 30.5, ARC-C 41.3, HellaSwag 50.4) close to the text-only LLaMA-3.2-1B baseline (MMLU 32.2, ARC-C 32.8, HellaSwag 41.2) despite being trained jointly on four tasks (Table 5). Comparisons against Moshi (7B), VITA (8x7B), and GLM-4-Voice (9B) show the 1.7B model achieving competitive or better ASR and comparable text-capability retention at a fraction of the parameter count, though many of those numbers are drawn from the competitors’ own reports rather than reproduced under identical conditions.

Novelty Assessment

The contribution is an engineering integration: ESPnet-SpeechLM does not propose a new architecture, tokenization method, or training objective. Its multi-stream LM variants (VALL-E-style, delay-interleave from MusicGen, UniAudio’s MultiScale-Transformer) and its codec and SSL tokenizers are all adopted from prior work. The genuine contribution is consolidating a previously fragmented stack (heterogeneous codecs, multi-stream architectures, multi-task data pipelines, and a 60+ metric evaluation suite) into one configurable, fully open pipeline, and demonstrating with Table 1 that no prior open-source speech LM codebase matches its combination of full open-sourcing across data, training, inference, evaluation, and weights simultaneously. The reported model results are competitive but not state-of-the-art on any individual benchmark; they function as a validation that the toolkit produces usable models rather than as a new performance frontier. The “Codec_SSL” frame-concatenation tokenization choice is presented as an empirical finding but is not systematically ablated against codec-only or SSL-only tokenization in the paper.

Field Significance

Moderate — this paper provides an open, end-to-end toolkit covering the full speech-LM development lifecycle (data preprocessing, multi-task pre-training, inference, and standardized evaluation via VERSA), filling a gap left by prior codebases that release only a subset of these components. Its value is primarily infrastructural rather than scientific: it lowers the barrier to building competitive speech LMs and provides a reproducible reference implementation with released recipes, datasets, and pre-trained weights, but the paper does not introduce a new model family or establish a new performance ceiling.

Claims

• supports: A speech language model can be initialized from a pre-trained text LLM and jointly trained on speech recognition, speech synthesis, text continuation, and audio continuation without substantially degrading the text-only capability of the base model.

  Evidence: The 1.7B multi-task model trained on ASR, TTS, TextLM, and AudioLM objectives scores MMLU 30.5, ARC-C 41.3, and HellaSwag 50.4, close to the text-only LLaMA-3.2-1B baseline (32.2, 32.8, 41.2) despite carrying three additional speech tasks. (§4.3, Table 5)

• supports: Concatenating neural codec tokens with self-supervised speech representations frame-by-frame is a viable tokenization strategy for both speech understanding and generation tasks within a single sequential model.

  Evidence: The “Codec_SSL” scheme (ESPnet-Codec combined with XEUS SSL tokens) is used for the headline ASR and TTS experiments and the paper reports it “behaves well in both speech understanding and generation” (§3.3).

• supports: A decoder-only autoregressive speech language model can match or exceed dedicated, larger ASR-only systems on English benchmarks while using substantially fewer parameters.

  Evidence: A 442M-parameter ESPnet-SpeechLM ASR model reaches average WER 5.4% across six English test sets, matching OWSM v3.1-medium (1.02B, 5.4%) and beating Whisper-small (244M, 6.4%) and Whisper-medium (769M, 5.7%). (§4.2, Table 3)

• complicates: Cross-system comparisons of speech language models reported in the literature are frequently not run under matched conditions, limiting how much can be concluded from any single performance table.

  Evidence: In the multi-task comparison (Table 5), competitor numbers for Moshi, VITA, GLM-4-Voice, and others are taken directly from their own published reports rather than reproduced by the authors, and the paper explicitly flags this with footnote markers. (§4.3, Table 5)

• complicates: Combining a codec tokenizer with a self-supervised tokenizer frame-by-frame is reported as an effective design choice but is not validated against single-tokenizer ablations in the same controlled setting.

  Evidence: The claim that Codec_SSL tokenization “behaves well” rests on a single line of justification without a paired ablation against codec-only or SSL-only tokenization on the same task and dataset. (§3.3)

Limitations and Open Questions

The multi-task model’s TTS quality (Proxy MOS 3.99, WER 6.0%) is noticeably weaker than the single-task TTS model trained on the same architecture (Proxy MOS 4.03, WER 3.1%), indicating a capacity or interference cost to joint multi-task training that the paper reports but does not analyze further. Most training and evaluation data is English-only (the multilingual text corpus is used only for the TextLM objective, not for speech tasks), so the demonstrated speech capabilities are not evidence of multilingual generalization. Competitor numbers in the multi-task comparison table are drawn from third-party reports under unmatched training data and conditions rather than reproduced by the authors, which the paper itself notes. As a system/demo paper, the contribution is the toolkit and its reference recipes rather than a novel architecture or training method; the headline numbers serve to validate functionality rather than push state of the art on any individual benchmark.

Wiki Connections

• Spoken Language Model — provides an open, end-to-end toolkit for building decoder-only speech LMs across ASR, TTS, and joint multi-task settings, addressing the lack of fully open training pipelines in the speech LM ecosystem.
• Autoregressive Codec TTS — implements and benchmarks multiple autoregressive multi-stream TTS architectures (VALL-E-style, delay-interleave) within a single configurable framework.
• Neural Audio Codec — supports multiple interchangeable codec tokenizers (ESPnet-Codec, DAC, EnCodec, UniAudio codec) and introduces a frame-by-frame codec+SSL concatenation tokenization scheme.
• Zero-Shot TTS — demonstrates zero-shot speaker-prompted TTS with SPK-SIM and Proxy MOS evaluation against CosyVoice, VALL-E, and VALL-E-2 baselines.
• Self-Supervised Speech — uses XEUS self-supervised tokens concatenated with codec tokens as a core tokenization strategy for its speech LM.
• RLHF Speech — includes built-in support for reinforcement learning from human feedback fine-tuning of speech language models as a configurable training option.
• VALL-E — implements VALL-E’s multi-stream autoregressive architecture as one of four supported multi-stream language model designs and compares against a third-party VALL-E implementation in the TTS benchmark.
• VALL-E 2 — used as a third-party zero-shot TTS baseline on LibriSpeech test-clean, underperforming the proposed system on WER and SPK-SIM.
• CosyVoice — used as the strongest TTS baseline in the zero-shot comparison, outperforming the proposed system on Proxy MOS while trailing it on WER and SPK-SIM.
• Moshi — cited as an open-source speech LM that releases inference code and weights but not full training data or code, motivating ESPnet-SpeechLM’s emphasis on end-to-end openness; also used as a multi-task comparison point.
• EnCodec — supported as one of several interchangeable neural codec tokenizers within the toolkit.
• GLM-4-Voice — compared against in the multi-task evaluation table on ASR and TTS metrics, and cited in Table 1 as a partially open-sourced speech LM codebase.
• UniAudio — its MultiScale-Transformer multi-stream architecture is one of the four supported multi-stream language model implementations, and it is compared against in Table 1’s codebase feature comparison.
• Whisper — used as the primary ASR baseline (small and medium variants) for evaluating the proposed system’s recognition accuracy.