EMNLP · 2025 · Conference
Chen Wang et al. (Institute of Automation, Chinese Academy of Sciences / University of Chinese Academy of Sciences) · → Paper · Demo: ✓ · Code: ✓
OpenS2S is a fully open-source, end-to-end large speech language model for empathetic speech-to-speech interaction, featuring a streaming interleaved decoding architecture and an automated data construction pipeline that synthesizes bilingual empathetic speech dialogues using LLMs and controllable TTS.
Problem
The most capable empathetic speech language models (GPT-4o, Kimi-Audio, GLM-4-Voice) are either closed-source, require millions of hours of pre-training data, or fail to release training data and code. This opacity prevents scientific study of empathetic behaviors, bias analysis, and reproducibility. Existing open-source models either ignore paralinguistic information (emotion, prosody) or achieve empathy only through resource-intensive pre-training. There is also no good automated pipeline for constructing diverse, emotionally annotated speech dialogue data at low cost.
Method
Architecture. OpenS2S has four components:
- Audio Encoder: Qwen2-Audio encoder extracts features at 25Hz; a speech adapter (2 CNN downsampling layers + FFN) reduces to 6.25Hz continuous representations.
- LLM: Qwen3-8B-Instruct; audio and text embeddings are concatenated as input.
- Streaming Speech Decoder: Initialized from Qwen3-1.8B, vocabulary extended by 16,384 speech tokens. Receives LLM final hidden states through a linear projection layer and generates speech tokens via interleaved decoding: for every M=4 LLM hidden states consumed, N=8 speech tokens are generated. Speech tokenizer is GLM-4-Voice’s Whisper-large-v3-based quantizer (12.5 tokens/sec). Cross-entropy loss is computed only on generated speech tokens.
- Token2Wav Decoder: Chunk-aware causal flow matching model (streaming, mel-spectrogram synthesis) + HiFi-GAN vocoder, both from GLM-4-Voice.
Three-stage training:
- Stage 1 (Speech Understanding Pre-training): Semantic alignment (BLSP-Emo behavioral alignment paradigm on ASR datasets) + emotional alignment (SER datasets). Only speech adapter trained; audio encoder and LLM frozen. Datasets: ~1.9M EN + comparable ZH ASR pairs, ~70k SER utterances.
- Stage 2 (Speech Generation Pre-training): TTS offline pre-training (vocab expansion + text-to-speech mapping), then interleaved streaming training. LM frozen; linear projection + speech decoder trained. ~10k hours Emilia data.
- Stage 3 (Empathetic Instruction Tuning): Speech encoder frozen; all other components fine-tuned on empathetic + general speech instruction data.
Automated empathetic data construction pipeline:
- 1,000 EN + 1,000 ZH seed audio samples from SER datasets, manually annotated with text, gender, age, emotion.
- Qwen3-32B-Instruct generates paralinguistic-sensitive instructions (e.g., age-sensitive queries) + voice cloned via CosyVoice2 → 50k EN + 50k ZH speech instructions.
- Qwen3-32B-Instruct generates empathetic text responses + emotion label for delivery → CosyVoice2 synthesizes speech responses.
- Additional 100k general bilingual speech-to-speech pairs from InstructS2S200K.
Key Results
On VoiceBench and URO-Bench (Table 2), compared to Qwen-2-Audio, GLM-4-Voice, Kimi-Audio, and LLaMA-Omni2:
- VoiceBench alpacaeval: OpenS2S (4.09) ranks second to Kimi-Audio (4.46); beats GLM-4-Voice (3.97) and LLaMA-Omni2 (3.96).
- VoiceBench ifeval: OpenS2S (42.89) substantially outperforms LLaMA-Omni2 (17.36) and GLM-4-Voice (25.92), but trails Kimi-Audio (61.1).
- URO-Bench UnderEmotion-en: OpenS2S (46.9) is competitive with Kimi-Audio (59.22) and beats LLaMA-Omni2 (39.46), using significantly less training data than Kimi-Audio.
- URO-Bench UnderEmotion-zh: OpenS2S (67.68) is close to all comparators.
OpenS2S is uniquely fully open (training data + pre-training code + fine-tuning code + model weights), in contrast to all compared models which release only some subset.
Novelty Assessment
The primary novelty is the combination of: (a) full open-source release including training data and code, (b) an automated empathetic speech dialogue data synthesis pipeline that avoids the need for large-scale human annotation, and (c) a streaming interleaved decoding architecture for low-latency empathetic speech generation. The architecture itself closely follows prior work — the interleaved decoding framework is inspired by Minmo and LLaMA-Omni2, the speech tokenizer and token2wav decoder are reused from GLM-4-Voice, and the audio encoder is Qwen2-Audio. The contribution is primarily systems and reproducibility-enabling rather than architectural novelty. The data pipeline is the most original technical element.
Field Significance
Moderate — OpenS2S provides the research community’s first fully open-source spoken conversational agent with explicit empathy support, releasing training data, pre-training code, fine-tuning code, and model weights that comparable systems withhold. Its technical architecture is assembled from existing components (GLM-4-Voice tokenizer, Qwen2-Audio encoder, interleaved decoding from Minmo and LLaMA-Omni2), making its primary contribution one of reproducibility and the automated empathetic dialogue data construction pipeline rather than architectural novelty.
Claims
- Competitive empathetic spoken conversational agent performance can be achieved with substantially less training data than resource-intensive proprietary systems, when combined with a targeted automated data construction pipeline. (§4.1, Table 2)
- Automated synthesis of empathetic speech dialogue data using LLMs for content generation and controllable TTS for speech rendering provides a scalable alternative to large-scale human annotation in building spoken conversational agent training corpora. (§3.2)
- Interleaved decoding, where speech tokens are generated in fixed-ratio bursts relative to consumed LLM hidden states, enables streaming speech generation in end-to-end spoken conversational agents without a separate parallel decoding module. (§2.1)
- Open-source release of training data and code for empathetic spoken conversational agents remains rare, despite broad availability of model weights across the ecosystem. (§1, Table 1)
Limitations and Open Questions
- Performance still trails Kimi-Audio on all evaluated benchmarks; the data volume gap (less data = lower ceiling) is acknowledged.
- The model simulates empathy through learned associations rather than cognitive understanding; subtle emotional nuances may be missed.
- Output speech emotion is constrained to a fixed young-female voice for all responses in the current SFT dataset, limiting response speaker diversity.
- Cultural and cross-linguistic empathy variation is not addressed; the model may exhibit culturally specific empathetic patterns.
- No human subjective evaluation of speech naturalness (MOS) is reported; evaluation is benchmark-based.
- Potential bias inheritance from the underlying Qwen3-32B-Instruct data is unanalyzed.
Wiki Connections
- spoken-language-model — OpenS2S contributes a fully open-source empathy-capable end-to-end system, with an automated data construction pipeline relevant to data scaling discussions
- speech-to-speech — direct contribution: end-to-end streaming speech-in, speech-out dialogue
- streaming-tts — the streaming interleaved decoding architecture links to this concept
- emotion-synthesis — empathetic speech response generation with emotion conditioning is a core design element
- instruction-conditioned-tts — CosyVoice2 is used with instruction prompts to synthesize emotionally expressive speech responses
- OpenOmni — in-corpus reference; contemporaneous open-source omnimodal speech LLM
- URO-Bench — in-corpus reference; benchmark used for empathy evaluation
- VocalNet — contemporaneous approach to speech LLM efficiency; where OpenS2S targets empathy and open-source reproducibility, VocalNet targets data efficiency and WER via MTP; complementary SCA advances from EMNLP 2025
- SpeechLM survey — contextualizes OpenS2S’s architecture within the broader aligned multimodal LLM paradigm