arXiv · 2023 · Preprint
Tang et al. (Tsinghua University / ByteDance) · → Paper · Demo: ? · Code: ✓
SALMONN connects dual auditory encoders (Whisper for speech and BEATs for non-speech audio) to a frozen Vicuna LLM via a window-level Q-Former, enabling a single model to perform speech recognition, audio captioning, emotion recognition, music understanding, and emergent cross-modal reasoning tasks from audio input.
Citation Stub
This paper is not a speech generation paper but is cited by the corpus. See Context in Speech Generation below for why it is relevant.
Context in Speech Generation
SALMONN provides a general-purpose audio-language model architecture that can process speech, audio events, and music within a single LLM-based framework. Its dual-encoder design, pairing Whisper’s supervised speech representations with BEATs’ self-supervised audio semantics, enables coverage of both speech-specific and general audio tasks without requiring separate specialist models. The window-level Q-Former bridge adapts the variable-length audio encoder outputs into a fixed token vocabulary compatible with instruction-tuned LLMs. The paper also introduces an activation tuning procedure that uses few-shot self-supervised data generation to unlock emergent cross-modal abilities such as speech-audio co-reasoning and audio-based storytelling, which are absent after standard instruction tuning alone. Spoken conversational agent and speech LM research can draw on SALMONN’s approach for audio understanding components, particularly its conditioning scheme for feeding diverse audio representations into LLM inference pipelines.
Wiki Connections
- spoken-language-model — SALMONN’s dual-encoder + Q-Former architecture for audio-conditioned LLM inference informs the design of speech LMs that combine SSL and supervised audio encoders with a text LLM backbone
- self-supervised-speech — BEATs, a self-supervised audio representation model, serves as one of SALMONN’s two encoder branches, contributing non-speech semantic understanding
- 2301.11325 (Whisper) — serves as SALMONN’s speech-specialist encoder for ASR-quality phonetic and linguistic representations
- 2303.08774 (GPT-4) — cited as an upper-bound reference for instruction-following capability across audio reasoning tasks
- 2306.12925 — cited as a related audio-LLM baseline in the evaluation comparison
- 2302.13971 (LLaMA) — underlying architecture family used by the Vicuna backbone
- 2305.11000 (SpeechGPT) — cited as an early speech LM for contextual comparison