EMNLP · 2025 · Conference
Pengchao Feng et al. (Shanghai Jiao Tong University) · → Paper · Demo: ? · Code: ✓
Proposes an end-to-end retrieval-augmented generation framework that embeds spoken queries directly into a shared speech-text vector space, letting speech-to-speech dialogue systems retrieve external textual knowledge without an intermediate ASR transcription step.
Problem
End-to-end speech-to-speech dialogue models (e.g., Mini-Omni, LLaMA-Omni, SLAM-Omni) respond directly to spoken input with spoken output, but they lack a mechanism for consulting external knowledge sources, which limits their accuracy on questions requiring up-to-date or domain-specific information. Retrieval-Augmented Generation (RAG) addresses this gap in text-based LLMs, but porting RAG to the speech modality is hindered by a modality gap: retrieval indices are built over text, while the query arrives as speech. The straightforward fix, transcribing the query with ASR before retrieving, adds latency from running a high-accuracy ASR model such as Whisper, which is a problem for real-time voice assistants. The paper asks whether retrieval accuracy and latency can be better balanced by retrieving directly from speech.
Method
The system has three stages. In indexing, source text documents are chunked and each chunk is embedded with a text encoder into a shared multimodal vector space, then stored with its text in a database. In retrieval, a spoken query is embedded directly with a speech encoder drawn from the same multimodal embedding model, and the top-k database chunks by cosine similarity to the query embedding are returned, with no intermediate transcription step. In generation, the retrieved chunks and the original spoken query are combined via a RAG prompt template (adapted from LangChain) and passed to an existing end-to-end speech-to-speech dialogue model, which produces the spoken response. The framework treats the dialogue model as a fixed, off-the-shelf component: no new generation architecture or generation-side training is introduced. The shared embedding space is provided by SONAR, a multi-encoder framework whose speech encoder is distilled to match a text-encoder teacher via an MSE alignment loss, so the speech encoder can be used for retrieval without speech tokenization. Experiments use GLM-4-Voice as the downstream speech-to-speech generator; no fine-tuning of GLM-4-Voice or SONAR is performed by the authors. The top 4 retrieved chunks are used per query in all experiments.
Key Results
On HotpotQA (English) and RGB (Chinese), the end-to-end retrieval system cuts retrieval latency to roughly one-fourth of the ASR-based cascade (about 0.08s vs. 0.3-1.2s), while retrieval F1 against ground-truth facts is 1-4 points lower than the cascade using various text embedding models (OpenAI, BCE, M-E5) (Table 1). Downstream answer accuracy improves substantially over a no-retrieval baseline (about 20% absolute on HotpotQA, about 43% on RGB-zh), but trails the ASR-cascade RAG systems by roughly 9 points on HotpotQA and roughly 15 points on RGB-zh, and trails an oracle-knowledge topline further still (Table 1). An ablation that swaps the SONAR speech encoder for an ASR (Whisper) + SONAR text-encoder pipeline yields nearly identical accuracy to the end-to-end system, indicating the accuracy gap versus ASR-cascade RAG stems mainly from the limited semantic alignment quality of the SONAR text encoder itself, not from using a speech encoder in place of ASR. Replacing SONAR with CLAP as the shared encoder degrades accuracy further, which the authors attribute to CLAP being optimized for coarse speech-text alignment rather than fine-grained text semantics (Table 2).
Novelty Assessment
The contribution is an engineering framework, not a new model architecture or training method: it wires a pretrained multimodal embedding model (SONAR) into a RAG pipeline ahead of an unmodified, pretrained end-to-end speech-to-speech dialogue model (GLM-4-Voice). The authors acknowledge contemporaneous work (SpeechRAG, SEAL, WavRAG) explores the same shared-embedding-space idea for speech retrieval; they position their distinguishing contributions as an explicit latency/accuracy trade-off analysis against ASR-cascade RAG and multilingual (English/Chinese) experiments. The released TTS-synthesized speech versions of HotpotQA and RGB-zh, plus the accompanying code, are the paper’s most concrete lasting artifacts, since the retrieval mechanism itself reuses an existing off-the-shelf encoder.
Moderate — the paper is a clear, honestly-reported empirical demonstration that end-to-end speech retrieval trades a bounded accuracy loss for a substantial latency win versus ASR-cascade RAG, and it isolates the source of that accuracy loss (text-encoder semantic alignment quality) via a controlled ablation. It does not advance the RAG framework itself (only the base LangChain-style pipeline is tested) or propose a new speech-text embedding model, so its significance is bounded to establishing this trade-off empirically for the S2S RAG setting rather than closing it.
Claims
- supports: Encoding a spoken query directly into a shared speech-text embedding space for retrieval can substantially reduce end-to-end retrieval latency compared to an ASR-then-retrieve cascade, without requiring speech tokenization.
Evidence: End-to-end retrieval with a SONAR speech encoder reduced measured retrieval time to about 0.08s from about 0.3-1.2s for ASR-based cascade retrieval on both HotpotQA (en) and RGB (zh), roughly a 4x speedup. (§4.3, Table 1)
- complicates: Retrieving directly from a speech query, bypassing ASR, currently incurs a downstream answer-accuracy cost relative to ASR-based text retrieval, because the accuracy gap traces to the quality of the shared text embedding space rather than to the use of a speech encoder per se.
Evidence: End-to-end RAG trailed ASR-cascade RAG systems by about 9 points answer accuracy on HotpotQA and about 15 points on RGB-zh; an ablation swapping the SONAR speech encoder for a Whisper+SONAR-text-encoder pipeline produced nearly identical accuracy, isolating the gap to the SONAR text encoder’s semantic alignment quality. (§4.3-4.4, Tables 1-2)
- refines: Not every joint speech-text embedding model provides sufficient semantic alignment for effective retrieval; encoders optimized for coarse audio-text matching underperform those distilled against a strong text-semantic teacher.
Evidence: Replacing the SONAR encoder with CLAP as the shared embedding degraded both retrieval F1 and answer accuracy on both benchmarks, attributed to CLAP preserving weaker fine-grained text semantic alignment than SONAR’s text-encoder-distilled embedding space. (§4.4, Table 2)
Limitations and Open Questions
The framework was evaluated with a single retrieval architecture (a native, LangChain-style RAG pipeline) and a single shared-embedding model (SONAR); the authors explicitly note it is unclear whether more advanced RAG designs (e.g., graph RAG) or stronger speech-text embedding models would narrow or close the accuracy gap versus ASR-cascade retrieval, so the reported trade-off should not be read as an upper bound on end-to-end speech retrieval.
The evaluation also relies on TTS-synthesized speech queries (via Azure TTS) rather than naturally spoken audio, which may not fully capture retrieval robustness to real spontaneous speech, disfluencies, or acoustic noise. Only two QA benchmarks in two languages (English, Chinese) were tested, and only one end-to-end speech-to-speech dialogue model (GLM-4-Voice) served as the generator, so generalization to other model families is untested.
Wiki Connections
- Speech-to-Speech — targets the end-to-end spoken dialogue sub-paradigm directly, adding a retrieval-augmentation layer in front of an existing end-to-end speech-to-speech dialogue model to address its lack of external-knowledge access.
- Spoken Language Model — evaluates its retrieval pipeline using an unmodified end-to-end spoken dialogue model as the downstream generator, grounding its responses in retrieved external knowledge within a real spoken-dialogue setting.
- Evaluation Metrics — contributes a controlled comparison methodology (retrieval latency, retrieval F1, and downstream answer accuracy across ASR-cascade, oracle, and end-to-end retrieval configurations) for evaluating knowledge-grounded speech dialogue systems.
- GLM-4-Voice — used unmodified as the end-to-end speech-to-speech dialogue model that generates responses in all experiments.
- SeamlessM4T — cited as an accurate, robust multilingual speech-text alignment model in related work, contrasted with the paper’s argument that high ASR accuracy does not guarantee good retrieval performance.
- Mini-Omni — cited as a representative end-to-end speech-to-speech dialogue system exemplifying the model class this framework’s retrieval augmentation targets.
- LLaMA-Omni — cited as a representative end-to-end speech-to-speech dialogue system exemplifying the model class this framework’s retrieval augmentation targets.
- SLAM-Omni — cited as a representative end-to-end speech-to-speech dialogue system exemplifying the model class this framework’s retrieval augmentation targets.