Abstract
Speech-to-speech systems accept speech and produce spoken responses, spanning conversational agents, full-duplex interaction, and speech-to-speech translation. Real-time interaction is now practical in several settings. Across 94 reviewed papers, systems still trade reasoning quality against latency, turn-taking, and sensitivity to how something is said.
Current State
- Cascaded ASR–language-model–TTS systems remain the strongest general baseline. Mature components still lead many spoken-instruction, reasoning, and role-playing comparisons (VoiceBench, AURA, SpeechRole). This is not a clean proof that cascades are intrinsically better: comparisons often also vary model scale, data, and component maturity.
- Modular speech output currently preserves language ability more reliably than joint speech training. Keeping a text-language-model backbone frozen or lightly adapted and attaching a streaming speech decoder reduces the reasoning and instruction-following loss seen in several jointly trained systems (Freeze-Omni, LLMVoX, NVDuplex).
- Full-duplex operation has not solved conversational timing. Systems can listen while speaking and stop quickly when interrupted, but fast response often comes with false yielding, premature takeovers, weak backchannels, or poorer semantic continuation ([[2507.23159|Full-Duplex-Bench v1.5]], FLEXI).
- Semantic competence is ahead of paralinguistic understanding. Emotion, speaker traits, ambient sound, and non-verbal behavior remain weaker than content understanding across recent diagnostic benchmarks (MTalk-Bench, VoxRole, EMO-Reasoning).
- Single-turn scores overstate conversational ability. Context recall, emotional coherence, information density, and bias can worsen as an interaction continues (2025.findings-acl.470, MTalk-Bench).
- Q4 moves useful work into the listening window. Stream RAG and related agents retrieve, reason, or plan tools from partial speech to lower response latency. Short chunks can trigger premature or incoherent actions, so cancellation and full-context fallback are part of the capability rather than implementation details.
- Direct translation benefits from structured and synthetic supervision, but multilingual reliability remains uneven. Multi-token unit prediction in MTP-S2UT and pseudo-parallel transfer in RosettaSpeech reduce reliance on recorded parallel speech; language choice, code-switching, voice fidelity, and low-resource coverage still produce large condition-specific gaps.
Method Landscape
| Approach | Why use it | Principal limitation |
|---|---|---|
| Cascaded ASR + LM + TTS | Strong task accuracy, debuggable intermediate text, replaceable components | Pipeline latency and loss of acoustic information during transcription |
| Modular LM + speech decoder | Preserves a strong text backbone while enabling streaming spoken output | Often restricted in voice, language, or expressive control |
| Native joint speech-text model | Direct token generation and a natural route to full-duplex interaction | Current systems often lose reasoning or knowledge relative to their text backbones |
| Dual-stream full duplex | Separates user and agent audio so the system can listen while speaking | Evidence remains concentrated in smaller systems and restricted interaction tests |
| Simultaneous speech translation | Optimizes cross-lingual content, voice transfer, and streaming together | Results depend strongly on language pair, latency policy, and proprietary evidence |
These approaches increasingly overlap. The useful distinction is not whether a system is called “end to end.” It is where the system couples recognition, reasoning, timing, and speech generation, and which parts can still be trained separately.
Key Trade-offs
- Task accuracy versus direct acoustic modeling. A cascade can use a stronger text reasoner, while a direct audio model can retain emotion, timing, and other information discarded by a transcript.
- Fast response versus stable floor control. Lower latency does not guarantee correct decisions about interruption, pauses, background speech, or backchannels.
- Semantic tokens versus acoustic fidelity. Text-aligned representations work well with language models; acoustic codec representations preserve more voice and paralinguistic detail.
- Joint adaptation versus capability preservation. Training more of the backbone may improve speech integration while risking forgetting of text reasoning and instruction-following.
- Evaluation scale versus evaluation validity. Automated judges enable broad comparisons but are least reliable for subtle, paralinguistic, or near-tied speech outputs.
- Speculative latency versus action accuracy. Reasoning before the user finishes saves time only when ambiguity and recovery costs are measured alongside successful early actions.
Open Questions
- Will end-to-end systems close the cascade quality gap when backbone scale, data, and components are matched?
- Can a full-duplex model respond quickly while also handling pauses, interruptions, and backchannels robustly?
- Which training strategy best preserves language ability while learning speech-specific representations?
- Can retrieval systems be designed for conversational memory rather than only external factual knowledge?
- Which evaluation protocol best combines semantic correctness, acoustic appropriateness, interaction quality, and multi-turn behavior?
- Can synthetic and pseudo-parallel data close multilingual and code-switched gaps without erasing voice or prosodic fidelity?
Go Deeper
Read Speech-to-Speech Systems: In Depth for the detailed research landscape, the evidence behind these assessments, disagreements about staged pretraining and retrieval, and a representative reading path.
Scope
This assessment covers 94 integrated papers through Q4 2025 and combines spoken dialogue, full-duplex interaction, and speech-to-speech translation. These tasks share a speech-in/speech-out interface but not always objectives or metrics. Surveys support taxonomy rather than independent replication, known duplicate publication records are treated as one evidence point, and several frontier systems rely on proprietary data or unmatched commercial comparisons.
This page is generated from wiki/_claims/speech-to-speech.yaml (digest date: 2026-09-20).
Complete structured claims and paper-level provenance remain in the source YAML.