arXiv · 2025 · Preprint
Nizar El Ghazal et al. (Orange Research) · → Paper · Demo: ? · Code: ✗
Compares context management strategies for a speech-LLM that consumes a full spoken multi-turn dialogue and outputs a structured JSON dialogue state, finding that feeding the entire spoken history (uncompressed) outperforms text-mixed and compressed alternatives on Spoken Dialog State Tracking.
Note on scope: this paper’s task is Spoken Dialog State Tracking (DST), a language-understanding task. The system does not synthesize speech, convert voices, or produce any audio output; its only generative step is an LLM autoregressively emitting a JSON string of slot-value pairs. It is included in this corpus for its architectural relevance to context management in spoken-input LLMs (a component shared with generative spoken conversational agents), not for any speech-generation contribution.
Problem
Spoken Dialog State Tracking (DST) — the task of condensing a multi-turn spoken conversation into a structured, machine-readable state (domains and slot-value pairs) — lags well behind written DST in performance. Cascade pipelines (ASR → correction → text DST) are vulnerable to error propagation from the ASR stage, particularly on proper nouns and domain-specific terminology common in DST slot values. End-to-end (E2E) speech-aware LLMs (speechLLMs) have been shown to mitigate this by consuming speech directly, but prior E2E approaches diverge in how they represent conversational history beyond the current turn: one line of work (Druart et al., 2023) combines the spoken current turn with the written previous dialogue state, while another (Sedláček et al., 2025) combines the spoken current turn with a written transcript of all previous turns. Both approaches still fall back on a written (text) representation for anything beyond the immediate turn. The paper asks what happens if the system relies on spoken representations for the entire conversation history instead, either in full or in a compressed form.
Method
The system has three components: a speech encoder, a connector, and an LLM. The speech encoder (W2vBERT) processes each dialogue turn (user and agent) and produces dense per-turn representations, which are down-sampled with a x6 stride and mapped into the LLM’s input embedding space by a single-layer Transformer connector. The LLM (OLMo 2 1B in the primary configuration, or Gemma2-9B-Instruct in a scaling variant) then autoregressively generates a JSON string containing the active domains and slot-value pairs representing the dialogue state, conditioned on a prompt built from these speech embeddings.
The paper’s central methodological contribution is a systematic comparison of three ways to represent dialogue history in this prompt. The “Multimodal Context” baseline (following Sedláček et al., 2025) concatenates the spoken embedding of only the current user turn with a written text history of prior turns, and additionally requires the model to predict an ASR transcription of the current turn as an auxiliary output. The “Full Spoken Context” strategy instead concatenates the speech embeddings of every prior turn (user and agent) with no written intermediary at any point, so the model consumes a purely spoken conversation history. The “Compressed Spoken Context” strategy keeps the same purely-spoken principle but replaces each turn’s full sequence of speech-encoder frames with a small set of N trainable query vectors, computed via a query-based attention-pooling TransformerDecoder (self-attention among queries, then cross-attention from queries to the per-turn speech sequence); setting N=1 gives single-vector attention pooling per turn.
Training proceeds in two stages: an ASR pretraining stage in which the LLM is frozen and only the speech encoder and connector are trained (via the LLM predicting the transcription from speech embeddings, with gradients propagated back through the connector and encoder) using a mix of Loquacious Medium, Fisher, and the SpokenWOZ train split (transcribed with Whisper-large-v3, since SpokenWOZ lacks ground-truth transcripts); and a DST fine-tuning stage in which the speech encoder is frozen and only the connector, the optional compression module, and a LoRA adapter (rank 16) on the LLM are trained with cross-entropy loss against ground-truth dialogue-state JSON. Inference uses beam search with 5 beams.
Key Results
On the SpokenWOZ test set with post-processing (time-expression canonicalization and fuzzy matching for open/proper-noun slots, matching prior work’s evaluation protocol), the Full Spoken Context system with OLMo2-1B reaches 39.32% Joint Goal Accuracy (JGA), ahead of the Compressed Spoken Context (10-query) variant at 36.49%, and both surpass the prior E2E speechLLM baseline using the same LLM scale (WavLM + connector + OLMo-1B, Sedláček et al. 2025, at 34.66%) and considerably outperform earlier cascade and E2E systems reported for SpokenWOZ (SPACE+WavLM-align: 25.65%; Whisper+T5 cascade: 24.10%; UBAR+GenWOZ: 25.90%) (Table 1, §3.3). Scaling the LLM to Gemma2-9B-Instruct pushes Full Spoken Context to 45.52% JGA, exceeding the prior Gemma2-9B-based SOTA reported by Sedláček et al. (2025) at 42.17% (Table 1, §3.3), though the authors note this comparison involves potential test-set contamination risk for Gemma2-9B given its undisclosed training data.
Without post-processing, on both SpokenWOZ dev and test splits, Full Spoken Context (36.89%/36.29%) and the 10-query Compressed Spoken Context (34.26%/33.51%) both outperform the Multimodal baseline (31.85%/32.06%), while single-query (1-query) compression underperforms the baseline (31.03%/30.99%), indicating that compressing an entire turn to one vector discards too much information (Table 2, §3.4). A per-slot-group analysis shows Full Spoken Context leads across categorical, time, open, and profile slot types, with profile slots (names, IDs, emails spelled out across turns) the hardest for all systems and worst for the 1-query variant. A per-turn analysis shows JGA for all systems declines sharply between turns 5–30 and approaches zero by turn 40, which the authors attribute to growing dialogue-state complexity combined with the strictness of the JGA metric and the limited capacity of the small LLM used. Increasing the number of compression queries from 1 to 3 yields a large JGA gain that then plateaus for higher query counts (Figure 4, §3.5).
Novelty Assessment
The underlying architecture (speech encoder + connector + LLM, with LoRA-based fine-tuning) is not new; it follows the same design used by Druart et al. (2023) and, especially, Sedláček et al. (2025). The paper’s genuine contribution is narrower and empirical: a controlled comparison isolating how much of the conversational history should be represented as spoken embeddings versus text, and whether that spoken history can be compressed without much accuracy loss. The attention-pooling compression module (trainable queries + cross-attention over per-turn speech frames) is a reasonable but standard mechanism (comparable to perceiver-style pooling used elsewhere), applied here for the first time to spoken dialogue history compression in DST. The result that full spoken context beats a mixed text/speech context, and that it can be substantially compressed with modest loss, is a useful data point for context management in spoken-input LLMs generally, but the paper’s own framing is explicit that its contribution is a comparative study rather than a new model family. It is important to be precise about what is and is not generative here: the system’s sole generative act is an LLM producing a JSON string; there is no speech synthesis, voice conversion, or any acoustic output anywhere in the pipeline.
Field Significance
Low — this paper’s contribution is to Spoken Dialog State Tracking, a spoken-language-understanding task, not to speech generation. Its relevance to spoken conversational agent design is architectural and adjacent: it provides evidence on how much and in what form spoken conversational history should be retained in the context of an LLM consuming multi-turn audio input, which is a design question shared with generative speech-LLM systems that also need to manage long spoken context windows. It does not advance, evaluate, or claim anything about synthesized speech quality, voice identity, or acoustic generation.
Claims
- supports: Feeding an LLM the raw spoken representation of the full multi-turn dialogue history, rather than a mixed representation that substitutes text for earlier turns, improves task performance in spoken-input LLM systems.
Evidence: Full Spoken Context reaches 36.89%/36.29% JGA (dev/test, no post-processing) versus 31.85%/32.06% for the Multimodal baseline that mixes the current spoken turn with a written history of prior turns, on SpokenWOZ. (§3.4, Table 2)
- supports: Per-turn attention-pooling compression of spoken history embeddings in a speech-LLM can retain most of the task performance of the uncompressed full history while sharply reducing context length, provided enough pooling queries are used.
Evidence: Compressed Spoken Context with 10 queries reaches 34.26%/33.51% JGA (dev/test) versus 36.89%/36.29% for the full uncompressed spoken history, while a single-query (1-vector) compression underperforms even the mixed-modality baseline (31.03%/30.99%); increasing queries from 1 to 3 yields the largest gain, with diminishing returns beyond that. (§3.4, Table 2; §3.5, Figure 4)
- complicates: Task performance in LLM systems consuming long multi-turn spoken context degrades sharply as dialogue length grows, regardless of the context-management strategy used.
Evidence: Joint Goal Accuracy for all context-management variants (multimodal, full spoken, compressed spoken) declines steeply between turns 5–30 and approaches zero by turn 40 on SpokenWOZ, attributed to growing dialogue-state complexity, the strictness of the JGA metric, and the limited capacity of the small LLM used. (§3.4)
- complicates: Consuming the full uncompressed spoken conversation history in an LLM-based system creates a memory and latency cost that grows with dialogue length, limiting practical deployment relative to compressed or text-based context representations.
Evidence: The authors note that attention cost scales quadratically with the number of spoken turns retained in context, causing GPU memory bottlenecks even with relatively small context windows on long dialogues, and that for the largest scaling experiment (Gemma2-9B) the full-context speech-embedding sequence had to be truncated to 1500 tokens to fit available compute. (§5 Limitations; §3.5)
Limitations and Open Questions
Warning
The proposed systems are evaluated only in an offline research setting; the paper explicitly states they remain far from delivery-level and does not evaluate end-to-end latency, streaming inference, or failure modes under production/streaming constraints, which are all prerequisites for real-world deployment of a Spoken DST system.
Beyond the streaming/latency gap, the empirical analysis is confined to a single benchmark, SpokenWOZ, which the authors describe as the main human-conversational-speech DST benchmark available; extending to other corpora such as Spoken MultiWOZ is flagged as future work but complicated by those corpora’s heavy use of synthesized (TTS) speech for training/validation splits, which could conflate DST robustness with robustness to TTS artifacts. The Full Spoken Context approach’s memory and compute footprint grows with dialogue length due to quadratic attention cost, and while the compression variant mitigates this, the authors note it may discard fine-grained cues such as named entities, numbers, or precise time slots. Code and trained models are not yet released; the authors state this as a future intention rather than a current deliverable.
Wiki Connections
- Spoken Language Model — the system’s LLM (OLMo 2 1B or Gemma2-9B-Instruct) is adapted via a connector and LoRA to consume an external multi-turn spoken dialogue signal directly, rather than generating or processing its own previously produced audio.
- Self-Supervised Speech — the speech encoder (W2vBERT) is a self-supervised pretrained speech representation model used as a frozen backbone during the DST fine-tuning stage, with only the connector, compression module, and an LLM LoRA adapter trained on top of it.