arXiv · 2025 · Preprint
Teleki et al. (Texas A&M University) · → Paper · Demo: ✗ · Code: ✓
Introduces DRES, a factorized evaluation framework that isolates how LLM backbones used in SpeechLLMs handle conversational disfluencies, and shows that structural robustness is governed by training objective rather than scale.
Problem
SpeechLLMs compose an acoustic encoder with a pretrained LLM backbone, and a common assumption in the field is that larger, more capable backbones will be more robust to the messiness of real conversational speech. Spontaneous speech is full of disfluencies (interjections, false starts, edits, and parentheticals) that are rare in the written text these backbones are pretrained on. Prior evaluation of this robustness has almost always been end-to-end: a speech signal goes in, a transcript comes out, and any deletion, insertion, or rewrite in that transcript could originate either from acoustic mis-transcription (ASR systems are known to under-transcribe short interjections) or from the language model’s own editing decisions. This entanglement means end-to-end benchmarks cannot attribute structural failures to the backbone LLM itself, and existing disfluency-detection literature (sequence-tagging methods on Switchboard) does not directly translate to the generative, open-ended editing behavior of modern LLM backbones.
Method
The paper reframes disfluency removal as a deletion-constrained repair task: given a disfluent transcript and its gold disfluency mask, the correct fluent output is a unique monotonic subsequence of the input, so any additional deletion or rewrite is a structural error with no paraphrastic freedom. Building on this, the authors introduce DRES (Disfluency Removal Evaluation Suite), which fixes the acoustic component of a SpeechLLM pipeline (S_A = 0) and feeds gold Switchboard conversational transcripts directly to a language backbone, isolating language-level editing behavior (D_θ) from acoustic suppression effects. Robustness is then defined via word-level precision (E_P), recall (E_R), and F1 (E_F) between the model-implied deletion mask and the gold mask, recovered through an alignment procedure from prior work, as well as category-specific Z-scores over the Shriberg disfluency categories (EDITED, INTJ, PRN).
Nine backbone families are evaluated (Table 1), spanning proprietary and open-source models, dense and mixture-of-experts architectures, instruction-tuned and reasoning variants, and parameter counts from 125M (MobileLLM) to frontier scale (GPT-4o, o4-mini). Models are tested under full-transcript and segmented-transcript (~4-sentence) conditions, under in-context learning with k in {0, 1, 3, 5} demonstration pairs, and, for two models (GPT-4o-mini and Llama-3B), before and after supervised fine-tuning on the disfluency-removal task, with generalization measured on GSM8K, MMLU, and CoQA. A k-means clustering analysis over the (E_P, E_R) plane is used to test whether models fall into distinct, statistically stable “editing policies” (under-deletion, over-deletion, balanced, poor).
Key Results
Clustering in precision-recall space recovers four stable regimes with moderate-to-strong separation (silhouette score 0.54, Davies-Bouldin Index 0.664; permutation test p ≈ 0.005), and 70-100% of models retain the same policy across in-context learning levels, indicating editing behavior is a backbone-intrinsic property rather than a prompting artifact. Proprietary GPT models cluster in the desired balanced region, while reasoning-oriented variants (o4-mini, Phi-4 reasoning) systematically shift into over-deletion, treating the task as paraphrasing rather than literal repair. Models handle overt EDITED repairs well but perform substantially worse on short INTJ/PRN markers (“uh,” “you know”), the opposite of the difficulty ranking reported for supervised sequence-tagging baselines. Segmenting long transcripts into shorter spans consistently improves F1 and shifts models toward the balanced region, even for models with 128k+ context windows. Scale improves raw F1 within a model family but does not change its editing policy. Fine-tuning GPT-4o-mini and Llama-3B on the disfluency task raises DRES F1 by more than 22 points (mid-70s to mid-90s and high-60s to above 90, respectively) but produces consistent negative deltas on GSM8K, MMLU, and CoQA (Table 2). All comparisons are internal to DRES’s own gold-transcript protocol rather than against a shared external end-to-end benchmark, since the paper’s explicit goal is to factor out acoustic effects that other benchmarks conflate.
Novelty Assessment
The contribution is an evaluation-methodology and empirical-benchmark one rather than an architectural one: no new model or training method is proposed for speech generation itself. The genuine novelty is the factorization that isolates backbone editing behavior from acoustic suppression, and the demonstration, via clustering statistics, that “editing policy” is a real, stable, backbone-intrinsic property rather than a byproduct of prompting or scale. The finding that reasoning-oriented training increases over-deletion, and that fine-tuning trades structural fidelity for general capability, are negative results relative to the common assumption that bigger and more capable backbones are automatically more robust to conversational speech. The evaluation is limited to English Switchboard and to a narrow deletion-only notion of structural fidelity, which the authors acknowledge does not capture broader normalization or repair behaviors.
Field Significance
Moderate — the paper does not change how speech generation systems are built, but it provides a controlled, statistically validated diagnostic showing that scale and reasoning capability do not straightforwardly translate into robustness to conversational structure for the LLM backbones that SpeechLLMs rely on. This is useful evidence for backbone selection and fine-tuning decisions in spoken conversational agent pipelines, though its impact depends on English-Switchboard-specific findings generalizing to other languages and disfluency phenomena, which the paper itself flags as untested.
Claims
- supports: Structural robustness of a language-model backbone to conversational speech disfluencies is governed primarily by training objective rather than parameter scale.
Evidence: Within each model family, larger variants achieved higher word-level F1 but retained the same over-deletion, under-deletion, balanced, or poor editing policy as their smaller siblings across the 125M-to-frontier size range tested. (§4.5, Finding 4, Figures 2, 5)
- complicates: Reasoning-oriented training objectives can bias a language model toward semantic abstraction at the expense of literal structural fidelity, a failure mode not visible in standard semantic benchmarks.
Evidence: o4-mini and Phi-4 reasoning variants clustered systematically in the over-deletion region of precision-recall space, deleting fluent content at higher rates than non-reasoning models of comparable scale. (§4.2, Finding 1, Figure 2)
- complicates: Fine-tuning a language-model backbone to specialize in one conversational-speech repair task can measurably degrade its general reasoning and knowledge performance.
Evidence: Fine-tuning GPT-4o-mini and Llama-3B improved DRES F1 by more than 22 points but produced consistent negative deltas on GSM8K, MMLU, and CoQA relative to the base models. (§4.6, Finding 5, Table 2, Figure 6)
- contradicts: Disfluency categories that are easiest for supervised sequence-tagging models to detect and remove are also the easiest for generative language-model backbones.
Evidence: Models handled EDITED (overt repair) structures well but underperformed on INTJ and PRN markers, reversing the difficulty ranking reported by prior BiLSTM- and BERT-based sequence taggers on the same disfluency categories. (§4.3, Finding 2, Figure 3b)
Limitations and Open Questions
The evaluation is conducted entirely on English Switchboard transcripts with acoustic suppression fixed to zero (S_A = 0); the authors note the deletion-constrained framework is language-agnostic in principle, but conversational repair patterns across other languages remain unvalidated. The deletion-only task also captures only a minimal notion of structural fidelity and does not model broader transcript normalization behaviors that real SpeechLLM deployments would need to handle. Because DRES deliberately factors out the acoustic encoder, its findings characterize backbone-only editing policy and do not by themselves predict end-to-end deployed robustness, where acoustic suppression and language-level editing interact.
Wiki Connections
- Evaluation Metrics — introduces DRES, a factorized precision-recall-based evaluation framework and a k-means clustering methodology for characterizing generative editing behavior, independent of acoustic transcription error.