arXiv · 2026 · Preprint
Arjun Chandra et al. (Boston University / Amazon AGI) · → Paper · Demo: ? · Code: ✓
Introduces TRACE, a training-free framework that lets text-only LLM judges reason over a structured textual blueprint of inexpensive audio signals (ASR, quality predictors, prosody/affect descriptors) to evaluate speech-to-speech systems, paired with a Human Chain-of-Thought (HCoT) re-annotation protocol that separates evaluation into content, voice-quality, and paralinguistics dimensions with typed ties.
Problem
Automatic evaluation of speech-to-speech (S2S) systems currently forces a choice between two flawed options: text-only LLM judges that operate on transcripts and are blind to nonlinguistic cues like sarcasm, emotion, and delivery; and Audio Language Model (ALM) judges that process raw audio directly but are expensive, opaque, and, as this paper shows, still often fail to reason correctly about paralinguistic cues. Compounding this, the human-preference benchmarks used to validate S2S judges (SpeakBench, S2S-Arena) collapse evaluation into a single overall pairwise label using either no-tie protocols (forcing a winner even when both responses are unacceptable) or untyped-tie protocols (which don’t distinguish “both good” from “both bad”), producing labels that can be gamed by content-only evaluators and that underweight delivery quality.
Method
The paper’s contribution has two parts. First, a Human Chain-of-Thought (HCoT) annotation protocol re-labels SpeakBench and the English subset of S2S-Arena with dimension-first pairwise judgments across three axes grounded in communication research: Content (C, linguistic), Voice Quality (VQ), and Paralinguistics (P, e.g., prosody, affect, emphasis), followed by an overall judgment. Each dimension (and the overall label) uses a typed-tie label space {1, 2, both-good, both-bad} rather than a forced winner: raters first decide whether each candidate is independently acceptable along that dimension, return a single winner if only one is acceptable, “both-bad” if neither is acceptable, and only perform a relative comparison (winner or “both-good”) if both are acceptable. This design follows ITU-T P.800/P.835/P.808 guidance for separating perceptual scales and decouples absolute acceptability from relative superiority. Each dataset is rated in three passes (blind overall-first, full HCoT dimension-first, and a repeated HCoT pass on a resampled subset) to measure inter-human agreement via Cohen’s κ.
Second, TRACE (Textual Reasoning over Audio Cues for Evaluation) is a two-stage, training-free auto-rater built on top of HCoT-style dimension-first judging. Stage 1 extracts a compact, structured “blueprint” of inexpensive audio signals for each candidate response: an ASR transcript (Whisper-large-v3) for Content, non-intrusive perceptual quality predictors (e.g., DNSMOS P.835) for Voice Quality, and lightweight prosody/affect descriptors (drawing on openSMILE/eGeMAPS-style feature sets) for Paralinguistics. Stage 2 passes this textual blueprint (not raw audio) to an LLM, which is prompted to output per-dimension decisions and reasoning in structured JSON, without requesting an overall score directly. A deterministic, dataset-specific fusion policy then maps the three dimension-wise decisions to a final overall label: a content-first policy for SpeakBench (mirroring its instruction-following intent, using VQ/P only as tie-breakers) and an acceptability-cap policy for S2S-Arena (where a response judged unacceptable on Content or Paralinguistics cannot receive an overall rating above that ceiling, since many failures in this delivery-centric dataset are paralinguistic).
Key Results
Validating HCoT, a simple multinomial logistic model built only from the human (C, VQ, P) dimension labels reconstructs the overall HCoT label at high accuracy on both datasets, and blind-vs-HCoT inter-human agreement on the typed 4-way overall label is substantial to almost-perfect (Cohen’s κ=0.651 on SpeakBench, κ=0.796 on S2S-Arena). On the original (non-HCoT) labels, a transcript-only LLM judge already matches or beats a full ALM judge (59.8% vs. 51.4% accuracy on SpeakBench), demonstrating that the original labels overweight content and motivating the re-annotation. Against HCoT ground truth, TRACE reaches 68.6% overall accuracy on SpeakBench and 57.0% on S2S-Arena, exceeding both the Audio Judge (61.1% / 47.5%) and the transcript-only LLM Judge (62.7% / 45.9%), with gains confirmed significant by two-sided McNemar tests (p<0.05 in every comparison, p<10⁻³ for most). TRACE is roughly 3x cheaper than the ALM judge on SpeakBench with GPT-4o (12.53 total) while remaining more accurate, and the ranking is preserved when swapping the backbone LLM from Gemini 2.5 Flash to GPT-4o. Mechanism probes show TRACE genuinely uses non-content cues rather than defaulting to content: it resolves content-tied pairs using voice-quality/paralinguistic signals roughly 23% of the time on SpeakBench (vs. 3-5% for baselines), and on S2S-Arena its acceptability-cap fusion policy suppresses fabricated winners on both-bad pairs (winner-on-bad rate 48.6% vs. 70.7% for the Audio Judge and 73.5% for the LLM Judge), which is the primary driver of its overall gain there — though this comes with a trade-off, as TRACE’s accuracy specifically on pairs with a legitimate winner (73.5%) trails both baselines (86.7% and 84.1%).
Novelty Assessment
The paper’s two contributions are evaluation-methodology, not generative-architecture, novelties. HCoT applies established subjective-testing principles (ITU-T P.800/P.835 dimension separation) to S2S preference labeling in a way not previously done for existing S2S benchmarks, and the typed-tie label space is a genuinely useful refinement over prior no-tie/untyped-tie protocols, directly diagnosed and motivated by a concrete failure mode (fabricated winners on both-bad pairs) rather than asserted abstractly. TRACE’s core idea, that a text-only LLM can substitute for an expensive ALM judge if given a sufficiently informative pre-extracted textual summary of audio cues, is a pragmatic engineering combination of existing off-the-shelf tools (Whisper, DNSMOS, prosody/affect feature extractors) rather than a new architecture, but the empirical demonstration that this decomposition outperforms both a raw-audio ALM judge and a transcript-only judge, at lower cost than the ALM, is a substantive and well-validated result.
Field Significance
high — this paper identifies and directly diagnoses a labeling-artifact problem (forced/untyped ties fabricating winners on unacceptable response pairs) that likely affects prior S2S human-preference benchmarks broadly, and demonstrates that a much cheaper text-only LLM judge, given structured audio-derived signals instead of raw audio, can match or exceed a full audio-language-model judge, with mechanistic evidence (counterfactual probes, ablations) for why the gain occurs rather than only reporting a top-line number.
Claims
- complicates: Human-preference benchmarks for speech-to-speech evaluation that collapse judgments into a single overall pairwise label, using no-tie or untyped-tie protocols, systematically overweight textual content and underweight paralinguistic delivery, and can fabricate a winner between two unacceptable responses.
Evidence: On original S2S-ARENA and SPEAKBENCH labels, a transcript-only LLM judge matches or exceeds a more expensive audio-language-model judge (59.8% vs. 51.4% accuracy on SPEAKBENCH), and on the re-annotated S2S-ARENA subset, paralinguistics is rated both-bad in ≈55% of examples, showing the original forced-winner protocol was fabricating superiority on unacceptable pairs. (§3.1, Table 1)
- supports: Prompting a text-only LLM judge with a structured textual blueprint of inexpensive, pre-extracted audio signals achieves higher agreement with human dimension-wise judgments than either a transcript-only LLM judge or a full audio-language-model judge, at substantially lower cost than the ALM.
Evidence: Against HCoT ground-truth labels, TRACE reaches 68.6%/57.0% overall accuracy on SpeakBench/S2S-Arena versus 62.7%/45.9% for a transcript-only LLM judge and 61.1%/47.5% for an ALM judge (McNemar p<0.05 in every comparison), while costing roughly one-third of the ALM judge on SpeakBench (12.53). (§4.2, Table 3; Table 4)
- supports: Separating a speech evaluation judgment into explicit content, voice-quality, and paralinguistics dimensions with a typed-tie label space produces a more reliable, internally consistent evaluation signal than a single collapsed overall pairwise rating.
Evidence: A multinomial logistic model built only from human (C, VQ, P) dimension labels reconstructs the HCoT overall label at high accuracy, and blind-vs-HCoT inter-human agreement on the typed 4-way overall label reaches substantial-to-almost-perfect levels (Cohen’s κ=0.651 SpeakBench, κ=0.796 S2S-Arena). (§4.1, Table 2)
- complicates: An automated judge’s overall accuracy advantage over baselines can be driven primarily by correctly rejecting pairs where neither candidate is acceptable, rather than by superior discrimination among genuinely competing candidates.
Evidence: On S2S-Arena (58% both-bad pairs), TRACE’s overall gain traces to reducing “winner-on-bad” fabrication to 48.6% (vs. 70.7% Audio Judge, 73.5% LLM Judge), even though TRACE’s accuracy specifically on pairs with a legitimate winner (73.5%) trails both baselines (86.7%, 84.1%). (§4.2, Figure 5)
- supports: Explicit reasoning over paralinguistic delivery cues, not just content, allows an evaluation judge to resolve semantic ties between candidate speech responses in a way that better matches human preference.
Evidence: In a content-controlled counterfactual probe on SpeakBench, TRACE uses voice-quality/paralinguistic cues to resolve content-tied pairs in ≈23% of cases, versus only ≈3-5% for the Audio Judge and transcript-only LLM Judge baselines. (§4.2, Figure 3)
Limitations and Open Questions
The paper’s own limitations note that all experiments are restricted to English-language datasets (SpeakBench, S2S-Arena), leaving generalization to other languages and cultural expressivity norms untested. The acoustic blueprint schema (which signals to extract for Content/VQ/Paralinguistics) was designed manually rather than learned, and may omit finer-grained attributes needed for edge cases; the authors propose data-driven schema induction as future work. TRACE also inherits error from its upstream automatic extractors (ASR, quality predictors, prosody descriptors) with no calibration or confidence weighting to mitigate propagated mistakes. On the winner slice specifically (pairs where a legitimate best response exists), TRACE trails both the Audio Judge and LLM Judge baselines, indicating the acceptability-cap fusion policy trades some winner-discrimination accuracy for its gains on both-bad rejection.
Wiki Connections
- Speech-to-Speech — evaluates and re-annotates two existing human-preference S2S benchmarks (SpeakBench, S2S-Arena), diagnosing labeling artifacts specific to pairwise S2S preference collection.
- Evaluation Metrics — introduces both a new dimension-first, typed-tie annotation protocol (HCoT) and a new training-free auto-rater (TRACE) for automatic S2S evaluation.
- Subjective Evaluation — the HCoT protocol collects genuine human dimension-first pairwise ratings (content, voice quality, paralinguistics) with measured inter-rater agreement, following ITU-T P.800/P.835/P.808 subjective-testing guidance.
- AudioJudge — SpeakBench and the ALM-judge baseline methodology this paper compares against originate from AudioJudge; TRACE is evaluated as a cheaper alternative to this audio-language-model judging approach.
- SageLM — a comparable multi-aspect, explainable LLM-based speech judgment system cited as related prior work on structured S2S auto-rating.
- VocalBench — a comparable benchmark for vocal conversational abilities, cited alongside S2S-Arena/SpeakBench as part of the broader S2S evaluation benchmark landscape.
- VoiceBench — cited as an early S2S benchmark relying on transcript-based general-knowledge assessment, representative of the objective-metric evaluation line TRACE complements.
- Moshi — cited as an example end-to-end S2S system whose outputs motivate the need for paralinguistics-aware automatic evaluation.
- LLaMA-Omni — cited as an example end-to-end S2S voice-assistant system representative of the systems these benchmarks and judges are designed to evaluate.
- PredGen — cited as related work on real-time speech interaction systems within the same S2S deployment context this evaluation framework targets.