arXiv · 2026 · Preprint

Thomas Thebaud et al. (Johns Hopkins University / Amazon) · → Paper · Demo: ? · Code: ✓

SPEARBench is an open, fully automatic benchmark that feeds real human question-answer dialogue contexts into speech-to-speech language models and scores their spoken answers against the original human answer across timing, overlap, turn-taking, dialect, emotion, stance and pitch dimensions.

Problem

Speech-to-speech (S2S) language models answer spoken queries with synthetic speech, often in streaming or full-duplex mode. Standard measures such as WER and UTMOS show whether the answer is intelligible and clean, but not whether it is conversationally natural: whether it arrives at the right time, avoids talking over the user, keeps a consistent dialect, matches the question’s emotion, or preserves interpersonal stance. The authors argue that existing dialogue benchmarks either need human annotators or cover one dimension (latency, or interruptions alone), and that commercial leaderboards are not open. SPEARBench aims to put many naturalness dimensions into one reproducible, automatic pipeline with a human reference condition.

Method

Data preparation (§III-A). The benchmark uses the dev and test sets of the Seamless Interaction corpus (dyadic English conversations of up to 10 minutes, with separate speaker channels, transcripts and diarization), both its naturalistic subset (non-actor prompted conversations) and its improvised subset (scenario-based, with annotated interpersonal stances and speaker relationships). A turn is a run of one speaker’s speech with no backchannel interruption longer than 0.5 s. A valid dialogue is any K ≥ 3 consecutive turns between two speakers where the second-to-last turn ends with a question mark. Turns 1 to K-2 form the context, turn K-1 is the question, and turn K is the human answer kept as reference. At least one valid dialogue was found in 70.91% of conversations (4.6 per conversation on average), giving 5,419 dialogues from 2,007 conversations: 22.77 h of context and question audio and 14.56 h of answers (Table I).

Inference (§III-B). Each model receives the context and question audio and produces a spoken answer. The benchmark stores the waveform and the answer start time relative to the end of the input. A negative start time counts as an interruption. Seven systems are evaluated (Table II): GPT-audio-1.5 (non-streaming), GPT-realtime-2, Gemini-2.5-flash-native-audio, Gemini-3.1-flash-live and Qwen3-Omni-30B-Instruct (full-duplex), and Qwen2.5-Omni-7B and Mini-Omni (0.5B) (half-duplex). Three of these have open weights.

Evaluation (§III-C). The pipeline chains previously published evaluator models into seven groups:

  1. Intelligibility and quality: WER and CER from whisper-large-v3 and Qwen3-ASR-0.6B, using each model’s own text output as the reference, plus UTMOS.
  2. Latency and interruptions: Silero VAD measures the gap from question end to voiced answer; overlap with the original speaker counts as an interruption, with its total duration reported.
  3. Turn-taking naturalness: the TurnNat evaluator (a DualTurn-based voice-activity predictor that scores the surprisal of local turn-taking patterns) scores the reconstructed two-channel interaction.
  4. Language and dialect: MMS language ID reports the share of English answers. Voxlect-English yields an 11-dimensional dialect profile (after merging British Isles categories). From these profiles the benchmark defines two new quantities: dialectal entrainment (least-squares regression coefficient β between question and answer profiles) and dialectal variance (trace of the profile covariance matrix).
  5. Emotional naturalness: the TRACE predictor takes Vox-Profile emotion embeddings and arousal/valence/dominance (AVD) of question and answer, an SBERT embedding of the conversation prompt, and a one-hot relationship vector, and outputs a relative naturalness score. Question-to-answer AVD correlations measure emotional entrainment.
  6. Interpersonal stance (improvised subset only): following StanceBench, GPT-audio acts as an audio-language judge on 10 stance dimensions (politeness, empathy, warmth and others). The benchmark reports the share of answers with the same, more negative, or more positive stance than the question.
  7. Explainable features: speaker-normalized pitch range per decile, answer duration and voiced ratio, following the reference-based prosody and rhythm evaluation of the same group.

No new model is trained. The curated dialogues, the evaluation code and a public results website are released.

Key Results

All numbers are from Table III, with the original human answer as the reference row.

  • Signal quality does not separate models from humans in the expected direction. All seven models score far higher UTMOS (3.72 to 4.36) than the human answers (2.22), and all seven have lower WER than humans (5.8% to 26.7% against 27.1%). GPT-audio-1.5 has the highest UTMOS (4.36, Qwen3-Omni 4.34) and the lowest WER (5.8%) and CER (1.4%); it is the only non-streaming system. Gemini-3.1-flash-live is the weakest on intelligibility (WER 26.7%, close to the human 27.1%) and the only model with higher CER than humans (23.1% against 17.2%). The human WER is computed against corpus transcripts of spontaneous conversational speech, while model WER uses each model’s own text output as reference, so the two are not strictly comparable.
  • Latency is longer than human for every model. Humans answer after 742 ms on average; models take 959 ms (GPT-audio-1.5) to 2,724 ms (Qwen3-Omni). Three of the four full-duplex models are slower than both half-duplex models (Qwen2.5-Omni 1,176 ms, Mini-Omni 1,243 ms); Gemini-3.1-flash-live (1,154 ms) is the exception.
  • Interruptions concentrate in two systems. Mini-Omni overlaps the speaker in 58.1% of dialogues (1,243 ms average overlap) and GPT-realtime-2 in 23.6% (431 ms). The other streaming models never interrupt. Humans in the corpus overlap in 33.8% of answers (521 ms), so zero overlap is not itself human-like.
  • Turn-taking naturalness is near or above human for recent full-duplex models (5.31 to 5.35 against 4.99 for humans; Gemini-3.1-flash-live highest at 5.35, Qwen3-Omni 5.34, GPT-realtime-2 5.33), and half-duplex Qwen2.5-Omni scores 5.23. Mini-Omni scores far lower (3.10), and GPT-audio-1.5 sits slightly below human (4.74).
  • Emotional naturalness is lower than human for every model (9.21 to 10.91 against 11.10; p < 10^-13, paired Friedman test, §IV-E). Mini-Omni has the lowest score. AVD entrainment shows the clearest gap: human answers correlate with the question on arousal (0.53), valence (0.36) and dominance (0.51), while models reach at most 0.25, 0.29 and 0.20 respectively. Qwen2.5-Omni shows no correlation on any dimension (-0.02 to -0.03).
  • Dialect. 98.1% to 100% of model answers are detected as English. Human answers have the largest dialectal variance (250.6, against 84.5 to 204.5 for models), consistent with models using a few fixed voices. The text reads the entrainment results as little or no dialect following by any model, but the model β values (0.28 to 0.54) bracket the human value (0.53), so β alone does not separate most models from humans in Table III.
  • Stance is largely preserved. Same-stance rates are 95.7% to 98.0% for models against 97.1% for humans; Mini-Omni has the most negative shifts (4.2%) and, per Figure 5, much lower Focus.
  • Prosody is flatter. Normalized pitch variation is 0.321 for humans and 0.118 to 0.299 for models (Gemini-3.1-flash-live closest). Figure 6 shows a smaller pitch range than humans across all deciles. Answer duration varies widely (6.6 s to 19.3 s against 8.8 s for humans): four models answer longer than humans and three shorter.

The comparison is fair in that all systems receive identical audio contexts, but timing depends partly on each vendor’s serving interface (for example GPT-realtime-2 runs with a semantic VAD), as the authors acknowledge.

Novelty Assessment

The contribution is integration and benchmark construction rather than new measurement science. Most evaluator components (TurnNat, TRACE, StanceBench, Vox-Profile, Voxlect, the prosody and rhythm features) come from earlier or companion papers by the same group and collaborators. The genuinely new parts are the question-anchored dialogue extraction protocol over Seamless Interaction, the dialectal entrainment and variance measures, the human-answer reference condition across all dimensions, and the side-by-side results for seven current commercial and open S2S systems. The dialect measures are not validated against human judgment, and the paper validates no part of the aggregate protocol against listeners; it relies on each component’s earlier validation.

Field Significance

Low — SPEARBench gives a reproducible, multi-dimensional automatic profile of how current speech-to-speech systems differ from human answers in the same dialogue context, and its main finding (high signal quality and intelligibility coexisting with flat prosody, weak emotional entrainment and non-human timing) is clearly documented. Its reach is limited by English-only two-speaker data and the absence of any human validation of the combined protocol.

Claims

  • complicates: Signal-level quality and intelligibility metrics do not indicate conversational naturalness in spoken dialogue systems, and can rank synthetic answers above human ones.

    Evidence: Every evaluated S2S model has higher UTMOS (3.72 to 4.36) than original human answers (2.22) and all have lower WER, yet all have lower emotional naturalness, weaker AVD entrainment and smaller pitch variation than humans. (§IV-A, §IV-E, §IV-G, Table III)

  • supports: Current speech-to-speech dialogue models fail to adapt the emotional intensity and dominance of their spoken answers to the conversational partner, even when they match its general valence.

    Evidence: Human answers correlate with the question on arousal (ρ = 0.53), valence (0.36) and dominance (0.51); the seven models reach at most 0.25 arousal and 0.20 dominance correlation while keeping valence correlations up to 0.29, and all models score significantly lower than humans on TRACE emotional naturalness (p < 10^-13). (§IV-E, Figure 4, Table III)

  • supports: Spoken answers from current speech-to-speech models show a narrower pitch range than human conversational answers.

    Evidence: Speaker-normalized pitch variation is 0.321 for human answers and 0.118 to 0.299 across the seven models, with a smaller range in every pitch decile. (§IV-G, Figure 6, Table III)

  • refines: Low response latency, zero overlap and predictive turn-taking naturalness capture different aspects of conversational timing and do not move together.

    Evidence: Recent full-duplex models score at or above the human reference on turn-taking naturalness (5.31 to 5.35 against 4.99) while answering 1.2 to 2.7 s after the question against 742 ms for humans, and humans themselves overlap in 33.8% of answers while four streaming models never overlap. (§IV-B, §IV-C, Table III)

  • complicates: Full-duplex operation does not by itself yield faster responses than half-duplex operation in deployed speech-to-speech systems.

    Evidence: Three of four full-duplex systems (2,137 to 2,724 ms) answer more slowly than both half-duplex systems (1,176 and 1,243 ms); only Gemini-3.1-flash-live (1,154 ms) is faster, and the authors note timing depends partly on serving interfaces. (§IV-B, §V, Table II, Table III)

Limitations and Open Questions

Warning

The whole protocol is automatic and has not been checked against human judgments of naturalness. Each score relies on a separately validated evaluator model, and the authors list human validation as future work.

  • The data is English-only and two-speaker. Language-consistency results therefore reduce to an English/non-English share.
  • Intelligibility comparisons mix references: model WER uses each model’s own text output as reference, while human WER is against corpus transcripts of spontaneous speech.
  • Timing depends on vendor serving interfaces (VAD settings, network), so latency and interruption values for proprietary APIs are partly deployment measurements, not model properties.
  • Dialect and pitch variability are constrained by the fixed voices each system offers, so lower dialectal variance may reflect the voice inventory rather than conversational behaviour.
  • The emotional naturalness score is only interpretable relative to the human condition and across systems, not as an absolute perceptual scale.
  • The stance judge (GPT-audio) comes from the same model family as two evaluated systems (GPT-audio-1.5, GPT-realtime-2), and the paper does not discuss possible self-preference.
  • Only single question-answer exchanges are evaluated; multi-turn consistency and backchanneling are not measured.

Wiki Connections

  • Spoken Language Model — the benchmark feeds real human dialogue audio into seven existing speech LMs and measures their spoken answers.
  • Speech-to-Speech — evaluates the speech output of streaming and full-duplex speech-to-speech dialogue systems against human answers in the same context.
  • TurnNat — the turn-taking naturalness evaluator used for SPEARBench’s turn-taking dimension, from the same group.
  • Reference-Based Prosody and Rhythm Evaluation — the source of SPEARBench’s explainable pitch and timing features, from the same group.
  • Vox-Profile — provides the speech emotion recognition embeddings and AVD estimates fed into the emotional naturalness predictor.
  • Qwen3-Omni — evaluated full-duplex system with the highest UTMOS among open models and the longest latency.
  • Qwen2.5-Omni — evaluated half-duplex system that shows no AVD entrainment and the lowest dialectal variance.
  • Mini-Omni — evaluated half-duplex system with the highest interruption rate and lowest turn-taking and emotional naturalness scores.
  • Gemini 2.5 — the report the paper cites for the evaluated Gemini-2.5-flash-native-audio system.
  • Full-Duplex-Bench v1.5 — prior interruption-focused benchmark that SPEARBench positions itself against as single-dimension.