ACL · 2026 · Conference

Feng Jiang et al. (Shenzhen University of Advanced Technology / The Chinese University of Hong Kong, Shenzhen / Nanjing University / Shenzhen Loop Area Institute / CentraleSupélec, Université Paris-Saclay / National University of Singapore) · → Paper · → Preprint · Demo: ? · Code: ✓

S2S-Arena is a speech-native, arena-style benchmark that ranks speech-to-speech models by pairwise judgments made directly on their spoken responses, so that paralinguistic expression in the generated speech (tempo, emotion, singing, role voices) counts toward the score instead of being lost in a transcript.

Problem

Benchmarks for LLM-based speech-to-speech systems mostly test speech understanding and score responses in the text modality: the output audio is transcribed and an LLM judges the transcript (Table 1 lists Dynamic-SUPERB, AudioBench, MMAU, SD-Eval, VoxDialogue, VoiceBench, AIR-Bench and MULTIVOX, all with LLM evaluators). Transcription discards exactly the cues that make spoken interaction expressive: prosody, emotion, speaking style and speaker traits. VStyle judges output style in speech but does not cover paralinguistic perception in the input. The authors argue there is no benchmark that jointly tests whether a model perceives paralinguistic cues in the input and expresses requested paralinguistic attributes in its own spoken output, judged in the speech modality.

Method

Four-level interaction protocol. Each query is assigned one of four levels of increasing paralinguistic demand (§3.1): L1 Instruction-Only (semantic task, paralinguistics ignored); L2 Paralinguistic Perception (the model must infer a cue such as the speaker’s age or emotion from the input and adapt the content of its reply); L3 Paralinguistic Expression (neutral input, but the instruction requests output attributes such as reciting a tongue twister at three speeds); and L4 Full Paralinguistic Interaction (perceive input cues and express matched paralinguistics, for example translating a cheerful request into Chinese with cheerful prosody).

Task design and seed data. Four application domains (Education, Social Interaction, Entertainment, Medical Consultation) and 19 tasks came out of an elicitation study with 19 researchers (§3.2; Tables 8 and 9), including pronunciation correction, emphasis and rhythm control, cross-lingual emotional translation, sarcasm detection, identity-based responses, singing, natural sound simulation, role-play and tongue twisters. The seed set (§3.3.1) has at least 10 scripts per task. Neutral samples are synthesized with Doubao-TTS; samples with strong paralinguistic cues are human-recorded or taken from RAVDESS, with eight background-noise categories added. Four native Mandarin annotators screened the audio, leaving 293 seed samples.

Self-instruction augmentation. GPT-4o generates new JSON scripts from 5-shot seed prompts (§3.3.2, Appendix A.2), which are voiced with Doubao-TTS, AudioX and Parler-TTS without further filtering. This adds 50 samples per task, 950 in total. A manual check of 100 random synthetic samples (25 per level) found 90% agreement with the intended difficulty level and 93% with the intended paralinguistic attribute. The full benchmark has 1,243 samples (L1 81, L2 329, L3 475, L4 358; Table 2). The augment set covers 100+ tasks and is 75.79% English, 24% Chinese, and 0.21% Japanese and Hindi (§3.4).

Speech-native arena judging. Each comparison concatenates the spoken instruction and two anonymous model responses into one audio prompt, and the judge picks a winner (no ties) on instruction alignment, paralinguistic expressiveness and output audio quality (§4.1). The judge was chosen in a pilot on the seed set: 19 human annotators made 432 pairwise comparisons through a web interface (10% double-labelled, 83.7% inter-annotator agreement; Appendix A.5), and two audio LLMs were compared against them (Table 3). Gemini 2.5-Pro agreed more closely with the humans and judges the full augment set. Model pairs are sampled with a Gaussian weight on their current Elo gap, centred at one third of the maximum gap, to avoid trivial or hopeless matches. Elo starts at 1000 and is updated with K = 32.

Systems evaluated. Six industry systems (GPT-4o-realtime, Doubao, FunAudioLLM, GLM-4-Voice, Qwen2.5-Omni 7B, Kimi-Audio) and four academic ones (SpeechGPT, Mini-Omni, Mini-Omni2, LLaMA-Omni). FunAudioLLM, GPT-4o-realtime, Doubao and Qwen2.5-Omni are queried through official APIs; the rest run locally (Appendix A.4). Moshi and LSLM are excluded because they lack an LLM backbone (Table 7).

Key Results

Overall ranking (1,001 Gemini-judged comparisons, Table 4). Qwen2.5-Omni has the highest Elo (1246.1), but the top three are within 15 points: GPT-4o-realtime 1239.2 and Doubao 1231.9. Doubao has the highest win rate (67.9% vs. 65.7% for GPT-4o-realtime and 59.0% for Qwen2.5-Omni), and GPT-4o-realtime has 140 wins, though Kimi-Audio has more (142) from more matches (288). A middle tier follows: GLM-4-Voice 1148.2, FunAudioLLM 1088.3, Kimi-Audio 1056.7. The academic systems trail: LLaMA-Omni 908.7, Mini-Omni2 727.4, SpeechGPT 677.1, Mini-Omni 676.4. A footnote reports 1,000 bootstrap resamples that keep the ranking order, with Spearman correlation 0.94 ± 0.03 between the original and bootstrap rankings.

The leader depends on how Elo is aggregated. In the per-domain macro average (Table 5), GPT-4o-realtime leads (1144.5), followed by Doubao (1136.9) and Qwen2.5-Omni (1101.4). The per-level macro average (Table 6) gives the same order (1144.2, 1136.8, 1110.1). Domain profiles differ: FunAudioLLM scores 1105.9 in Entertainment but 876.2 in Medical, and Qwen2.5-Omni’s best domain is Social (1155.9) while its Education score is 1096.7.

The gap widens with paralinguistic difficulty (Table 6). At L1, academic and industrial systems overlap (LLaMA-Omni 977.7, Kimi-Audio 980.6, Mini-Omni 985.8). At L3, GPT-4o-realtime reaches 1241.7 against 920.2 for LLaMA-Omni and 721.5 for SpeechGPT. At L4, Doubao (1205.8) is the top system and LLaMA-Omni scores 942.4. The authors link component choices to these differences: larger LLM backbones help L1 instruction following, larger encoders help L2 perception (LLaMA-Omni 965.2 vs. Mini-Omni 803.0), vector-quantized input shows no clear benefit (Kimi-Audio 949.7 vs. Qwen2.5-Omni 1109.1 at L2), and flow-matching decoders are presented as important for expressive output.

Judge validation (Table 3). Gemini 2.5-Pro agrees with human pairwise judgments at Cohen’s κ = 0.6553 (82.87% agreement). Qwen2.5-Omni as a judge reaches κ = 0.4667 (73.15%). The human pilot ranking (Table 10, seed set) puts GPT-4o-realtime first (1365) ahead of a Whisper + GPT-4o + CosyVoice cascade (1207) and the same cascade with a SenseVoice encoder (1025). The authors read the cascade gap as evidence that the speech encoder matters.

Novelty Assessment

The main contribution is the evaluation design: pairwise preference judgments made on audio, with an explicit difficulty ladder separating perception (L2) from expression (L3) and their combination (L4). Arena-style Elo ranking and self-instruct data expansion are borrowed from text-LLM evaluation, and several earlier speech benchmarks already probe paralinguistic perception. What is new is the requirement that judgments see the generated audio, plus a judge choice backed by human agreement data. The architectural analysis in §5.2 is correlational. It compares ten systems that differ in data, backbone, encoder and decoder all at once, without controlled ablations, so its component-level conclusions are hypotheses rather than findings. The benchmark data is small (293 human-curated plus 950 synthetic samples), and most queries are voiced by TTS systems.

Field Significance

Moderate — The paper provides a speech-native evaluation protocol for speech-to-speech systems in which output prosody, emotion and style affect the score, and shows that automatic audio judges vary considerably in how well they track human preference (κ 0.66 vs. 0.47). Its leaderboard is a 2024-2025 snapshot. Its more durable contributions are the L1-L4 framing, showing that differences between systems grow once expressive output is required, and the evidence that pairwise speech-native judging with a validated audio LLM is feasible.

Claims

  • supports: Differences between speech-to-speech systems grow as tasks move from semantic instruction following to producing requested paralinguistic attributes in the output speech.

    Evidence: At L1 the strongest academic systems are within the range of industrial ones (LLaMA-Omni 977.7, Mini-Omni 985.8 vs. Kimi-Audio 980.6), while at L3 GPT-4o-realtime reaches 1241.7 against 920.2 for LLaMA-Omni and 721.5 for SpeechGPT, and at L4 Doubao reaches 1205.8 vs. 942.4 for LLaMA-Omni. (§5.2, Table 6)

  • supports: Audio-capable LLM judges can approximate human pairwise preferences over spoken responses well enough for large-scale ranking, but agreement varies substantially across judge models and must be validated before use.

    Evidence: Against human pairwise judgments on the seed set, Gemini 2.5-Pro reaches Cohen’s κ 0.6553 (82.87% agreement) while Qwen2.5-Omni reaches κ 0.4667 (73.15%); human inter-annotator agreement on a 10% double-labelled subset is 83.7%. (§4.1, Table 3; §Appendix A.5)

  • complicates: A single leaderboard position for a speech-to-speech system depends on the aggregation scheme and task domain, so one aggregate score does not capture expressive capability.

    Evidence: Qwen2.5-Omni has the highest sequential Elo (1246.1) but ranks third on the domain and level macro averages, where GPT-4o-realtime leads (1144.5 and 1144.2); FunAudioLLM ranges from 876.2 (Medical) to 1105.9 (Entertainment). (§4.2, Table 4; §5.1, Table 5; §5.2, Table 6)

  • complicates: Attributing expressive speech-to-speech quality to specific components such as the speech decoder type is weakly supported when comparisons span systems that differ in data, backbone and encoder at the same time.

    Evidence: The paper credits flow-matching decoders for expressive generation using Doubao (1205.8) vs. LLaMA-Omni (942.4) at L4, but lists Doubao’s backbone, encoder and decoder as unknown; the remaining contrasts (for example Kimi-Audio vs. Qwen2.5-Omni for vector quantization) also differ in more than one component. (§5.2, Table 6; §Appendix A.1, Table 7)

Limitations and Open Questions

Rankings rest on one automatic judge and mostly synthetic queries

The 1,001 main-arena comparisons are judged entirely by Gemini 2.5-Pro (κ 0.6553 against humans on the seed set), and 950 of the 1,243 queries are voiced by TTS systems without filtering. The authors note in §Limitations that synthetic speech may favour models adapted to that distribution. The human pilot (Table 10) is on a different subset and a different model set: there LLaMA-Omni ranks last (714, below SpeechGPT 849 and Mini-Omni 841), while in the Gemini-judged arena it is the strongest academic system (908.7 vs. 677.1 and 676.4, Table 4). Human and automatic rankings therefore do not agree on every system.

  • The judge sees one concatenated audio prompt and returns a single win/loss covering instruction alignment, expressiveness and audio quality, so the scores cannot separate paralinguistic expression from content quality or audio fidelity. No per-criterion scores are reported.
  • Arena sampling favours pairs with moderate Elo gaps, and match counts are uneven (138 to 288 per model, Table 4). Sequential Elo also depends on match order; the bootstrap footnote addresses ranking stability but no confidence intervals per model are given.
  • The benchmark covers utterance-level, short-range interaction only. Persona consistency, long-term emotional dynamics and discourse coherence are out of scope (§Limitations). Full-duplex behaviour such as turn-taking and barge-in is not evaluated.
  • Seed-data quality control and the human pilot rely on Mandarin-speaking annotators, while 75.79% of the augment set is English (§3.3.1, §3.4, Appendix A.5).
  • Commercial systems are accessed through APIs whose versions may change (GPT-4o-realtime is pinned to the 2024-10-01 preview), which limits reproducibility of the ranking.

Wiki Connections

  • Spoken Language Model — Benchmarks ten LLM-based speech systems by feeding them real spoken queries and judging their spoken responses, with an analysis of backbone, encoder and decoder choices.
  • Speech-to-Speech — Evaluates the generated speech of speech-to-speech dialogue systems directly in the audio modality, including requested tempo, emotion and singing in the output (L3/L4).
  • Subjective Evaluation — Uses 19 human annotators making 432 pairwise preference judgments on generated spoken responses to build a pilot ranking and validate the automatic judge.
  • Evaluation Metrics — Validates audio-LLM-as-judge pairwise scoring against human judgments (Cohen’s κ 0.6553 for Gemini 2.5-Pro vs. 0.4667 for Qwen2.5-Omni).
  • Qwen2.5-Omni — Evaluated system with the highest sequential Elo, also tested as a candidate judge with lower human agreement.
  • GLM-4-Voice — Evaluated industrial system leading the middle tier (Elo 1148.2), listed with a quantized Whisper encoder and a flow-matching plus HiFi-GAN decoder.
  • Kimi-Audio — Evaluated system whose weak L2 perception score is used to argue that vector-quantized input gives no clear benefit.
  • FunAudioLLM — Evaluated system with a strong Entertainment but weak Medical domain profile, and the basis of the SenseVoice cascade in the human pilot.
  • LLaMA-Omni — Strongest academic system in the automatic arena but last in the human pilot.
  • SpeechGPT — Evaluated academic system with discrete mHuBERT units, lowest at L3.
  • Mini-Omni — Evaluated academic system with a 0.5B backbone, competitive at L1 but weak on paralinguistic levels.
  • VoiceBench — Prior voice-assistant benchmark that the paper contrasts as judging spoken responses through transcripts only.