arXiv · 2026 · Preprint
Haechan Kim et al. (KRAFTON / KAIST / Seoul National University) · → Paper · Demo: ✗ · Code: ?
Builds a Korean evaluation suite for speech LMs (two spoken QA benchmarks transferred from VoiceBench and OpenAudioBench through rulebook-guided “hypertranslation” and normalization, plus an audio understanding benchmark built from native Korean ASR corpora), and shows that English scores do not predict Korean performance and that Korean spoken QA and audio understanding rankings diverge.
Problem
Benchmarks for speech LMs are overwhelmingly English. The obvious way to extend them, a cascade of transcript translation, text normalization, and TTS, silently breaks items whose validity depends on the source language: case-sensitivity instructions have no Korean equivalent, letter-frequency constraints and English adjective-order (BBH hyperbaton) questions do not transfer, and a single normalization error on a number or unit can make a spoken question unanswerable. For audio understanding, translating transcripts does not help at all, because speaker age, gender, accent, and turn structure are properties of the waveform. Source benchmarks also carry their own label errors that a translation pipeline would carry over. Multilingual speech resources such as FLEURS, CoVoST 2, and SD-QA cover ASR, translation, or dialectal QA but give no procedure for transferring modern speech LM evaluation sets.
Method
The paper proposes two human-agent construction frameworks and instantiates both for Korean.
Spoken QA transfer (KVoiceBench, KOpenAudioBench). Four stages:
- Ground-truth correction. For sub-benchmarks with deterministic answers (multiple choice and short answer), a reviewer LLM (GPT-5.4) checks each source label and proposes corrections; a meta-reviewer LLM (Gemini Pro) independently verifies them, and only confirmed corrections are applied. Unresolved items are dropped.
- Hypertranslation. An LLM agent (Claude Sonnet 4, with six parallel sub-agents) classifies samples as direct translation, format conversion, equivalent replacement, deletion, or error correction, and escalates ambiguous cases to a human expert as structured questions. The expert’s decisions are codified into a released rulebook (two consultation rounds, 10 decision points). Examples: option labels A/B/C/D become Korean consonant labels, answer aliases are rebuilt in Korean and deduplicated, case instructions are removed, letter-frequency targets are remapped to Korean letters, and BBH hyperbaton is redesigned as Korean particle and verb-ending grammar judgments.
- Speech-friendly normalization. A second rulebook, also built through a human-agent loop, covers 11 categories (counter-dependent native vs. Sino-Korean numerals, units where ”/” means “per” only inside unit expressions, acronyms, chemical formulas, URLs, and so on). Claude Sonnet 4 applies it.
- TTS synthesis. Normalized text is synthesized sentence by sentence with Qwen3-TTS, conditioned on Korean male and female reference audio and transcripts. Whisper-large-v3 transcribes the output, and samples above a WER threshold (value not reported) are re-synthesized.
Audio understanding construction (KMMAU). Rather than transferring English audio, the framework turns native Korean ASR corpora (KSS, KMSAV, Seoul Corpus) into multiple-choice items, picking the construction method by capability: rule-based from speaker metadata (age, gender, number of speakers), rule-based from transcriptions (word order, word frequency), LLM-generated with human review (fact extraction, topic summary), and fully manual annotation by listeners (role/profession, general counting). Gender items have 2 options, age 3, and all others 4.
The resulting suite has 12,345 samples: KVoiceBench 7,306 (9 subsets), KOpenAudioBench 2,835 (4 subsets), and KMMAU 2,204 (9 capabilities). Eight speech LMs that support both languages are evaluated with a single run each (temperature 0.7, top-p 0.95). Spoken QA follows each source sub-benchmark’s metric: accuracy for multiple-choice and short-answer, GPT-5.4 judge scores (100-point scale) for open-ended tasks with the original VoiceBench judge prompt translated into Korean, prompt/instruction-level accuracy for IFEval, and rule-based refusal rate for AdvBench. KMMAU, MMAU, and MMAU-Pro report accuracy.
Key Results
Curation statistics. Ground-truth correction found 221 label errors in 4,308 deterministic English source items (5.1%), concentrated in Web Questions (134 errors, 13.4%) with near-zero rates on OpenBookQA (0.2%) and BBH (0.1%) (Table 2). Hypertranslation flagged 653 of 10,083 source samples (6.5%) for redesign or deletion, and 578 of 10,719 source samples (5.4%) were rejected overall (§4.4).
English vs. Korean spoken QA (Table 1). The table below defines drop as English score minus Korean score, so a positive number means lower Korean performance.
| Model | VoiceBench → KVoiceBench | Drop | OpenAudioBench → KOpenAudioBench | Drop |
|---|---|---|---|---|
| Raon-Speech | 76.8 → 66.6 | 10.2 | 70.2 → 52.1 | 18.1 |
| Qwen2.5-Omni | 66.7 → 49.0 | 17.7 | 66.7 → 39.2 | 27.5 |
| Audio Flamingo 3 | 41.6 → 18.8 | 22.8 | 38.9 → 12.6 | 26.3 |
| Step-Audio 2 Mini | 50.3 → 32.0 | 18.3 | 59.6 → 31.0 | 28.6 |
| InteractiveOmni | 62.4 → 20.0 | 42.4 | 66.7 → 11.5 | 55.2 |
| Fun-Audio-Chat | 73.6 → 50.1 | 23.5 | 72.4 → 43.1 | 29.3 |
| HyperCLOVA X 8B Omni | 48.7 → 45.1 | 3.6 | 57.4 → 45.1 | 12.3 |
| MiniCPM-o 4.5 | 76.1 → 39.5 | 36.6 | 74.8 → 35.7 | 39.1 |
Every model scores lower in Korean on both pairs, but the size of the drop ranges from 3.6 to 42.4 points on VoiceBench and 12.3 to 55.2 on OpenAudioBench. Raon-Speech has the highest Korean scores on both benchmarks (16.5 points above second-place Fun-Audio-Chat on KVoiceBench and 7.0 above second-place HyperCLOVA X 8B Omni on KOpenAudioBench). Raon-Speech is a KRAFTON model and the authors are KRAFTON-affiliated. The English leaders are close: Raon-Speech 76.8 vs. MiniCPM-o 4.5 76.1 on VoiceBench, while MiniCPM-o 4.5 leads OpenAudioBench (74.8 vs. 72.4 for Fun-Audio-Chat and 70.2 for Raon-Speech) yet falls to 39.5 and 35.7 in Korean. HyperCLOVA X 8B Omni shows the smallest drop, partly because its English scores are among the lowest.
Task-family effects (Table 5). The paper calls reasoning (BBH/KBBH) relatively robust; this holds for most models (Raon-Speech 85.1 → 83.5, Qwen2.5-Omni 52.9 → 49.9; Audio Flamingo 3 and HyperCLOVA X even score higher on KBBH), but InteractiveOmni collapses from 39.7 to 8.3. Multiple-choice knowledge tasks fall sharply for most models (e.g., OpenBookQA → KOpenBookQA: Qwen2.5-Omni 83.3 → 31.0, MiniCPM-o 4.5 87.7 → 25.2), while Raon-Speech keeps 74.8. Safety refusal changes unevenly: Audio Flamingo 3 drops from 48.7% refusal on AdvBench to 5.7% on KAdvBench and MiniCPM-o 4.5 from 98.9% to 48.5%, while HyperCLOVA X 8B Omni (85.8 → 83.2), Qwen2.5-Omni (99.4 → 95.9), Raon-Speech (96.7 → 87.3), and Fun-Audio-Chat (95.2 → 84.8) change less, and Step-Audio 2 Mini rises (56.2 → 64.3). Qwen2.5-Omni has the highest KAdvBench refusal rate.
Korean audio understanding (Table 1, Table 6). KMMAU ranks models differently from Korean spoken QA. Raon-Speech (71.8) and Fun-Audio-Chat (70.4) are within 1.4 points, Step-Audio 2 Mini is third (64.8) despite 32.0 on KVoiceBench, and Qwen2.5-Omni and MiniCPM-o 4.5 tie at 62.9. HyperCLOVA X 8B Omni, second on KOpenAudioBench, scores 31.6 on KMMAU. Capability leaders vary: Raon-Speech has the highest word-order accuracy (98.7, with Step-Audio 2 Mini at 94.0), Fun-Audio-Chat has the highest fact extraction (91.9), role/profession (92), and general counting (69.2), MiniCPM-o 4.5 has the highest topic summary (98) and number of speakers (36), and Qwen2.5-Omni has the highest gender accuracy (98.9). Number-of-speakers accuracy is at most 36% for every model on a 4-option task. The comparison with MMAU and MMAU-Pro is explicitly a cross-benchmark reference, not a paired language gap: InteractiveOmni (66.1 MMAU / 44.1 MMAU-Pro vs. 30.7 KMMAU) and HyperCLOVA X 8B Omni (53.2 / 40.5 vs. 31.6) are the two models whose KMMAU score falls well below their MMAU-Pro score (by 13.4 and 8.9 points); most other models score higher on KMMAU than on MMAU-Pro.
Novelty Assessment
The contribution is evaluation infrastructure, not a model. Translating benchmarks is standard practice in text NLP (XNLI, MLQA, MMLU-ProX), and the TTS and ASR components (Qwen3-TTS, Whisper) are off the shelf. What is new is making the transfer procedure explicit and auditable for speech: a released hypertranslation rulebook that redesigns items rather than translating them, a separate normalization rulebook targeted at TTS failure modes, a two-LLM label audit applied before transfer, and a capability-dependent recipe for building audio understanding items from native ASR corpora instead of translated audio. The curation statistics (about 5% of items needing correction, redesign, or removal) give the cascade-transfer critique some quantitative weight. The construction relies heavily on commercial LLM agents, and the paper does not ablate the rulebooks against a naive translate-and-synthesize baseline, so the size of the validity gain over a plain cascade is argued rather than measured.
Field Significance
Moderate — The paper releases a Korean spoken QA and audio understanding suite for speech LMs, together with rulebooks that can be reused for other target languages. Its evaluation demonstrates that English speech LM leaderboards give a poor guide to non-English spoken QA, refusal behaviour, and audio understanding, so it serves as evidence that multilingual speech LM claims need target-language benchmarks.
Claims
- complicates: Speech LM performance on English spoken QA benchmarks does not reliably predict performance on the same tasks posed in another language, and the size of the cross-lingual drop varies widely between models.
Evidence: All eight models score lower on KVoiceBench and KOpenAudioBench than on VoiceBench and OpenAudioBench, but drops range from 3.6 (HyperCLOVA X 8B Omni) to 42.4 (InteractiveOmni) points on VoiceBench; MiniCPM-o 4.5 has the highest OpenAudioBench score (74.8) but falls to 35.7 in Korean. (§5.2, Table 1)
- supports: Transferring speech benchmarks across languages through a plain translate-normalize-synthesize cascade produces a non-trivial fraction of invalid items, so language-specific redesign and label auditing are needed.
Evidence: An LLM reviewer and meta-reviewer confirmed 221 label errors in 4,308 deterministic English source items (5.1%, 13.4% in Web Questions); hypertranslation flagged 653 of 10,083 items (6.5%) for redesign or deletion, e.g., case instructions and English adjective-order questions; 578 of 10,719 source samples were rejected overall. (§3.1.1, §3.1.2, §4.4, Table 2, Table 3)
- complicates: Spoken question answering and audio understanding probe different capabilities of speech LMs, so a model ranking on one does not transfer to the other.
Evidence: Step-Audio 2 Mini scores 32.0 on KVoiceBench but ranks third on KMMAU (64.8), while HyperCLOVA X 8B Omni ranks second on KOpenAudioBench (45.1) but scores 31.6 on KMMAU. (§5.2, Table 1)
- complicates: Safety refusal behaviour of speech LMs learned or measured in English does not carry over consistently to spoken harmful requests in another language.
Evidence: Refusal rate drops from 48.7% to 5.7% for Audio Flamingo 3 and from 98.9% to 48.5% for MiniCPM-o 4.5 on AdvBench vs. KAdvBench, while Qwen2.5-Omni stays at 95.9% and Step-Audio 2 Mini rises from 56.2% to 64.3%. (§5.2, Table 5)
- complicates: Counting the number of speakers in conversational audio remains a weak point of current speech LMs.
Evidence: On KMMAU’s 4-option number-of-speakers items, the best of eight models reaches 36% accuracy (MiniCPM-o 4.5) and the lowest 21%, while the same models reach up to 98.9% on gender. (§5.2, Table 6)
Limitations and Open Questions
Warning
The spoken QA benchmarks use TTS-synthesized Korean speech from two reference voices, so they test comprehension of clean synthetic speech rather than natural spoken variability, and the transfer pipeline is not compared against a naive cascade baseline. All results are single runs, open-ended items are scored by an LLM judge, and the top-ranked model (Raon-Speech) comes from the authors’ own organization.
- The WER threshold used to trigger re-synthesis is not reported, and the synthesized audio receives no listening test or quality metric beyond that check.
- KMMAU is not paired with an English counterpart, so its comparisons with MMAU and MMAU-Pro are cross-benchmark references rather than controlled language gaps; the paper says so explicitly.
- Benchmark construction depends on commercial LLMs (GPT-5.4, Gemini Pro, Claude Sonnet 4), and the label-correction step prioritizes precision, so some source errors likely remain.
- Extending the framework to another language requires a native speaker to rebuild both rulebooks; only Korean is demonstrated.
- The cited reference for Step-Audio 2 Mini is the earlier Step-Audio report, and the Raon-Speech technical report is cited without an arXiv identifier, which limits checking those systems’ training data for Korean exposure.
Wiki Connections
- Spoken Language Model — Evaluates eight existing speech LMs by feeding them Korean speech (synthesized questions and native ASR-corpus audio) and measuring spoken QA, safety, and audio understanding.
- Evaluation Metrics — Proposes rulebook-guided procedures for transferring speech LM benchmarks to a new language and for building audio understanding items from native ASR corpora.
- VoiceBench — Source benchmark for KVoiceBench; its sub-benchmarks are label-audited, hypertranslated, and re-synthesized in Korean.
- Baichuan-Audio — Introduces OpenAudioBench, the source benchmark for KOpenAudioBench.
- MMAU — English audio understanding benchmark used as a cross-benchmark reference and as the naming template for KMMAU.
- MMAU-Pro — Second English audio understanding reference benchmark in the evaluation.
- Qwen3-TTS — TTS system used to synthesize the Korean spoken QA audio from reference voices.
- Qwen2.5-Omni — Evaluated model; has the highest refusal rate on KAdvBench.
- MiniCPM-o 4.5 — Evaluated model; highest OpenAudioBench score but one of the largest English-to-Korean spoken QA drops.
- Fun-Audio-Chat — Evaluated model; second on KVoiceBench and KMMAU and leader on several contextual KMMAU capabilities.