arXiv · 2025 · Preprint
de Seyssel and Dhekane (Apple) · → Paper · Demo: ✗ · Code: ✗
Introduces a three-axis taxonomy, evaluation aspect, foundation model capability, and task/protocol requirement, for systematically matching speech evaluation tasks to the speech foundation models they can meaningfully assess.
Problem
Evaluation practice for speech foundation models is disjointed. Long-standing benchmark suites (SUPERB, SLUE, LeBenchmark and their multilingual variants) apply fixed, fine-tuning-head-based task templates that assume a representation-extraction interface, a setup well suited to self-supervised encoders like wav2vec 2.0 and HuBERT but not to generative or interactive systems. As the field has diversified into spoken language models, multimodal speech-text systems, and full-duplex dialogue agents, existing survey and taxonomy efforts each cover only a slice of this space (self-supervised encoders, spoken language models, or large audio-language models specifically) and organize evaluations by benchmark purpose rather than by what a model can or must expose. This leaves no principled way to determine, given a model with known capabilities, which evaluations are feasible and meaningful for it, and which would be misaligned or uninformative.
Scope and Coverage
The paper proposes a three-axis taxonomy for organizing the speech model evaluation landscape. Axis 1 (Evaluation Aspect) captures what an evaluation measures: linguistic, understanding/reasoning, indexical, paralinguistic, acoustic/signal fidelity, codec reconstruction, and aesthetic. Axis 2 (Foundation Model Capability) captures what a model must expose to attempt a task: speech representation extraction, discrete unit extraction, sequence probability estimation, speech generation, speech continuation, multimodal (speech+text) processing, and real-time streaming/interaction. Axis 3 (Task/Protocol Requirement) captures what the evaluation setup itself demands: supervised ground-truth data, fine-tuning, zero-shot/direct evaluation, signal-level metrics, evaluation via auxiliary models, or human evaluation. Sections 3.1-3.4 walk through each axis and illustrate it by classifying a broad catalogue of existing evaluation tasks and benchmark suites, tabulated in Appendix A (Tables A.2-A.6), including SUPERB, SLUE, LeBenchmark, ML-SUPERB, FLEURS, XTREME-S, Codec-SUPERB, the Zero Resource Speech Benchmark, SALMon, MAD-Speech, EmphAssess, StressTest, Full-Duplex-Bench, and S2S Arena. Section 4 then applies the taxonomy prescriptively: given a model’s exposed capabilities, the framework filters which evaluations are feasible, and separately identifies capability/aspect combinations for which no adequate benchmark currently exists. As a conceptual and organizational contribution, the paper reports no original experiments, trained models, or new datasets of its own.
Key Results
Applying the taxonomy to the current benchmark landscape (§4.2), the paper finds that linguistic and acoustic-fidelity aspects are comparatively well covered by long-established metrics (WER/CER-style transcription accuracy, PESQ/STOI/MCD-style reconstruction quality), while paralinguistic/prosodic aspects (emphasis and stress transfer, expressive timing), indexical aspects (speaker, accent, and dialect preservation in generative output), and aesthetic/perceptual quality beyond mean opinion score remain comparatively underexplored and unstandardized. Cross-referencing model capability requirements (Table A.4) against protocol requirements (Table A.5) shows most existing benchmarks assume either a representation-extraction interface (SUPERB-style frozen-feature probing) or a generation interface, while composite capabilities such as full-duplex, real-time interaction are served by very few benchmarks; the paper cites Full-Duplex-Bench as one of the sole examples targeting turn-taking, backchanneling, and interruption handling directly. The paper also documents that evaluations relying on auxiliary models, such as ASR-based WER on synthesized speech or LLM-based scoring pipelines used in AudioBench and VoiceBench, trade scalability for an added dependency on the auxiliary model’s own biases and limitations (§3.3).
Novelty Assessment
The contribution is conceptual: it organizes an already fragmented evaluation landscape rather than introducing a new benchmark, metric, or model. Its genuine novelty lies in explicitly separating what a model must expose (Axis 2, Foundation Model Capability) from what an evaluation protocol demands (Axis 3, Task/Protocol Requirement), a distinction the paper argues prior surveys of self-supervised encoders, spoken language models, or large audio-language models do not make explicit, since those organize benchmarks primarily by model family or benchmark purpose rather than by capability-task alignment. The taxonomy itself is not empirically validated: the paper reports no experiment testing whether practitioners applying the framework select more appropriate evaluations, and the illustrative classification tables in Appendix A reflect the authors’ own judgment calls rather than a systematic literature-coding exercise with reported inter-rater agreement. The paper is explicit that its catalogue is illustrative rather than exhaustive and that its scope excludes language coverage entirely.
Field Significance
Moderate — this paper offers a framework that jointly organizes speech evaluation by what an evaluation measures, what a model must expose to attempt it, and what the protocol demands, addressing fragmentation across self-supervised-encoder, generative, and interactive-dialogue benchmark suites that prior surveys treated separately. Its value is primarily as a reference vocabulary and diagnostic tool for identifying evaluation coverage gaps (paralinguistic, indexical, and full-duplex evaluation) rather than as new empirical evidence about any specific model or metric.
Claims
- supports: Evaluation protocols for speech foundation models must be matched to the specific capabilities a model exposes, since a task that assumes a capability the model lacks yields an infeasible or misleading evaluation.
Evidence: The Foundation Model Capability axis (Axis 2) enumerates seven capability types and shows, for example, that encoder-only representation models cannot be meaningfully assessed with generation-based distributional metrics such as Fréchet Audio Distance, while generative models without exposed frame-level representations cannot be probed with intrinsic tasks like ABX discrimination or sWUGGY. (§3.2)
- complicates: Current speech evaluation benchmarks systematically underrepresent paralinguistic/prosodic, indexical, and aesthetic aspects relative to linguistic and acoustic-fidelity aspects, leaving expressive and social dimensions of speech comparatively unmeasured.
Evidence: Section 4.2 documents that paralinguistic/prosodic evaluation remains rare and unstandardized (limited to recent efforts like EmphAssess and StressTest), indexical aspects are typically treated as nuisance classification rather than integral evaluation dimensions, and aesthetic evaluation relies mainly on MOS and a small set of automatic predictors with little cross-domain validation. (§4.2)
- complicates: Automatic evaluation via auxiliary scoring models inherits the biases and limitations of the auxiliary model, so measured performance can reflect properties of the scorer rather than the system under test.
Evidence: The Task/Protocol Requirement axis identifies “Evaluation with Auxiliary Models” (e.g., ASR-based WER on synthesized speech, or ASR+LLM pipelines in AudioBench and VoiceBench) as scalable but introducing an additional layer of dependency that must be considered when interpreting results. (§3.3)
- refines: Interactive, full-duplex spoken dialogue capabilities such as streaming, turn-taking, and backchanneling require a distinct real-time-interaction evaluation axis that current benchmarks address far less thoroughly than offline generation or recognition tasks.
Evidence: Full-Duplex-Bench is identified as one of the few benchmarks explicitly targeting the Real-Time Streaming and Interaction capability, and §5 flags interactive/full-duplex evaluation as a systematic blind spot alongside meaning-preservation for discrete or latent semantic units. (§3.2, §5)
Limitations and Open Questions
Warning
The taxonomy abstracts away from language coverage entirely: it classifies evaluation tasks rather than individual benchmark instances, so it does not by itself reveal which evaluations are linguistically portable, even though the authors acknowledge that most existing datasets, metrics, and evaluation pipelines concentrate heavily on English. (§5)
The illustrative Appendix A tables (representative evaluation tasks and benchmarks) are explicitly not exhaustive and reflect the authors’ own classification choices rather than a validated coding scheme; new tasks and benchmarks continue to appear faster than any static catalogue can track. The taxonomy has also not been tested prescriptively: the paper does not report a case study in which a practitioner applied the framework to select evaluations and observed improved evaluation outcomes as a result.
Wiki Connections
- Evaluation Metrics — the paper’s central three-axis taxonomy classifies the full landscape of speech evaluation metrics and benchmarks by what they measure, what model capability they require, and what protocol they demand.
- Subjective Evaluation — treats human evaluation as both a distinct Evaluation Aspect (Aesthetics: MOS, preference tests) and a distinct Task/Protocol Requirement, and finds aesthetic/perceptual evaluation comparatively underdeveloped relative to automatic metrics.
- Spoken Language Model — defines spoken language models as a core class of speech foundation models and analyzes which evaluations (sequence probability estimation, speech continuation, multimodal reasoning) are appropriate given their exposed capabilities.
- Neural Audio Codec — treats codec reconstruction as a distinct evaluation aspect and reviews Codec-SUPERB style signal-level metrics (PESQ, STOI, F0 correlation, Mel distance) used to assess codec resynthesis quality.
- Full-Duplex-Bench — cited as one of the few existing benchmarks that targets the Real-Time Streaming and Interaction model capability, illustrating the taxonomy’s identified gap in standardized full-duplex evaluation.