arXiv · 2025 · Preprint
Kubis et al. (Adam Mickiewicz University / Samsung R&D Institute Poland) · → Paper · Demo: ? · Code: ✓
C3T (Cross-modal Capabilities Conservation Test) is a benchmark that quantifies how well speech-aware LLMs preserve language understanding capabilities when accessed via speech, measuring both fairness across speaker demographics and robustness across text and audio modalities.
Problem
Existing benchmarks for multimodal LLMs with audio input focus on foundational speech capabilities such as ASR accuracy and audio scene classification, but they do not systematically assess whether language understanding capabilities transfer faithfully from text to speech modalities. In particular, no prior benchmark tracks whether a model treats speakers of different age, gender, or accent equitably when answering language understanding tasks — a failure mode that raw accuracy metrics can mask entirely. Adapting textual benchmarks naively by recording text samples is also problematic because many standard LLM tasks (multi-step mathematical problems, long exam questions) are implausible to speak aloud, and a single-speaker recording cannot reveal demographic bias.
Method
C3T builds on tasks drawn from BIG-Bench-Hard (a subset of the Open LLM Leaderboard v2) that are filtered to retain only prompts plausible for voice communication. Out of 154,536 candidate samples, 2,470 text prompts from eleven BIG-Bench-Hard tasks survive manual and automatic filtering that removes overly long or symbol-heavy items.
To synthesize speech for a diverse pool of speakers without requiring costly human recordings, the authors train a voice cloning TTS model designed after Voicebox. The model operates via Conditional Flow Matching (CFM), mapping Gaussian noise to a target audio distribution. Its backbone is a transformer augmented with rotary positional embeddings (RoPE) in place of both the ALiBi attention bias and the convolutional positional encoding used in the original Voicebox. For accent evaluation, 29 accent groups from the ESLTTS dataset each contribute one synthesized audio prompt per task instance. For age and gender evaluation, speakers from the GLOBE dataset yield 14 prompt variants per task instance (seven age ranges, two genders, restricted to Australia, England, and Scotland accents).
The benchmark introduces three complementary metrics beyond standard exact-match accuracy. Overall fairness measures the fraction of tasks for which all speakers receive the same model answer. Conditional fairness disaggregates this by speaker characteristic (age, gender, or dialect). Cross-modal robustness requires a task to be fair across speakers and also produce the same answer as the text-input condition. The evaluation covers Audio Flamingo 3, Qwen2-Audio, Ultravox, and Voxtral Mini.
Key Results
Transitioning from text to speech input causes an exact-match accuracy drop of 4–13 percentage points across all tested models, with Ultravox performing best in both modalities (Table 1). Overall fairness scores are considerably higher (0.43–0.83), because models often produce the same wrong answer for all speakers. When evaluation is restricted to tasks that the model answers correctly for at least one speaker (the constrained fairness condition), performance collapses: the best constrained fairness score is 1.6% (Ultravox), meaning that in more than 98% of solvable tasks, at least one speaker demographic receives an incorrect answer. Conditional fairness scores for age, gender, and dialect follow the same pattern, with fewer than 5% of tasks yielding consistent results under the constrained condition. Audio Flamingo 3 achieves the highest overall fairness (0.83) but shows a 13-point gap between fairness and cross-modal robustness (0.70), indicating that modality inconsistency persists even when all speakers are treated equally.
Novelty Assessment
The primary novelty is the benchmark design and its metric framework. The idea of using a voice cloning TTS model to scale speaker diversity in evaluation is practically well-motivated and effectively sidesteps the cost of recording large speaker pools. The TTS model itself is an engineering adaptation of Voicebox (replacing ALiBi and convolutional positional encoding with RoPE) rather than a novel architecture, and the authors do not claim otherwise. The counterfactual fairness framing applied to speech is borrowed from the textual LLM fairness literature. The contribution is thus measurement-oriented: a reusable benchmark design that exposes a gap existing evaluations overlook.
Field Significance
Moderate — C3T fills a genuine gap in the evaluation of speech-aware LLMs by introducing fairness and cross-modal robustness metrics alongside a scalable, TTS-based speaker simulation methodology. Its results demonstrate that high overall fairness scores can mask near-total failure on constrained solvable tasks, which has implications for how robustness claims in spoken conversational agent evaluations should be interpreted. The benchmark’s forward-compatible design (new tasks can be added without re-recording) is a practical advantage for future evaluations.
Claims
-
supports: Raw accuracy metrics are insufficient to surface demographic fairness failures in speech-aware LLMs.
Evidence: Models achieve overall fairness scores of 0.43–0.83, but when constrained to tasks they can solve for at least one speaker, constrained fairness drops below 5% across all models, a discrepancy invisible to accuracy-only evaluation. (§7, Table 1)
-
complicates: Speech-aware LLMs that perform robustly at the population level may still exhibit systematic speaker-group inconsistencies.
Evidence: Audio Flamingo 3 achieves an overall fairness of 0.83 but a constrained fairness of 0.002, and its cross-modal robustness trails overall fairness by 13 percentage points, indicating modality inconsistency even when all speaker groups receive the same answer. (§7, Table 1)
-
supports: Voice cloning provides a scalable alternative to human recording for constructing diverse-speaker evaluation benchmarks.
Evidence: C3T uses a Voicebox-based conditional flow matching TTS to synthesize 14–29 speaker variants per task instance, covering age, gender, and accent groups from ESLTTS and GLOBE, replacing a recording effort that would otherwise be infeasible at scale. (§3, §6)
-
refines: Cross-modal degradation in speech-aware LLMs reflects not only ASR errors but also speaker-characteristic-dependent response variation that survives correct transcription.
Evidence: The benchmark explicitly tests scenarios where transcription is correct but model output differs across speakers, isolating demographic sensitivity from recognition error as distinct failure modes. (§1, §5)
Limitations and Open Questions
The benchmark is English-only and the speaker pool for age and gender evaluation is limited to three accent regions (Australia, England, Scotland). TTS-synthesized speech may not fully capture the acoustic variability of natural recordings, and the degree to which voice cloning artifacts influence model behavior is not characterized. The evaluation relies on exact-match comparison, which restricts applicable tasks to those with short, unambiguous answers and prevents testing open-ended reasoning capabilities. The custom fairness and robustness metrics are not directly comparable to existing benchmark scores, limiting cross-study synthesis.
Wiki Connections
- Evaluation Metrics — C3T introduces fairness and cross-modal robustness metrics specifically for speech-aware LLMs, extending standard exact-match evaluation with speaker-demographic disaggregation.
- Zero-Shot TTS — the benchmark’s speaker simulation pipeline relies on a zero-shot voice cloning TTS model based on the Voicebox architecture to generate speaker-diverse audio prompts.
- Spoken Language Model — C3T evaluates spoken language models (Audio Flamingo 3, Qwen2-Audio, Ultravox, Voxtral Mini) on the degree to which their language understanding capabilities survive the text-to-speech-input transition.
- Flow Matching — the voice cloning TTS component uses Conditional Flow Matching as its generative backbone, following the Voicebox design.