arXiv · 2026 · Preprint

Jonggeun Lee et al. (Seoul National University) · → Paper · Demo: ✗ · Code: ✓

SpeakerSleuth is a 1,818-instance, human-verified benchmark that tests whether large audio-language models (LALMs) can judge speaker consistency across multi-turn dialogues. Across 12 LALMs it finds poorly calibrated detection, weak turn-level localization, and a strong bias toward textual coherence over acoustic evidence. Discrimination by comparing candidates works much better.

Problem

Multi-speaker dialogue synthesis (podcasts, voice-overs, conversational agents) has to keep each speaker’s identity stable across many turns. Failures such as speaker confusion, timbre drift, and voice-quality variation often show up only when turns are heard together: a single generated utterance can sound fine on its own. The usual tool for speaker consistency is cosine similarity between speaker-verification embeddings (ECAPA-TDNN, WavLM). It compares only pairs of utterances, needs hand-set thresholds, and does not judge a whole dialogue at once. LALM-as-a-judge has been proposed for speech quality assessment, but no benchmark has tested whether LALMs can actually tell speakers apart from acoustics in a dialogue setting, or how they compare with embedding methods.

Method

The benchmark breaks speaker consistency into three tasks, each tied to a step in a generate, verify, and regenerate workflow:

  • Detection: a binary consistent/inconsistent judgment over all of a target speaker’s turns. This tests the model’s absolute decision threshold.
  • Localization: identifying which turn(s) are inconsistent, or “None”.
  • Discrimination: filling a masked turn by choosing among three candidates (classification), or ranking all three by similarity to the target speaker.

In the primary setting, models receive only the target speaker’s 5 audio turns plus a reference clip of at least 3 seconds from outside the segment. Interlocutor turns are withheld so that acoustic cues are isolated from text.

Construction. Dialogues come from four English sources: Bazinga (multi-party TV/movie dialogue, 201 Season-1 episodes from 14 series), AMI (evaluation-set meetings: spontaneous, overlapping, noisy, mostly non-native speakers), Behavior-SD (synthesized dialogues with fillers and backchannels), and DailyTalk (studio two-speaker conversational TTS data). Each segment has exactly 5 target-speaker turns, with at most 20 turns overall. Three scenarios are built per dialogue from identical content:

  • S1 (Fully Consistent): the original audio.
  • S2 (Gender Switch): one randomly chosen turn is voice-converted to an opposite-gender speaker.
  • S3 (Similar Speaker): one turn is converted to the most ECAPA-TDNN-similar other speaker.

FreeVC performs the conversion in the primary benchmark. An extended benchmark mixes in CosyVoice3, OpenVoice, and YourTTS. Qwen3-32B filters out segments that lack conversational coherence when taken out of context. Three expert annotators from the research team check audio quality, the naturalness of converted turns, and whether each scenario is valid. The final benchmark has 606 dialogues × 3 scenarios = 1,818 instances: 10.2 hours of audio from 197 speakers.

Systems evaluated. There are 12 LALM judges, all prompted at temperature 0 with no adaptation:

  • Proprietary: GPT-4o-audio and Gemini-2.5-Pro/Flash/Flash-Lite.
  • Open: Qwen2.5-Omni-3B/7B, Qwen3-Omni-30B-A3B, MiniCPM-o-2.6, Gemma-3n-E4B, Phi-4-multimodal, OmniVinci, and Audio-Flamingo-3.

There are also 6 embedding baselines: three aggregation rules (Pairwise Similarity, Centroid Distance, Reference Comparison), each with an ECAPA-TDNN or WavLM backbone and thresholds set in preliminary validation.

Metrics.

  • Detection: per-scenario accuracy and Balanced Accuracy, which averages S1 with the mean of S2 and S3.
  • Localization: macro-averaged per-instance precision, recall, and F1, plus a Balanced F1.
  • Discrimination: classification accuracy, NDCG@1/@2, and Exact Match against the ground-truth order S1 ≻ S3 ≻ S2.

VC Quality Ranking (§7.4). This add-on task ranks four candidates: the source clip plus OpenVoice, YourTTS, and CosyVoice3 clones. Three human annotators ranked 485 samples, with Kendall’s W = 0.860 on 50 shared samples, and their rankings serve as ground truth.

Key Results

  • Detection is poorly calibrated (§6.1, Table 2, Figure 5). LALMs cluster along the anti-diagonal:
    • MiniCPM-o-2.6 and Audio-Flamingo-3 nearly always answer “consistent” (S1 85.3% / 99.2%, S2 0.7% / 1.3%).
    • Gemini-2.5-Flash-Lite and Qwen2.5-Omni-7B lean the other way.
    • Gemini-2.5-Pro is the best LALM at 64.7% balanced accuracy, but reaches only 39.3% on S3 (similar speaker).
    • Embedding methods do about as well (Pairwise WavLM 64.9%), but show their own backbone-specific bias: ECAPA-based methods over-flag, while WavLM-based methods under-flag.
  • Localization is weak (§6.1, Table 2).
    • Phi-4-multimodal flags essentially nothing (S2/S3 F1 0.1%).
    • Gemma-3n-E4B flags nearly everything (recall 95%, precision 19%, which is chance for 5 turns), with near-identical S2 and S3 scores.
    • Only Gemini-2.5-Pro (S2/S3 F1 62.5/47.5) and Gemini-2.5-Flash (62.3/45.0) combine useful precision with an S2→S3 drop that tracks acoustic difficulty.
    • The best Balanced F1 overall is 61.7% (Pairwise WavLM).
  • Discrimination works much better than detection (§6.2, Table 2).
    • Gemini-2.5-Pro reaches 81.5% classification accuracy, 92.6% NDCG@2, and 71.5% Exact Match. Gemini-2.5-Flash reaches 75.6%, 88.3%, and 61.6%.
    • Pairwise/Centroid embedding methods are near-perfect at choosing the best candidate (93–99%), but their Exact Match is only 55–59%.
    • Embeddings are best at finding the top match, while Gemini-2.5-Pro is better at ordering all three candidates.
  • Textual context hurts (§7.1, Table 3). When interlocutor turns are given as text, most models’ S1 accuracy rises and their S2/S3 accuracy collapses. For example, Gemini-2.5-Flash-Lite drops 67.0 points on S2 and 66.0 on S3, so models miss even gender switches in textually coherent dialogue. Two exceptions: Gemini-2.5-Pro degrades on all scenarios, and Phi-4-multimodal shows the inverted pattern.
  • Removing the reference clip makes several models default to “consistent” (§7.2, Table 3). For example, Qwen2.5-Omni-7B moves from S1 32.0 → 98.5 and S2 72.3 → 2.8.
  • Robustness checks.
    • 10-turn dialogues: on average, −1.2 points for detection, −4.2 for localization, and −3.1 for discrimination, with rankings largely preserved (§7.3, Table 4).
    • Swapping in the 4-VC benchmark: average absolute changes of 1.9, 2.0, and 2.8 points (Appendix D.1, Table 10).
    • Longer clips monotonically improve S2/S3 performance for the top three LALMs (§7.3, Figure 6).
    • Judging each turn separately against the reference pushes most models’ balanced detection to about 50% (Appendix C, Table 8).
  • VC Quality Ranking (§7.4, Table 5). Gemini-2.5-Pro leads with 48.0% accuracy and 16.8% Exact Match, against a random baseline of 25.0% / 4.2%. Most other models are only modestly above random, and GPT-4o-audio is well below it (7.3%).

Two caveats affect how these numbers should be read. First, the “inconsistent” turns are voice-converted, so detecting them partly means detecting VC artefacts as well as identity change. Annotators screened converted turns for naturalness, and the 4-VC variant moves headline numbers only slightly. Second, embedding-method thresholds were tuned in preliminary validation, while LALMs run zero-shot with fixed prompts. Comparisons between the two families therefore reflect different amounts of tuning.

Novelty Assessment

The contribution is an evaluation design and benchmark, not a model. The new parts are:

  • the three-task breakdown of consistency judgment (absolute detection, turn localization, relative discrimination);
  • controlled scenario construction that holds dialogue content fixed and varies only the acoustics of one turn (gender switch vs. most-similar speaker);
  • the ablations isolating textual context, the reference anchor, and per-turn vs. whole-set presentation.

The finding that LALMs over-weight text has been suggested by prior attention-imbalance work, which the paper cites. What is new is a clean behavioural test in which the text is coherent by construction and so carries no signal. The scenarios are built with standard tools (FreeVC, ECAPA-TDNN nearest-speaker selection, an LLM coherence filter). The evaluated models are used off the shelf with fixed prompts, and no prompt-sensitivity analysis is reported.

Field Significance

Moderate. SpeakerSleuth gives the first controlled measurement of how well LALM judges handle dialogue-level speaker consistency, a dimension that pairwise embedding similarity covers poorly. Its main lesson is a caution for LALM-as-a-judge evaluation of multi-speaker synthesis: current models are badly calibrated as absolute judges and are swayed by text. They still show usable relative acoustic discrimination, which points toward comparative rather than absolute judging protocols. The paper’s arXiv comment states acceptance at ACL 2026 (Main); no Anthology record was available at ingest, so the page uses the arXiv version date.

Claims

  • complicates: Large audio-language models used zero-shot as judges lack calibrated absolute decision thresholds for speaker consistency, so binary consistent/inconsistent verdicts are unreliable even when the models can perceive the acoustic difference.

    Evidence: Detection balanced accuracy is below 60% for most of the 12 LALMs, with models split between always-consistent (Audio-Flamingo-3: S1 99.2%, S2 1.3%) and over-flagging behaviour. The best model, Gemini-2.5-Pro, reaches 64.7% but only 39.3% on similar-speaker substitutions. (§6.1, Table 2, Figure 5)

  • supports: Relative, comparative judgment protocols give more reliable LALM-based speaker similarity assessment than absolute binary judgments.

    Evidence: Gemini-2.5-Pro reaches 81.5% discrimination accuracy and 71.5% ranking Exact Match versus 64.7% balanced detection accuracy. Removing the reference clip, or comparing turns individually against a single reference, collapses detection toward “consistent” or toward about 50%. (§6.2, §7.2, Table 2, Table 3, Appendix C, Table 8)

  • complicates: Large audio-language model judges weight textual coherence over acoustic evidence, so adding conversational text context can hide speaker-identity errors in synthesized dialogue.

    Evidence: Adding interlocutor turns as text (coherent by construction) raises S1 accuracy but collapses S2/S3 detection for most models. For example, Gemini-2.5-Flash-Lite drops from 70.3% to 3.3% on gender switches and GPT-4o-audio drops from 32.8% to 6.3%. Localization F1 shows the same pattern. (§7.1, Table 3, Appendix E, Table 12)

  • refines: Speaker-embedding similarity is strong at picking the closest-matching voice but weak at threshold-based detection, turn localization, and full ranking, and its biases depend on the backbone.

    Evidence: Pairwise/Centroid ECAPA-TDNN reach 99.2% discrimination accuracy but only 58.6% Exact Match. ECAPA-based methods over-flag (Reference ECAPA S1 8.5%), WavLM-based methods under-flag, and the best localization Balanced F1 is 61.7%. Optimal thresholds vary widely across configurations. (§6.1, §6.2, Table 2, Appendix B.2, Figure 7)

  • complicates: Automatically ranking voice-cloning outputs by speaker similarity in agreement with human listeners remains largely unsolved for current audio-language models.

    Evidence: On 485 human-ranked samples (Kendall’s W = 0.860) comparing source audio with OpenVoice, YourTTS, and CosyVoice3 clones, the best LALM (Gemini-2.5-Pro) reaches 48.0% accuracy and 16.8% Exact Match versus a 25.0% / 4.2% random baseline. Most models are near random. (§7.4, Table 5)

Limitations and Open Questions

Warning

All inconsistent turns are produced by voice conversion (FreeVC in the primary benchmark), and all dialogues are English. Results describe how judges detect VC-substituted turns in English dialogue. They do not directly measure the timbre drift or speaker confusion that dialogue TTS systems produce.

  • Data and coverage.
    • DailyTalk contributes 567 instances but only 2 speakers, so speaker diversity is uneven across sources.
    • Individual acoustic factors (noise, reverberation, recording quality) are not isolated, and demographic effects (accent, age) are not analyzed (§Limitations).
  • Evaluation setup.
    • LALMs are evaluated with a single fixed prompt per task at temperature 0, and prompt sensitivity is not reported.
    • Embedding thresholds come from preliminary validation on unspecified data, which favours the embedding baselines somewhat.
    • Audio verification relies on three annotators from the authors’ own team.
  • Open questions.
    • Whether fine-tuning, or attention re-balancing during multimodal fusion, can fix the text-over-acoustics bias. The paper raises this but does not test it.
    • Whether comparative protocols can be turned into calibrated absolute verdicts.

Wiki Connections

  • Evaluation Metrics — Introduces a task suite and metrics (balanced detection accuracy, balanced localization F1, NDCG/Exact Match discrimination) for dialogue-level speaker consistency, and benchmarks LALM judges against embedding-similarity metrics. Inconsistent turns are built with off-the-shelf voice conversion systems (FreeVC, plus CosyVoice3, OpenVoice, YourTTS in the extended set).
  • Subjective Evaluation — Measures how well automatic LALM judges agree with human rankings of voice-cloning similarity (three annotators, Kendall’s W = 0.860), and finds weak agreement.
  • SpeechLLM-as-Judges — Represents the LALM-as-a-judge paradigm for speech quality that this benchmark stress-tests on speaker consistency.
  • Behavior-SD — One of the four source datasets, supplying the synthesized-dialogue condition.
  • ECAPA-TDNN — Serves both as an embedding baseline backbone and as the selector of the most similar speaker for S3 scenarios.
  • CosyVoice 3 — One of the voice conversion models in the extended benchmark and one of the cloning systems ranked in the VC Quality Ranking task.
  • OpenVoice — Used to generate inconsistent turns in the extended benchmark and as a voice-cloning candidate in VC Quality Ranking.
  • Qwen3-Omni — Evaluated as an LALM judge, and the third-best on discrimination (60.4% accuracy).
  • Qwen2.5-Omni — Its 3B and 7B variants are evaluated as judges and show opposite detection-threshold biases.
  • Audio Flamingo 3 — Evaluated as an LALM judge, and an extreme case of always predicting “consistent” (S1 99.2%, S2 1.3%).