arXiv · 2026 · Preprint
Nicolas M. Müller et al. (Fraunhofer AISEC / Resemble AI) · → Paper · Demo: ? · Code: ✓
A crowdsourced real-or-fake listening game with 1,768 participants and 35,532 judgments over 138 TTS and VC systems, replicating a 2021 protocol, finds that human accuracy on synthesized speech has barely moved while accuracy on genuine speech has fallen by 8.6 percentage points: listeners have become more suspicious of real audio rather than worse at hearing synthesis artifacts.
Problem
Prior human studies of audio deepfake perception evaluated at most 13 attacks, all from systems that predate 2021 (the earlier study by the same group used ASVspoof 2019 LA attacks). Since then, codec language models, flow-matching TTS, and commercial voice-cloning APIs have changed what synthetic speech sounds like, but nobody had measured how humans perceive this newer generation at scale, or whether exposure to better deepfakes has changed how listeners treat authentic recordings.
Method
The study is a direct replication of the 2021 setup on a much larger and more modern stimulus pool. A public web game presents one English clip per round; participants may replay it freely, then label it real or fake, after which the interface reveals the ground truth and the prediction of an ML detector (immediate feedback). Each round is bona fide with probability 0.5; otherwise an attack is drawn with an active-learning weight proportional to one minus its running human accuracy, so easily detected systems are shown less often. Before playing, participants report age, self-rated IT skill (1 to 5), and whether English is their native language; no registration is required.
Bona fide audio comes from LJSpeech, In-the-Wild, and ASVspoof 5. Fake audio comes mostly from ASVspoof 5 and the English subset of MLAAD, covering 138 TTS and VC systems grouped into families used as an analysis axis: Seq2Seq (e.g. Tacotron 2), VITS, XTTS, flow-matching (e.g. F5-TTS, CosyVoice), diffusion (e.g. Grad-TTS, StyleTTS 2), AR-LM over codec tokens (e.g. VALL-E, Bark, Chatterbox), voice conversion (e.g. RVC, OpenVoice V2), commercial APIs (e.g. ElevenLabs, Resemble AI), ASVspoof 5 attacks, and an “Other” group shown in the per-family breakdown. The reference detector is an in-house Wav2Vec 2.0 front end with an AASIST back end, trained on public and internal data that are not further specified. After excluding participants with fewer than 5 rounds and attacks with fewer than 10 judgments, 1,768 participants and 138 attacks remain. Accuracy is reported with 95% bootstrap confidence intervals; the IT-skill comparison uses a Mann-Whitney test. The anonymized per-round dataset (without demographics) and analysis code are released.
Key Results
- Skepticism shift (Table 1, Figure 1). Overall human accuracy fell from 72.8% (2021) to 68.7% (2026). The drop comes almost entirely from bona fide audio: accuracy on real samples fell from 72.7% to 64.1%, while accuracy on fake samples moved only from 72.9% to 71.2%.
- Per-family human accuracy (§4.1, Figure 3). Commercial APIs were hardest to detect (61.3%), followed by AR-LM systems (65.9%); these two families together contain 65 models and 8,929 samples. The figure shows VITS and XTTS at about 69%, diffusion at about 71%, Other at about 72%, VC at about 74%, Seq2Seq at 75.4%, flow-matching at 76.8%, and ASVspoof 5 attacks at 85.9%. Human accuracy on real audio (about 64%) sits below every fake family except commercial APIs. In the discussion, commercial plus AR-LM samples together are detected at 65.4% versus 74.9% for the remaining “legacy” families (§5), so the stable overall fake accuracy hides a shift in composition.
- ML detector (Table 1, Figure 3). The reference detector reached 94.5% overall (95.5% in 2021), more than 20 points above humans in both periods. Its weakest families were AR-LM (83.7%) and diffusion (84.7%); the figure shows it at 99% or above on several families (including real audio) and 94% on commercial APIs, so the families hardest for humans and for the detector only partly overlap.
- Demographics (§4.2, Figure 2). Age and native-English status showed no meaningful association with accuracy. Self-rated IT skill levels 1 to 4 were indistinguishable (median about 67%); level 5 scored about 4 points higher (median 72%, p < 0.001).
- Learning (§4.2, Figure 4). Accuracy was 67.0% in participants’ first five rounds and 71.0% for those who played more than 15 rounds, rising over roughly the first 20 rounds and then plateauing; active participants fell from 1,768 to 136 by round 50.
The 2021 versus 2026 comparison is not fully controlled: the stimulus pools, bona fide sources, and participant populations differ, so the real-sample drop may partly reflect different bona fide material as well as changed listener attitudes. The paper attributes it to listener skepticism but does not test alternative explanations.
Novelty Assessment
The protocol is not new: it reuses the 2021 game interface and active-learning sampler from the same group. The contribution is scale and timeliness, with roughly ten times more attacks and nearly four times more participants, covering post-2021 systems, plus the specific observation that the change since 2021 shows up on real rather than fake samples. The per-architecture-family breakdown is a useful descriptive axis, though families are coarse and uneven in size. The released per-round judgment dataset is a secondary contribution. There is no new metric or modelling method, and the reference detector is an in-house model with unspecified training data.
Field Significance
moderate: The paper provides a large human-perception data point on modern synthesized speech and reframes the deepfake risk around listeners’ distrust of authentic audio rather than only their failure to catch fakes. It also gives TTS researchers a crowd-level signal that codec language model and commercial outputs are perceptually closer to real speech than flow-matching or seq2seq outputs in this stimulus pool, although the comparison is confounded by which specific systems and samples each family contains.
Claims
- supports: Human listeners cannot reliably separate modern synthesized speech from real recordings, with accuracy well below automatic spoofing detectors on the same material.
Evidence: Across 35,532 judgments, overall human accuracy was 68.7% versus 94.5% for a Wav2Vec 2.0 + AASIST detector; the detector also outperformed humans by more than 20 points in the 2021 study. (§4, Table 1)
- complicates: Improved speech synthesis may degrade listeners’ trust in genuine speech more than it degrades their ability to spot synthetic speech, so fake-only detection accuracy understates the perceptual impact of synthesis quality.
Evidence: Compared with the 2021 study, accuracy on fake samples changed from 72.9% to 71.2% while accuracy on real samples fell from 72.7% to 64.1%, and real-audio accuracy ranked below most fake families. (§4, Table 1, Figure 1, Figure 3)
- supports: Speech from autoregressive codec language models and commercial voice-cloning services is harder for human listeners to tell apart from real speech than speech from seq2seq or flow-matching TTS.
Evidence: Human accuracy on fake samples was 61.3% for commercial APIs and 65.9% for AR-LM systems, versus 75.4% for Seq2Seq and 76.8% for flow-matching families; commercial plus AR-LM averaged 65.4% against 74.9% for the other families. (§4.1, §5, Figure 3)
- complicates: The synthesis families that are hardest for human listeners to detect are not necessarily the ones hardest for automatic detectors, which supports combining human and machine judgment.
Evidence: The detector’s weakest families were AR-LM (83.7%) and diffusion (84.7%), while it scored 94% on commercial APIs, the family where human accuracy was lowest (61.3%). (§4.1, §5, Figure 3)
- refines: Listener background matters little for detecting synthesized speech, while short practice with feedback gives a modest, quickly saturating gain.
Evidence: Age and native-English status showed no significant effect; only self-rated IT experts scored about 4 points higher; accuracy rose from 67.0% (first five rounds) to 71.0% (beyond 15 rounds) and plateaued after about 20 rounds. (§4.2, Figure 2, Figure 4)
Limitations and Open Questions
- The 2021 and 2026 conditions differ in stimuli, bona fide sources, and participant pools, not only in time. The real-sample decline is read as rising skepticism, but no analysis rules out an effect of the new bona fide material (for example In-the-Wild recordings, which vary in channel quality).
- Participants were self-selected through an open, anonymous web game and skew young; playback equipment and browser compression were uncontrolled; overlap with 2021 participants cannot be ruled out (§Limitations).
- English only.
- The active-learning sampler shows hard attacks more often, which yields uneven per-attack counts and makes per-family averages depend on which systems each family contains. Family boundaries are coarse (e.g. XTTS and VITS are separate from AR-LM and VAE families, and ASVspoof 5 is grouped by source rather than architecture).
- Feedback after every round, including the detector’s prediction, may change listener behaviour relative to real-world exposure to deepfakes.
- The detector is in-house, trained on unspecified public and internal data, and could have seen material related to the test sources, so its accuracy is a reference point rather than a reproducible baseline.
- No naturalness or quality ratings were collected, so the study cannot say whether perceived quality explains detectability across families.
Wiki Connections
- Subjective Evaluation — Contributes a large crowdsourced real-or-fake listening study of synthesized speech from 138 systems, with a longitudinal comparison to a 2021 replication.
- VALL-E — Named as a representative of the AR-LM family, the second-hardest family for human listeners.
- VALL-E 2 — Cited for its human-parity claim in zero-shot TTS, which motivates revisiting human detection of modern synthetic speech.
- F5-TTS — Representative of the flow-matching family, whose samples humans detected comparatively well.
- CosyVoice — Listed as a flow-matching family example among the stimulus systems.
- OpenVoice — Representative of the voice conversion family of stimuli.
- Tacotron 2 — Representative of the Seq2Seq family, among the easier families for human listeners.
- Grad-TTS — Representative of the diffusion family, which was also among the harder families for the ML detector.