Deepfake detectors lose half their accuracy in the wild: what to ask vendors

Lab results say deepfake detection is a solved problem. Real-world results say otherwise: when researchers tested leading open-source detectors on media actually circulating online, they lost roughly half of their headline accuracy compared with academic benchmarks. For banks, insurers and platforms buying detection technology, that gap is now a procurement risk.

The benchmark-to-reality gap

The Deepfake-Eval-2024 benchmark measured detectors against content collected from real social media and fraud contexts. As summarised in this review of real-world detector performance, AUC fell by about 50% for video, 48% for audio and 45% for images versus earlier academic test sets.

Other studies point the same way:

• A 2026 generalization study (arXiv 2603.05852) found speech deepfake detectors degraded sharply on real-world audio across languages and acoustic conditions.
• An in-the-wild evaluation (arXiv 2507.21905) found fewer than half of tested detectors beat an AUC of 60%, and simple JPEG compression made results worse.
• A benchmark of open-source image detectors (arXiv 2602.07814) saw a 37-point accuracy gap on unseen generators, from 75.0% for the best model to 37.5% for the worst.

Most of these studies tested open-source models, and they use different metrics, so the numbers are not directly comparable. The direction, however, is consistent.

Why benchmarks flatter detectors

A June 2026 paper, "What Do Deepfake Benchmarks Measure?", offers a sobering explanation. The authors showed that a simple linear probe on frozen, general-purpose representations comes close to the performance of purpose-built detectors across video, image and audio. Their conclusion: many benchmarks largely reward general modality understanding rather than forensic understanding.

In practice, that means a high benchmark score may say little about whether a detector spots the artifacts of a new generator, a compressed phone call or a real-time voice clone routed through a contact center.

What this means for fraud teams

Production audio looks nothing like a clean research dataset. Calls are compressed, noisy, narrowband and often streamed live. Attackers also iterate quickly, so a detector tuned to last year's generators can fail silently on this year's. The risk is twofold: missed fraud that passes as genuine, and false positives that block real customers and erode trust in the control.

It also changes how detection should be bought. A vendor slide showing 99% accuracy on a public dataset is a starting point for a conversation, not evidence of fitness for your channels. Ask for results on data that resembles your own traffic, and insist on seeing how the system behaves when it meets something it has never seen before.

Questions to put to any detection vendor

• Was the model tested on unseen generators, and how did accuracy change?
• Is performance reported on telephony-grade, compressed and multilingual audio, not just studio recordings?
• What are the false positive and false negative rates at your operating threshold, not just AUC?
• How often is the model retrained as new voice and video generators appear?
• Does performance hold across accents, languages, age groups and demographics?

Build for generalization, not leaderboards

The safest posture is layered. Combine deepfake detection with liveness checks, device and channel signals, and continuous verification, so no single score decides an outcome. Run your own evaluation on your own traffic, and re-test on a schedule. A detector that earns trust in your environment is worth far more than one that tops a public leaderboard.

See how Corsound AI Deepfake Detect is built for real-time, real-world audio and video, and talk to our team about validating detection against your own call and onboarding data.