SWE-bench, WebArena Exploited via Environmental Manipulation
Researchers at UC Berkeley demonstrated that every major AI agent benchmark — including SWE-bench, WebArena, OSWorld, and others — can be fully exploited to achieve near-perfect scores without solving a single task, using trivial environmental manipulation rather than genuine capability. The attacks include pytest hook injection, config file leakage, DOM manipulation, and reward component bypassing, with zero LLM calls required in most cases. This represents a systemic integrity failure in the evaluation infrastructure underpinning AI deployment decisions across industry and research.