OpenAI and Paradigm's EVMbench measures AI agents on detecting, patching, and exploiting smart contract bugs

What EVMbench actually tests

OpenAI built EVMbench with Paradigm to evaluate AI agents on three separate smart contract security tasks: detecting vulnerabilities, patching them, and exploiting them. The benchmark draws on 117 curated vulnerabilities from 40 audits, most sourced from open Code4rena audit competitions.

It also pulls scenarios from the security auditing of Tempo, an L1 built for high-throughput, low-cost stablecoin payments. OpenAI frames this as extending the benchmark into payment-oriented contract code, where it expects agentic stablecoin payments to grow.

The three modes are graded differently. Detect scores recall against ground-truth vulnerabilities and their audit rewards. Patch requires eliminating exploitability while preserving functionality, verified by automated tests. Exploit runs end-to-end fund-draining attacks on a sandboxed chain, graded by transaction replay and on-chain verification.

The gap between breaking and fixing

The headline number is exploit-mode performance: GPT-5.3-Codex via Codex CLI scores 71.0%, up sharply from GPT-5's 33.3% roughly six months earlier. But detect recall and patch success both remain below full coverage, and OpenAI is candid that a large fraction of vulnerabilities stay hard to find and fix.

The behavioral explanation is worth noting. Agents do best when the objective is unambiguous — in exploit mode they iterate until funds are drained. In detect mode they sometimes stop after finding one issue rather than auditing exhaustively. In patch mode, removing a subtle bug without breaking intended functionality is the sticking point.

That asymmetry matters more than the top-line score. The defensive tasks OpenAI wants to encourage — comprehensive auditing and safe patching — are precisely the ones where the agents are weakest, while the offensive task is where they excel.

The engineering behind reproducible grading

To make exploit results trustworthy, OpenAI wrote a Rust-based harness that deploys contracts, replays agent transactions deterministically, and restricts unsafe RPC methods. Exploit tasks run in an isolated local Anvil instance, not live networks, and every vulnerability is historical and publicly documented.

They also red-teamed the exploit environments to find and patch ways an agent might cheat the grader, and used automated task-auditing agents alongside Paradigm's domain expertise to check environment soundness. For patch mode, they verified each vulnerability was exploitable and fixable without compilation-breaking changes.

The stated limitations are specific rather than boilerplate. Sequential transaction replay puts timing-dependent behaviors out of scope. The clean Anvil state is not a mainnet fork, only single-chain environments are supported, and some cases require mock contracts. In detect mode, OpenAI concedes it cannot reliably tell whether extra issues an agent flags are real findings humans missed or false positives.

Why a dual-use benchmark ships with grant money attached

EVMbench is explicitly dual-use: the same capability that audits a contract can drain it. OpenAI's response is to release the tasks, tooling, and evaluation framework while pairing measurement with defensive commitments — expanding the private beta of its Aardvark security agent, offering free scanning to open-source maintainers, and committing $10M in API credits through its Cybersecurity Grant Program for defenders, especially in open source and critical infrastructure.

The practical takeaway for anyone deploying smart contracts is that the exploit-detect gap is a warning, not reassurance. A 71% exploit score against historical Code4rena bugs signals that attacker-side automation is arriving faster than defender-side reliability. Teams securing on-chain assets should treat AI-assisted auditing as a supplement that still needs exhaustive human review — because the benchmark itself shows agents stop early on exactly the audits that matter.