Graduating from CyberGym Level 1
It is time for the security industry to graduate from CyberGym Level 1.
CyberGym has been one of the most impactful cybersecurity benchmarks since its launch in June 2025. Compared to prior benchmarks, it represented a jump in scale and a step in the right direction for the industry: 1,507 tasks versus approximately 200 challenges for NYU CTF, 40 for Cybench, and 40 for CVE-Bench. Every task was derived from vulnerabilities found in real C/C++ projects through Google’s fuzzing service.
At launch, the task was hard for everyone. The top agent-model combinations hit only about 20%, while the best single model at the time, Claude Sonnet 4, reached a mere 17.9%. In April 2026, Mythos shocked the industry by scoring 83.1%.
The top of the leaderboard has since compressed, with multiple companies now claiming performance above 90%.
The benchmark has multiple difficulty levels (ranging from Level 3 to Level 0, with Level 3 being the easiest and Level 0 the hardest), and the industry is mostly still publishing Level 1 results.
In Level 1, the agent receives the description of a vulnerability that has already been reported and patched. The task is to locate the known issue in the code and prove it. The setup is intentionally not a novel discovery task. The agent is given the relevant pre-patch source code, context about the vulnerable component, and an interface for submitting PoC inputs to the vulnerable program. That is a useful capability to measure, but it captures only one part of the work security teams need done.
While CyberGym was a great step forward for the industry, it is time to graduate from CyberGym Level 1 as the primary measure of cyber capability, in favor of newer benchmarks.
At depthfirst, we believe cyber capability evaluations should be rigorous and representative of real-world scenarios.
Eventually every public benchmark saturates, and Level 1 is now there. Once a dataset is fixed and open, systems are optimized for it, scores compress at the top, and the aggregate number stops being informative.
A benchmark can be saturated before it reaches 100%. In the case of CyberGym, not every remaining failure cleanly reflects an agent’s inability to understand or reproduce the target vulnerability. During our research, we noticed certain “failed detections” stemming from harness-specific constraints, sanitizer or crash-trace mismatches, PoCs that triggered a different bug, or inconsistencies among the vulnerability description, patch, and ground truth. This creates a noisy practical ceiling, where the difference between 90% and 95% may reflect benchmark-specific optimization more than a meaningful improvement in real-world security capability. This does not diminish the benchmark’s contribution, but it does make small differences at the top of the leaderboard harder to interpret.
We encourage the industry to move away from CyberGym Level 1 to a more comprehensive set of benchmarks that test finding an unknown vulnerability, judging reachability, producing a reliable PoC, generating a patch, and verifying the patch fixes the issue without regressions. Measuring the false-positive rate per confirmed issue should also be part of the evaluation.
We have seen recent research efforts moving in this direction. CyberGym-E2E, published by the same research group at UC Berkeley, pushes the evaluation environment even beyond Level 0, covering discovery, PoC generation, and patch generation across 920 tasks in 139 projects. While it still has limitations in language coverage and supported bug classes, it is far more representative of the tasks application security teams face and provides a better evaluation of cyber capabilities.
Building benchmarks is hard, and public benchmarks can have short half lives once they are widely adopted. Kudos to the CyberGym group for continuing to innovate and improve the benchmark.
We have built something similar at depthfirst for our internal evaluations and we plan to share more about our methodology soon.