The human control group for SWE-bench
Are you better
than AI?
AI coding agents are ranked on how many real GitHub issues they can fix. Nobody measured the humans. Take the exact same tasks, graded by the exact same tests, and find out where you land.
100 validated tasks15 min each86+ models comparedscored by swebench 5.0.2
On these 100 tasks · vals.ai runs
Full table →Claude Opus 56m 51s / task99%
Claude Fable 54m 21s / task99%
GLM 5.310m 29s / task98%
DeepSeek V4 Pro 08133m 05s / task97%
GPT 5.6 Sol2m 23s / task97%
GPT 5.6 Luna2m 51s / task97%
1
Get a real issue
A bug report from Django, sympy, scikit-learn and friends, picked from the tasks SWE-bench rates as “<15 min fix”.
2
Fix it in your browser
The whole repository at the original commit: file tree, full-text search, a real editor. Fifteen minutes on the clock.
3
Get graded, compare
Your diff runs through the official harness on a public GitHub runner. See which models you beat on exactly the tasks you did.
A fair fight, mostly
We kept everything that can be kept identical, and we're upfront about what isn't. Results are reported as they are, whichever side they flatter.
Identical
- Tasks
- Real GitHub issues from SWE-bench Verified, at the original commit
- Grading
- The official SWE-bench harness, same Docker images, same hidden tests
- Bar to pass
- Every FAIL_TO_PASS and PASS_TO_PASS test must pass
Different
- Time
- You get 15 minutes. Agents ran until they decided they were done.
- Execution
- You can read, search and edit, but not run code. Agents had a shell.
- Hints
- Neither side sees hints or the tests. Same as the leaderboard.