SWE-bench Verified
Resolution rate on 500 human-validated real GitHub issues. Measured by Vals AI: Mini-SWE-agent. This benchmark is archived as saturated and is no longer run on new releases.
41models scored
measured 2026-09-01
measured 2026-09-01
Why this is archived
Vals archived this benchmark as saturated and no longer runs it on new releases. Scores move several points with the harness, and most figures quoted elsewhere are vendor self-reports rather than standardized runs.
Coverage
SWE-bench Verified ranks 88 models; this table covers the 41 of them whose pricing we have verified against the provider's own page, so 47 scored models are not shown.
Source
Vals AI — SWE-bench Verified, read 2026-11-20
Standings
| Model | Maker | Score | Blended $/MTok | Score per $ |
|---|---|---|---|---|
| 1Claude Opus 5 | Anthropic | 97.00% | $10.00 | 9.7 |
| 2DeepSeek V4 Pro 0813 | DeepSeek | 96.40% | $1.98 | 48.7 |
| 3gpt-5.6-sol | OpenAI | 96.20% | $8.00 | 12.0 |
| 4Grok 4.6 | xAI | 95.60% | $3.00 | 31.9 |
| 5gpt-5.6-terra | OpenAI | 95.40% | $4.50 | 21.2 |
| 6GLM-5.3 | Z.ai | 95.40% | $2.15 | 44.4 |
| 7Claude Fable 5 | Anthropic | 95.00% | $20.00 | 4.8 |
| 8Kimi K3 | Moonshot | 93.40% | $6.00 | 15.6 |
| 9gpt-5.6-luna | OpenAI | 93.00% | $0.45 | 206.7 |
| 10GLM-5.3-Flash | Z.ai | 92.00% | $0.237 | 387.4 |
| 11Claude Opus 4.8 | Anthropic | 88.60% | $10.00 | 8.9 |
| 12Grok 4.5 | xAI | 86.60% | $3.00 | 28.9 |
| 13GLM-5.2 | Z.ai | 82.80% | $2.15 | 38.5 |
| 14gpt-5.5 | OpenAI | 82.60% | $11.25 | 7.3 |
| 15Claude Opus 4.7 | Anthropic | 82.00% | $10.00 | 8.2 |
| 16Gemini 3.8 Flash | 80.00% | $1.50 | 53.3 | |
| 17Claude Sonnet 5 | Anthropic | 79.60% | $4.00 | 19.9 |
| 18Gemini 3.6 Flash | 79.60% | $1.50 | 53.1 | |
| 19Gemini 3.1 Pro Preview Preview 02/26 Preview | 78.80% | $4.50 | 17.5 | |
| 20Claude Opus 4.6 Thinking | Anthropic | 78.20% | $10.00 | 7.8 |
| 21gpt-5.4 xhigh | OpenAI | 78.20% | $5.63 | 13.9 |
| 22Kimi K2.7 Code | Moonshot | 78.20% | $1.71 | 45.7 |
| 23Claude Sonnet 4.6 | Anthropic | 77.40% | $6.00 | 12.9 |
| 24Claude Opus 4.5 Thinking | Anthropic | 76.40% | $10.00 | 7.6 |
| 25GLM-5.1 | Z.ai | 76.40% | $2.15 | 35.5 |
| 26Kimi K2.6 | Moonshot | 76.20% | $1.71 | 44.5 |
| 27gpt-5.2 | OpenAI | 75.80% | $4.81 | 15.8 |
| 28Gemini 3.5 Flash-Lite | 75.00% | $0.85 | 88.2 | |
| 29Gemini 3 Flash Preview 12/25 Preview | 75.00% | $1.13 | 66.7 | |
| 30gpt-5.4-mini | OpenAI | 73.00% | $1.69 | 43.3 |
| 31Grok 4.3 | xAI | 71.40% | $1.56 | 45.7 |
| 32GLM-5 | Z.ai | 71.40% | $1.55 | 46.1 |
| 33Claude Sonnet 4.5 Thinking | Anthropic | 70.00% | $6.00 | 11.7 |
| 34gpt-5.4-nano | OpenAI | 69.80% | $0.463 | 150.9 |
| 35gpt-5.1 | OpenAI | 69.80% | $3.44 | 20.3 |
| 36GLM-4.7 | Z.ai | 69.40% | $1.00 | 69.4 |
| 37gpt-5 | OpenAI | 69.00% | $3.44 | 20.1 |
| 38Claude Haiku 4.5 Thinking | Anthropic | 66.60% | $2.00 | 33.3 |
| 39Gemini 3.1 Flash-Lite Preview | 62.80% | $0.563 | 111.6 | |
| 40gpt-5-mini | OpenAI | 60.80% | $0.688 | 88.4 |
| 41Gemini 2.5 Pro | 54.40% | $3.44 | 15.8 |