Index/Models/SWE-bench Verified

SWE-bench Verified

Resolution rate on 500 human-validated real GitHub issues. Measured by Vals AI: Mini-SWE-agent. This benchmark is archived as saturated and is no longer run on new releases.

41models scored
measured 2026-09-01
Why this is archived

Vals archived this benchmark as saturated and no longer runs it on new releases. Scores move several points with the harness, and most figures quoted elsewhere are vendor self-reports rather than standardized runs.

Coverage

SWE-bench Verified ranks 88 models; this table covers the 41 of them whose pricing we have verified against the provider's own page, so 47 scored models are not shown.

Source

Vals AI — SWE-bench Verified, read 2026-11-20

Standings

ModelMakerScoreBlended $/MTokScore per $
1Claude Opus 5 Anthropic 97.00% $10.00 9.7
2DeepSeek V4 Pro 0813 DeepSeek 96.40% $1.98 48.7
3gpt-5.6-sol OpenAI 96.20% $8.00 12.0
4Grok 4.6 xAI 95.60% $3.00 31.9
5gpt-5.6-terra OpenAI 95.40% $4.50 21.2
6GLM-5.3 Z.ai 95.40% $2.15 44.4
7Claude Fable 5 Anthropic 95.00% $20.00 4.8
8Kimi K3 Moonshot 93.40% $6.00 15.6
9gpt-5.6-luna OpenAI 93.00% $0.45 206.7
10GLM-5.3-Flash Z.ai 92.00% $0.237 387.4
11Claude Opus 4.8 Anthropic 88.60% $10.00 8.9
12Grok 4.5 xAI 86.60% $3.00 28.9
13GLM-5.2 Z.ai 82.80% $2.15 38.5
14gpt-5.5 OpenAI 82.60% $11.25 7.3
15Claude Opus 4.7 Anthropic 82.00% $10.00 8.2
16Gemini 3.8 Flash Google 80.00% $1.50 53.3
17Claude Sonnet 5 Anthropic 79.60% $4.00 19.9
18Gemini 3.6 Flash Google 79.60% $1.50 53.1
19Gemini 3.1 Pro Preview Preview 02/26 Preview Google 78.80% $4.50 17.5
20Claude Opus 4.6 Thinking Anthropic 78.20% $10.00 7.8
21gpt-5.4 xhigh OpenAI 78.20% $5.63 13.9
22Kimi K2.7 Code Moonshot 78.20% $1.71 45.7
23Claude Sonnet 4.6 Anthropic 77.40% $6.00 12.9
24Claude Opus 4.5 Thinking Anthropic 76.40% $10.00 7.6
25GLM-5.1 Z.ai 76.40% $2.15 35.5
26Kimi K2.6 Moonshot 76.20% $1.71 44.5
27gpt-5.2 OpenAI 75.80% $4.81 15.8
28Gemini 3.5 Flash-Lite Google 75.00% $0.85 88.2
29Gemini 3 Flash Preview 12/25 Preview Google 75.00% $1.13 66.7
30gpt-5.4-mini OpenAI 73.00% $1.69 43.3
31Grok 4.3 xAI 71.40% $1.56 45.7
32GLM-5 Z.ai 71.40% $1.55 46.1
33Claude Sonnet 4.5 Thinking Anthropic 70.00% $6.00 11.7
34gpt-5.4-nano OpenAI 69.80% $0.463 150.9
35gpt-5.1 OpenAI 69.80% $3.44 20.3
36GLM-4.7 Z.ai 69.40% $1.00 69.4
37gpt-5 OpenAI 69.00% $3.44 20.1
38Claude Haiku 4.5 Thinking Anthropic 66.60% $2.00 33.3
39Gemini 3.1 Flash-Lite Preview Google 62.80% $0.563 111.6
40gpt-5-mini OpenAI 60.80% $0.688 88.4
41Gemini 2.5 Pro Google 54.40% $3.44 15.8

Get an email when the index changes

Price changes, new tools, acquisitions and shutdowns, sent when the data is updated.