WhatisSWE-bench,andhowshouldyoureaditsscores?

What is SWE-bench?

SWE-bench is a benchmark that tests whether an AI system can fix real bugs in real open-source repositories. Each task is a GitHub issue paired with the repository at the commit just before a human fixed it. The model gets the issue text and the code, produces a patch, and the patch is graded by running the tests that shipped with the human fix. The original paper (Princeton, ICLR 2024) built 2,294 tasks from 12 popular Python repositories. The best model at the time, Claude 2, resolved 1.96% of them.

The grading rule is strict and mechanical. The Verified dataset card lists two test sets per task:

• FAIL_TO_PASS: tests tied to the issue that fail before the fix and must pass after it.
• PASS_TO_PASS: tests that passed before and must still pass afterward.

The SWE-bench harness applies the model’s patch inside a Docker container, adds the hidden test changes and runs them. A task counts as “resolved” only if both conditions hold. The headline score is the percentage of tasks resolved.

The model never sees the tests. That is the strength of the design and the source of most of its problems: if the hidden tests check a function name the issue never mentioned, a correct fix fails, and if the tests are thin, a wrong fix passes.

What are the SWE-bench variants?

The variants differ in task count, language coverage and how hard they try to stay out of training data. “Bash only” is not a dataset; it is a fixed-scaffold filter on swebench.com.

A few details matter when you read a number:

• Pro has three splits. Scale’s launch post describes 731 public tasks, 858 held-out and 276 commercial tasks from private codebases, across 41 repositories. Public tasks come from copyleft (GPL-style) repos on the theory that they are less likely to sit in training data. After the October 2026 audits described below, Scale cut the public set to 642 tasks. A Pro score from before that date was run on a different task set.
• Live is the contamination answer. Because SWE-bench-Live keeps adding issues created after models’ training cutoffs, a high score on recent months is harder to explain by memory. Its lite and verified splits are frozen, so check which split a result used.

Why do scores for the same model differ?

Because a SWE-bench score measures a system: the model, the agent scaffold around it, the number of turns and attempts it gets, and the money spent per task. Change any of those and the number moves.

The cleanest evidence is on swebench.com’s own Verified board, which lists the same open weights under several agents (checked November 2, 2026):

Kimi K2 spans 27.4 points, from 43.8% in the bash-only setup to 71.2% inside the Lingxi agent. Qwen3-Coder 480B gains 14.2 points moving from bash-only to OpenHands. Claude 4.5 Opus moves less, 76.8% to 79.2%, which fits a common reading: stronger models need less help from the scaffold.

Budget is the second lever. Scale’s Pro leaderboard states that most entries ran with uncapped cost and a 250-turn limit, while older grayed-out entries ran with capped cost and 50 turns. The swebench.com bash-only data also records spend per task: MiniMax M2.5 resolved 75.8% of Verified at about $0.07 per task, against $0.75 for Claude 4.5 Opus at 76.8%.

Harness effects are smaller when the model is strong and the harnesses are mature. An imec benchmark from September 2026 ran Qwen3.8-27B through Claude Code, Codex and Pi on 64 SWE-bench Pro tasks and got 52%, 48% and 44%. GLM-5.3-Flash landed at 50% to 52% across the same three, but Codex used roughly half the tokens and GPU cost of the other two. The Quesma audit of Blitzy’s 66.5% on Pro makes the opposite point: a heavy harness that plans and verifies beat GPT-5.4’s self-reported 57.7%, and the authors concluded that “harnesses matter more this year than just switching between frontier models.”

Finally, vendor numbers and leaderboard numbers are different things. Mistral’s Devstral Small 2 card reports 68.0% on Verified; the bash-only run on swebench.com shows 56.4%. Qwen’s Qwen3.8-27B card reports 61.7 on SWE-bench Pro using the Claude Code harness, and says Qwen “corrected problematic tasks” before re-running baselines, so that 61.7 is on Qwen’s own refined version of Pro.

What are the known problems with SWE-bench?

There are four well-documented problems: flawed tests, contamination, leaks in the environment, and a gap between “tests pass” and “a maintainer would merge it.”

1. Flawed tests. The SWE-Bench+ paper (2024) found that 32.67% of successful patches in its sample had the fix already written in the issue or its comments, and 31.08% passed only because the tests were too weak. In February 2026 OpenAI said it would stop reporting Verified; The Decoder reported that its audit found at least 59.4% of the audited problems had tests that rejected correct solutions, often by enforcing implementation details the issue never described. The audited subset was the hardest 27.6% of the dataset, which models often failed.
2. Contamination. The same OpenAI post said GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash “could reproduce some original fixes from memory.” A December 2025 study found Claude models were 6 times better at naming the files to edit on Verified than on comparable Python benchmarks, given only the issue text.
3. Environment leaks. In September 2025, GitHub issue #465 showed that agents could run git log inside the evaluation containers and read future commits, sometimes including the fix. The images were rebuilt on September 11, 2025. SWE-bench Pro hit a similar problem: The Inference reports that a May 2026 audit caught Claude Opus agents reading answers from git history in more than 12% of reviewed rollouts, and that graders rejected correct patches 24% of the time and accepted wrong ones 8.5% of the time. Scale responded by removing 89 tasks, rewriting 529 problem statements and turning off network access, and graded its own fix.
4. Passing is not merging. METR had four maintainers of scikit-learn, Sphinx and pytest blind-review 296 AI patches that had passed the grader. Roughly half would not have been merged, and adjusted merge rates ran about 24 points below grader scores.

Saturation sits on top of all four. On Hacker News, SWE-bench co-creator Ofir Press wrote in April 2026 that Verified “is now saturated at 93.9%” and pointed to Multilingual and Multimodal as still unsaturated.

What do developers report in practice?

Practitioners mostly treat SWE-bench as a coarse filter and trust their own repos more. These are community reports, not our tests:

• In an r/LocalLLaMA thread on Terminal-Bench scores, commenters called Verified “increasingly contaminated and unreliable”, preferred SWE-bench Pro and named Terminal-Bench 2.0 the most accurate benchmark they had found.
• In a December 2025 r/LocalLLaMA thread on SWE-rebench results, a commenter said MiniMax M2.1 fixed an Android UI bug that Sonnet could not, despite the ranking, and summed it up as “real life vs numbers.” Others in the same thread were surprised how well the 24B Devstral Small did for its size.
• On Hacker News, a widely upvoted reaction to the SWE-Bench+ paper noted that filtering problem tasks dropped SWE-Agent with GPT-4 from 12.47% to 3.97%, which “matches my intuition” about real coding ability. Another commenter disputed some of the paper’s examples, so the critique is not settled either.

The pattern: use the leaderboard to shortlist, then decide on your own code.

How do open-weight models score on SWE-bench?

On swebench.com’s bash-only Verified board, the best open-weight models sit within a few points of closed frontier models, but the board’s most recent runs date from February 2026. Snapshot, checked November 2, 2026, all in mini-SWE-agent:

For reference, Claude 4.5 Opus scored 76.8% in the same setup. On the harder Multilingual board, GLM 5 reached 69.7% against 72.7% for the top entry, Gemini 3 Flash.

SWE-bench Pro is far less flattering. On Scale’s public leaderboard, the top entries are Muse Spark 1.1 at 61.50 and GPT-5.4 (xHigh) at 59.10. The listed open models trail: Qwen3-Coder 480B at 38.70, MiniMax 2.1 at 36.81, Kimi K2 at 27.67, gpt-oss-120b at 16.20 and DeepSeek V3.2 at 15.56. Those open-model rows are older runs and may predate the 642-task cleanup; the page does not date them.

Most of the models in the first table do not fit on a single consumer GPU. For which open models you can actually run, and on what hardware, see the best local LLM for coding.

How should you use SWE-bench when picking a local coding model?

Use it to narrow the field, compare like with like, then test on your own repository. A practical checklist:

1. Name the variant and date. “70% on SWE-bench” means nothing. Verified, Pro before October 2026 and Pro after it are three different tests.
2. Prefer bash-only or same-harness comparisons. If two numbers come from different scaffolds, the gap may be the scaffold. The DeepSeek Coder vs Qwen Coder comparison shows the same effect across vendor tables.
3. Discount vendor cards. Treat a model card’s score as an upper bound under the vendor’s best harness, and look for an independent run.
4. Favor fresh or private tasks. SWE-bench-Live, Pro’s commercial split and Multilingual are harder to memorize than Verified.
5. Run your own harness. The score you will see depends on the agent you use. Our local LLM coding agent guide covers which harnesses connect to local models and why tool calling and context length matter more than a leaderboard point.
6. Check cost and turns, not just accuracy. A local model that needs 3x the tokens to reach the same answer may still be slower in practice.

Benchmarks for refusals have the same shape of problem; LLM refusal rate benchmarks covers how to read those.

FAQ