AI coding benchmarks: compare the task before the score

AI coding benchmarks: compare the task before the score

SWE-bench, LiveCodeBench, EvalPlus, and Aider measure different coding tasks and should not be merged into one ranking.

A coding-model score only answers the question encoded by its task, repository, harness, and metric.

No current public table establishes which model “actually writes better code” in the abstract. Code synthesis, code repair, repository navigation, tool use, and reviewable integration are different tasks. A model may do well on one while failing the deployment a team cares about.

What each benchmark can show

The official SWE-bench repository describes an evaluation built around real GitHub issues and repository-level patches. The LiveCodeBench repository documents a contamination-aware coding evaluation. EvalPlus extends HumanEval and MBPP with stronger testing, while Aider’s leaderboard documentation explains its own coding comparison. These are four different instruments, not four votes in one overall poll.

Run a comparison your team can reproduce

Choose candidate models that meet the legal, hardware, and access constraints first. Then lock the exact model ID, revision, precision, provider route, system prompt, tool permissions, context limit, temperature, and budget. Use a small set of representative tasks: one bounded implementation, one bug fix, one multi-file change, and one task that should decline or ask for clarification.

For each run, retain the prompt, tool trace, patch, tests, elapsed time, token or account usage, and review verdict. Score accepted change rate and review burden separately from raw test pass rate. This does not replace public benchmarks; it makes their claims testable under the conditions that matter to the deployment.

The coding benchmark is a test of a particular task and harness, not a field-wide model ranking. LiveCodeBench’s authors report that relative model performance changes across tasks, so a leader on one evaluation need not lead on another. Compare a model’s run configuration and its accepted output against the same task before importing its score into a buying decision.

For background on a particular metric, see SWE-bench Verified’s blind spots. Model serving choice remains separate from benchmark rank; MLX vs llama.cpp on Apple Silicon covers a different deployment question.

A sourced comparison with a limited meaning

Aider’s leaderboard includes a gpt-5 (high) entry with 88.0% and a reported cost of $29.08, and a Kimi K2 entry with 59.1% and $1.24. Both rows use the diff edit format, but they specify different model routes: OpenAI for the first and OpenRouter for the second. These are results recorded by the benchmark maintainer, not a current API price quote or a new Groundy test. The higher-scoring row is also more expensive in that recorded run. (leaderboard documentation)

That gives a useful screening question: does the higher success rate justify its cost on the tasks you need completed? It does not supply the answer for a different repository, current model release, provider price or agent. Keep the exact row label and recorded setup with the number. A new model sharing a family name has not inherited this result.

A team can turn the same idea into a more relevant pilot. Select representative issues whose acceptance criteria are known before the agent runs. Give each candidate the same repository revision and allowed tools. Record both the proportion of accepted changes and the total cost of the attempt set, including failures. Dividing that cost by accepted changes is a declared local metric; it is not the same as the leaderboard’s reported run cost.

If a candidate needs human intervention, retain that intervention in the result. A patch that eventually passes after extensive repair should not silently become an autonomous success. Likewise, a refusal on an underspecified task can be the right outcome. The acceptance rule has to state what counts before a score is calculated.

Decision rule

Use public benchmarks to screen candidates, then choose the model and workflow that produces the best accepted change rate at an acceptable review, latency, security, and cost burden on representative work. A fixed, reproducible test beats a rolling winner table.