A team picking a coding model from the patch-style leaderboards is answering the wrong question if the job is 0-to-1 work. CLI-Tool-Bench, a structure-agnostic benchmark for ground-up CLI tool generation, hands seven frontier LLMs an empty workspace and a spec, and the best of them clears only 43.8%1 of tasks. Editing competence and architecture competence are separate evaluation surfaces, and only one of them shows up on the leaderboards most teams actually consult.
What does CLI-Tool-Bench measure that other benchmarks miss?
CLI-Tool-Bench measures whether an LLM can architect and ship a working command-line tool from a specification, starting from nothing, rather than whether it can locate and patch a defect inside a mature codebase. The distinction sounds semantic until you look at what the dominant benchmarks actually test. SWE-bench and its successors present the model with an existing repository, a failing test or issue description, and a defined location in the code where the fix belongs. The structure is given. The task is file-filling.
The CLI-Tool-Bench paper, authored by Ruida Hu and classified under cs.SE and cs.AI, names two limitations in that paradigm. First, existing benchmarks rely on predefined structural scaffolds, which reduces generation to filling in a skeleton someone else designed. Second, they depend on rigid white-box unit testing, which forces the generated code to conform to specific internal implementations rather than validating end-to-end behavior a user would actually observe. A model can pass both and still be unable to decide how many modules a tool needs, where argument parsing lives, or how to decompose a spec into components.
CLI-Tool-Bench inverts the setup. Each task gives the agent an empty workspace and a real CLI specification drawn from one of 94 real-world repositories spanning multiple programming languages and complexity levels. The agent must handle repository planning and dependency management autonomously. Nothing scaffolds the decomposition.
Evaluation is black-box. The generated software runs in isolated sandboxes, and its system-level side effects and terminal outputs are compared against human-written oracle implementations using a multi-tiered equivalence metric. The grader does not care whether the code is organized the way the reference is organized. It cares whether running tool --flag value produces equivalent behavior. That is the difference between grading a building’s floor plan and grading whether the doors open.
The paper is a preprint: arXiv moderates submissions but does not peer review them, and CLI-Tool-Bench’s v2 landed on July 17, 2026, eight days before this writing, with v1 dating to April 8, 2026. Treat its specific numbers as provisional. The structural critique, however, stands on its own regardless of what revision three says.
How well do frontier models actually score?
The best result across seven state-of-the-art LLMs is a 43.8%1 overall success rate, meaning the strongest model tested fails to ship a working CLI from scratch more than half the time. That number deserves a second read if you have spent the last year watching coding-agent leaderboards climb. On patch-style benchmarks, frontier models have been posting scores that make autonomous software engineering look like a near term staffing question. On greenfield CLI generation, the same tier of model is a coin flip with bad odds.
The paper’s evaluation runs each model against the same 94-repository task set under the same empty-workspace conditions, so the 43.8%1 ceiling is a like-for-like comparison across frontier systems rather than a single cherry-picked run. The per-model ordering is in the preprint, and given that v2 is eight days old and the result is not peer reviewed, quoting a ranked list here would age badly. What matters for model selection is the shape of the result, not the podium: no model in the current frontier cohort reliably architects greenfield CLI software end-to-end.
The counterintuitive detail is what the 43.8%1 ceiling implies about where the difficulty lives. These are the same models that pass white-box unit tests against existing codebases at rates teams now consider good enough to deploy. Handed a scaffold, they fill it. Handed an empty directory, they flounder. The scaffold itself was carrying a large share of the architectural work, and benchmarks that provide it were silently crediting the model with a skill it never demonstrated.
There is also a spend dimension. The evaluation found that higher token consumption does not necessarily yield better task performance. More on that below, but for model selection the immediate consequence is that you cannot buy your way past the 43.8%1 ceiling by letting the agent reason longer.
Why do agents fail at greenfield work?
Agents fail at 0-to-1 generation on architecture and scope, not on function-finding: they default to monolithic code structures that pass nothing end-to-end, and spending more tokens does not fix the decomposition problem. This is the paper’s most useful finding for practitioners, because it inverts the mental model most teams bring from patch benchmarks.
In a patching task, the hard part is localization: finding the function where the bug lives, understanding its contract, and writing a fix that satisfies the surrounding code. Models have gotten genuinely good at this. In a greenfield task, localization does not exist yet because there is no code to locate anything in. The hard part is deciding what files should exist, what each one owns, and how execution flows between them. According to the CLI-Tool-Bench results, agents exhibit a strong tendency to generate monolithic structures: one large file, all the logic inline, argument parsing interleaved with business logic and output formatting.
Anyone who has reviewed a junior engineer’s first solo project recognizes the shape. The failure is not ignorance of the language or the libraries. It is the absence of decomposition discipline, the thing seniority is supposed to supply.
The token finding sharpens this. The intuition behind test-time compute scaling is that more reasoning tokens let a model explore more of the solution space and converge on better answers. On this task class, the paper reports that higher token consumption did not reliably improve outcomes. Agents that burned more tokens appear to have spent them producing larger monolithic structures rather than better decompositions. Verbosity is not architecture. A model that writes a sprawling single-file program has not thought harder than one that writes well-factored modules; it has just committed to more surface area that must behave correctly end-to-end.
This matters because several agent products market extended reasoning as the lever for harder coding tasks. For greenfield CLI work, at least as this benchmark measures it, that lever is not connected to the outcome.
Should you trust patch leaderboards for model selection?
Not for 0-to-1 or prototyping work. Patch-style leaderboards measure whether a model can edit code inside a structure it was given, and the CLI-Tool-Bench results show that skill does not transfer to deciding the structure in the first place. If your evaluation pipeline for a coding model consists of SWE-bench variants, you have measured editing competence and called it coding competence.
The right frame is two evaluation surfaces. Surface one: can it edit code? Can it find the defect, respect existing conventions, and land a patch that passes the project’s tests? This is what mature-codebase benchmarks measure, and they measure it reasonably well. Surface two: can it architect code? Given only a spec and an empty workspace, can it choose a decomposition, manage dependencies, and produce something whose observable behavior matches a reference implementation? That is what CLI-Tool-Bench measures, and the 43.8%1 ceiling says the answer today is “sometimes.”
The two surfaces are not correlated strongly enough to infer one from the other. A model that tops a patching leaderboard can still ship an unmaintainable CLI from scratch, because the leaderboard never asked it to make a single architectural decision. Teams that bet on agents for prototyping, scaffolding new services, or generating internal tools are betting almost entirely on surface two, and most of them selected the model using surface one data.
This split has gotten more consequential as agents move from autocomplete to autonomous operation. The ANCHOR paper (arXiv:2607.10455, ICML 2026), which audits the alignment of autonomous CLI agents, notes that these systems already execute hundreds of actions across multi-hour sessions: writing code, running shell commands, browsing the web, managing cloud infrastructure. A clarification is in order, because the names are easy to conflate: ANCHOR studies agents that operate a terminal, while CLI-Tool-Bench evaluates models that build CLI tools. Different surfaces. But ANCHOR’s observation about session length and action volume is the load-bearing context for why greenfield generation quality matters. When an agent runs autonomously for hours, the code it generates at the start becomes the substrate everything else runs on. A monolithic, badly scoped foundation is not a style problem at that point; it is an operational risk multiplier.
How should you pick and govern a model for greenfield work?
Run a structure-agnostic end-to-end check against your own spec, treat editing and architecture as separate selection criteria, and budget human review for decomposition because agents will default to monolithic output. None of this requires waiting for the benchmark ecosystem to mature.
The practical procedure follows directly from how CLI-Tool-Bench is built. Take two or three CLI-shaped tasks from your actual backlog: the internal migration tool, the log-scraping utility, the codegen wrapper. Strip them down to a spec and an empty directory, the same setup the benchmark uses. Run each candidate model against the tasks in a sandbox, and grade the output the way the benchmark does: execute it, compare observable behavior against a reference you trust, and ignore whether the code is organized the way you would have organized it. Then look at the structure anyway, because you are the one who will maintain it. If the agent hands you a single file with all the logic inline, that is the monolithic failure mode showing up in your own evaluation, and it predicts pain in the next iteration even if the behavior checks pass today.
On the governance side, the token finding changes the cost model. If higher token consumption does not improve greenfield outcomes, then paying for extended reasoning on 0-to-1 generation tasks is buying latency, not quality. Route the budget differently: put the savings into a human decomposition pass, where an engineer sketches the module boundaries before the agent fills them in. That is, notably, the same division of labor that makes patching benchmarks look easy. The scaffold was always the valuable part. You can supply it yourself.
It is also worth keeping the deployment context in view. Modern coding agents run inside elaborate inference harnesses (Claude Code, Codex, OpenClaw are the named examples in coverage of harness-based agent training) that drive multi-turn reasoning and tool use. CLI-Tool-Bench targets exactly that regime, which means its results are relevant to the tools teams already run rather than to some hypothetical future agent. The 43.8%1 ceiling is a statement about the stack you can buy today.
What can’t this benchmark tell you yet?
CLI-Tool-Bench cannot tell you which model to pick from a ranked list, cannot replace scaffolded-editing benchmarks, and cannot be treated as a vetted result, because it is a 94-repository preprint whose second revision is eight days old. Those are the honest limits, and they constrain how the 43.8%1 figure should travel.
Scale first. Ninety-four repositories is a deliberate, curated set, and it is enough to establish that the failure mode is real and architectural, but it is not enough to produce stable per-model rankings. The specific ordering across the seven evaluated models, and the exact 43.8%1 ceiling, come from a single non-peer-reviewed preprint that was revised on July 17, 2026. Preprints on arXiv pass moderation, not review. A v3 could shift the numbers substantially, and independent replication does not yet exist.
Scope second. The benchmark is deliberately structure-agnostic and black-box, which is its strength as a measure of 0-to-1 architecture and its weakness as a general evaluation. It cannot tell you how a model performs at the patching, refactoring, and test-conformance work that makes up most of professional software engineering. The scaffolded, white-box benchmarks it criticizes still measure real things; they just measure a different surface than their marketing implies. Treat CLI-Tool-Bench as a complement, not a replacement. A model portfolio chosen with both surfaces in view beats one chosen from either leaderboard alone.
The durable takeaway survives any revision of the scores. Patching benchmarks do not measure ground-up architecture. Agents default to monolithic code when nobody supplies the decomposition. More reasoning tokens do not fix that. If your agents are going to scaffold projects, generate internal tooling, or run multi-hour autonomous sessions of the kind ANCHOR documents, then your model selection process needs a greenfield check you run yourself, on specs that look like your work, graded on end-to-end behavior. The leaderboards will not warn you. They were never measuring the thing you are about to ship.
Frequently Asked Questions
Does CLI-Tool-Bench evaluate autonomous agents that operate a terminal?
No. CLI-Tool-Bench measures whether a model can architect and ship a CLI tool from a spec. It does not test agents that execute shell commands, browse the web, or manage cloud infrastructure. The ANCHOR paper (arXiv:2607.10455) evaluates those autonomous terminal operators, which is a completely different evaluation surface.
How does CLI-Tool-Bench differ from SWE-bench in its evaluation method?
SWE-bench uses predefined structural scaffolds and rigid white-box unit testing to verify that a model can locate and patch a defect inside an existing codebase. CLI-Tool-Bench provides an empty workspace and uses black-box differential testing. It executes the generated software in isolated sandboxes and compares terminal outputs against human-written oracles, ignoring internal code structure.
What is the practical cost implication of the token consumption finding?
The benchmark found that higher token consumption does not monotonically improve task performance for greenfield generation. Agents that burned more tokens tended to produce larger monolithic structures rather than better decompositions. This means paying for extended reasoning on 0-to-1 tasks buys latency, not quality, and teams should budget human review for decomposition instead.
Can the 43.8% success rate be used to rank specific LLMs today?
No. The 43.8% ceiling and the per-model ordering come from a single non-peer-reviewed preprint (arXiv 2604.06742, v2) that was revised on July 17, 2026. The specific numbers are provisional and may shift in future revisions or with independent replication. The structural critique regarding monolithic defaults is more durable than the specific ranking.
