A nearly self-hosted coding-agent factory is an assemble-it-yourself proposition: sandboxed agents that clone repositories, patch code, run tests, and open pull requests on infrastructure you control. The decision it forces is about ownership: with no hosted vendor in the loop, your team inherits the safety and secrets burden, and on evidence fetched 2026-08-22 that burden is unsolved.
What does a self-hosted factory actually demonstrate?
A working factory demonstrates that the loop closes, and nothing beyond that. An agent picks up a task, receives a repository, changes code, runs tests inside a sandbox, and opens a pull request, with no vendor runner in the path. That is a pipeline with state, credentials, and failure modes, which puts it well ahead of a prompt in a chat window. What it does not provide is any number a buyer needs: no cost per merged pull request, no failure or rollback rates, no security assessment of the sandbox, no measurement of how much reviewer time each PR consumed. A demonstration, not an evaluation.
Verification status matters for what follows. The source set behind this article, accessed 2026-08-22, contains no working factory’s internals, no managed-agent pricing, and no cost-per-merged-PR measurement. So the analysis rests on the pattern itself and on three arXiv benchmarks: ATBench for unsafe agent trajectories, MaliciousSkillBench for malicious agent-skill detection, and ReguSim for rule grounding. Vendor claims are labeled as vendor claims throughout.
What does a self-hosted coding-agent factory look like?
Stripped to stages, the pattern is five components on your own infrastructure: an ephemeral sandbox, a scoped repository clone, an agent loop with tool access, a test harness, and a pull-request writer with deliberately limited permissions. The engineering weight sits in the boundaries between stages, because every boundary is a policy decision you now own.
The sandbox is per task and disposable. Nothing persistent lives inside it: no standing credentials, no shared cache a later task can read, no network reach beyond what the task needs. The clone is scoped to one repository with a token that expires when the sandbox does. The agent loop gets a tool surface you define, and the size of that surface is the single most consequential choice in the whole design. The test harness supplies the objective signal, and the PR writer gets permission to open pull requests and nothing more: no merge rights, no push access to protected branches, merge authority stays with a human.
The benchmark record explains why the tool surface deserves that emphasis. Across ATBench’s 1,000 trajectories, agents invoked 1,954 tools drawn from pools spanning 2,084 available tools. Real agent runs touch a lot of machinery, and every tool is a thing the agent can misuse or a channel something else can reach it through. In a hosted product, someone else curates that surface and absorbs the escapes. In this architecture, the curation and the escapes are both yours.
How often do agent trajectories go unsafe?
In ATBench, 497 of 1,000 evaluated agent trajectories are labeled unsafe, against 503 labeled safe, averaging 9.01 turns and 3.95k tokens per trajectory. That near-even split is how the benchmark was constructed, not a rate observed in anyone’s production fleet.
The construction is still informative, because it shows what unsafe looks like mechanically: a trajectory in which the agent, mid-task, takes an action with harmful potential using the tools available to it. Two details sharpen the picture. The trajectories are short, about nine turns on average, and the benchmark’s long-context delayed-trigger protocol is built so that risk emerges across multiple stages of a trajectory rather than in the opening move. And the tool breadth cited above means the unsafe action rarely requires exotic machinery; it happens with the ordinary tools a coding agent already has.
What this cannot give you is your own denominator. A factory operator’s review load is a function of the actual unsafe-action rate of their agents on their repositories, and no production figure of that kind appears in the fetched sources. ATBench establishes that unsafe trajectories are common enough to study systematically and short enough to miss. Your rate is a telemetry problem, and telemetry is one more thing the self-hoster has to build.
Can automated screening replace the vendor’s safety layer?
No. In MaliciousSkillBench, the strongest detector, a word TF-IDF SVM, scores 0.932 Macro-F1 under random splits but falls to 0.665 when evaluated on sources it has never seen, and at that operating point it flags 62.4% of benign skills as malicious.
The benchmark is built for exactly this question: 9,740 agent skills, 7,505 malicious and 2,235 benign, evaluated under splits of increasing difficulty. The same detector holds 0.916 Macro-F1 under structural disjoint splits, where it sees new skills but familiar sources, then drops to 0.665 under Source-Disjoint evaluation, where the skill sources themselves are new. The reading is uncomfortable but plain: word-level features learn the vocabulary of the training sources, and when the source distribution moves, the signal goes with it. Novelty erases the classifier.
The second failure mode is the operating point. On held-out sources, the paper reports, the best detector still recovers 95.6% of malicious skills, which sounds like a working gate. But with 62.4% of benign skills flagged alongside them, the queue behind the gate buries the reviewers it was supposed to protect. A self-hosting team would tune the threshold down to a tolerable false-positive rate, and the detector would give back recall to a degree the paper does not report at that operating point, because nobody runs it there.
This lands directly on the factory builder. A self-hosted setup also decides which skills, extensions, and tool packages its agents may load, which is a supply-chain screening problem identical in shape to the one MaliciousSkillBench constructs. In a Devin-class product, that screening queue is staffed by someone else, at quality you cannot see. Self-hosted, the queue and the staffing are yours, and this benchmark is the best public estimate of what the gate achieves on sources it wasn’t trained on.
Do written rules bind agent behavior?
Partially, and only as mitigation. In ReguSim’s simulated trading environment, making the rules visible to the agent reduced rejected actions but did not eliminate them, for both DeepSeek V4 Pro and Gemini 3.5 Flash.
Two constraints before leaning on that result. ReguSim is a controlled simulation in financial compliance, not production coding telemetry, so its findings transfer to coding agents by analogy rather than measurement. And “rejected actions” is the study’s own accounting of rule violations caught in its environment, not a general misbehavior rate.
Even with those caveats, the direction of the finding matches what operators already suspected: a rule in the prompt behaves like a soft prior, not a constraint. The agent is less likely to violate a rule it can read, and nothing in the architecture enforces the reading. The design consequence for a factory is to move enforcement out of the prompt and into the harness, where permissions, allowlists, and merge gates live. That is the same set of components the self-hoster just took ownership of, which is the pattern of this whole decision: every control the benchmark says is unreliable in prompt form has to be rebuilt as infrastructure, by you.
What does the buy side offer once you discount the marketing?
On the verified record, a Devin-class product demonstrably offers consolidated model access and an absorbed operations layer; the quality of its safety gate is as invisible from outside as a homegrown gate’s is from inside.
The neutral facts first: Devin is Cognition Labs’ autonomous software-development agent, and the company raised $21 million in early 2024 at a $350 million valuation, per Wikipedia’s profile. The vendor’s desktop page advertises unlimited access to SWE-1.7, billed as “the fastest coding model in the world,” and claims more than 1 million developers and over 4,000 enterprise customers. The speed claim has no independent benchmark in the fetched sources, and the user counts are self-reported with nothing audited behind them. (The speed claim sits on the page that sells the model.)
The unverifiable numbers are not the real issue, though. The product you cannot inspect is the safety gate: its recall on novel sources, its false-positive rate, the number of humans who staff its review queue. MaliciousSkillBench’s numbers are the best available public estimate of what a 2026-era automated gate does on unfamiliar sources, and there is no evidence any vendor clears that bar, only that vendors absorb the operational cost of failing to. “Vendor absorbed” and “solved” are different claims, and the marketing conflates them.
GitHub Copilot’s coding agent and Claude Code on managed runners sit between full SaaS and full self-hosting, trading control for less operational surface. Neither could be priced or safety-characterized from the fetched sources, so they appear here as named options rather than comparators.
Build or buy: which axes actually differ?
The options diverge on who owns each control, not on whether the control is hard. The table below maps the decision axes the evidence actually speaks to.
| Decision axis | Self-hosted factory | Devin-class hosted agent |
|---|---|---|
| Blast radius when an agent misbehaves | Bounded by sandboxes you build and operate | Bounded by vendor sandboxing you cannot inspect |
| Unsafe-action detection | Your pipeline; best public detectors degrade to 0.665 Macro-F1 on unseen sources | Absorbed by vendor; recall and FPR undisclosed |
| Policy enforcement | Your harness and prompts; rules reduce but do not eliminate violations | Absorbed by vendor; method undisclosed |
| Secrets management | Yours: scoped tokens, egress control, rotation discipline | Vendor-side perimeter you trust blindly |
| Human review throughput | The ceiling; your screening FPR sets queue depth | Also the ceiling; the equivalent rate is invisible |
| Model access | Whatever you wire in and pay for per token | Unlimited SWE-1.7 claimed on the desktop tier |
| Data boundary | Your infrastructure, your logs | Vendor infrastructure, contractual controls |
| Cost per merged PR | Computable from your own bills and telemetry | Not verifiable from public sources fetched here |
Two rows deserve emphasis because they are the ones buyers get wrong. Review throughput caps both paths; a hosted agent that opens PRs faster than your team merges them has merely relocated the bottleneck, with a false-positive rate you can no longer measure. And the detection row is symmetric in a way neither side advertises: the self-hosted gate fails publicly on your own dashboards, the hosted gate fails privately on someone else’s, and the benchmark record says both are failing somewhere.
What about cost per merged PR?
Nobody public has the number, and this article will not invent one. The fetched sources contain no GitHub Copilot coding-agent pricing, no Claude Code managed-runner costs, and no cost-per-merged-PR measurement from any self-hosted factory, commercial or hobbyist.
What can be said is directional. The token side of self-hosting has become cheap enough that the argument usually reframes itself as seat-based subscription versus raw compute, and that framing is intuitive but incomplete. The denominator that decides unit economics is reviewer minutes per merged PR, compounded by the false-positive load of whatever screening sits in front of review. At MaliciousSkillBench’s 62.4% benign false-positive rate on novel sources, screening cost can plausibly dominate token cost, and no pricing page on either side of the market quotes that figure.
The practical move is instrumentation from day one: reviewer minutes per PR, PRs merged per reviewer-week, tokens consumed per PR, and sandbox incident counts. Any team that collects those numbers for a quarter will hold a better build-vs-buy dataset than anything in the public record summarized here, because as of this writing the public record is empty.
Should you self-host your coding agents?
Self-host if what you want is control: a hard data boundary, a bounded blast radius you can audit, and a safety gate whose failures show up in your own telemetry. Do not self-host on the theory that safety is solved and only the token bill matters; the August 2026 benchmarks say safety is not solved, on either side of the fence. Treat the move as a transfer of safety ownership first and a cost decision second.
Two variables decide it, and both are measurable inside your organization before you commit. The first is review throughput: if your team cannot absorb the PR volume an agent factory generates, including the false positives any screening layer adds, the factory’s output ceiling is your reviewers, and that is true whether the gate runs on your cluster or a vendor’s. The second is secrets discipline: per-task scoped credentials, expiry, and egress denial are unglamorous work, and ATBench’s tool-surface numbers describe exactly how much machinery an agent swings around while it has them.
Build the factory and you sign up, knowingly or not, to run its safety program. That is the actual product of self-hosting: not cheap tokens, but the controls and the accountability that come with them. The vendor alternative does not remove the failure modes MaliciousSkillBench and ReguSim document. It hides them.
Frequently Asked Questions
How does the 62.4% false-positive rate in MaliciousSkillBench translate to reviewer workload?
If a self-hosted factory generates 100 benign PRs per week, a detector with that false-positive rate flags 62.4 of them as malicious, forcing reviewers to manually clear 62.4 false alarms before touching real threats. This overhead can exceed the time spent reviewing actual code changes, making the screening layer a primary driver of unit economics rather than token costs.
Why does the TF-IDF SVM detector fail on Source-Disjoint evaluation?
The model relies on word-level features that memorize the vocabulary of training sources. When evaluated on unseen sources, the distribution of malicious patterns shifts, causing the classifier to lose signal. This indicates that automated safety gates built on static lexical features are brittle against novel attack vectors or new skill repositories, requiring continuous retraining or more robust structural analysis.
What specific telemetry metrics should a self-hosting team track to validate safety controls?
Teams should instrument reviewer minutes per merged PR, PRs merged per reviewer-week, and sandbox incident counts. These metrics reveal the true cost of the safety layer by quantifying the human effort required to handle false positives and unsafe trajectories, providing a baseline for comparing self-hosted operations against vendor-absorbed costs that are otherwise invisible.
How does ReguSim’s finding on rule grounding apply to coding agent permissions?
ReguSim shows that visible rules act as soft priors rather than hard constraints, meaning agents can still violate policies even when rules are explicitly stated in the prompt. For coding agents, this implies that prompt-based instructions alone are insufficient for security; enforcement must be moved into the infrastructure layer, such as scoped tokens and network egress controls, to prevent unauthorized actions.
