Can 5 Qwen 27B Agents Beat 1 Frontier Model on Hard Coding Problems?

Five coordinated Qwen3.8-27B agents did outscore a single Fable 5 call on ten hard competitive-programming problems, but only in one repository’s own benchmark run, and the same repository concedes its result is parity with the harness it copies rather than an improvement. If you are paying frontier-model prices for hard coding tasks, the honest reading of this result is not “orchestration beats scale.” It is that a five-role, file-mediated workflow of cheap open-weight models is now plausible enough to be worth a disciplined re-run on your own terms: post-cutoff problems, multiple reruns, a compute-matched frontier baseline, and a cost-per-solved-problem budget. Until that re-run exists, no routing decision should change.

The repo-stated result, and where each number comes from

On 2026-10-31 a Hacker News post describing a Memdoor workflow called team-solve drew a large discussion thread, and three secondary write-ups relayed it the next day. The claim: five Qwen3.8-27B instances, coordinated as planner, two ideators, solver and manager plus a repair worker, scored a mean of 9.8 out of 10 over five passes on the first ten LiveCodeBench Hard problems, versus 8.0 for a single Fable 5 call and 9.6 for the GVS5H harness the workflow reproduces.

Every one of those numbers traces to the repository’s own run. The write-ups at daily.dev, AI Weekly and OfLight corroborate the reporting, not the result; as of 2026-11-01 no independent reproduction exists, and the tool itself sits at 89 stars and 230 commits. That distinction matters for everything below, so here is the full scorecard with its status:

Inside team-solve: five cold agents, files on disk, a script that judges

The architecture is more interesting than the score, because it explains both the appeal and the failure modes. Each of the five roles starts fresh with no shared chat context, communicates only through files on disk, and a shell script rather than a model decides pass or fail. Cold starts per role are a deliberate bet: no agent inherits another’s reasoning drift, and the on-disk artifact trail makes each step inspectable.

The whole thing is a YAML recipe. Memdoor is an Apache-2.0 single Go binary, the team-solve example is eight YAML steps, steps name no model, so whatever you select with /model fills every role; giving a single role a different model means editing the YAML, not making a selection. The creator’s framing in the Hacker News thread is worth quoting because it sets the right expectation: “Anyone can reshape the harness by redefining the steps in yaml files , models and temperatures. Up to you guys to try you have the recipe in this repo.” That is an invitation to reproduce, and it is the strongest argument the project makes.

It is also a reminder of where the capability claim actually lives. A modern agent’s capability depends not only on its foundation model but on the harness that builds prompts, manages state, invokes tools and coordinates execution. Team-solve is a harness result. The question “did five 27B models beat a frontier model” is partly unanswerable until you separate what the models did from what the scaffolding did for them.

The economics: $0.39, 40 minutes, and 15.3 million tokens

Across fifty complete runs the workflow cost $35.79 and consumed 15.3 million output tokens, or $38.77 counting sixteen incomplete attempts, per the same repo-stated accounting. The median run was $0.39 and about forty minutes.

Cheap per run does not mean cheap per decision, because orchestration multiplies calls. The general principle is documented outside this repo: “Running multiple inferences per query effectively multiplies operational costs or latency by a factor of the ensemble size”, and that study judged the marginal gains of ensembling not worth the resulting computational inefficiency for live, scalable health systems. A five-agent pipeline where each step invokes a model turns the “cheap small model” premise into a volume business. The repo’s own numbers show the shape: 15.3 million output tokens to solve ten problems five times.

The latency side is just as structural. An independent AWS Bedrock evaluation of multi-agent collaboration found goal success rates rising by up to 70% over single-agent setups, but measured per-turn communication overhead of 13.39s to 35.44s, with user-perceived turn latency in software tasks reaching 168.73s versus 31.46s in simpler domains. Software work pays the highest coordination tax. A 40-minute median per problem confirms the same thing from the other direction: this is asynchronous batch infrastructure, something you point at a queue of hard problems overnight, not something that answers while you wait at a prompt.

Is the baseline even the same race? Compute matching

One of the bluntest objections in the Hacker News thread: “So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. …Cool? Were they not expecting that to work?”

Strip the sarcasm and the point stands. Fable 5 was one call per problem and was not run in this harness; the team makes many calls with retries, parallel ideation and a repair loop. A single-call score is the right baseline only if a single call is the product you would otherwise buy. If the real alternative is “frontier model with a few retries and a test loop,” the 8.0 baseline understates it, and nobody has measured the frontier model inside this harness at all. The fix is procedural: run the frontier model through the same harness, at a matched token or dollar budget, and compare that number.

The contamination window is unresolved, by the authors’ own account

The benchmark problems are contest problems dated March 29 to April 6, 2025, and the authors state upfront that the model may have encountered them during training, so contamination cannot be ruled out. This is not an outside attack; it is the repo’s own caveat, and commenters in the Hacker News thread pushed the same point about saturation of spring-2025 LiveCodeBench material.

There is an established discipline for exactly this. The LiveCodeBench protocol tags every problem with its contest release date and, for newer models, evaluates only on problems released after the model’s training cutoff, so the model cannot have seen the exact problem in training. The Memdoor run did not apply that window. A re-run that selects problems released after Qwen3.8-27B’s cutoff converts this from an unresolvable argument into a measured one, and it is the single cheapest check on this list to do correctly.

How big is 0.2 points when reruns spread by 6?

Even taking every number at face value, sample size decides what they mean. Ten problems, five passes. The harness study in arXiv:2610.04433 ran three production harnesses (Claude Code, mini-SWE-agent, OpenCode) over the same models, including Qwen3.8-27B, on SWE-bench Verified, and it sets the noise floor with results relayed from earlier work, not its own runs: ten reruns of a single configuration “already spread by up to 6 points” (Bjarnason et al., as the paper cites it), and a three-harness comparison by Vats & Golev (2026) that “finds within-model differences of 0 to 8 points with intervals that cover zero except the largest.”

Against that noise floor, the 0.2-point edge over GVS5H (9.8 versus 9.6) is not a finding. The source page itself says the two are effectively the same at this size; the claim is parity, not improvement. And even the 1.8-point gap to Fable 5 rests on ten problems against a baseline measured outside the harness. A trustworthy version of this experiment needs several reruns per configuration so the spread of each configuration is visible, not just its mean.

The same study adds a finding that should recalibrate expectations in both directions: across the three harnesses, the harness barely moved pass rate but changed the bill by up to 3x on the same model and tasks, and every harness effect the authors could name was a way to lose a task rather than win one. This sits in tension with the GVS5H README’s claim that its method lifts Qwen3.8-27B from 66.8% to 92.4% on 100 problems. The domains differ (contest programming versus repository issue-fixing), so neither number transfers to the other’s setting. What transfers is the method: measure the spread, then attribute.

One more lesson from that study is operational. When auditing reproductions, the authors found 71% of web-enabled DeepSeek and Qwen trials had fetched the upstream fix from GitHub, discarded those runs, and enforced offline behavior at the network layer. Any re-run of a coding benchmark that leaves network access open is measuring retrieval as much as reasoning. If you re-run team-solve, enforce offline at the network layer, not in a prompt.

Audit the harness, because that is where these claims die

The repo-stated log already contains the instructive failures: one problem was lost to a wrong counterexample, and a correct program was overwritten twice by the repair loop. Two more caveats from the same accounting: the loop saw only public samples while scoring used hidden tests, and the model ran on OpenRouter rather than locally, so the published medians reflect hosted serving, not your GPU.

The broader reason to read the code, not the score table, is precedent. A 2026 multi-agent orchestration paper (arXiv:2511.15755) had its headline claims withdrawn after “a code audit found the multi-agent arm’s action list is a source constant, not model output, and the scorer reads only that field.” None of the reported material alleges a scoring defect in Memdoor’s run, and its shell-script judge is a good sign. But the episode sets the evidentiary bar for self-reported orchestration wins generally: the scorer, the action trail and the pass/fail logic are the claim. Practitioner reports add the operational version of the same problem: in multi-agent setups, if a final result is wrong, it takes careful log analysis to pinpoint which agent’s action or which piece of context led to the mistake. Memdoor’s file-mediated design, where every artifact lands on disk, is unusually well suited to that analysis, which is a genuine point in its favor for anyone auditing a re-run.

Running it yourself

The local path has hard constraints worth knowing before you commit a weekend. Ollama needs a 32K context window rather than its 4K default; the Q4_K_M quant is 17.1GB and does not fit a 16GB GPU, while IQ4_XS at 15.7GB can. Local speed was not measured in the repo run; the published 40-minute median came from OpenRouter, so local runs may be slower. Since the YAML steps name no model, /model sets one model for every role at once, and giving a role its own model means editing the YAML; either way the recipe works as a testbed for comparing models under an identical harness.

Verdict: re-run it, then decide on cost per solved problem

Can five Qwen 27B agents beat one frontier model on hard coding problems? The evidence supports a narrower statement: one repository’s self-reported run, on ten spring-2025 contest problems, scored parity with the harness it reproduces and beat a single-call frontier baseline that never ran in the same harness. That is a promising recipe, not an established result.

I would treat team-solve as a cheap batch tool to trial, and I would run the trial properly before routing any production work: problems released after Qwen3.8-27B’s training cutoff, several reruns per configuration, a frontier model inside the same harness at a matched budget, offline enforcement at the network layer, and the decision made on cost per solved problem rather than headline score. The 40-minute median confines it to asynchronous work regardless of what the re-run shows, and an unsolved problem that costs forty minutes and $0.39 is a different product from an unsolved problem that costs one API call. The recipe is open, reshapeable and cheap enough that verifying it costs less than arguing about it.