Five Qwen3.8-27B instances did not beat Fable 5 on LiveCodeBench Hard in any sense a buyer should bank on. What the project actually showed, by its own numbers, is parity with its own harness and a lead over a Fable 5 baseline that was never run under the same conditions. The cost side is more concrete: the team-solve workflow spent $35.79 in output tokens for 50 runs, which works out to roughly $0.72 per run by simple division, but that figure excludes the GPU or serving cost of running five concurrent 27B models. If you are an engineering lead weighing orchestrated open-weight ensembles against a frontier coding subscription, the honest reading is: a promising parity signal at a measurable token cost, with the infrastructure bill and the benchmark validity both still open questions.
What the headline number actually measures
The claim circulating in 2026 coverage is that a Memdoor workflow built from five Qwen3.8-27B instances plus a repair worker scored a mean of 9.8 out of 10 over five passes on the first ten LiveCodeBench Hard problems, against 9.6 for the authors’ own harness and 8.0 for Fable 5 alone, per the project’s writeup as reposted on daily.dev. All three figures are project-reported; nothing in the available coverage describes independent replication.
Two qualifiers shrink the claim considerably. First, the gap against the authors’ own harness, 9.8 versus 9.6, is statistically nothing at this sample size. The project’s own materials concede that the two scores are “effectively the same at this size; the claim is parity, not improvement,” according to the oflight explainer, the only piece of the available coverage that carries the parity caveat alongside the numbers. Second, the Fable 5 baseline is not a like-for-like comparison. As the same explainer puts it: “Fable 5 was one call per problem and was not run in this harness; the team makes many calls.” The 8.0 measures what a single unassisted API call scores, not what Fable 5 scores given the same delegation loop, the same retry budget, or the same harness. What the 9.8-versus-8.0 gap measures is closer to “delegation and iteration help on hard competitive programming problems” than “Qwen3.8-27B is a better coding model than Fable 5.”
That distinction matters because it determines what you would be buying if you adopted the setup. You would not be buying a better model. You would be buying an orchestration pattern that lets a cheaper model spend more calls per problem.
Inside the team: five roles, no shared context
The architecture, as documented by af.net’s realtime coverage, is a planner, two parallel ideators, a solver, and a manager, with a designated repair worker alongside. The coordination model is the interesting engineering choice: “The agents never share a chat context window; instead, all roles interact strictly via files on disk, with each agent re-entering cold with only the specific files it requires.”
Cold re-entry through files is a deliberate trade. It avoids the context-rot problem of one ever-growing conversation, and it means each agent’s prompt contains only what its role needs, which keeps per-call token counts bounded. The cost of that discipline is exactly what the run economics show: many calls per problem, each rebuilding its working picture from disk. The workflow’s results come from spending calls, not from any single call being smarter. This is also why the comparison against a one-call Fable 5 baseline says little about relative model quality.
Memdoor itself (per the oflight explainer) is an Apache-2.0 coding and automation agent shipped as a single Go CLI binary, with no Docker or Python dependency, running locally or over SSH. That lowers the barrier to re-running the experiment, which, as we will get to, is the single most important thing a reader can do with this result.
The real bill: $35.79, and what it leaves out
Here is the project-reported cost picture, all from the daily.dev repost of the project’s numbers:
| Figure (project-reported) | Value | What it covers |
|---|---|---|
| Score, five-instance team | 9.8/10 mean, 5 passes, 10 problems | Output-token spend and orchestration, not model quality |
| Score, authors’ own harness | 9.6/10 | Effectively parity at this sample size |
| Score, Fable 5 baseline | 8.0/10 | One call per problem, different harness |
| Output-token cost, 50 runs | $35.79 (15.3M output tokens) | Output tokens only |
| Cost including cut attempts | $38.77 | Adds 16 attempts cut by timeout, fix, or pause |
| Derived per-run cost | ~$0.72–$0.78 | Simple division of the above by 50 runs |
| Median run time | 40 minutes of model time | Time spent in model calls; wall-clock per problem may be longer |
| Longest run | 4 hours, $0.39 per the project repost | Tail duration; coverage differs on whether $0.39 is a tail or median cost |
Dividing $35.79 by 15.3 million output tokens implies roughly $2.34 per million output tokens, consistent with hosted pricing for a 27B open-weight model rather than self-hosted GPU cost; the oflight explainer notes the model ran on OpenRouter, not locally. So treat the token spend as a metered API bill for the ensemble’s output, not as the total cost of ownership. It excludes input tokens (the file-based, cold re-entry design re-reads files constantly, so input spend could be substantial), the compute serving five concurrent 27B instances, and the ops time to stand the thing up.
The frontier side of the comparison is simply missing from the available evidence. No Fable 5 per-task API pricing appears in the project writeup or the coverage around it, so a completed cost-versus-subscription comparison cannot be built from this material. Any article that tells you the ensemble is “cheaper than Fable 5” is filling that gap from somewhere other than the data.
Latency and failure modes: the 40-minute median and the 4-hour tail
The latency profile is the second decision axis, and it cuts against the ensemble for most interactive work. A typical run took a median of 40 minutes of model time, and the longest run took 4 hours, per the project’s reported numbers. Both figures are model time, not wall-clock, so elapsed time per problem can run longer still. Sixteen of the attempts in the 50-run exercise were cut short by a timeout, a fix, or a pause, which is why the cost figure has two versions, $35.79 clean and $38.77 inclusive.
Forty minutes median latency is fine for a batch task: an overnight sweep of hard problems, a research benchmark, a queue of well-specified issues. It is a poor fit for the interactive loop where most coding-assistant subscriptions earn their seat price. And the 4-hour tail matters more than the median if you are planning throughput, because tail runs tie up all five instances at once. A team-of-five architecture has a failure mode a single API call does not: the run is only as fast as its slowest role, and every role shares the same serving capacity.
Why the comparison is not like-for-like
Three validity problems stack on top of each other, and all three come from the project’s own disclosures rather than hostile reading.
One call versus many. The Fable 5 baseline made one call per problem in a different harness; the Qwen team makes many calls with file-based state and a repair worker, per the oflight explainer. A fairer baseline would give Fable 5 the same multi-call budget and the same harness. Until someone runs that, the 8.0 is a lower bound on the frontier model, not a measurement of it.
Contamination is unresolved. The authors flag this themselves: the benchmark problems “date from March 29 to April 6, 2025, and the model may have encountered them during training, meaning data contamination cannot be ruled out,” as quoted in af.net’s coverage. LiveCodeBench exists partly to mitigate contamination by dating problems, per its design paper, but a March, April 2025 window is old enough that a model trained or refreshed in 2026 could plausibly have seen these problems or their discussions. This does not prove contamination; it means the result cannot exclude it.
Sample size. Ten problems, five passes. The source page itself treats 9.8 and 9.6 as the same score at this size. An ensemble-vs-baseline gap of 1.8 points on ten problems could be real, could be noise, and could be contamination. The evidence cannot settle which.
What five concurrent 27B instances actually cost to serve
Suppose you decide the token bill is acceptable and you want to self-host rather than pay per-token. Now the hard part starts, because serving cost is not a property of the model; it is a property of the model, the hardware, the configuration, and the load.
The most useful anchor here is a concurrency-aware methodology for LLM infrastructure cost estimation (arXiv:2606.11690), which found that effective cost per output token “spans $0.21 to $15.25 per million output tokens” across six configurations and a full load range. That is roughly a 70x spread for serving cost depending on how well your deployment is utilized. An idle or poorly batched five-instance deployment lands near the expensive end; a saturated, well-batched one lands near the cheap end. The $35.79 project figure tells you nothing about where your deployment will fall on that spread. You have to measure it.
Serving and routing choices are not a rounding error in this calculation. A measurement study of hosted open-weight LLM APIs (arXiv:2605.02821) found that, under observed feasibility constraints, routing lowered Qwen3-32B cost by 37.8% and raised DeepSeek-V3.2 average throughput by about 90% relative to direct official access. Different models than Qwen3.8-27B, but the mechanism transfers: for open-weight deployments, the provider, routing, and configuration you choose can move cost and throughput as much as the choice of model itself. Five concurrent 27B instances is a serving-engineering problem before it is a modeling problem, and the headline benchmark omits that entire bill.
Where open-weight ensembles hold up, and where gains are “not universal”
It would be too strong to dismiss the ensemble approach. The GVS5H method this workflow reproduces reports, via its README as cited by the oflight explainer, Qwen3.8-27B rising from 66.8% to 92.4% on 100 problems under the GVS5H method. That is a large gain, and it is project-reported and comes with the README’s own warning that gains are not universal. Treat it as evidence that the orchestration pattern can be worth a lot on some problem sets, not as a transferable multiplier.
There is also adjacent independent evidence that open-weight models can reach API-level quality on narrow tasks. A study on political-science text classification (arXiv:2605.19275) found that in a task-specific oracle comparison across 34 tasks, local models matched or exceeded the best API model on 9, with the best API model ahead of the best local model by 0.015 F1 on average. That is a different domain entirely, text classification rather than competitive programming, so it cannot validate the LiveCodeBench claim. What it does show is that the general pattern, “an open-weight model chosen per task can sit within noise of the frontier API,” has been observed independently somewhere. Whether it holds for the hard tail of coding work is precisely what the Memdoor result leaves unproven.
Before you trust any of it: the re-run checklist
Everything actionable in this article reduces to verification work you can do yourself, and Memdoor being a single Apache-2.0 Go binary makes the loop cheap to start.
- Run fresh problems. Re-run at least ten LiveCodeBench Hard problems from after the model’s training window, not the March 29, April 6, 2025 set. This is the only way to address the contamination caveat the authors themselves raised.
- Give the frontier model the same harness. Run Fable 5 (or your actual frontier API) with the same multi-call budget, retry logic, and file-based state. A single-call baseline measures delegation, not model quality.
- Benchmark your serving tier. Measure your effective cost per output token under realistic concurrency before extrapolating. The $0.21–$15.25 per-million-token spread means any borrowed number is worthless for your infrastructure.
- Measure the tail, not the median. A 40-minute median with a 4-hour tail determines your queue depth and GPU reservation, and the tail is what breaks throughput plans.
- Cap the prompt budget per problem. Decide in advance how many calls a problem may consume, so the ensemble’s economics stay comparable to a per-task frontier price you can actually obtain.
Verdict: a parity signal worth a bounded pilot
I would treat Memdoor’s result as a project-reported parity signal and pilot it only where per-task cost and throughput matter more than latency: batch jobs, benchmark sweeps, well-specified issue queues with tolerant deadlines. The reported economics, roughly $0.72–$0.78 per run in output tokens at a 40-minute median, are plausible for that shape of work. For interactive coding assistance, a 40-minute median and 4-hour tail do not compete with a single frontier call regardless of the token price.
What I would not do is read this as evidence that five 27B open-weight instances replace a frontier coding subscription. The comparison that would prove that, same harness, same call budget, fresh problems, independent replication, real frontier pricing on the other side of the ledger, has not been run, or at least not published anywhere in the available record. Buried in the project is a practical upside: you can run it yourself, and the reported bill for the full 50-run exercise was $35.79 to $38.77 in output tokens. If your re-run survives the checklist above, you will have something no current coverage of this result can offer: numbers you actually trust, measured on your own infrastructure, against a baseline that was allowed to fight back.
Frequently Asked Questions
What is the reported output-token cost for the 50-run exercise?
the team-solve workflow spent $35.79 in output tokens for 50 runs, which works out to roughly $0.72 per run by simple division, but that figure excludes the GPU or serving cost of running five concurrent 27B models.
Why is the comparison against Fable 5 not like-for-like?
Fable 5 was one call per problem and was not run in this harness; the team makes many calls.
What is the median model time for a typical run?
A typical run took a median of 40 minutes of model time, and the longest run took 4 hours, per the project’s reported numbers.
Join the discussion
Share a useful perspective or ask a question about this article.