When a coding agent writes against an internal library its model never saw in training, it invents APIs that don’t exist and misuses the ones that do. The default fix is context-window stuffing with docs, and it has a hard ceiling: the paper itself reports that even oracle retrieval leaves models making recurring errors. MEMCoder (arXiv:2604.24222) treats memory as something built from execution rather than retrieval. Run the code, watch what fails, accumulate the lesson. The payoff is an 18.41 percentage-point lift over RAG baselines on private-library code generation.
Why do coding agents hallucinate internal APIs?
Internal libraries sit outside the public pretraining corpus by definition. The weights carry a strong prior for numpy, requests, and the long tail of well-indexed open source, but nothing for a proprietary payments client, an internal feature store, or a team’s fork of a tensor library. When the model is asked to call an API it has never seen, it does what language models do: it emits the statistically most plausible signature, which is frequently wrong or fabricated.
Hallucinated and misused library calls are a documented failure mode in practical code generation, not a fresh discovery (arXiv:2409.20550). What the MEMCoder paper adds is a precise accounting of where the residual errors sit after you have done the obvious thing and handed the model the documentation.
Why does documentation retrieval hit a ceiling?
The central finding is that static documentation tells the model what APIs exist but not how to use them correctly. The paper’s authors make this precise with a Vanilla-versus-Oracle comparison: in the Oracle setting, every required API document is handed to the model directly, removing retrieval error entirely. Even under that condition, the models still make recurring errors on NdonnxEval (arXiv:2604.24222 HTML). Better retrieval alone does not produce reliable code.
What do the three failure levels look like?
The authors categorize residual failures under Oracle retrieval into three levels, each with a representative case from the ndonnx benchmark:
- API-level errors. The model treats
ndonnx.Arrayas a Python sequence and callslen(x), which raisesTypeError: object of type 'Array' has no len(). It also fabricatesndx.average, an API absent from the docs. Documentation prevents neither. - Cross-API errors. The model chains
equalintowhereintodivide, but the mask shape fromwheredoes not line up with the row-wise values feedingdivide, producing a malformed denominator. Each individual API is correct; the composition is not. - Task-level errors. The model invokes
sum,divide,multiply, andwherecorrectly in isolation but misreadssumas a row mean, so the scaling factor and final output miss the task objective.
The point of the taxonomy is that these errors recur. The same misuse reappears across generations of one task and resurfaces in related tasks, because each RAG query starts from scratch and the model never accumulates familiarity with the library.
How does execution memory close the loop?
MEMCoder is a training-free layer that sits on top of an existing RAG pipeline. The loop has four stages: generation, execution, reflection, and update. The model writes code, the code runs against the real library, the feedback is analyzed, and the distilled lesson is written into an external memory that the next generation can retrieve.
The memory is multi-level, mirroring the failure taxonomy: usage guidelines are accumulated at the API level, the cross-API level, and the task level. During generation, MEMCoder retrieves both the static API documentation and relevant historical memories to guide the next attempt. After execution, it analyzes the feedback to refine whichever guideline the failure implicates.
How large is the improvement?
Across two benchmarks and multiple models of varying scale, MEMCoder adds an average of 18.41 percentage points to pass@1 over its RAG backbones (arXiv:2604.24222). The gains hold across model scale, and MEMCoder also outperforms existing self-evolving memory methods adapted to the same task.
Where does the bottleneck actually move?
The paper reframes agent memory for internal libraries as a verification problem rather than a retrieval problem. Once you accept that even perfect retrieval leaves recurring errors, the marginal dollar is better spent on execution than on a bigger embedding index. That is a real reframe for anyone building coding-agent infrastructure: the thing you must provide is a sandbox that can actually run the generated code and return structured feedback, not a fancier vector store over the docs.
This is also why the approach generalizes awkwardly. Execution memory requires executable feedback, which the authors note is available in test-driven workflows and can be approximated with LLM-generated tests (the CodeT line of work). If your internal library cannot be exercised in a sandbox, or your tasks have no executable check, the closed loop has nothing to learn from and MEMCoder collapses back toward ordinary RAG.
What does this mean for tool builders?
Three concrete consequences fall out of the result.
First, frameworks that treat memory purely as retrieval over static docs are now competing against a closed-loop alternative that posts double-digit point gains. The differentiator is whether you can execute and learn from the outcome, not how much internal context you can stuff. Vendors selling context-window maximization as the answer to internal-API hallucination should expect this comparison.
Second, sandboxed execution graduates from optional capability to prerequisite. An agent that cannot run code against the real library cannot build execution memory. That raises the bar for deployment: isolated runtimes, deterministic test signals, and the plumbing to feed runtime errors back into the prompt loop become load-bearing rather than nice-to-have.
Third, the cost model flips. RAG spend is per query and denominated in tokens. Execution memory spends per attempt, denominated in sandbox compute and in the extra reflection calls that update the memory. Enterprise libraries with long build cycles and flaky suites will cost more per lesson learned than the benchmark proxies suggest, and noisy reflection is a real risk: every mis-attributed error or misleading guideline that lands in memory gets retrieved and reused on later attempts.
One caveat the authors are explicit about: NdonnxEval and NumbaEval use public libraries (ndonnx and numba-cuda) as proxies for genuinely private ones, because real proprietary codebases are not available to researchers. How well execution memory transfers to large, messy internal stacks with weak test coverage remains an open question. The mechanism is sound; the generalization needs validation outside benchmark proxies.
Frequently Asked Questions
Which model sizes showed the 18.41-point improvement?
The average gain comes from experiments across Qwen2.5-7B, Qwen3-32B, and GLM-4-Plus. Because the lift holds at smaller scales, execution memory could let teams use cheaper models for internal-library coding tasks.
How is MEMCoder different from EvolveMem?
MEMCoder evolves usage guidelines from code execution feedback, while EvolveMem evolves the retrieval infrastructure itself. The 18.41 percentage point gain belongs to the execution-memory design, not to retrieval redesign.
What runtime infrastructure does a team need to deploy execution memory?
You need a sandbox that executes generated code against the actual library and returns compilation or runtime feedback to a reflection stage. That usually means containerized runtimes, reproducible test signals, and prompt plumbing that writes distilled guidelines back into the multi-level memory.
When does perfect documentation retrieval still fail to help?
On NumbaEval, oracle retrieval adds only 0.54 to 4.27 percentage points, compared with 21.30 to 32.60 on NdonnxEval. That gap shows RAG’s ceiling depends heavily on the task type, not just on retrieval quality.
What could make execution memory cost more than it saves?
Noisy reflection can poison memory with mis-attributed errors, and because those guidelines are reused across later tasks, one bad lesson compounds. In enterprises with long build cycles and flaky suites, the per-lesson sandbox compute and reflection-token spend can exceed the benchmark cost assumptions.
