How to Catch LLM Hallucinated Package Imports Before They Ship

If your pipeline executes code written by an LLM coding agent, treat every package name it suggests as attacker-registerable until your own check says otherwise. That is the practical finding from a cluster of 2026 preprints: current frontier code models still invent package names at measured rates, and a share of those invented names can be registered by anyone on the public registries the models were tested against. The detection approaches that catch these phantom imports are all partial, which is why the decision that matters is not whether to check, but which check to run and where in the pipeline to put it.

One caveat up front, because it shapes how much weight the numbers can bear: every quantitative result below is author-reported from unreviewed arXiv preprints observed in November 2026. None of it is peer-reviewed, and the static-analysis paper evaluates Python only, a scope limit I flag where it constrains what you can infer.

The threat model: a hallucinated import is a name an attacker can register

Typosquatting has always depended on a human making a typo. An LLM coding agent removes that dependency: the model itself invents plausible-sounding package names, and it does so consistently enough that an attacker does not need to guess what a developer might mistype. If a model hallucinates some-package in response to a common prompt, an attacker who registers some-package on PyPI or npm is waiting at exactly the address the generated code points to. The industry term for this is slopsquatting, and the important property is determinism: models hallucinate the same names repeatedly, so the bait can be placed in advance.

That property is what makes this a pipeline problem rather than a vigilance problem. A developer can be told to double-check imports; a CI job running thousands of agent-generated diffs a week cannot rely on that.

What the 2026 frontier-cohort replication measured

The most decision-relevant paper is a replication study on the 2026 frontier-model cohort, which re-ran earlier hallucination methodology against “five frontier code-capable LLMs released between October 2025 and March 2026: Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2.” Across 199,845 paired Python and JavaScript prompts validated against PyPI and npm master lists, the replication’s authors report overall hallucination rates between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini). The paper’s own title says the threat remains, and the numbers back it: model generations have not closed this gap.

The sharper finding concerns the shared hallucinations. The study’s authors “identify a set of 127 package names (109 on PyPI, 18 on npm) that all five evaluated models invent identically.” After coordinated disclosure with PyPI Security and npm, “53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry’s existing defenses.” Read that carefully: even after the registries were told exactly which names to watch, dozens of them are still open. Registry-side defense helps, but it is not a substitute for checking on your side.

These counts date quickly. Registries act, models get superseded, and the 53-name window could narrow or shift by the time you read this. The durable conclusion is structural: models converging on identical phantom names is a persistent property, and registry defenses have not eliminated the residual set.

What static analysis catches, and its ceiling

The natural first instinct is static analysis: scan generated code, extract import statements, flag anything suspicious. An empirical analysis of static analysis methods for library-hallucination detection quantifies what that buys. The authors report that “static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset.”

That range is enormous, and the spread itself is the finding. Which model generated the code and what the dataset looks like moves your catch rate between 14% and 85%, per the same paper. You cannot budget against the top of that range.

More important is the ceiling. Through manual analysis, the authors “identify cases a static method could not plausibly catch, which gives an upper bound on their potential from 48.5% to 77%.” Even a perfect static method, in the paper’s assessment, cannot exceed that bound, because some hallucinations are only visible with information static analysis does not have. So static analysis is worth running, but it is a floor layer, not a solution.

Here is the evidence limitation that matters most for generalization: the paper’s evaluation is Python-only. The authors state it plainly: “Our evaluation is limited to Python,” with the comparison run on the DS-1000, ODEX and BigCodeBench benchmarks and Mypy and Pyright among the methods tested. The 14–85% catch rates and the 48.5–77% ceiling in that paper are therefore author-reported Python figures, and transfer to other languages, stacks or registries is untested. Verify against your own generated-code corpus before treating any point in those ranges as an expectation.

Why a registry-existence check misses the worst cases

The simplest deterministic check is “does this package name exist on the registry?” It feels sufficient. A two-layer detector study on local coding LLMs shows why it is not. The authors found that “half of the flagged hallucinations are packages already registered on PyPI, as low quality lookalikes of well known projects, caught by the classifier rather than the deterministic layer (e.g., pil, faiss, tabula, haystack).”

Half. The most attack-relevant cases, names that exist on the registry precisely because someone registered a lookalike, sail through an existence check. A deterministic layer tells you whether the name resolves; it cannot tell you whether the thing that resolves is the library the model meant, or whether that library is any good. The classifier layer, trained to recognize lookalikes and low-quality doppelgangers, is what caught these. If your current mitigation is “we validate imports against PyPI,” this finding should change your plan.

The same study supplies two more facts that shape pipeline design. First, hallucination scales with prompt provenance: the study’s authors report rates of 0–10% on routine coding prompts versus 40–73% on adversarial slopsquat-bait prompts. Routine use is a low-probability hazard; an agent processing untrusted input (issue text, user prompts, scraped content) is operating in a different regime entirely. Second, on fallback architecture, the same paper reports that “when primary and fallback share a model family, approximately 84% of primary failures recur on the fallback, motivating cross family pairing.” A second opinion from a sibling model is mostly the same opinion. If your agent pipeline retries with a fallback model, that fallback should come from a different family.

Layered detection: deterministic check, classifier, calibrated confidence

Putting the evidence together, the defensible architecture has three components:

1. A deterministic registry-validation layer. Cheap, fast, catches names that do not exist at all. Necessary, and demonstrably insufficient alone.
2. A trained classifier. This is what catches the already-registered lookalikes, per the two-layer study above. It is the layer that addresses the attack-relevant half of the problem.
3. A calibrated confidence signal. A binary flag forces every borderline case into a human’s queue or out of it. A Bayesian calibration proposal for hallucinated-import detection describes a layer that “emits a Beta-posterior probability per detection, derived from a 3-category epistemic taxonomy.” A per-detection probability lets you set policy: auto-block above one threshold, route to review in a middle band, pass below another. Whether that calibration holds up outside the authors’ evaluation is untested in the supplied evidence, but risk-ranked triage is strictly more useful at review time than a flat yes/no when volume is high.

One clarification on what this layering does and does not do: it reduces the set of bad imports that reach install. It does not vet the packages that pass. A hallucination-free dependency list can still contain genuinely malicious or abandoned packages; that is a separate, older problem that dependency scanning already exists for.

Generation-time mitigation: what model editing buys

A different strategy is to fix the model rather than check its output. The BOUND model-editing approach edits model knowledge to suppress package hallucinations. The authors report it “reduces Package-HR by 65.4% on average across the three models, compared with 37.8% for Full-FT” (full fine-tuning). In a pip-install recommendation setting, the same paper reports reductions of 35.7% (Sample-HR) and 34.0% (Package-HR) with a 12.8% improvement in valid output rate.

Two observations keep this in proportion. First, even if BOUND’s 65.4% reduction applied to the replication’s 4.62–6.10% base rates, roughly 1.6–2.1% of package references would remain hallucinated: better, not solved. That residual is cross-study arithmetic, not a measured outcome: BOUND was evaluated on DeepSeekCoder, Qwen3 and Llama-3.1 in package-recommendation tasks, not on the frontier cohort that produced those base rates. Second, model editing applies to models you control. If your agents call hosted frontier APIs, you cannot edit them, and the generation-time lever is not yours to pull. For teams running local or fine-tunable coding models, BOUND-style editing is a worthwhile reduction layer; for everyone, it is an argument about why vendors should do this, not a control you can deploy.

Where does the check belong?

The placement decision follows from two facts: every detection layer is partial, and the cost of a miss is a malicious package installed into your environment. That pushes the check to the last point where failure is cheap to stop, which is before install, at review or CI, where a human or a policy can act on the flag.

A few practical consequences of this framing. Exposure depends on prompt provenance: if your agents process adversarial or untrusted input, you are in the 40–73% regime from the local-LLM study, and a review-time gate is not optional in any serious sense. If agents only handle routine internal prompts, the same study’s 0–10% range still applies to thousands of generated imports a month. And the calibrated-signal layer matters most exactly at review time, where the question is “which of these flags deserves a human’s ten minutes,” not “is anything flagged.”

The verdict, with its limits

Gate generated imports with a layered check at review or CI: deterministic registry validation, a trained classifier, and a calibrated confidence signal that lets policy decide what blocks and what queues for a human. Pair agent fallbacks across model families. Do not rely on generation-time mitigation or any single static check; the measured ceiling on static methods and the already-registered lookalikes that bypass existence checks rule both out as sole controls. That verdict follows from the measured partiality of every layer and the fact that residual phantom names are demonstrably registrable; the inference is mine, the measurements are the authors’.

What this does not establish, and what you should verify in your own stack: which off-the-shelf tools implement each layer; the runtime cost of the registry-probe, classifier and calibration layers (the static-analysis paper times only its own tools, at a mean of 1 second for Mypy, roughly 2.5 seconds for Pyright and its grammar, and 4 seconds for the o3-mini baseline; nothing in the supplied evidence times the rest); false-positive rates on real repositories, which none of the supplied evidence reports; and applicability outside the Python and JavaScript settings these papers evaluate. The static-analysis paper’s catch rates and ceiling are Python-only figures, so they are the weakest generalization in this article. And all five sources are unreviewed preprints with author-reported numbers; if any of these findings fails peer review or independent replication, the layering logic survives, but the specific rates should be re-derived against your own generated-code corpus before you set policy thresholds.

Frequently Asked Questions

What is the maximum detection rate for static analysis of library hallucinations?

The authors report that “static analysis tools can detect 16-70% of all errors, and 14-85% of library hallucinations, with performance varying by LLM and dataset.”

How many hallucinated package names remain registrable after registry disclosure?

After coordinated disclosure with PyPI Security and npm, “53 of these (41 on PyPI, 12 on npm) remain registrable by an attacker after each registry’s existing defenses.”