Local Coding LLMs Hallucinate Packages: Slopsquatting Defenses Compared

A preprint posted to arXiv on 24 August 2026 reports that local coding LLMs fabricate package names at rates from near zero on routine prompts to 73%¹ on adversarial ones, and proposes a two-layer detector that produced hallucination-free code on 76%¹ of test runs. Every number in that sentence is preprint-reported and unreplicated as of 2026-08-28. The practical consequence still stands: running your coding model offline does not move you out of the supply-chain blast radius, because the attack lives on the public registry, not on your network. Dependency safety becomes a CI property.

What did arXiv 2608.23897 actually measure?

The paper measures how often local coding LLMs invent package names that an attacker can pre-register, and whether a pipeline of deterministic checks plus a trained classifier can catch those inventions before install. The preprint (v1, 14 pages) is titled “Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs,” and its target is precisely the assumption many teams made when they moved coding assistants on-prem: that air-gapping the model air-gaps the risk.

The threat model is slopsquatting. A model suggests pip install some-plausible-sounding-package, the package does not exist, an attacker registers it first with a malicious payload, and the next developer (or agent) to trust the suggestion installs the attacker’s code. The attack requires no network access to the model’s host. It requires only that the model’s output eventually reaches a machine with registry access, which in any real workflow it does.

The study ran 300 curated prompts¹ against local coding models and measured hallucination rate as a function of prompt adversariality. On routine coding prompts, rates sat between 0 and 10%¹. On prompts deliberately baited to induce slopsquatting, rates climbed to 40-73%¹, scaling almost linearly with how adversarial the prompt was. That spread is the finding that matters for defense design: any control tuned to the routine-prompt regime is calibrated against the wrong threat.

The authors then built a detector pipeline: a LangGraph state machine that retries generation at escalating temperatures, routes persistent failures to a stronger fallback model, and gates every suggested package through two verification layers. Across the 300 prompts¹, the pipeline produced hallucination-free code on 76%¹ of runs. The primary model exhausted its retry budget on 28.7%¹ of runs; intra-model retries recovered roughly a quarter of those, and the cross-model fallback recovered a further 16.5%¹ of the remainder.

Two caveats belong in this section rather than at the end. The abstract names neither the primary nor the fallback model, and the evaluation is Python/PyPI only. And per arXiv’s own description of its moderation policy, submissions are checked for topicality and scholarly value but not peer reviewed, with contents presented “as is.” Treat every rate in this article as a single team’s measurement, not an established figure.

What does the deterministic existence check catch?

The first defense layer answers exactly one question: does this package name exist on PyPI, and is it what it claims to be? It is deterministic, cheap, and catches the cleanest form of the attack: a hallucinated name that nobody has registered yet, which is precisely the name an attacker would pre-register tomorrow.

The layer’s non-obvious component is the import-name reconciler. Python has a longstanding split between the name you import and the name you install: import cv2 comes from pip install opencv-python, import PIL comes from pip install Pillow. A naive gate that checks the import statement against the registry would either false-positive on every legitimate mismatch or, worse, get trained by its users to ignore mismatches entirely. The reconciler maps import names to distribution names without opening a bypass around the existence check, so cv2 resolves to opencv-python and gets verified under the name that actually controls what code lands on disk.

For teams building this gate themselves, the reconciler is the part most likely to be skipped, because the failure mode is silent. The check passes, the developer stops reading, and the gap between import namespace and distribution namespace becomes a permanent blind spot.

The check also cannot catch a name that already exists and is malicious or junk. That is not an edge case. It is half the problem, which is where the second layer comes in.

Why does the classifier layer catch what existence checks miss?

Half of the hallucinations the pipeline flagged were packages that already existed on PyPI as low-quality lookalikes of well-known projects, and the deterministic existence check is structurally incapable of catching any of them. The paper names examples like pil, faiss, tabula, and haystack: real, registered packages whose names shadow famous ones (PIL/Pillow, faiss, tabula-py, haystack-ai). A model that emits pip install pil has produced a command that runs, installs something, and passes every existence check you point at it.

The second layer is a Random Forest classifier trained on ten features derived from the package name and its PyPI metadata. The paper does not enumerate all ten in the abstract, but the category is clear: provenance and reputation signals, the kind of metadata that separates a maintained distribution with a decade of release history from a squat registered last month. This is the layer that turns “does it exist” into “should it exist in your dependency tree.”

The 50/50 split between the two failure classes is the strongest argument in the paper for layered defense. An existence-only gate catches the not-yet-registered hallucinations and waves through every registered lookalike. A classifier-only gate catches lookalikes but has nothing to say about a name that has no metadata at all. Each layer’s blind spot is the other layer’s core competency, and the measured catch distribution is close enough to even that dropping either layer abandons roughly half the observable attack surface.

Why does fallback routing need a different model family?

When the primary model fails and the pipeline falls back to a second model from the same family, approximately 84%¹ of those failures recur on the fallback. Same-family fallback is close to no fallback.

This is the finding with the most direct architectural consequence, and it makes sense on reflection. Models from one family share training data recipes, tokenizers, and often the same instruction-tuning lineage, so the latent associations that make a model invent fastapi-utils-extended tend to be shared property. Routing a retry to a sibling checkpoint re-rolls correlated dice. The paper’s pipeline instead routes to a stronger cross-family fallback, and that routing recovered 16.5%¹ of the runs that survived the primary’s entire retry budget, on top of the roughly quarter recovered by intra-model retries at escalating temperatures.

The counterweight: the paper reports that the weaker primary refused 6 of 10 direct slopsquat baits with no detector assistance at all, which the authors read as evidence that recent instruction tuning provides a baseline defense. A baseline is not a gate. Four of ten baits went through, and the adversarial-prompt rates of 40-73%¹ describe what happens when the bait is less direct.

Does model choice move the needle on hallucination rate?

Model choice is a weak control against slopsquatting, because hallucination rate tracks prompt adversariality far more strongly than it tracks model quality. The same preprint measures 0-10%¹ on routine prompts and 40-73%¹ on baited ones across its local-model set; picking a better model moves you within a band, while the attacker’s prompt moves you between bands.

The independent replication sharpens this picture from a different angle. arXiv 2605.17062v3, updated 9 August 2026, measured package hallucination across 199,845² paired Python/JavaScript prompts on five cloud frontier models released between October 2025 and March 2026. Rates ran from 4.62%² (Claude Haiku 4.5) to 6.10%² (GPT-5.4-mini). The original Spracklen et al. study from USENIX Security ‘25, which both 2026 papers build on, reported a spread from 5.2%² on commercial models to 21.7%² on open-source ones, so the 2026 frontier cohort compresses the inter-model spread by roughly an order of magnitude.

Read carefully, that is an argument against spending your security budget on model selection. If the best and worst frontier models sit within 1.5 percentage points of each other, the marginal safety from switching models is small compared to the marginal safety from a verification gate that works regardless of which model generated the import.

The replication also quantified a cross-model attack surface no single-model study can see: 127 package names (109 PyPI, 18 npm) that all five evaluated models invent identically. Hallucinated names are not random noise; they are stable, predictable outputs that an attacker can enumerate in advance. After coordinated disclosure with PyPI Security, 53 of those names (41 PyPI, 12 npm) remained registrable by an attacker as of the v3 update. The range shrank; the registrable surface did not disappear.

One caution on comparison. The two studies do not corroborate each other: the 0-73%¹ range is local models across adversariality levels, the 4.62-6.10%² range is 2026 cloud frontier models, and neither paper tested the other’s model set. Anyone citing the low frontier numbers to dismiss local-model risk is comparing two different experiments.

What does PyPI-only evidence mean for npm teams?

Every defensive measurement in the detector paper comes from Python and PyPI; nothing in it has been tested against npm, and npm is the larger target. npm describes its registry as the largest software registry in the world, with more than two million packages and over 17 million developers. The replication study confirms npm is not theoretically exposed but measurably exposed: 18 of its 127 cross-model hallucinated names were npm packages, and 12 of those remained registrable after disclosure.

The defense pattern transfers better than the numbers do. A registry existence check is trivial to implement against npm’s API. The lookalike-classifier layer does not transfer as a trained artifact, since the Random Forest was trained on PyPI metadata features, but the feature category (registration age, maintainer history, download provenance, name distance from popular packages) has obvious npm analogues. The import-name reconciliation problem mostly vanishes in JavaScript, where the module specifier and the package name coincide far more often, though scoped packages and exports remapping keep it from being free.

What npm teams should not do is import the paper’s rates. The 76%¹ hallucination-free pipeline figure, the 84%¹ same-family recurrence rate, the adversariality scaling curve: all measured on PyPI. JavaScript’s flat namespace and heavier squatting history plausibly shift those numbers in either direction, and the honest position as of 2026-08-28 is that nobody has published the measurement.

How do the defense layers compare?

The evidence supports a layered stack in which each component covers a failure the others structurally miss, and in which model choice is the weakest layer despite being the one teams most often treat as the control.

Two alternative gate designs deserve mention as boundaries of this comparison. Work on unanimous LLM juries for generated SQL shows that committees of strong models can cut false accepts while still passing most correct outputs, but committee composition matters enough that the design carries its own fragility. And hallucination detection in general remains unsolved: the SHROOM-Visions 2026 shared task drew 27 teams and over 600³ submissions, with the best systems reaching roughly 0.51 intersection-over-union on span detection. The detector paper’s approach works partly because package names are a narrow, verifiable domain with a ground-truth oracle (the registry). Do not generalize its architecture to free-form code correctness.

Where should dependency verification actually live?

Dependency safety for coding agents belongs in CI and in the agent’s install sandbox, not at the network boundary, because the attack transits public registries that every build must be allowed to reach. Air-gapping the model changes where the weights run. It does nothing about the fact that the generated requirements.txt eventually meets pip install on a machine with registry access, and that meeting is where the slopsquat fires.

The concrete stack the evidence supports, in order of what it catches per unit of effort:

1. Lockfile-only installs inside the agent sandbox. The agent’s environment installs from a reviewed lockfile, not from names in model output. New dependencies require the gate, not the agent.
2. Registry existence plus provenance verification before any install. Every generated import resolves through the import-name reconciler to a distribution name, then passes existence and provenance checks at install time, against a current registry snapshot.
3. Cross-family fallback routing for generation retries. Same-family fallbacks repeat roughly 84%¹ of primary failures, so pipeline resilience requires a fallback from a different lineage.
4. Model choice as a tiebreaker, not a control. Fine for the routine-prompt band, irrelevant against baited prompts.

Teams that skipped these gates on the theory that local models are safe by construction are the ones this preprint reprices. The cost of an air-gapped coding setup now includes the verification pipeline, and that cost applies retroactively to every team already running one.

What are the strongest limitations of this evidence?

Every local-model number cited here comes from a single non-peer-reviewed preprint with 300 curated prompts, an n=24 user study (mean satisfaction 4.4/5, with 21 of 24 stating adoption intent), and primary and fallback models the abstract never names. The 76%¹ hallucination-free figure, the 84%¹ recurrence rate, and the adversariality curve are all one team’s unreplicated measurements. The only independent replication in circulation measured cloud frontier models, not local ones, so the local rates could move substantially when someone re-runs the experiment with named models and a published harness.

Context for calibrating trust in single preprints: arXiv’s moderation process classifies submissions by topicality and scholarly value, does not peer review them, and presents contents “as is” without warranty. That does not impugn this paper, which is an empirical systems study with code and data deposited under Zenodo DOI 10.5281/zenodo.22087562. It does mean the default reading mode for anything on arXiv is to treat every number as a claim awaiting replication.

The verdict holds even under that discount. The threat model does not depend on any specific rate: the replication independently confirms that hallucinated package names are stable, cross-model, and partly attacker-registrable after disclosure, with 53 names still available as of its v3 update. The defense layers (existence check, import reconciliation, lookalike classification, cross-family fallback) are cheap, deterministic, and grounded in the structure of the attack rather than in any one measurement. Build the gates now; revisit the percentages when the replication literature catches up.

Frequently Asked Questions

Can the Random Forest classifier from the preprint be applied directly to npm packages?

No, the classifier is trained specifically on PyPI metadata features and cannot be ported directly to npm. While the underlying feature categories like registration age and maintainer history have npm analogues, the trained artifact itself is registry-specific. Teams using npm must build their own classifier or rely on existence checks, as the paper provides no cross-registry model weights.

How does the 76% hallucination-free rate compare to general hallucination detection benchmarks?

The 76% success rate is high because package names are a narrow domain with a deterministic oracle (the registry). In contrast, the SHROOM-Visions 2026 shared task for general vision-language hallucination detection saw best systems reach only 0.51 intersection-over-union on span detection. This highlights that the detector’s efficacy relies on the verifiability of package names, not general code correctness.

What is the operational risk of using same-family models for primary and fallback agents?

Using same-family models creates a correlated failure mode where approximately 84% of primary hallucinations recur on the fallback. This renders the fallback layer nearly useless for catching model-specific biases. To achieve the 16.5% recovery rate reported in the study, pipelines must pair models from different lineages to ensure independent error profiles.

Does the 4.62-6.10% hallucination rate on frontier models validate local model safety?

No, the two studies measure different model sets and prompt types, so the rates are not comparable. The frontier replication tested cloud models on routine prompts, while the local study measured adversarial baiting. Citing the low frontier rates to dismiss local risk ignores that local models hit 40-73% hallucination rates on adversarial prompts, a regime the frontier study did not test.