A wheel on PyPI is verified only when rebuilding it from the published source archive produces a byte-identical file to the one PyPI serves, and by that standard most of the Python ecosystem fails. The “No Snake Oil” preprint, posted 2026-07-24, rebuilt 12,1801 popular PyPI releases with two independent tools and found that only 15.4% to 19.1% matched bit-for-bit. A normalized equivalence check lifts that to 60.2% to 78.9%1, which still leaves a fifth to two-fifths of sampled releases that nobody outside the original uploader can account for.
What does “reproducible wheel” actually mean here?
A reproducible wheel, in this paper’s operational sense, is a wheel that a third party can regenerate from the sdist on PyPI and get back a byte-identical artifact to the wheel PyPI distributes. That definition is stricter than it sounds, and the paper is careful to separate it from a weaker notion, source equivalence, where the rebuilt wheel differs only in ways a normalization tool can prove are cosmetic.
PyPI distributes packages in two forms, source archives (sdists) and pre-built wheels, as described in the Python Package Index overview. The reproducibility question is whether those two artifacts for the same release are reconcilable: does the wheel actually come from the sdist sitting next to it, or from something else? Before this measurement, the honest answer for most of the ecosystem was “presumably.” Pip installs the wheel when one exists, hash-pinning verifies that the wheel you downloaded is the wheel the pin recorded, and nothing in the standard toolchain ever asks whether the wheel could have been produced from the published source.
The paper introduces a second bar precisely because the first one is too strict to be useful as a trust signal. Byte identity fails on non-deterministic build metadata that has nothing to do with tampering. The preprint’s source-equivalence metric, implemented in a tool the authors call daleq4py, normalizes those differences through a kernel of provenance-preserving datalog rules and asks the narrower question: setting aside cosmetics, is the content the same? Both numbers matter, and confusing them is the fastest way to misread this paper.
How was the measurement done?
The study took 12,1801 popular PyPI releases and rebuilt each one from its sdist using two independent rebuild systems, macaron and oss-rebuild, then compared the outputs against the wheels actually published on PyPI. Using two rebuilders matters because a rebuild failure could mean the package is non-reproducible, or it could mean the rebuild tool got the environment wrong, and cross-checking two implementations separates those failure modes.
The sample skews toward well-maintained, popular releases, which the authors acknowledge and which readers should keep in mind before extrapolating. Popular packages have CI, multiple maintainers, and release automation; the long tail of PyPI has a maintainer running python setup.py bdist_wheel on a laptop that hasn’t been rebooted since the last point release. Whatever non-reproducibility rate the popular tier shows, the broader ecosystem almost certainly shows worse.
The toolchain context also moves. The current CPython release is 3.14.6 as of this writing, and each interpreter and build-backend release changes what ends up embedded in a wheel. Reproducibility is not a property a package earns once; it is a property that decays every time the build stack underneath it shifts.
How many PyPI wheels actually rebuild?
The headline numbers from the preprint: 15.4%1 of macaron rebuilds were byte-for-byte identical to the published wheels, and 19.1%1 of oss-rebuild rebuilds were. Under daleq4py’s source-equivalence check, those figures rise to 60.2% for macaron and 78.9%1 for oss-rebuild.
| Rebuild tool | Byte-identical to PyPI wheel | Source-equivalent (daleq4py) |
|---|---|---|
| macaron | 15.4% | 60.2% |
| oss-rebuild | 19.1% | 78.9% |
Two observations survive a close read. First, the byte-identity rate is dismal but mostly boring: the gap between ~16% byte-identical and ~70% source-equivalent1 is made of timestamps, paths, and archive ordering, the known pathologies of Python’s build stack. Second, the gap between the two rebuild tools is itself informative. oss-rebuild beats macaron by 3.7 points on byte identity and 18.7 points on source equivalence, per the paper’s reported figures. That spread is not noise; it reflects how much of “is this package reproducible” depends on how faithfully the rebuild environment reconstructs the original build. Provenance of the verifier changes the verdict.
The residual is the part worth losing sleep over. Under the more generous oss-rebuild measurement, 21.1%1 of sampled releases are neither byte-identical nor source-equivalent. Under macaron, 39.8%1. That residual is the unexplained trust gap: wheels whose content cannot be accounted for by the published source, for reasons that range from unlisted build steps and patched vendored dependencies to, in the worst case, actual tampering. The paper does not claim the residual is malicious. It claims the residual is unverifiable, which from a supply-chain standpoint is the operative fact.
What do pip-audit and hash-pinning actually guarantee?
Hash-pinning guarantees that the artifact you install today is bit-identical to the artifact someone recorded when the pin was written. It says nothing about where that artifact came from. This is the assumption the paper puts under pressure: the entire hash-pinning control verifies a chain that starts at “PyPI served this file” and stops there. Whether the file could have been produced by anyone other than the original uploader, from the source anyone can read, is a question the control never asks.
Pip-audit operates one layer over, matching installed versions against known-vulnerability databases. It answers “is this version known to be bad?” Hash-pinning answers “is this the same bytes as before?” Neither answers “are these bytes what the source produces?” The paper’s framing reframes supply-chain assurance from verifying the distribution channel to verifying the artifact itself. PyPI’s upload pipeline is the root of trust in the pin-and-audit model, and it is a root that nobody independently checks, because until now there was no measured baseline for what checking would even find.
Sigstore-style signing improves the attestation story, but attestation and reproduction answer different questions. A signature tells you who built the wheel and that they stood behind it at build time. A reproduction tells you the build was faithful to the source. A compromised or coerced maintainer account signs beautifully. The two controls compose; neither substitutes for the other.
This is the gap the measurement makes concrete. Before the preprint, “trust PyPI’s upload pipeline” was an untested premise disguised as a control. After it, teams have a number for how much of the popular ecosystem can currently be moved from “trusted” to “verified,” and it is somewhere between 60% and 79%1 depending on whose rebuild harness you believe.
How much of your own dependency tree is checkable?
The practical use of the paper is coverage math: for a given dependency tree, what fraction of pinned wheels can be independently rebuilt and source-equivalence-checked today? The population-level answer from the study is 60.2% to 78.9%1 for popular releases. The answer for a specific tree will differ, and computing it is now an engineering task rather than a research one. For most teams the bottleneck is rebuild infrastructure: faithfully reconstructing each package’s original build environment is what makes the workflow expensive to run.
The workflow the paper implies is straightforward to state and annoying to run. Take the resolved dependency set, fetch each release’s sdist and wheel, run the rebuild, run the equivalence check, and bucket every dependency into one of three states: byte-identical, source-equivalent, or unexplained. The first two buckets are verifiable in the strong and moderate senses. The third bucket is where policy has to exist, because today it usually doesn’t: an unexplained wheel in a pinned requirements file passes every control the standard toolchain applies.
The burden shift is the real consequence. Under the pin-and-audit model, supply-chain assurance was outsourced to PyPI by construction. Under a rebuild-and-verify model, assurance lands on whoever runs the rebuilds, which means the consuming organization or a verification service it trusts. That is more work, and it is also the first model in which the trust claim is actually checkable. Teams that already vendor dependencies or run internal indexes are closest to being able to do this; the rebuild step slots into the mirroring pipeline they already operate.
One honest complication: rebuild coverage is a moving target. The percentages in the paper will drift as maintainers adopt deterministic build practices in response to exactly this kind of measurement, and as build backends change defaults. A coverage report is a snapshot. The preprint is the first baseline, not a permanent scoreboard.
Where does this measurement fall short?
The strongest limitation is baked into the sample. Twelve thousand one hundred eighty popular releases is a large measurement of the healthy end of PyPI, and reproducibility rates measured there almost certainly overstate the ecosystem as a whole. Popular packages have the CI infrastructure and maintainer attention that deterministic builds require; the long tail does not. A direct measurement of that long tail would almost certainly produce worse numbers, but no such study exists yet. Treat the 60.2, 78.9%1 source-equivalence range as an upper bound on what a random dependency tree would show, not an estimate of it.
The second limitation is interpretive rather than methodological. Byte-for-byte identity conflates mundane non-determinism with tampering, and any headline built on the ~16%1 figure alone overstates supply-chain risk by an order of magnitude. The paper’s own daleq4py metric is the correction, which is why the honest way to cite this work is always as a pair of numbers: 15.4, 19.1% byte-identical, 60.2, 78.9% source-equivalent1, per the preprint. Citing the first without the second is how a build-hygiene finding turns into a compromised-ecosystem story, and the evidence does not support that story.
There is also a venue caveat worth one sentence. The paper is a preprint hosted on arXiv, which as of July 2026 operates as an independent nonprofit; preprints are not peer-reviewed, and the specific percentages should be treated as a well-documented first measurement awaiting replication rather than settled facts. The methodology (sdist-to-wheel rebuild comparison with a normalization layer) is the durable contribution. The exact figures are the timely one, and they will change.
So what should a Python team do with this?
Treat PyPI wheels as verifiable-not-trusted. Run from-source rebuilds against your dependency tree, report coverage in terms of daleq4py-style source equivalence rather than byte identity, and write explicit policy for the residual that fails both checks. The paper’s practical contribution is that this is now quantifiable: you can state, with numbers, what fraction of what you ship can be independently regenerated from source, and that fraction is the first honest supply-chain metric the Python ecosystem has had.
The counterweight stays attached to the recommendation. Most byte-identity failures are timestamps and paths, not adversaries; the popular-release sample flatters the ecosystem; and the rebuild verdict depends on the rebuild tool’s provenance, with macaron and oss-rebuild disagreeing by nearly nineteen points on source equivalence. The residual unexplained gap, somewhere between a fifth and two-fifths of popular releases depending on the harness, is the real finding. It is not evidence of compromise. It is evidence that for a large slice of the packages everyone pins and audits, the pinned hash has never once been checked against the source it claims to come from.
Frequently Asked Questions
How does the No Snake Oil measurement differ from standard hash-pinning verification?
Hash-pinning only confirms that the downloaded wheel matches the hash recorded at pin time, assuming PyPI’s upload pipeline is trustworthy. The No Snake Oil study shifts the trust boundary by independently rebuilding wheels from source archives to verify that the published artifact could actually be produced by anyone, not just the original uploader. This exposes a gap where a pinned hash is valid but the wheel’s provenance remains unverified.
What is the practical impact of the 18.7 percentage point gap between macaron and oss-rebuild?
The gap reveals that rebuild tool provenance significantly affects verification outcomes, with oss-rebuild achieving 78.9% source equivalence compared to macaron’s 60.2%. This variance suggests that a package’s reproducibility status may depend on the specific build environment reconstruction fidelity of the tool used. Teams should treat the lower bound as the conservative trust signal and investigate packages that fall below even the more permissive tool’s threshold.
Why is the 12,180-release sample size potentially misleading for long-tail packages?
The study sampled popular releases, which typically have better CI infrastructure and maintainer attention than the long tail of PyPI. This selection bias means the 60.2% to 78.9% source equivalence rates likely overstate the ecosystem’s overall reproducibility. Long-tail packages often lack automated build checks, so their actual verification gap is probably wider than the headline figures suggest.
How should teams interpret the 21.1% to 39.8% residual unexplained gap?
This residual represents wheels that are neither byte-identical nor source-equivalent to their published sources, indicating unverifiable build steps or potential tampering. While most byte-identity failures stem from benign non-determinism like timestamps, the residual gap includes packages where the content cannot be accounted for by the source alone. Teams should treat this residual as a policy trigger for manual review rather than an immediate compromise indicator.
