Cloudflare’s cache-transcoding post, surfaced in feeds on 2026-09-04, proposes storing one Zstandard-compressed object per asset and transcoding on demand for clients without zstd support, and claims petabytes of cache storage savings [unverified, self-reported]. The verifiable core is the trade: one stored encoding cuts storage and origin refreshes, while every hit from a client that never sends Accept-Encoding: zstd pays a re-encode at 105 to 290 MB/s, five to fifteen times slower than the zstd decode before it, per zstd’s benchmark table. Your baseline encoding and your client mix decide whether it pays.
What did Cloudflare announce, and what can actually be verified?
A Cloudflare post proposing Zstandard compress-at-rest with Pingora-side transcoding [unverified] appeared on 2026-09-04, and its petabytes-saved headline is a self-reported figure without a published denominator. The proposal’s mechanics are legible without the post, because the underlying convention is standard: an HTTP cache that varies storage on Accept-Encoding holds one copy per negotiated encoding, a gzip copy, a brotli copy, sometimes both plus an uncompressed copy for clients that send nothing. Collapsing those variants into a single zstd-compressed object cuts storage and cuts the number of variants an origin refresh has to refill. The cost lands on the response path: a client that asks for gzip gets a zstd decode followed by a gzip re-encode, per hit, on the cache’s CPU.
The savings figure fails the first test of a storage claim, which is a denominator: petabytes out of what total footprint, measured against which baseline encoding, at which compression level. zstd spans negative levels to 22, and its ratio moves across that entire range, so a savings figure quoted without a level is not a number. Every Cloudflare-specific figure and every Pingora-specific mechanism below is treated as a vendor claim, and none of them is load-bearing for the analysis that follows.
What the public record offers is narrower: zstd’s own benchmark numbers in the README, the format’s documented properties in Wikipedia’s zstd entry, and release-level engineering detail in zstd’s release notes. What it does not offer is operator evidence from Varnish, Nginx, or Fastly shops running compress-at-rest in production. The decision framework below is built from zstd’s measurements and HTTP mechanics, not from CDN case studies.
How much storage does zstd save at rest?
About 5% versus a zlib-stored object and about half a percent versus brotli, at level 1 on the Silesia corpus, according to zstd’s own benchmark table, so the density case depends almost entirely on what your cache stores today. The conditions matter: Silesia is a general-purpose corpus of desktop files, run on a Core i7-9700K at 4.9 GHz under Ubuntu 24.04 with lzbench and gcc 14.2.0. It is not a CDN object store, and one desktop CPU is not an edge server. It is also the best public table available, and it is zstd’s own.
| Codec and level | Ratio on Silesia | Compression | Decompression | Where it sits in the transcode path |
|---|---|---|---|---|
| zstd 1.5.7, level 1 | 2.896 | 510 MB/s | 1,550 MB/s | The stored, at-rest encoding |
| brotli 1.1.0, level 1 | 2.883 | 290 MB/s | 425 MB/s | The alternative at rest; a fallback re-encode target |
| zlib 1.3.1, level 1 | 2.743 | 105 MB/s | 390 MB/s | The gzip re-encode, the bottleneck of the fallback path |
zstd 1.5.7, --fast=4 | 2.146 | 665 MB/s | 2,050 MB/s | The speed-first profile when CPU costs more than bytes |
Against zlib, the case is domination rather than a tradeoff. zstd 1.5.7 at level 1 reaches ratio 2.896, compressing at 510 MB/s and decompressing at 1,550 MB/s; zlib 1.3.1 at the same level reaches 2.743, at 105 MB/s and 390 MB/s. The zstd-stored object is about 5.3% smaller (34.5% of the original versus 36.5%), fills roughly five times faster, and decodes roughly four times faster. The only axis where zlib wins is universal client support, which is exactly the axis the transcoding proposal taxes.
Against brotli, the density argument evaporates. brotli 1.1.0 at level 1 reaches ratio 2.883, within half a percent of zstd’s 2.896. zstd’s real margins are speed: 510 versus 290 MB/s encoding, 1,550 versus 425 MB/s decoding. A cache that already stores brotli gains roughly 0.45% density by migrating to zstd-at-rest. That is a rounding error dressed up as a storage strategy. The honest argument for zstd over a brotli cache is CPU economics and variant collapse, not bytes saved.
Three caveats keep these numbers honest. First, a ratio quoted without its level misleads: zstd’s levels run from negative values (fastest) to 22 (best ratio), with compression speed varying by a factor of more than 20 across the range, per Wikipedia’s zstd entry. Second, Silesia is a desktop-file corpus; a CDN’s object mix skews toward minified text and already-compressed media, and recompressing the latter spends CPU for zero bytes. Third, levels 20 and above (the --ultra range) require substantially more memory, per the zstd manual, which matters if compression happens on capacity-constrained edge hardware.
What does transcoding cost per cache hit?
The transcode for a non-zstd client costs a zstd decode at roughly 1,550 MB/s plus a re-encode at 105 MB/s for gzip or 290 MB/s for brotli, per zstd’s benchmark table, and the re-encode is the entire bill. Per megabyte of object, the decode costs about 0.65 ms of one core; the gzip re-encode at zlib’s 105 MB/s costs about 9.5 ms; the brotli re-encode costs about 3.4 ms. A gzip fallback therefore costs on the order of 10 ms of single-core time per megabyte per hit, against effectively zero CPU to serve pre-stored compressed bytes straight from cache. On the gzip path the decode step is 6 to 7% of the total; for brotli it is closer to 16%. Calling it transcoding hides where the cost sits: this is a re-encode with a fast prefix.
The decode half stays cheap no matter how you store the object. zstd’s decompression speed is roughly constant across compression levels, a property the README notes is shared by most LZ algorithms such as zlib and lzma; across zstd’s full level range, compression speed varies by a factor of 20 or more while decompression varies less than 20%, per Wikipedia’s zstd entry. That is the property compress-once-decode-per-hit economics rest on: store at level 19 for density and the decode still runs near 1,550 MB/s. The re-encode gets no such relief, because it is bounded by the target codec’s encoder, and it runs on every hit from every client in the tail.
The 1.5.7 release notes (20 February 2025, 500+ commits) improve level-1 compression speed by 10 to 20% on 4 to 32 KB blocks, citing RocksDB’s typical ~16 KB blocks as the beneficiary. HTML documents, JSON responses, and JS chunks sit in exactly that size range, so the small-block work is CDN-adjacent by construction. The zstd manual adds two standard mitigations for repeated operation: reusing a compression context across operations cuts repeated-compression cost, and dictionary training dramatically improves compression of small records.
The structural mitigation is to cache the transcoded fallback variant on first request instead of transcoding every hit. That returns you to two stored encodings for the assets the tail actually requests, which reads like defeat until you notice the policy is now per-asset and demand-driven rather than a global convention: popular assets with tail traffic materialize a gzip variant, and the zstd-only long tail never pays the re-encode at all.
Which clients can receive zstd today?
The zstd content coding has been standardized since 2018, when RFC 8478 defined it alongside the application/zstd media type and the .zst extension, and RFC 8878 retained the registration when it obsoleted that document in February 2021, per Wikipedia’s zstd entry. Standardization says nothing about deployment. Which browsers advertise Accept-Encoding: zstd, and since which version, is not a question the format’s documentation can answer, and it is the question the tail-tax arithmetic hangs on.
If Safari and older Android WebViews do not advertise zstd, the fallback path is a first-class path rather than a rarity, and the petabytes arithmetic inherits whatever share of hits those clients represent. Nothing in zstd’s engineering fixes this, because it is not a zstd problem: it is a client-mix question that each operator answers from their own logs.
The number that prices this trade is the zstd share of hits in your own access logs, which is one week of logging Accept-Encoding away. Any claim that transcoding stays “rare” is a claim about that distribution on your traffic, and it is checkable locally before you commit an architecture to it.
Should your cache store one encoding per asset?
Store one zstd-compressed object per asset when your baseline is gzip-stored variants, when storage and origin-refresh costs dominate your edge CPU budget, and when most of your clients already advertise zstd; skip the migration if your cache already stores brotli and density is the motivation. The decision rules, keyed to the numbers above:
- Baseline gzip. Roughly 5.3% density, about 5x faster fills, about 4x faster decodes on the zstd path, per zstd’s benchmark table. The strongest case, modulo the tail.
- Baseline brotli. Roughly 0.45% density. The remaining argument is operational: one variant per asset instead of N, one
Accept-Encodingaxis to normalize, one object per origin refresh. You are buying simplicity, not storage. - Client mix. Set your threshold from logs, not from vendor slides. Below it, the re-encode tax scales linearly with tail hits, on the order of 10 ms of a core per megabyte per hit for gzip clients.
- Object mix. Text-class assets only. Recompressing already-compressed payloads spends CPU for no bytes, and Silesia’s ratios will not transfer exactly to your object mix.
- CPU placement. The decode is cheap enough to run anywhere; the re-encode belongs on whichever tier has CPU provisioned for it.
A stored-bytes format is a long-term commitment, because a format change at rest is a fleet-wide migration, not a config toggle. zstd’s credentials here are about as good as the field offers: the format is stable and documented in RFC 8878, the reference implementation is dual-licensed BSD OR GPLv2, and multiple independent implementations already exist, per the README. Compress-at-rest asks you to trust the format for years; a format that has been stable and RFC-documented since 2018 has the track record for it.
When does the trade flip in tiered cache?
Collapsing per-encoding variants into one zstd object multiplies the distinct objects a fixed cache footprint can hold, so upper tiers gain effective capacity while the per-hit transcode tax lands lower, at the tier facing non-zstd clients. The arithmetic is direct: a cache storing three encodings per asset holds roughly three times as many distinct assets in the same bytes after collapsing to one. More resident objects means higher hit rates on the long tail, which is the core job of an upper tier. Origin refreshes drop in step, since a miss refills one variant instead of N, and if compression happens on your side of the fill it runs at 510 MB/s at level 1 rather than zlib’s 105 MB/s, per the README’s benchmark.
CPU placement is the design lever. The decode half of the transcode is cheap enough to run on any tier; the re-encode belongs where CPU is provisioned, which in a tiered topology is usually the opposite end from where density matters most. Dense zstd storage up high, transcoding at the edge tier with the CPU budget, and the fallback variant optionally cached once materialized.
One quadrant deserves a caution sign. Compress-on-response spends CPU once per fill per encoding; compress-at-rest spends it once per fill plus once per non-zstd hit. The tax scales with hits, not with misses, so high hit rate plus a long tail is the expensive configuration, and it is also the configuration a well-tuned tiered cache tends toward. The better your hit rate, the more the tail costs you, until the tail shrinks. That dynamic, more than any single benchmark number, determines whether a multi-year commitment to this architecture pays.
What’s the verdict, and what would change it?
The evidence supports zstd-at-rest over a gzip-stored cache, does not support it over brotli on density grounds, and cannot confirm Cloudflare’s savings figure; the client mix is the variable that decides the fallback tax. Store one zstd object per asset where storage and origin-refresh costs dominate and most clients send Accept-Encoding: zstd. The fallback path costs a cheap decode (about 1,550 MB/s) plus a re-encode (105 to 290 MB/s) that is the real bottleneck, so a long browser-support tail converts a storage win into a per-hit CPU tax. Versus a gzip cache, the roughly 5% ratio case is real. Versus brotli it is marginal, and the argument has to come from operational simplicity instead.
The limitation is symmetrical with the analysis: every number here is either zstd’s own Silesia run on one desktop CPU or format documentation. No Cloudflare post, no Pingora internals, and no Varnish, Nginx, or Fastly operator data is in the public record. The petabytes figure has no independent verification, and a vendor’s self-reported savings on the vendor’s own cache deserve the default discount until the denominators appear.
Four things would change the answer. Safari’s and older Android’s Accept-Encoding status at the time you decide, since that sets the tail share. A public benchmark on a CDN object-store corpus rather than Silesia. Operator-published before-and-after numbers with baselines and levels stated. Per-level ratio curves on web-asset mixes, so the density claim stops resting on a desktop-file corpus.
The durable story, if the economics hold anywhere, is the convention shift: from compress-on-response toward compress-at-rest, with Accept-Encoding handled as capability negotiation instead of a vary-normalization chore. Whether transcoding stays on the rare path or becomes the norm is a question about Safari’s roadmap, not about zstd.
Frequently Asked Questions
How does zstd’s compression speed compare to xz in package management contexts?
Arch Linux switched from xz to zstd in 2020, accepting a 0.8% increase in package size in exchange for 14x faster decompression. Canonical’s 2018 tests found zstd at level 19 produced packages about 6% larger than xz but decompressed significantly faster, illustrating the explicit trade-off between density and CPU cost.
What specific latency optimization does zstd v1.5.6 offer for web traffic?
The ‘Chrome Edition’ release introduced the ZSTD_c_targetCBlockSize parameter, which splits compression blocks to improve time-to-first-byte. This optimization costs approximately 2% in compression speed at level 8 and less than 0.1% in compression efficiency, a trade-off designed specifically for HTTP streaming scenarios.
Why is zstd’s decompression speed stable across different compression levels?
Decompression speed varies by less than 20% across zstd’s full level range, a property shared with zlib and lzma. This stability allows operators to store objects at high levels like 19 for maximum density without incurring a per-hit CPU penalty, as the decode cost remains near 1,550 MB/s regardless of the compression level used at rest.
How does zstd 1.5.7 handle small block sizes relevant to CDN assets?
Version 1.5.7 improves level-1 compression speed by 10% to 20% for blocks between 4 and 32 KB, a range that covers typical HTML documents, JSON responses, and JavaScript chunks. The release also makes the CLI multi-threaded by default, capped at 4 threads, to better utilize modern edge hardware during the compression phase.
