Cloudflare has put a name and an algorithm behind a problem most platform engineers already know by feel: a Raft or etcd control plane is only as fast as its slowest leader link. In an engineering post published today, the company describes Meerkat, an experimental consensus service for its internal global control plane, powered by QuePaxa, a 2023 consensus algorithm from Tennage, Băsescu and coauthors. Everything about Meerkat in this article is author-reported from that single vendor post: it describes an experimental internal service, with no production benchmarks, SLOs or client-library details disclosed. The performance figures cited below for QuePaxa are prototype numbers from the original paper, relayed by a third-party summary, and have not been independently benchmarked.
That evidence limitation matters, but the underlying decision does not depend on Cloudflare’s specific numbers. If your control plane spans continents and its progress is tied to one leader sitting behind a volatile intercontinental link, you have three realistic options: keep Raft and tune the topology, move to a multileader Paxos variant, or adopt a leader-free protocol with a timeout-free core and pay for the correctness work that choice demands. Cloudflare chose the third. Whether you should depends less on the algorithm’s headline throughput than on which constraint is actually binding in your deployment, and on whether you can fund the verification burden a young implementation carries.
The failure mode Cloudflare is designing against
Leader-based consensus ties the whole system’s write progress to one node. When that node sits behind a degraded link, every follower waits on it, and standard timeouts make the recovery slow rather than immediate. Cloudflare reports experiencing exactly this in production, and its account is worth quoting because it describes the failure precisely:
“We have seen this happen firsthand: in 2020, the partial failure of a core Cloudflare network switch caused a consensus-powered system to degrade and the dashboard to operate 80x slower than usual, for six hours. Critically, the system lost no data. But it became unavailable for writes until the issue was resolved, which is unacceptable.” — Choosing QuePaxa, an unusual consensus algorithm
Two details in that account do the analytical work. First, safety held: no data was lost. The failure was liveness, the system refusing to accept writes for hours while a partially failed network component confused the assumptions a leader and its election timeouts rely on. Second, the trigger was a partial failure. A clean crash is the case Raft handles well; a switch that is alive but slow, lossy or intermittently partitioned is the case where leader election timeouts become the bottleneck rather than the remedy. If your WAN pain looks like this, the problem is latency variance and asynchrony, not round-trip count, and that distinction drives everything below.
What QuePaxa actually changes
QuePaxa, published at SOSP 2023 by Tennage, Băsescu, Ford and colleagues, combines a randomized asynchronous consensus core with a one-round-trip fast path for a preferred leader, according to a third-party summary of the paper. It is a crash-fault protocol, not a Byzantine one. The paper’s prototype results, as relayed by the same summary: roughly 584,000 commands per second on a LAN and 250,000 per second across a WAN under normal conditions, with liveness maintained under attacks and misconfigurations that stall Raft or Multi-Paxos. Treat those as paper-reported prototype numbers, not independent benchmarks.
Cloudflare’s own summary of why the design fit its problem is direct:
“The most notable thing about QuePaxa is how it can efficiently reach consensus in a single round trip under favorable network conditions, but also gracefully degrades performance during latency spikes, connectivity issues, and server failures. QuePaxa achieves this by solving the exact problems we just discussed through three key design choices: having an optional leader…” — Cloudflare blog
The mechanism behind that claim is documented in the QuePaxa Rust crate, a sans-IO, no_std replicated log explicitly positioned as an alternative to Raft and Multi-Paxos:
“QuePaxa’s distinguishing property is that liveness never depends on timeouts: a leader gives a one-round-trip fast path, but when it is slow, partitioned or gone, any replica can drive any slot with randomized priorities and the log keeps moving — no election, no stall waiting for a timeout to fire.”
The crate tolerates crash and recovery of any minority of replicas. The important nuance: the safeguard here is the randomized asynchronous core, not the mere absence of a leader. The Mandator/Sporades paper makes this boundary explicit:
“All leader-based and leaderless partially synchronous protocols fail at guaranteeing liveness in the face of network asynchrony and DDoS attacks. In contrast, Sporades continues to commit requests in the presence of network asynchrony.”
So “go leaderless” is not, by itself, robustness. A leaderless protocol built on partial synchrony assumptions loses liveness under the same hostile conditions that stall Raft. What you are evaluating in QuePaxa is a specific liveness mechanism, randomized priorities letting any replica drive any slot, and that is the property to verify rather than the marketing category.
Option A: keep Raft, fix the topology
The strongest counterweight to “a WAN footprint means leader-free consensus” is that leader-based protocols are not static. CD-Raft, an optimized Raft for cross-domain sites, attacks the same pain from inside the Raft family. It reduces write operations to a single cross-domain round trip through a Fast Return strategy and adds an Optimal Global Leader Position strategy, on the finding that leader position materially affects total global latency. The reported results against unmodified Raft, from the CD-Raft paper:
“For example, compared to Raft, CD-Raft reduces the average latency by 39.81% and tail latency by 49.44% in a write-only workload. In a typical read/write-balanced workload, CD-Raft reduces the average latency by 32.90% and the tail latency by 49.24%.”
Those percentages come from the CD-Raft paper’s cross-domain setting and have not been replicated against QuePaxa, so do not read them as a head-to-head result. What they establish is narrower and still useful: if your WAN pain is dominated by where the leader sits and how many intercontinental round trips a write requires, topology tuning can recover a large fraction of the latency without leaving a mature, well-tooled algorithm family. Raft’s operational ecosystem, runbooks, client libraries, failure-mode folklore, is a real asset that a 2023 algorithm does not yet have.
Option B: multileader Paxos
A second path stays in the Paxos family while removing the single-leader bottleneck. WPaxos is described by its authors as “a multileader Paxos protocol that provides low-latency and high-throughput consensus across wide-area network (WAN) deployments,” achieving throughput by pipelining so new commands can begin phase 2 before earlier slots have committed. Multiple leaders placed near clients let most writes commit locally, and the protocol has years of literature behind it. The tradeoff is that multileader designs introduce conflict handling when leaders contend over the same objects, which moves complexity from network tuning into the protocol’s conflict-resolution path, a cost that will recur in the verdict.
Mapping the decision to your deployment
The evidence supports a framework with four axes: what binds your latency, how much asynchrony you must survive, what round-trip count you can reach by tuning, and how much correctness engineering you can fund.
| Your situation | Best-supported option | Why | What it costs |
|---|---|---|---|
| Writes stall because the elected leader sits far from most replicas; latency is stable but long | Tuned Raft (leader placement, single cross-domain RTT, as in CD-Raft) | 39.81% average and 49.44% tail write latency reduction reported without changing algorithm family | Operational tuning work; still inherits leader-timeout failure modes |
| High WAN throughput with regional write locality needed | Multileader Paxos (WPaxos) | Low WAN latency and high throughput within a mature family | Conflict handling between leaders; more protocol complexity than Raft |
| Partial failures and latency spikes regularly defeat election timeouts; writes must keep moving under asynchrony | QuePaxa (crate, paper summary) | Liveness independent of timeouts; any replica can drive any slot with randomized priorities | Formal verification, simulation, fault injection, crash-recovery engineering on a young implementation |
| Hostile or adversarial network conditions are the threat model | None of the above alone; the Sporades paper shows partially synchronous protocols, leader-based or leaderless, lose liveness under asynchrony and DDoS | The differentiator is an asynchronous core, not leaderlessness | Evaluating protocols against their synchrony assumptions, not their category |
Two inferences in that table deserve to be flagged as inferences rather than measured findings. The mapping from “latency variance is binding” to “QuePaxa fits” follows from the protocol’s documented mechanism, but none of the papers cited here benchmarks QuePaxa against tuned Raft or WPaxos. And the row ordering reflects relative evidence maturity, not a performance ranking.
The cost Cloudflare accepted, and what you should verify
Cloudflare is unusually candid about what choosing a young algorithm requires:
“Choosing QuePaxa meant choosing a less-traveled path. It has not been battle-tested as extensively as Raft or Paxos, and we’re also extending the original crash-stop design to support crash recovery. This demands careful engineering and stronger evidence of correctness than usual systems.That is why we are formally verifying our QuePaxa implementation and Meerkat as a whole, alongside extensive simulation and fault-injection testing.” — Cloudflare blog
Read that as a price list. Formal verification, simulation, fault-injection testing, and extending the protocol from crash-stop to crash recovery: that is the correctness budget, and it lands on top of normal integration work. There is also a subtler second-order cost. Leader-based consensus concentrates operational pain in tuning: timeouts, leases, region pinning. Leader-free and multileader designs redistribute that pain into conflict resolution, where any replica can drive a slot and priorities resolve contention. Conflict-resolution paths are harder to audit and to reason about under failure, which raises the cost of correctness review for the whole control plane. This is my inference from the documented mechanisms, not a measured result, but it is consistent with why Cloudflare is paying for formal verification rather than relying on soak time.
Before adopting any young consensus implementation, QuePaxa’s crate included, several items belong on your checklist rather than assumed solved, whatever the documentation already covers:
- Crash recovery. The original design is crash-stop; Cloudflare is extending it. Confirm the state of that work in whatever you adopt.
- Snapshotting and membership changes. The crate’s documentation covers both, with guide chapters on snapshots and on membership and leaders, and a simulator that exercises membership changes. Both are mandatory for a production log and both are historically where young implementations harbor bugs, so verify the depth of that coverage against your requirements.
- Client libraries and integrations. The crate documents a C FFI binding, the quepaxa-ffi crate, and examples/kv-tokio, a complete networked, fsynced key-value store with a load generator; production-grade client SDKs are undocumented. A sans-IO,
no_stdcore is a solid engineering foundation, but the surrounding surface is what your team will live with. - Testing maturity. Ask what simulation and fault-injection infrastructure exists beyond the vendor’s own, and whether any deployment outside the original authors has survived a real incident.
The verdict
For a control plane spanning continents with volatile latency, adopting QuePaxa is defensible only if you can fund the correctness budget Cloudflare describes: formal verification, simulation, fault injection, and crash-recovery engineering. If your pain is dominated by leader placement and round-trip count, the lower-risk path is tuned leader-based consensus: optimal leader placement and single-cross-domain-RTT writes as demonstrated by CD-Raft, or multileader Paxos for throughput with regional locality. QuePaxa’s throughput and liveness claims remain prototype-reported until independently benchmarked, and Meerkat itself is an experimental internal service described in a single vendor post.
My judgment, with the evidence boundaries stated: if losing write availability for hours during a partial network failure would cost you more than a verification program, and you have the systems-engineering depth to run one, QuePaxa’s timeout-free liveness is the only property among the options here that directly addresses that failure mode. If not, tune the Raft you have. The 2020 incident Cloudflare describes is real motivation for taking that failure mode seriously.
Join the discussion
Share a useful perspective or ask a question about this article.