Meta’s ads retrieval team says it now scores every eligible ad with near ranking-level model quality at more than one billion ads per second, on GPUs, in production. Before treating that as a signal to move your own embedding retrieval off CPUs, two qualifications matter. First, every number in that announcement comes from a single vendor-authored Meta engineering post describing a joint collaboration with NVIDIA as hardware partner, with no independent replication. Second, the independent systems research that bears on this question says GPU vector search pays off only under specific conditions, and those conditions, not the GPU itself, are what Meta engineered. Whether retrieval should leave your CPUs depends on whether your workload clears the same gates Meta had to clear: throughput high enough that compute dominates over data movement, an index that fits in GPU memory, and an update path that avoids CPU-GPU round-trips.
What Meta actually claims
The Meta engineering post published October 28, 2026 reports several distinct results, and keeping them separate matters because they rest on different mechanisms:
- Production throughput: “the system scores every eligible ad with near ranking-level model performance at retrieval scale and surfaces the top candidates at over one billion ads per second.”
- Model efficiency: the retrieval mixture-of-experts model reaches ranking-level matching quality at “~300x fewer inference-time FLOPs than the 2024 Meta Andromeda and over 10 billion ads/sec on a single Grace Hopper Superchip.”
- Pipeline fusion: consolidating previously segmented distributed workloads onto a single GPU pipeline cut network traffic by roughly 100x, because only compact request payloads cross the wire instead of intermediate tensors.
- Kernel co-design: in the joint Meta-NVIDIA collaboration launched in early 2026, “each component achieved 2x to 5x speedup over the baseline,” with the gating kernel for the retrieval MoE’s interaction network improving 5.3x.
One clarification before going further: Andromeda performs exact scoring of every eligible ad, not approximate nearest-neighbor search. That is a different algorithm class from the Faiss or HNSW indexes most search and recommendation platforms run. ANN retrieval trades recall for speed by visiting a small fraction of the catalog; Andromeda’s design makes brute-force-style scoring cheap enough to apply to everything. The techniques transfer. The workload does not map one-to-one.
Technique 1: Retrieval mixture-of-experts, and what it costs you
The headline efficiency claim, ~300x fewer inference FLOPs than the 2024 Andromeda, comes from replacing dense two-tower scoring with a mixture-of-experts model where a gating network activates only a slice of the model per ad. That is what makes scoring a billion ads per second computationally feasible at all.
But adopting MoE retrieval changes what capacity planning means, and this is where the first independent check lands. MoE models are wide: total parameters grow with expert count even though per-query compute stays flat. If the experts fit in GPU HBM, you get the FLOP savings. If they do not, and you offload experts to CPU DRAM, the economics invert. Research on offloading-based MoE inference finds that “the latency incurred by loading experts can constitute over 80% of the total end-to-end latency”. The model that was supposed to be cheap to run becomes a model you spend most of your time fetching.
For a platform engineer, this reframes the adoption question. You are no longer asking “how many CPU cores do I need for scoring throughput” but “does my model fit in accelerator memory, and if not, what does expert routing cost me?” Meta can size Grace Hopper nodes around its model. A mid-size platform running a retrieval MoE on whatever GPU inventory it has may find the binding constraint is HBM capacity and expert-placement balance, not FLOPs. The ~300x figure describes compute; it says nothing about the memory regime required to realize it.
Technique 2: Fusing the pipeline onto one GPU
The roughly 100x network-traffic reduction is arguably the most transferable finding in the post, because it is about architecture rather than hardware. Meta’s previous design segmented retrieval work across distributed stages, shipping intermediate results between them. The redesign fuses those stages into a single GPU pipeline so that only compact request payloads travel over the network.
The mechanism behind the win shows up in Meta’s own bottleneck statement: “At billions of ads per second throughput, the memory bandwidth of passing ad tensors through memory hierarchy becomes the new bottleneck. Intermediate tensors between GPU kernels account for terabytes per second of memory traffic.” Once scoring compute got cheap, thanks to the MoE, the cost of moving tensors dominated. This is the recurring pattern in accelerated systems: remove one bottleneck and data movement becomes the next one.
The transferable lesson is not “buy GPUs.” It is “count your transfers.” If your retrieval path moves query vectors, candidate sets, or intermediate embeddings between hosts or between CPU and GPU memory per request, that movement, not scoring, is likely your ceiling. You can audit this on a CPU architecture today.
Technique 3: Co-designed kernels, and what 2x to 5x does not tell you
The NVIDIA collaboration produced 2x to 5x speedups per pipeline component, with the 5.3x gating-kernel improvement as the standout. NVIDIA wrote optimized kernels; Meta integrated and benchmarked. Two cautions apply. The gating kernel’s baseline is disclosed: the 5.3x takes it from 2.4 billion to over 10 billion ads per second, at ~85% of the GPU’s peak HBM bandwidth (Meta engineering post). The baselines for the remaining components, and the occupancy figures, are not disclosed, so 2x against what is unknown there. And vendor-authored kernel benchmarks from a hardware co-design partnership are the least independent numbers in the entire announcement. Treat them as evidence that kernel-level optimization headroom existed, not as a number you can expect on your own models.
The counter-evidence: residency and transfer overhead
This is where independent systems research sharpens the decision. To GPU or Not to GPU: Vector Search in Relational Engines tested GPU vector search across interconnects and reached a blunt conclusion: “Even when using NVLink, with current data-owning vector indexes, running vector search on a GPU does not pay off unless the vector index is resident on the GPU.”
The paper’s underlying measurements explain why. Citing MaxBench, it reports that, for relational analytics, “data transfers account for 67–98% of runtime on PCIe, but on NVLink the interconnect bottleneck disappears and execution becomes GPU bound”; for vector search on PCIe, it cites a survey where transfers consume 40–97% of total latency (To GPU or Not to GPU). Read those findings together: on PCIe, you can spend nearly all your runtime moving data; on NVLink, the interconnect stops being the excuse, and yet GPU search still does not pay unless the index lives on the GPU. Residency, not bandwidth, is the gate.
This is exactly the condition Meta satisfied by construction. Andromeda’s fused pipeline keeps model and tensors on one Grace Hopper Superchip, where CPU and GPU share a high-bandwidth coherent memory fabric. The reported results are consistent with the arXiv findings rather than in tension with them: Meta engineered away precisely the costs the independent work documents. What that means for you is that the Andromeda numbers are not a preview of what happens when you point a GPU at your existing CPU-resident index. They are a preview of what happens when you rebuild the serving architecture around GPU residency.
The mutability problem nobody puts in the headline
Ads retrieval indexes churn constantly: ads expire, campaigns start, budgets pause. If your index updates continuously, a second independent finding matters. SIVF, a GPU-resident IVF index for streaming vector search, documents that “current GPU indices rely on a CPU-GPU Roundtrip pattern for eviction. That is, to delete even a single vector, the system must transfer the entire index state from VRAM to host memory, perform the compaction on the CPU, and re-upload the modified index to the device.”
A full index round-trip per deletion is not a tail cost; for a streaming catalog it is the dominant one. SIVF’s own results quantify the cost and the headroom from eliminating it: the standard Faiss baseline freezes for 355 ms per update on SIFT1M and over 1.1 s on GIST1M, while SIVF’s in-VRAM updates cut per-step latency to about 2.2 ms and 4.2 ms respectively, reaching 108.5 million vectors per second deletion and 4.07 million vectors per second ingestion on 12 GPUs. Those gains are relative to the standard Faiss GPU baseline, not to CPU ANN, and the 12-GPU figures are a distributed setup most platforms will not run. But the finding stands on its own: GPU index residency is necessary and not sufficient. Your update path has to be GPU-native too, or eviction round-trips hand the advantage back.
Where the thresholds sit
Putting the vendor claims and the independent findings together produces a decision frame organized around three gates:
| Gate | Evidence for the threshold | Below it | Above it |
|---|---|---|---|
| Throughput: does compute dominate? | Meta’s wins appear at billions of scores/sec; PCIe transfers consume 67–98% of relational runtime and 40–97% of vector-search latency when movement dominates | CPU-resident ANN serving; GPU adds transfer cost without compute relief | Scoring compute is the bottleneck GPUs actually relieve |
| Residency: do model and index fit in GPU memory? | GPU search “does not pay off unless the vector index is resident on the GPU”; MoE expert loading can be over 80% of latency when offloaded | Offloaded experts or host-resident index erase the GPU advantage | GPU-resident index plus HBM-resident model (retrieval MoE is Meta’s lever for shrinking the compute side) |
| Mutability: is the update path GPU-native? | Existing GPU indexes round-trip the full index through host memory to delete one vector | Frequent ingestion/deletion forces constant CPU-GPU eviction traffic | Streaming-capable GPU index designs (SIVF-class) or near-static catalogs |
Two honest qualifications on this table. First, the crossover evidence is batch-shaped rather than catalog-shaped: there are no precise catalog-size curves, because Meta reports its own hyperscale operating point and the arXiv work establishes conditions rather than a size curve. Second, the CPU-versus-GPU measurements that do exist favor GPUs once residency is satisfied. The independent paper benchmarks FAISS with CPU HNSW and GPU CAGRA and reports that “CAGRA, the current GPU graph-based index SOTA, reports 33–77× higher throughput than CPU HNSW at 90–95% recall,” and that batch size decides the copying strategies: for IVF, copying just the index structure to the GPU becomes better than CPU between 10–100 queries, while for CAGRA even copying the data-owning index only improves over CPU above a 10^3-query batch (To GPU or Not to GPU). Below those batch sizes, the copy-per-request strategies lose to CPU; only an already-resident index wins at low query counts. What you can measure is which side of each gate your workload sits on: profile whether your serving time is compute or transfer, check your index size against available HBM, and count mutations per second.
The practical verdict
Move retrieval scoring onto GPUs when you clear all three gates: throughput high enough that compute, not host-to-device movement, dominates your serving time; a model and index compact enough to stay GPU-resident, where retrieval MoE is the reported mechanism for shrinking the compute side at ~300x fewer inference FLOPs; and a streaming-update path that avoids CPU-GPU eviction round-trips. If you run billion-scale candidate scoring with a catalog and model that fit in accelerator memory, Meta’s architecture is a credible, vendor-reported template.
If you run a moderate-QPS service over a catalog in the millions, with a CPU ANN index that meets latency today, no evidence says GPU compute against that host-resident index helps, and the independent work says the naive version of the move, GPU compute against a host-resident index over PCIe, will hurt. That is not a verdict against GPUs for a millions-scale catalog: the same paper benchmarks millions-scale data, roughly 2.4 million reviews and 750,000 images, and reports a GPU-resident configuration that “beats cpu on every (index, query) pair by 2×–69×,” plus a DGX-Spark unified-memory system where the GPU beats CPU by roughly 2.5–15× for exhaustive search and 1.5–9× for ANN (To GPU or Not to GPU). What those wins require is residency or unified memory, not a faster card aimed at host memory. I would treat any GPU retrieval proposal that does not specify index residency and the eviction path as unfinished: those two lines determine the answer more than the accelerator choice does.
Limitations worth keeping
Every Andromeda figure in this article comes from one Meta engineering post describing a joint collaboration with the hardware vendor, with no independent replication and with most component speedup baselines undisclosed. The workload, exact scoring of every eligible ad over billion-scale catalogs with sparse user signals, is a specific regime; content and product search, where approximate CPU ANN is standard, may behave differently. No cost, energy, latency-SLO, or memory-sizing figures exist in the evidence, so total cost of ownership is an open question this article cannot close. The durable findings, that transfers dominate below a throughput threshold, that residency is the gate, that expert loading and eviction are the hidden MoE costs, come from independent work and will outlive this announcement. The numbers attached to Meta’s deployment should be treated as one vendor’s report of its own system, useful as a ceiling and an architecture sketch, not as a forecast for your stack.
Frequently Asked Questions
What are the three conditions required for GPU retrieval to be beneficial?
Whether retrieval should leave your CPUs depends on whether your workload clears the same gates Meta had to clear: throughput high enough that compute dominates over data movement, an index that fits in GPU memory, and an update path that avoids CPU-GPU round-trips.
Does GPU vector search work if the index is not stored on the GPU?
Even when using NVLink, with current data-owning vector indexes, running vector search on a GPU does not pay off unless the vector index is resident on the GPU.
What is the cost of updating a GPU-resident index with standard tools?
current GPU indices rely on a CPU-GPU Roundtrip pattern for eviction. That is, to delete even a single vector, the system must transfer the entire index state from VRAM to host memory, perform the compaction on the CPU, and re-upload the modified index to the device.
Join the discussion
Share a useful perspective or ask a question about this article.