Which RAG Architecture Should You Run? A 2026 Survey Maps the Options

Most teams running retrieval-augmented generation keep re-arguing the same question: how much machinery does the pipeline actually need? A new survey on arXiv, Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications (observed 2026-10-08), tries to map the whole design space. Read it as a map, not a verdict: it is a single, unreplicated preprint that consolidates other people’s results rather than running its own head-to-head comparisons, so nothing in this article rests on the survey alone. Every decision-shaping number below comes from the primary study that produced it.

The short answer, supported by that primary evidence: default to hybrid retrieval plus reranking over contextually enriched chunks, escalate to LLM augmentation or agentic sub-queries only for the minority of real queries that a measured deferral policy routes there, and ship a cheap retrieval-confidence signal before you add hops. The reason this order matters is that the costs are asymmetric. Each extra stage buys accuracy on some query types and silently fails on others, and the failures land in your observability and staffing budget, not your p50 latency chart.

Measure your real query mix before copying a benchmark

The strongest single finding in this research packet is about how badly synthetic workloads mislead architecture choices. In a production RAG deployment study, benchmarks built from synthetic queries indicated that LLM augmentation was necessary for over 90% of queries (production study). When the team measured actual user traffic, only 27.8% of real queries needed it (production study); a retrieval-only hybrid stage returned sources for the remaining 72.2%. The authors state the implication plainly: “Practitioners should validate the augmentation need on real user traffic before assuming benchmark results carry over to their production setting.”

That is a threefold discrepancy between what a benchmark says you need and what your users actually ask for. Real users, in that deployment, submitted short keyword lookups against an entity-rich index; the synthetic suite was full of complex questions that no real person had asked. If you sized your pipeline against the synthetic mix, you would be paying for always-on augmentation that nearly three quarters of your traffic never uses.

So the first selection criterion is not a technique at all. It is instrumentation: log your production queries, classify them, and find out what fraction genuinely requires anything beyond hybrid retrieval. Everything downstream in this article assumes you have done that measurement or are willing to.

What each stage measurably buys

The cleanest stage-by-stage accounting comes from a metadata-driven financial QA study that compared pipeline configurations head-to-head on the same corpus. Moving from naive RAG (plain vector lookup over plain chunks) to a full hybrid-plus-reranking pipeline with a Cohere reranker lifted Claim Recall from 45.9 to 50.7 and Context Precision from 20.0 to 23.0. Chunking strategy mattered independently: contextual chunks, where generated chunk metadata is prepended to the chunk text before embedding, produced a higher F1-score in every head-to-head comparison in the study, and for the hybrid-plus-reranking configuration F1 rose from 38.9 to 44.1.

Two things are worth noticing in those numbers. First, the gains are real but modest in absolute terms: roughly five recall points for the full stack upgrade. That is the honest size of what reranking and hybrid retrieval buy on this corpus, and it is large enough to matter for a financial QA product while being small enough that a team with weak evaluation infrastructure might not detect it at all. Second, the cheapest change in the study, swapping plain chunks for contextual chunks, won every F1 comparison it entered, though the same study cautions that “they also led to a decrease in Claim Recall in the most advanced pipelines.” If you are going to spend effort anywhere before adding stages, the evidence points at your chunking.

There is also a result that cuts the other way, and it deserves equal weight. A calibrated-fusion study tested whether a sophisticated thermodynamic fusion method could beat simple reciprocal rank fusion for combining graph and vector retrieval. It could not: 7 wins against 11 losses on MuSiQue (p=0.48), not a statistically significant separation (calibrated-fusion study). More machinery is not a default win, and some upgrades buy nothing measurable. When a vendor demo shows an elaborate fusion or scoring layer, this null result is the question to bring.

Latency economics: deferral beats always-on

The production study above also measured what a smart escalation policy is worth. Instead of running HyDE-style LLM augmentation on every query, the team built a post-retrieval cascade: serve retrieval-only results first, and escalate to augmentation only when a binary condition says the first stage fell short. The cascade “improves quality by +0.140 Composite Overall points over Always-HyDE, reduces latency by 31.8%, and serves 72.2% of real user queries without LLM augmentation” (production study).

Read that again, because it inverts the usual assumption. The deferred design was not a quality-for-latency trade. It was better on both axes simultaneously, because always-on augmentation was actively hurting quality on queries that never needed it. The authors describe the design as “workflow-agnostic and adaptable to any ordered set with a binary escalation condition,” which means the pattern transfers even where the specific augmentation technique does not.

The practical consequence: if you are currently running query rewriting, HyDE, or any LLM-in-the-loop pre-retrieval step on all traffic, the measured alternative is to gate it. The gate itself can be simple; what it requires is that you know your escalation rate and can observe what the escalated queries look like.

The multi-hop tax, and the failures nobody alerts on

Multi-hop questions, the kind where the answer requires chaining facts across documents, are where plain vector retrieval structurally breaks. A practitioner analysis of thousands of real RAG queries put it concisely: “Multi-hop queries require multiple steps of retrieval to get to the right information,” with simple semantic search failing to link related concepts across documents.

The calibrated-fusion study quantifies the gradient. Its vector-only baseline reached R@5 of 69.8% overall, but the hop-count breakdown tells the real story: 77.9% on 2-hop queries, already above HippoRAG 2’s 74.7% (R@5 diagnostic), then 68.9% on 3-hop and 46.2% on 4-hop. At two hops, you may not need graph machinery at all. At four hops, vector retrieval is returning incomplete evidence for more than half of queries, and something else has to do the work.

The worse problem is not the degradation but the silence. A September 2026 study on multi-hop failure modes measured what happens downstream when retrieval misses. On MuSiQue with an LLM-judge pipeline, the failure-mode study found that 39.5% of test queries failed to retrieve all gold passages into the top five, and the system returned a ranked list anyway, with no signal that the evidence was incomplete. The authors give this a name, Confident Wrong Answer Rate, and a product interpretation (confidence-scoring paper):

In a deployed multi-hop fact-verification system (HoVer dense pipeline, CWAR == 31.7%), over one third of verified claims are incorrect with no downstream signal of unreliability — the system returns a ranked list with the same interface confidence as correct results.

A Confident Wrong Answer Rate of 31.7% (HoVer dense-pipeline figure) means that in a deployed verification system, a third of answers were wrong yet returned with full interface confidence. This is the finding that should change how you budget. Multi-hop pipelines shift the bottleneck from retrieval quality to failure diagnosis: the system fails often enough to matter and tells you about it never. The same study offers the cheap countermeasure, a Retrieval Confidence Score computed in under a millisecond from ANN scores already available at retrieval time, with no LLM call required for the abstention decision. Shipping that signal before adding hops is, in our reading, the most valuable observability work in this entire design space.

When agentic RAG earns its keep

The case for agentic designs, where the system generates sub-queries and iterates rather than retrieving once, is best made by a fintech study comparing baseline and agentic pipelines: “Unlike B-RAG, which often fails when the exact match is missing, A-RAG’s sub-query generation and iterative re-ranking modules better synthesize partial context across sources, producing correct responses even when the originating chunk is not directly retrieved.”

That is a genuine upside, and it targets exactly the failure the previous section quantified. The price is measured too: “A-RAG’s average query latency was 5.02 seconds, significantly higher than B-RAG’s 0.79 seconds” (fintech study), roughly 6.4 times the baseline, in exchange for strict retrieval accuracy of 62.35% against the baseline’s 54.12% (69.41% against 58.82% adjusted) (fintech study). What the study does not report is dollar cost. That multiplier is the reason to gate the loop: treat agentic escalation as a tool for the queries your deferral policy identifies as hard, measured on your own traffic, rather than as the new default tier.

One more boundary comes from the anchor survey itself. Consolidating comparative studies, it reports that “retrieval outperform[s] fine-tuning on sparse domains while underperforming on deeply specialized ones.” If your corpus is a deeply specialized domain where the model’s parametric knowledge is the bottleneck, the whole retrieval-machinery question may be secondary to a fine-tuning decision. That finding is cited within the survey to a referenced comparative study rather than tested directly; we would not bet a roadmap on it, but we would let it trigger an evaluation before assuming retrieval is the right lever.

A selection framework

Composing the primary studies gives a decision structure keyed to things you can actually measure. Every cell below is inference from heterogeneous settings (financial QA, MuSiQue, HippoRAG2, HoVer, and one production deployment), not a head-to-head result on a single corpus, so treat the thresholds as starting points to validate, not constants.

The eval maturity axis underlies all of it

The survey’s most consequential observation is not about any technique. Consolidating the field’s recent work, it reports that practical bottlenecks such as hallucination persistence, unclear attribution, and retrieval irrelevance have been codified into taxonomies of known RAG failures (survey). Every row in the table above presumes you can detect the failure it guards against. A five-point recall gain is invisible without per-query claim-level evaluation. A 39.5% silent-failure rate (failure-mode study) is invisible without a confidence signal. An agentic pipeline’s wins are invisible without tracing which sub-query recovered which fact.

This is where the architecture choice lands on your org chart rather than your infrastructure diagram. The modular pipelines raise the accuracy ceiling, measured, but they push cost into monitoring, abstention logic, and error recovery. A team that cannot staff that observability work will get more production value from a boring hybrid-plus-reranking tier with good chunking and a confidence score than from an agentic stack it cannot diagnose.

Practical verdict and honest limits

Run hybrid retrieval with reranking over contextual chunks as the default tier. Gate LLM augmentation behind a measured deferral policy rather than running it always-on. Before adding multi-hop or agentic capability, ship a sub-millisecond retrieval-confidence signal and an abstention path, because the measured silent-failure rates, roughly a third of queries returning incomplete evidence with full interface confidence, are the kind of bug that erodes user trust without ever paging anyone. Escalate to agentic sub-queries only for the slice of real traffic that demonstrably fails on missing exact matches, and budget for what that escalation costs: the single measurement in these studies puts the agentic loop at 5.02 seconds per query against a 0.79-second baseline (fintech study).

Now the limits, plainly. The anchor is one unreplicated survey preprint consolidating other studies’ results. The quantitative deltas come from five heterogeneous settings that were never run head-to-head on a shared corpus, so the framework above is composed inference, not a benchmark. The agentic upside is a single study’s measurement, with latency reported but dollar cost not. The RRF null result is a standing warning that some upgrades buy nothing. And none of these sources reports staffing or monitoring costs, which this article argues are the binding constraint. What the evidence does support is the order of operations: measure your real query mix, fix chunking and reranking, gate augmentation, instrument confidence, and only then reach for hops. Teams that follow that order will, at minimum, be spending their complexity budget where the measurements say it pays.

Frequently Asked Questions

What is the measured impact of using a deferral cascade instead of always-on LLM augmentation?

The cascade “improves quality by +0.140 Composite Overall points over Always-HyDE, reduces latency by 31.8%, and serves 72.2% of real user queries without LLM augmentation” (production study).

How accurate is vector-only retrieval for multi-hop questions?

Its vector-only baseline reached R@5 of 69.8% overall, but the hop-count breakdown tells the real story: 77.9% on 2-hop queries, already above HippoRAG 2’s 74.7% (R@5 diagnostic), then 68.9% on 3-hop and 46.2% on 4-hop.