Multi-Hop RAG: When Decomposing Questions Beats Retrieving Once

If your RAG assistant answers “what year was X founded” reliably but fumbles “was the founder of X born before the CEO of Y,” the fix is probably not a bigger context window or a better retriever. The evidence in a recent cluster of preprints points to a narrower answer: route the question. Compositional questions get decomposed into self-contained sub-questions backed by a shared answer cache; single-hop lookups keep direct retrieval. The trade is roughly doubled per-question latency on decomposed queries, offset only when sub-answers get reused across similar questions.

Two caveats belong up front, before any numbers. First, the headline gains come from a single October 2026 preprint, DeFT-RAG, and every accuracy figure in it is author-reported and unreplicated. Second, the strongest cost evidence comes from a different paper entirely, a component ablation with a local 7B model, so the combined picture in this article is synthesized across systems that were never run head-to-head. That makes this a decision guide built on indicative numbers, not a benchmark recap.

Why one-shot retrieval misses compositional questions

A multi-hop question requires chaining evidence across documents: find entity A, use a fact about A to identify entity B, then answer a question about B. The DeFT-RAG authors frame the two mainstream strategies as failing at opposite ends. Graph-based RAG retrieves once from the entities the question names and, in their words, “rarely recover[s] the full evidence chain.” Iterative methods recover the chain step by step, “but their accuracy is limited by the many tasks they ask the LLM to perform, such as planning the sub-questions, deciding their dependencies, rewriting them as answers arrive, and keeping track of what is known” (DeFT-RAG, arXiv:2610.37826).

This is not only a benchmark observation. A coverage audit of the live assistant You.com found it included core answers about 51% of the time but covered background or follow-up sub-questions only about 25% of the time. Production RAG assistants already fail exactly the sub-question coverage that decomposition claims to fix. If your users’ complaints cluster around comparisons, chained evidence, and “it answered half my question,” that is the failure mode under discussion, not a retrieval recall problem.

What DeFT-RAG changes

DeFT-RAG’s contribution is less “decompose the question” (iterative RAG has done that for years) and more how the decomposition is packaged. Sub-questions are written to be self-contained, so each can be retrieved and answered without carrying the whole conversation state. And the answers are kept: the paper argues that iterative methods “discard the intermediate sub-questions and answers” once a question is answered, calling this wasteful because “multi-hop questions over the same corpus often share intermediate facts” (DeFT-RAG). A shared QA cache stores those intermediate answers so later questions can reuse them.

The reported results are strong. According to the authors, DeFT-RAG “achieves the best accuracy among thirteen baselines with both gpt-4o-mini and Qwen3.6 as the backbone, while giving the LLM fewer input tokens than any other iterative method” across MuSiQue, 2WikiMultiHopQA and HotpotQA (DeFT-RAG). Note the scope of that token claim: fewer input tokens than other iterative methods. It is not a claim to beat every one-shot or long-context pipeline on tokens.

In a cache experiment across 1,000 MuSiQue questions, the authors report 23.1% of sub-questions answered from the cache and 12.9% fewer LLM calls, without lowering accuracy (DeFT-RAG). That number matters, but read its conditions: MuSiQue questions are constructed from shared underlying passages, so their sub-questions overlap by design. A production query log with diverse, one-off questions may see far less reuse. The cache is the economic heart of the proposal, and its hit rate outside benchmark corpora is the paper’s least transferable number.

The cost ledger: decomposition is not free

The clearest independent price tag comes from a component ablation of a multi-hop QA pipeline built on a local 7B model. Its authors report that “decomposition requires an extra LLM call and roughly doubles per-question latency; the no-decomp condition runs at 2,546 ms versus 5,642 ms for the full pipeline.” That is one decomposition call plus its retrieval round, before any iterative follow-up.

Worth noting what that pipeline achieved without any proprietary API: EM=55.0% and F1=63.5% on a 5K-question HotpotQA sample, in the same ablation study. So the latency doubling buys real capability, on a stack a team can self-host. But doubling per-question latency is a user-facing cost. On an interactive assistant, going from roughly 2.5 to 5.6 seconds changes whether the feature feels responsive, and that cost lands on every decomposed question whether or not the cache ever pays out.

Here is the honest accounting, assembled from non-comparable systems, so treat it as indicative rather than measured:

No row in that table is a knockout. The decision turns on your query mix and your tolerance for latency, which is why the failure triage matters more than the leaderboard.

Failure triage: what decomposition fixes, and what it amplifies

Decomposition fixes a specific failure: the evidence chain that a single retrieval cannot reach. If your evaluation set shows correct answers when the right passages are manually assembled, but wrong answers when the retriever runs once, that is a retrieval-coverage gap, and decomposition (or iterative retrieval generally) addresses it.

Decomposition amplifies two other failures, and the multi-hop literature names both. RT-RAG (arXiv:2601.11255) lists them as the two critical modes: “(1) inaccurate query decomposition, the generation of poorly structured or imprecise sub-questions that fail to correctly reflect the logical requirements of the original question; (2) error propagation, where an early incorrect step leads to flawed synthesis and invalidates the final answer.” A decomposed pipeline takes your model’s ability to write correct sub-questions as a dependency. If the model garbles the logic of the original question at step one, every downstream retrieval is confidently answering the wrong thing, and the final synthesis inherits the error with a veneer of rigor.

A third trap is subtler: the final answer can be right while the reasoning is wrong, so accuracy metrics alone can mislead your triage. On the CofCA counterfactual multi-hop benchmark, GPT-4 produced only 36.3% correct reasoning chains across the dataset, alongside a significant gap between factual and counterfactual performance. A system can score well on HotpotQA-style end answers while its chains are broken, because short answers (yes/no, a single entity) can be reached from partial or flawed reasoning. If you credit decomposition with an accuracy gain, check chain quality on a sample before you trust it.

And do not assume an intervention transfers across models. On the KinshipQA benchmark, humans reached 89.0% accuracy versus 50.7% for LLMs under identical rule access, and an in-context rule intervention helped one class of system by +17.1% while hurting another, already above ~60% baseline, by −13.4%. Pipeline changes have system-dependent effects. DeFT-RAG’s gains with gpt-4o-mini and Qwen3.6 backbones are evidence about those pairings, not a guarantee for your model.

Routing by question structure

The strongest design conclusion in this evidence cluster is that decomposition should be conditional. RT-RAG routes each query into one of three patterns: Parallel, for independent sub-questions merged at the end; Sequential, where each answer feeds the next sub-question; or Direct, where no decomposition is needed. That routing is the natural shape for a production system too, because it confines the latency doubling and the error-propagation risk to the traffic that actually needs chaining.

A practical routing policy, grounded in the evidence above:

• Single-hop lookups and entity questions: direct retrieval. The ablation’s no-decomposition condition (2,546 ms, one retrieval, arXiv:2606.21553) is the right shape for this traffic.
• Comparisons and multi-entity questions: parallel decomposition. Sub-questions are independent, so errors propagate less and cached sub-answers are most reusable.
• Chained questions (“the founder of the company that…”): sequential decomposition with self-contained sub-questions. This is DeFT-RAG’s core case, and where the cache matters most, because bridge entities recur across questions over the same corpus.
• Anything where the model’s decomposition quality is suspect: fall back to direct retrieval rather than forcing a plan. Inaccurate decomposition is a documented failure mode (RT-RAG), and a bad plan is worse than a modest single retrieval.

This routing recommendation is an inference from the packet’s combined evidence, not a configuration any single paper tested end-to-end. It is the only configuration consistent with both the upside claims and the measured costs, but it has not been run head-to-head against either pure strategy.

The sufficiency threshold: when better retrieval is the cheaper fix

Before restructuring the pipeline, it is worth pricing the alternative, because the evidence says retrieval improvement saturates. In the SpectRAG token-budgeted retrieval study, a large retrieval advantage, Recall@5 of 64.7 versus LightRAG’s 49.2, “translates into only a small answer-accuracy gain.” The authors attribute this to a retrieval-and-generation gap: multi-hop answers are short and often reachable from partial evidence, so “beyond a sufficiency threshold, additional retrieved evidence yields diminishing returns on final accuracy.”

The practical consequence runs both ways. If your retriever is below that sufficiency threshold for your corpus, fixing recall is cheaper than decomposition: no extra LLM call, no latency doubling, no cache to operate. If you are already at the threshold and still failing compositional questions, more retrieval investment is the wrong tool, and the remaining errors are structural, which is where decomposition earns its cost. A recall evaluation on your failing questions, not a gut feeling, should decide which side of the threshold you are on.

There are also middle paths between one-shot retrieval and full question decomposition. Tree of Reviews, a tree-based dynamic iterative retrieval framework, beat the ITER-RETGEN baseline on retrieval metrics by 13.3%, 12.0% and 1.5% across three multi-hop datasets by expanding retrieval dynamically rather than decomposing the question up front. If your latency budget cannot absorb decomposition, iterative retrieval expansion is a real alternative worth prototyping against.

What I would do with this

For a team whose assistants handle single-document lookups but fail compositional questions, I would not rebuild around DeFT-RAG’s exact design on the strength of one preprint. I would build the routing skeleton: a lightweight classifier or model call that separates compositional traffic from direct lookups, decomposition only for the compositional path, and a shared sub-answer cache treated as an experiment with its own hit-rate metric from day one. Accept roughly doubled latency on the decomposed path as the price of admission, because on current evidence the cache (23.1% of sub-questions served, 12.9% fewer calls, author-reported on benchmark traffic, DeFT-RAG) is the only measured offset, and your production reuse rate is unknown until you measure it.

What this evidence cannot claim: that DeFT-RAG’s gains replicate outside its authors’ runs; that production cache hit rates will resemble a benchmark corpus built from shared passages; that long-context one-shot RAG, simply stuffing more retrieved text into a large context window, would not match decomposition for your workload, since no source here prices that third option; and that the synthesized cost model is anything more than indicative, since its latency comes from a local-7B ablation while its token and call counts come from gpt-4o-mini and Qwen3.6 runs. The durable part is the decision shape: route by question structure, price decomposition in calls and latency, gate retrieval spending at the sufficiency threshold, and monitor sub-question quality and cache hit rate as the new bottlenecks. Those survive whatever happens to this particular preprint in peer review.

Frequently Asked Questions

How much does decomposition increase per-question latency?

Its authors report that “decomposition requires an extra LLM call and roughly doubles per-question latency; the no-decomp condition runs at 2,546 ms versus 5,642 ms for the full pipeline.”

What is the sufficiency threshold in retrieval?

The authors attribute this to a retrieval-and-generation gap: multi-hop answers are short and often reachable from partial evidence, so “beyond a sufficiency threshold, additional retrieved evidence yields diminishing returns on final accuracy.”

What are the main failure modes of decomposition?

RT-RAG (arXiv:2601.11255) lists them as the two critical modes: “(1) inaccurate query decomposition, the generation of poorly structured or imprecise sub-questions that fail to correctly reflect the logical requirements of the original question; (2) error propagation, where an early incorrect step leads to flawed synthesis and invalidates the final answer.”