Running Text-to-SQL on a Small Local LLM: What Retry Loops Fix

If your team cannot send database schemas or rows to a frontier API, the evidence now supports a concrete middle path: serve a SQL-specialized model in the 1B, 8B range locally, wrap it in an automatic retry loop that switches prompts or models on failure and runs an optional syntax-correction step, and treat that loop as your first reliability lever rather than your last. That is the central finding of an October 2026 study, arXiv:2610.17305, which evaluates exactly this setup under a controlled inference budget. The catch, and it is a real one, is that retries repair the cheap error classes while the expensive ones, schema linking and query decomposition, persist. Whether the loop substitutes for a frontier API depends on which of those classes dominates your workload, and the only way to know is to measure on your own schema.

What the October 2026 study actually measured

The anchor paper, Reliable Text-to-SQL in Resource-Constrained Settings with Small LLMs via Iterative Retry Workflows, posted this month, frames the problem the way a platform team would: given a fixed inference budget and open-weight models between 1B and 8B parameters, how much reliability can orchestration buy? Its answer is a set of three workflows described in the paper as “lightweight inference workflows that automatically retry with alternative prompts and/or models when failures occur, including an optional syntax-correction step.”

That design choice matters more than it might appear. A retry loop that only re-runs the same prompt is mostly a temperature lottery. Switching the prompt, switching the model, and adding a re-prompted syntax-correction step are three different repair mechanisms aimed at three different failure causes, and the study treats them as composable rather than as alternatives. The correction step is an LLM call, not a rule-based pass, and that matters for cost: in all three workflows, the same model that produced the failed query is used once, conditioned on that query, to generate a corrected version, so a failed attempt can cost two generations rather than one.

A second finding from the same study shapes everything downstream: according to the authors, “SQL-specialized models are consistently more reliable than general-purpose and chat-oriented models at small scales.” Model selection inside the 1B, 8B band is therefore a measured decision, not a matter of grabbing whichever 7B chat model is already on your GPU server. This does not mean any model labeled “SQL” will work on your schema; it means that within the evaluated set, specialization was the reliable axis.

The failure-mode map: what a retry loop repairs and what it cannot

The practical value of a retry loop depends on what is actually failing. Text-to-SQL errors cluster into recognizable classes, and they respond very differently to a second attempt:

The lower two rows are the argument. In the SQL-of-Thought multi-agent study, the authors report that “reasoning models demonstrated clear advantages in nearly all specialized agentic tasks, including schema linking, query planning, chain of thought, and clause identification,” while non-reasoning models “often failed to decompose queries correctly and produced valid but logically wrong SQL.” Small models in the 1B, 8B range are, in this literature, overwhelmingly the non-reasoning kind.

That last phrase deserves attention: valid but logically wrong SQL is the worst failure mode for a retry loop, because nothing in the loop can see it. A syntax error announces itself by throwing. A query that joins the wrong tables and returns a plausible-looking number passes every check the retry workflow runs. If you build this system, execution success is not your acceptance metric; it is your retry trigger, nothing more.

One honest limitation of the current evidence: the study does not report per-error-class repair rates, so the clean split in the table above is a synthesis across papers rather than a single measured breakdown. The direction is well supported; the magnitudes per class are not yet published.

Choosing inside the small-model class

The study-reported finding that SQL-specialized models beat general and chat-tuned models at small scale (arXiv:2610.17305) has a mechanical explanation worth stating. At 7–8B parameters, a model’s capacity budget is tight. A chat-tuned model spends some of that budget on conversational behavior you do not need; a model fine-tuned on SQL corpora spends it on join syntax, aggregation patterns, and schema-grounded generation. At frontier scale, general capability can cover the gap. At 7B, the trade is visible in the output.

For calibration on how much headroom exists: SQLForge-LM reports, as study-reported figures, execution accuracy of 85.7% on Spider Dev and 59.8% on BIRD Dev (SQLForge), which its authors describe as narrowing the gap with closed-source methods. For historical context on the leaderboard itself, DAIL-SQL reached 86.6% execution accuracy on the Spider test set, against a prior state of the art of 85.3%, both test-set leaderboard figures (DAIL-SQL / Spider benchmark evaluation); on the Spider development split, the comparison table in the SQLForge paper places DAIL-SQL at 83.5 and DIN-SQL at 82.9 (SQLForge). These numbers come from different pipelines and are not directly comparable, but they bound the conversation: a well-built open pipeline is within a few points of the best published Spider numbers, while BIRD remains much harder for everyone.

The economics, honestly stated

The frontier-API alternative carries a concrete price tag in at least one measured configuration. In the SQL-of-Thought framework, the authors report that “a single run on the Spider benchmark takes about 42.58$ for Claude Opus 3, and 44.2$ for GPT models” (SQL-of-Thought). That is one multi-agent framework’s cost for one benchmark run, not a universal API price, and a single production query costs far less. But the number is useful in the other direction: evaluation itself is not free. If you plan to iterate on prompts, schemas, and workflows against a benchmark-sized suite, per-run costs in the tens of dollars compound quickly, and every one of those runs ships your schema to a third party.

The local path trades that spend for orchestration code: a retry controller, a syntax checker, an execution sandbox, and an evaluation harness. That is real engineering, and it is yours to maintain. For a team already serving models internally, it is also the kind of cost that is fixed rather than per-query, which is usually the right shape for a feature with unpredictable usage.

Latency: the cost of retrying at P95

A retry loop multiplies inference calls. A query answered on the first attempt costs one generation; a query that exhausts three retry workflows costs several, and tail latency is what your users notice. The retry study itself does not publish serving numbers, so treat this axis as unverified for that specific system.

Adjacent evidence suggests the problem is schedulable rather than fundamental. Hexgen-Flow, a serving framework built for agentic text-to-SQL workloads, reports (study-reported) P95 tail-latency reductions of 1.42–1.56× and throughput gains of 1.49–1.81× against state-of-the-art serving systems (Hexgen-Flow). Those gains come from request scheduling across the multi-call structure of agentic workloads, which is precisely the structure a retry loop creates. The caveat: these are Hexgen-Flow’s numbers for its own system, not a property you get automatically from any local deployment. If you adopt retries, budget the serving work, or at least cap retries per query and route first-attempt failures to a slower path rather than blocking the interactive one.

Benchmark honesty, and a procedure you can reproduce

Every accuracy number above was measured on Spider or BIRD, though not on one split: SQLForge’s figures and the retry study’s results come from development sets, while the DAIL-SQL leaderboard figures are Spider test-set results (SQLForge). Both benchmarks have a known structural bias. The Beyond SELECT authors put it directly: “Many datasets such as Spider (Yu et al., 2018) and Bird (Li et al., 2024) are tailored to specific scenarios or applications and usually emphasize “Select” operations, failing to capture the full spectrum of user questions and SQL queries encountered in real-world scenarios.” Whether contamination affects these benchmarks is not settled by any cited source, so treat it as an open risk rather than a finding.

The implication is not that the benchmarks are useless. It is that a retry-loop win on Spider Dev is evidence about Spider Dev. A reproducible procedure for your own decision looks like this:

1. Baseline on the standard sets. Spider’s development split has 1,034 instances across 200 databases, with 8,659 training instances (Spider benchmark evaluation); BIRD’s official development set has 1,534 samples (ErrorLLM). Run your candidate model single-shot, then with the retry workflow, on both. This makes your numbers comparable to the literature.
2. Build a private eval from your own schema. A few hundred questions over your actual tables, including your dialect’s quirks (window functions, dialect-specific date handling, whatever your warehouse makes awkward), labeled by error class: syntax, schema linking, decomposition, value binding.
3. Score retry repairs by class, not in aggregate. The aggregate number will flatter the loop, because syntax fixes are easy wins. The schema-linking row is the one that decides whether you need a bigger model.

Step 3 is where most evaluations go soft, and it is the step that maps directly onto the build-or-pay decision.

When retries stop substituting for scale

Two results bound what a runtime retry loop can ever give you, and both deserve more attention than they typically get.

First, the strongest retry-flavored result in the literature is not a runtime loop at all. RetrySQL trains models on retry data, and its authors report (study-reported) that “coding models trained with our RetrySQL paradigm are competitive with much larger closed-source proprietary LLMs when used as a part of the full text-to-SQL pipeline,” with OpenCoder 1.5B gaining roughly 4 percentage points of execution accuracy on BIRD and 3.1 points on Spider (RetrySQL). The lesson: retry benefits are real, but the durable version of them lives in weights, not in the loop.

Second, durability has a measured ceiling. In the crystallization study, which asks how much of test-time repair converts into lasting first-attempt accuracy, a verified memory bank of corrected episodes lifted held-out first-attempt accuracy by 4.34 percentage points, equal to “44.4% of the on-demand repair headroom measured on the same questions” (study-reported, arXiv:2608.07213). That ratio is descriptive rather than a probability, and its 95% CI of [24,65]% is wide: the retained share of repair value could sit anywhere from about a quarter to about two-thirds. Note where even that retained value lives: in a bank of corrected episodes retrieved for later questions over the same database, not in the model’s weights. In plain terms: at the point estimate, more than half of what a repair mechanism fixes on demand evaporates when you ask the model to get it right the first time without such a memory. If your product needs first-attempt correctness, for latency or for trust, a retry loop is a patch with a known half-life, and the training path or a stronger base model is the durable fix.

The decision

Build the retry loop first. The mechanism is validated in-domain by the October 2026 study, the base-model choice is a measured lever rather than a guess, the orchestration cost is engineering you control rather than per-query spend plus data egress, and the latency overhead appears schedulable based on adjacent serving work. Choose an SQL-specialized model, cap the retry budget, and instrument the failure classes separately from day one.

Then measure before you rule anything out. If your private evaluation shows failures concentrated in syntax and prompt sensitivity, the loop is doing its job and the frontier API is hard to justify. If schema-linking and decomposition errors dominate, retries will not save you; your money is better spent on schema retrieval and pruning, on the RetrySQL-style training path, or, if your privacy constraints allow any egress at all, on a larger model for the hardest slice of queries.

The limits of this advice are worth stating plainly. The retry-workflow result rests on a single study on standard benchmarks, its per-class repair rates are not reported, Spider and BIRD under-represent the production query mix, and the cost and latency figures come from specific systems rather than general prices. What the evidence does settle is narrower and still useful: for a privacy-constrained team, a retry loop around a specialized small model is the correct first build, and schema linking is where you should expect it to stop working.

Frequently Asked Questions

What is the cost of running the SQL-of-Thought framework on the Spider benchmark?

In the SQL-of-Thought framework, the authors report that “a single run on the Spider benchmark takes about 42.58$ for Claude Opus 3, and 44.2$ for GPT models” (SQL-of-Thought).

How does the size of the Spider development split compare to the BIRD development set?

Spider’s development split has 1,034 instances across 200 databases, with 8,659 training instances (Spider benchmark evaluation); BIRD’s official development set has 1,534 samples (ErrorLLM).

What execution accuracy did SQLForge-LM report on Spider and BIRD development sets?

SQLForge-LM reports, as study-reported figures, execution accuracy of 85.7% on Spider Dev and 59.8% on BIRD Dev (SQLForge), which its authors describe as narrowing the gap with closed-source methods.