Can Text-to-SQL Agents Enforce Row-Level Security? A New RBAC Benchmark

A text-to-SQL agent can return perfectly valid SQL that the asking user is not allowed to run, and a new arXiv benchmark now measures exactly that gap. The practical answer to the title’s question: not on its own. Models demonstrably generate queries referencing columns a user’s role forbids, so authorization has to stay in the warehouse, through per-user credentials or a user-scoped service account, rather than in the prompt.

What the new benchmark actually measures

The paper, Benchmarking Text-to-SQL under Role-Based Access Control (arXiv:2607.22115, accessed 2026-09-24), introduces what the authors describe as an automated framework for synthesizing realistic roles and fine-grained RBAC policies on top of arbitrary text-to-SQL benchmarks. Rather than building a new dataset from scratch, the framework layers access control onto existing tasks and evaluates compliance on LiveSQLBench with full CRUD workloads, meaning inserts, updates and deletes sit alongside the reads that dominate most text-to-SQL evaluation.

The metric matters more than the construction. The authors are explicit: “Our benchmark focuses on explicit RBAC compliance, i.e., whether generated SQL references only authorized resources.” That is a different axis from execution accuracy, the number most text-to-SQL leaderboards report. A model can score well on accuracy and still produce a query that answers the user’s natural-language question by reading a column that user’s role cannot see. The benchmark counts the second failure directly.

This is a correction to how the field has evaluated natural-language database access, where enterprise features like user-aware filtering have been discussed but rarely scored as a first-class metric.

The failure mode in numbers

The clearest quantified result concerns where violations concentrate. The paper reports that under the Full-Schema setting, “a large number of violations involve accessing known unauthorized columns,” and that DeepSeek-Coder records an average of 494.6 such cases.

Two things are worth holding onto there. First, these are not edge cases where authorization is ambiguous; the columns are known to be unauthorized, and the model references them anyway. Second, the Full-Schema setting is doing work. When the model sees the entire schema in its context, it reaches for forbidden columns often enough to generate hundreds of violations. That makes schema exposure itself a security variable, not just a context-length or accuracy variable.

The paper tested that variable directly. Its Table 6 comparison on BIRD switches models from Full-Schema to Role-Schema, where only role-permitted schema elements are shown. DeepSeek-Coder’s known-unauthorized accesses fall from 494.6 to an average of 244.8 cases, but violation-wrong cases rise from 436.0 to 560.6, and GPT-5-mini’s violation rate climbs 14.1 percentage points under the filtered schema. The paper’s own summary: restricting schema visibility “reduces explicit data leakage but does not effectively enforce RBAC policies, as models continue to hallucinate and rarely refuse unauthorized queries.” Hiding the schema is harm reduction, not enforcement.

And the pattern is leaderboard-wide, not a single model. Table 3 scores 13 models across Spider, BIRD and LiveSQLBench: on Spider, DeepSeek-V3.2-Reasoning posts a 2.11% violation rate and GPT-5 2.61%, while Snowflake-R1-7b reaches 48.68% despite strong execution accuracy. The paper singles out the same model’s 65.10% violation rate on BIRD, and on LiveSQLBench the worst rate reaches 82.03%. Commercial models generally violate less than open-weight ones, but none is clean: many solutions “with high benchmarking scores under an unrestricted setting suffer sharp performance degradation once access constraints are in place, due to frequent RBAC violations.”

Why valid SQL breaks the warehouse’s access-control assumption

Role-based access control works by attaching permissions to roles rather than individuals: users inherit access rights through their roles, as GeeksforGeeks’ RBAC explainer summarizes the standard model. Wikipedia’s definition calls RBAC a policy-neutral mechanism defined around roles and privileges. The word “policy-neutral” is the load-bearing one: the warehouse does not know your intent, it only knows which role executed the query.

That is precisely the assumption a broad service account breaks. If the agent connects to Snowflake, BigQuery or Postgres as one shared identity with wide grants, then from the warehouse’s perspective every query is authorized, because the warehouse applies the service account’s role, not the asking user’s. Enforcement silently moves from the database’s policy engine to whatever the prompt and schema filter happen to accomplish. The benchmark’s violation counts are, in effect, a measurement of how well that prompt-layer enforcement performs. The DeepSeek-Coder figure suggests it performs poorly under full schema exposure.

This is the mechanism behind a familiar audit problem. When a compliance team asks “who accessed this column,” the warehouse log answers honestly: the service account did. The question of whether the underlying user was entitled to see it has no database-side answer at all.

Three wiring options for an AI analyst

Given that, the deployment decision reduces to where authorization lives.

1. Per-user credentials. The agent executes each query under the asking user’s own warehouse identity. The database applies that user’s role, row and column policies exactly as it would for a human analyst in a BI tool. The model’s job shrinks to writing correct SQL; if it writes non-compliant SQL, the warehouse rejects or filters it. Operationally this is the heaviest option, since it requires identity passthrough from the application into the warehouse connection, but it is the only option where the benchmark’s failure mode is structurally impossible rather than merely discouraged.

2. User-scoped service accounts. A middle path: service accounts exist, but each is constrained to one user’s authorized scope rather than the whole corpus. IBM’s RBAC implementation guidance states the rule directly: to perform retrieval, “the service account should be constrained to act only within a certain user’s authorized scope, not on the entire corpus.” This preserves some operational convenience of service accounts while keeping the warehouse as the enforcement point. It is also the vendor-side counter to the default many teams drift into, one broad account with full table grants.

3. Permission-aware query rewriting. The agent (or a layer in front of it) knows the user’s role and rewrites or filters generated SQL to reference only authorized resources before execution. This is what the benchmark’s Full-Schema finding warns about: if the model sees the whole schema and rewriting is the only defense, the rewriter must catch every unauthorized reference the model emits, and the measured violation counts show how frequently models emit them. Rewriting is a useful belt; the evidence does not support it as the only suspender.

The last column is our read of the audit posture, and it follows from IBM’s logging guidance: “Logging should record which role (human or agent) exercised which assigned permissions to support investigations into misbehavior or anomalies.” A broad service account collapses every role into one identity, destroying exactly the attribution that guidance asks logging to preserve.

What the benchmark cannot see

The strongest caveat comes from the paper itself: “It does not model inference-based leakage, such as deriving restricted attributes from correlated authorized ones.” An agent can pass this benchmark cleanly, reference only authorized columns, and still leak restricted information by inference. If salary is forbidden but a correlated authorized column approximates it, explicit compliance catches nothing.

So a good compliance score is not a security proof. It is closer to a linter result: useful as a regression signal that your schema exposure or prompt handling has gotten worse, silent about entire classes of leakage. This pattern is familiar from other agent benchmarks, where a passing score can hide how agents actually fail once the evaluation’s fixed assumptions stop matching deployment reality. Read RBAC-compliance scores the same way: as a floor test, not a ceiling.

The verdict

Keep authorization in the warehouse. Run text-to-SQL agents under per-user credentials, or under service accounts constrained to each user’s authorized scope per IBM’s rule, and treat any RBAC-benchmark score as a measure of explicit unauthorized references rather than proof of permission-aware generation or inference-proof safety. If you run the agent on a broad service account, the warehouse never applies the asking user’s role, the prompt becomes your only access control, and the benchmark’s violation counts are a preview of what that control is worth.

One nuance worth keeping: the paper does test the fixes, and their limits sharpen that verdict rather than soften it. Section 6.3 of the paper evaluates supervised fine-tuning and few-shot prompting as remedies. Fine-tuning Llama3-SQLCoder-8B on role-aware Spider data cuts its violation rate from roughly 50% to 5.55% in-domain, but the gain is bought with over-refusal and it does not travel: on BIRD and LiveSQLBench the fine-tuned models stay well below GPT-5’s compliance while refusing far more legitimate queries. Few-shot prompting is worse, producing non-monotonic, model-dependent results that actively regress some models. The paper’s conclusion: “Common mitigation strategies, including schema restriction, prompt-based policy specification, and supervised fine-tuning, do not reliably eliminate these failures.”

Gaps before you ship

Two limitations bound everything above. First, the wiring options are stated at the architecture level; the concrete mechanics of row access policies, row-level security syntax and column masking on Snowflake, BigQuery or Postgres still need checking against each vendor’s documentation before you write DDL. The benchmark’s main release covers column-level and operation-level RBAC, and its authors leave row/cell-level policies out of that release. They do report a row-level feasibility study in the paper’s appendix: five Spider databases extended with row-level predicates, 298 query-role instances, and violation rates from 50.34% (DeepSeek-Coder) down to 2.15% (GPT-5), with the note that “RBAC compliance errors remain nontrivial under row-level predicates.” Treat that as proof the framework extends, not as a full row-level track.

Second, the leaderboard answers the comparative questions but not your deployment question. It ranks which models violate least (DeepSeek-V3.2-Reasoning and GPT-5 on Spider, Claude-Sonnet-4.5 on BIRD) and shows how schema filtering shifts the failure mix, yet no model posts a zero violation rate, so none clears the compliance bar on its own. The paper’s deployment conclusion matches the verdict above: “reliable RBAC is not solved by model selection alone,” and “robust deployment must also combine language reasoning with external, deterministic access-control enforcement.” What is left for you is a test harness pointed at your own schema and roles. The benchmark gives you the right question to ask your agent before production. The wiring decision determines whether a wrong answer is a logged error or a disclosure.

Frequently Asked Questions

Does restricting schema visibility effectively enforce RBAC policies?

The paper’s own summary: restricting schema visibility “reduces explicit data leakage but does not effectively enforce RBAC policies, as models continue to hallucinate and rarely refuse unauthorized queries.” Hiding the schema is harm reduction, not enforcement.

What is the main limitation of the new RBAC benchmark?

The strongest caveat comes from the paper itself: “It does not model inference-based leakage, such as deriving restricted attributes from correlated authorized ones.” An agent can pass this benchmark cleanly, reference only authorized columns, and still leak restricted information by inference. If salary is forbidden but a correlated authorized column approximates it, explicit compliance catches nothing.