NVIDIA's Free Agent Model on Vercel: The Prompt Retention Catch

On 2026-10-14, Vercel announced that NVIDIA’s Nemotron 3.5 Lightning is available free on AI Gateway as nvidia/nemotron-3.5-lightning-free, wired into Claude Code, Codex, and Cursor with a single CLI command once the latest Vercel CLI is installed. Buried in that same changelog is one line that changes what “free” means: “Note: The provider may retain prompts for model improvement.” That note gives no scope, no duration, and no opt-out.

The practical consequence is straightforward: the economics of this endpoint shift the cost of agent inference from dollars to data exposure. For some workloads that trade is fine. For prompts that carry secrets, customer data, or unreleased proprietary code, it is not, and “free” stops being neutral the moment your coding agent’s context window fills with material you would not email to a stranger. The right place to enforce the split is not a policy memo; it is the gateway routing rule, which is where the decision actually executes.

What shipped, and what the note actually says

Per the changelog, Nemotron 3.5 Lightning is “an open-weight reasoning and chat model for long-running agents, sub-agent work, chatbots, and RAG systems,” served by NVIDIA through Vercel’s AI Gateway. The vendor-stated specifications: 30B parameters with 3B active per token in a hybrid Mixture-of-Experts design that interleaves Mamba-2 and MoE layers with a few attention layers, a 1M token context window, and support for tool calling and structured outputs. All of that is changelog text, with no independent benchmark behind it.

Setup is deliberately low-friction. According to Vercel, users of Claude Code, Codex, and Cursor can select the model after installing the latest Vercel CLI and running npx vercel ai-gateway setup. The model is also callable through the AI SDK and the Chat Completions, Responses, and Messages APIs. On pricing, the changelog states that “AI Gateway reflects provider pricing with no markup and does not charge a platform fee on inference, including on Bring Your Own Key (BYOK) requests,” and the Lightning listing itself is free.

Two clarifications matter before any routing decision. First, “open-weight” does not mean private here. This is a hosted endpoint served by NVIDIA; you are sending prompts to someone else’s infrastructure, not running weights on your own. Second, the retention note is the entire data-governance documentation for this listing. The changelog does not state what “retain” covers (prompts only, or completions and metadata too), how long retention lasts, or whether any opt-out exists. That is not an accusation; it is the current state of the documentation.

What a coding agent prompt actually contains

To judge the retention note, you have to know what would be retained. Coding agent prompts are not chat messages. A recent study of delegation contracts in coding-agent work, arXiv:2606.17099, describes what flows between a supervising process and an agent: “the task T assigned to the agent (objective, scope, non-goals, success criteria); the authority A granted to it (files it may modify, commands it may run, actions that are forbidden); the work package W it returns (artifact plus evidence: changed files, commands, test results, limitations).”

Read that list as a data inventory. Objectives and success criteria describe your roadmap. Files and commands describe your codebase and infrastructure. Test results describe your failure modes. A long-running agent session on a 1M-token-context model concentrates all of it into one provider-visible stream, which is exactly the material a “may retain prompts for model improvement” clause would capture.

The same paper supplies the pressure that makes a free endpoint tempting in the first place. In its author-reported 64-run, blinded-review experiment, explicit delegation contracts “cost +13% agent tokens and +38% wall-clock time,” per arXiv:2606.17099, and the author concluded that on small tasks with capable models, contracts “buy reviewability rather than correctness.” Those figures are single-study and author-reported, but the direction is intuitive: the more disciplined your agent harness, the more tokens you burn, and the more attractive a zero-dollar model listing looks. The retention note is the price list attached to that discount.

It is also worth noting that data exposure through model endpoints is a formally probed risk class in this exact model lineage, not a hypothetical. NVIDIA’s Nemotron-4 340B technical report describes Garak scans covering “training data membership inference […] training data replay […] chat exfiltration” among other categories, “with nominal to good performance.” NVIDIA’s own security process treats prompt and channel leakage as something to test for. Your routing rule should treat it the same way.

A routing checklist for the free endpoint

The changelog does not define retention scope, so the conservative reading is that anything in the prompt stream may be kept and used for model improvement. Under that reading, the deciding question for each workload is: if this prompt stream showed up in a future training set, would I care?

The green rows are where the endpoint earns its keep. The yellow row is genuinely a judgment call, and I would treat the absence of scope documentation as a reason to default to “no” for anything you would be unhappy to see reproduced. The red rows are not close calls.

One row deserves a caveat. The yellow judgment call assumes the only issue is retention. But hosted endpoints carry a second, quieter risk: the thing behind the alias can change. A study of gateway-mediated agent systems, arXiv:2609.31358, puts it plainly: “A configured identifier is not necessarily an immutable weights snapshot: only the OpenAI identifier here contains an explicit snapshot date. No random seed was specified, and hosted services may change behind aliases.” That was written about a different system, but the mechanism is general. Pin your rule to the data class, not to an assumption that today’s nvidia/nemotron-3.5-lightning-free behaves like tomorrow’s.

Where the split should live: the gateway, not the wiki

The reason this decision is operational rather than rhetorical is that Vercel’s gateway already has the enforcement machinery. Per the changelog, “AI Gateway provides a unified API for calling models, tracking usage and cost, and configuring retries, failover, and performance optimizations for higher-than-provider uptime. It includes built-in custom reporting, budgets for API keys, routing rules, and more.”

Routing rules and per-key budgets are the two levers that matter. One caveat before the pattern: the changelog confirms that routing rules, budgets for API keys, and custom reporting exist, but it does not document that routing rules bind to a specific key or can include and exclude individual model listings. Verify that granularity in the AI Gateway documentation, or in the coding agents guide the changelog links, before relying on it. If routing rules can be scoped per key, the pattern is:

1. Issue separate API keys by sensitivity class. One key whose routing rules permit only the free NVIDIA listing, used for public-code and synthetic work. A separate key for production agent work that excludes it entirely.
2. Set budgets on the paid key, and watch them. A cap by itself is what creates a ran-out-of-allowance moment; paired with custom reporting, the budget becomes spend visibility, so the sensitive-work key gets topped up before anyone reroutes to the free endpoint mid-sprint.
3. Use custom reporting to audit the split. If prompts classified as sensitive start appearing against the free key, that is a routing bug to fix, not a training issue to memo.

The gateway is also where the no-execution and data boundaries for agent tools get enforced in the architecture described in arXiv:2609.31358, which models the gateway as the layer that constrains what an agent can do. The same logic applies to what an agent may expose. If your sensitivity rule lives in a document, compliance depends on every engineer remembering it at the moment they pick a model from a dropdown. If it lives in the routing configuration, compliance is the default.

One honest gap: the changelog documents routing rules and budgets as gateway features, but it does not document per-listing data-retention policies or Zero Data Retention options for any model on the gateway, paid or free. The angle of routing to “listings with Zero Data Retention support” is the right instinct, but the changelog does not say which listings, if any, carry that guarantee. Before you route sensitive work to any paid alternative, verify its retention terms directly with the provider rather than assuming a paid price implies a stronger privacy posture.

What the Nemotron lineage corroborates, and what it does not

The capability claims deserve separate scrutiny, because they are doing persuasive work in the changelog. NVIDIA’s own Nemotron 3 technical report introduces “the Nemotron 3 family of models—Nano, Super, and Ultra,” built on “a Mixture-of-Experts hybrid Mamba–Transformer architecture” with “context lengths of up to 1M tokens,” and commits to openly releasing model weights. That corroborates the architecture class and the long-context ambition. It also corroborates the open-weights framing.

What it does not corroborate is the name. The Nemotron 3 report names no “Nemotron 3.5 Lightning.” The closest independent citation of a 30B/3B-active model in this lineage comes from a separate NVIDIA-authored paper, arXiv:2606.26493, which describes a system “Built on Nemotron-3-Nano-30B-A3B—an open-weight 30B hybrid Mamba-Transformer MoE model.” A 30B hybrid MoE in the family clearly exists. Whether “Nemotron 3.5 Lightning” is that model under a serving name, a new variant, or something else is not established by the changelog or by the Nemotron 3 report. The Nemotron 3 report also states, as a vendor-reported result, that Nemotron 3 Nano’s base model shows improved predictions up to 1M tokens on code data, which lends plausibility to the long-context coding claims without verifying them for this listing.

There is one more lineage detail with practical value. Nemotron Nano 2’s report documents inference-time thinking-token budget control, letting users cap how many reasoning tokens the model generates before answering. If Lightning inherits anything like that knob, it would matter for cost-shaping on agent workloads, but inheritance is inference on my part; the changelog does not mention it.

So the verification ledger looks like this: architecture family, corroborated in spirit by NVIDIA’s own papers; the 30B/3B-active configuration, corroborated under a different model name; the 1M context, tool calling, structured outputs, and the “3.5 Lightning” name itself come from the vendor changelog alone. Treat the spec sheet as a hypothesis to test against your own workload before you route anything to it for its capabilities rather than its price.

What the changelog does not establish

Since the entire news premise rests on one vendor document, it is worth listing its silences plainly:

• Quality. No benchmark, no eval, no comparison against the models your agents currently use. Free and bad is expensive in agent loops.
• Rate limits and throughput. The changelog documents no quotas, concurrency limits, or behavior under load. A free tier that throttles a long-running agent mid-task can cost more wall-clock than it saves in tokens.
• Retention scope. Prompts only, or completions and metadata too? How long? Which teams or vendors touch the data? Any opt-out? The note answers none of these.
• Duration of “free.” The changelog announces availability, not a pricing commitment. Hosted listings change; plan as if the economics can move.
• Other listings’ retention posture. The changelog documents no Zero Data Retention support for any gateway listing, so the paid alternative’s terms need their own verification.

None of this makes the listing unusable. It makes it a listing you adopt with a routing rule and a verification step, rather than a default you inherit because setup took one command.

The decision

Route low-sensitivity agent workloads (public-code exploration, synthetic data, prototypes, learning experiments) to nvidia/nemotron-3.5-lightning-free and take the savings. Keep prompts containing secrets, customer data, or unreleased proprietary code on endpoints whose retention terms you have actually read, until NVIDIA or Vercel documents what “may retain prompts for model improvement” covers. Encode the split as an AI Gateway routing rule with separate keys and budgets, once you have verified that routing rules can be scoped per key, because the gateway is where the decision executes and a policy memo is where it goes to be forgotten.

I would also run one bounded verification step before widening the green list: point a synthetic but representative agent task at the endpoint, check output quality and latency against your current model, and only then decide whether the vendor’s 1M-context, tool-calling claims hold for your workload. That is a bounded check, not an assurance: the changelog does not establish that any specific sensitive workload is safe on this endpoint, or that the retention clause is narrower than its plain wording. The split above is a precaution grounded in an unscoped disclosure, not a quantified risk assessment, because the numbers needed for one do not yet exist.

The durable lesson outlasts this listing. Every free or discounted hosted endpoint makes the same trade in some form, and the retention note is the tell. Reading it belongs in the routing review, next to price and rate limits, not in legal boilerplate nobody opens.