On 2026-08-11, a preprint called SKILLER proposed moving agent skills out of prompt files and into model weights, claiming 4.3 to 20.4-point absolute gains on Qwen3.5-9B. The practical answer for teams running skill libraries: keep most skills as versioned text, and move only narrow, high-frequency, executor-stable skills into weights when you control the model and can afford the training loop.
Why did skill storage become a decision?
Skill storage became a decision because the incumbent approach has a known structural cost and, as of August 2026, the alternative finally has benchmark numbers attached to it. The incumbent approach stores the skill as text, and for that text to shape behavior it has to be in the context window, where every token is billed on every call. When the context fills up, skills fall out. The cost is per-call, recurring, and proportional to how much procedural knowledge you have accumulated.
SKILLER (v1 submitted 2026-08-11, v2 on 2026-08-14, 15 pages, 8 tables, submitter Chenhao Dang) frames skills as a standardized packaging of procedural knowledge that constrains a model’s behavior space, then asks why that packaging has to ride in the context window at all. Its mechanism is language-level reinforcement learning: a strong model acts as both actor and critic, the small-model agent system is treated as the environment, and every RL signal propagates as natural language. The output is a small model whose skills live in its weights, where they cost zero tokens per call and cannot be evicted by a long conversation.
The cost pressure behind the paper is explicit. The authors argue that running skill-constrained agents through popular harnesses such as Codex and OpenClaw on strong closed-source models is prohibitively expensive, while open-source models deployable on consumer-grade GPUs cut that cost drastically. The closed-source ledger they are pointing at is real: Claude Pro runs $17 per month billed annually ($20 monthly) and includes Claude Code, while Max starts at $100 per month with 5 to 20 times the usage. One seat is fine. A fleet of skill-driven executors billing per token is a different conversation, and that is the conversation SKILLER wants to have.
Meanwhile, the context-loaded camp is not standing still. Skills are a first-class primitive on Anthropic’s platform, listed in the navigation alongside Claude Code, Claude Code for Enterprise, and Claude Cowork. The vendor with the most to lose from weight-stored skills is investing in the text-file version. Both sides are spending money. That is what makes it a decision rather than a default.
What does each storage location actually cost?
There are three places a skill can live, and each one moves cost to a different point in the lifecycle: prompt files pay per invocation, retrieval pays per invocation plus index maintenance, and weights pay once at training time and never again per call.
Prompt files put the full skill text in context on every call. The per-call cost is the skill’s token count, every time, whether or not the skill turns out to be relevant to the task. The upside is total transparency: the skill is a file, it diffs, it reviews, it reverts, and it ports to any model that reads text.
Retrieval stores skills outside the context and injects the top-k relevant fragments. The per-call cost drops from “the whole library” to “the retrieved slice,” but never reaches zero, and you inherit the retrieval failure modes: the wrong skill surfaces, or the right one does not. The retrieval leg has fresh evidence, and it is humbling. In a matched MemoryArena comparison, a structured memory backend (MemoryLake) averaged 20.5% suite-level success against 13.6% for the best comparator across Mem0, a text-embedding-3-small vector RAG setup, and a long-context control. But the long-context control was the only system to register any web-shopping success at all (1 of 150), every system scored zero on travel planning, and the authors flag modest sample sizes, overlapping confidence intervals, and no paired significance tests. MemoryLake led on three of the five domains and ranked third among the four systems on the other two. No storage backend dominated across workloads.
Weights move the skill into the model itself. Per-call cost is zero tokens. The cost arrives earlier, as training infrastructure: SKILLER needs a strong model to generate and critique trajectories, and it needs environments to run those trajectories in. That second requirement is easy to underestimate. PhoneWorld, which builds controllable training environments from real GUI trajectories (34 apps, 16 domains, executable tasks with verifiers), shows what the infrastructure line item looks like: replacing just 10K auxiliary AndroidWorld steps with PhoneWorld supervision raised all four evaluation benchmarks at once, by +14.7 on AndroidWorld and +52.5 points on PhoneWorld itself. Environment quality, not trajectory volume, was the binding constraint. If you cannot build or rent verifiable environments for your domain, the weights route is closed to you regardless of its per-call economics.
How does the token math actually work out?
The crossover question reduces to one comparison: context-loaded skills cost s × f tokens forever, while weight-stored skills cost a one-time training bill T amortized over the model’s lifetime, where s is the skill’s token footprint, f is invocation frequency, and T includes the critic model, the trajectory environments, and the eval runs.
For a prompt-file skill, cumulative spend grows linearly: s × f × days. A 2,000-token skill invoked 500 times a day is a million tokens a day of skill overhead alone (illustrative numbers, since no published measurement exists). The skill ships on day one with zero upfront cost, which is why everyone starts here.
For retrieval, replace s with k, the retrieved fragment size, where k is usually much smaller than the full library but rarely zero, and add index upkeep plus the failure cost of missed retrievals, which the MemoryArena numbers suggest is not hypothetical.
For weights, the per-call term disappears and the expression becomes T / N, where N is total invocations over the model’s useful life. T is large and front-loaded: a strong-model critic in the loop, environment infrastructure of the PhoneWorld kind, and evaluation passes. The weights route wins when N is enormous and the executor is fixed, because T / N falls toward zero while s × f does not. It loses when N is small, when the skill changes weekly (every edit is a retrain, not a commit), or when the executor gets swapped, because T does not transfer.
Two structural facts bend the curves further. First, skill edits: a prompt-file skill is a pull request; a weight-stored skill is a training run. If your skills churn, the per-call savings never catch up with the retraining bill. Second, model upgrades: when the executor model revs, context-loaded skills carry over untouched and weight-stored skills have to be re-extracted. The weights route is a bet that your executor, your skill set, and your call volume all stay stable long enough to amortize T. That is a real bet, and most teams’ skill libraries do not look like it.
What did SKILLER measure, and where do its gains compress?
SKILLER’s headline claims are real but bounded: across five benchmarks on Qwen3.5-9B and Qwen3.5-4B, it outperforms three open-source and one closed-source skill generation or evolution method, and it matches strong closed-source models only on single-skill tasks, not multi-skill libraries.
The absolute gains are 4.3 to 20.4 percentage points for the 9B executor and 1.8 to 13.3 points for the 4B executor. The lower bound nearly halves between the two model sizes. Skills baked into weights appear to compete for capacity with everything else the small model knows, and the smaller the model, the tighter that budget gets. If your deployment target is the cheapest open model that runs on a consumer GPU, the paper’s own numbers say the technique delivers least exactly where its cost argument is strongest.
The closed-source parity claim needs its qualifier attached every time it is repeated: parity on single-skill SkillsBench tasks. A production skill library is a multi-skill system, and the preprint offers no measurement of multi-skill interference in weights, that is, whether skill twelve degrades skill three when both live in the same parameters. Prompt-file libraries have an analogous problem (context crowding), but it is visible: you can count tokens. Interference inside weights is not visible, and nobody has quantified it.
Two adjacent results bracket the claim. On the supportive side, ART tunes vision-language-action models with a tool-injection framework on a 30K-trajectory dataset described as much smaller than baselines, achieving a 20% higher success rate than mainstream baselines and landing in CVPR Findings 2026. Weight-level capability injection can be data-efficient. But ART is robotics, not agent harnesses; treat it as evidence the mechanism generalizes, not that the benchmark numbers do. On the cautionary side, a study of roughly 9,000 trajectories from six language-model agents finds that cross-task and within-task behavioral consistency are distinct axes, consistency is not reducible to success rate, and a frontier-versus-open-source consistency gap persists even under a task-difficulty control. SKILLER’s economics assume an open-weight executor can stand in for a closed model. Success-rate parity on single-skill tasks does not establish behavioral parity, and the consistency literature says the gap is exactly where the assumption gets shaky.
What leaves code review when skills move into weights?
When a skill moves into weights, its provenance shifts from a readable, diffable file to training-run logs, and that is the least reversible cost in the whole comparison.
A markdown skill library is an audited artifact. Every behavioral constraint is a line of text with a git blame. A reviewer can ask “when did we start telling the agent to skip the migration check?” and get a commit hash. A weight-stored skill answers that question with a training run ID, a critic-model version, and a trajectory dataset. The behavior is in the parameters; the explanation is in a log file, if the log file survived.
The value of keeping behavioral criteria as text is not hypothetical. RubricForge, which optimizes judging rubrics for agent evaluation, shows the mechanism from the evaluator’s side: because its rubric is human-readable text, every verdict is attributable to named criteria. On the same setup it cut the false-pass rate to 0.115 versus 0.173 for a G-Eval-style judge on tau-bench (173 labeled trajectories from 220 rollouts, one frozen 7B model as both agent and judge), a reduction the paper attributes to inducing the rubric from ground-truth-labeled trajectories, not to readability. The same paper is honest about the limits: its raw-agreement edge was not statistically significant (McNemar p = 0.248). Readable artifacts are not a magic accuracy guarantee. They are what make failures debuggable after the fact.
Scale that from a judge’s rubric to an agent’s operating procedure. A team running regulated or customer-facing agents has to answer “why did the agent do that?” with something better than “the RL loop converged that way.” Context-loaded skills answer it natively. Weight-stored skills answer it forensically, at best. If your audit requirement is real, this axis alone can settle the routing decision regardless of the token math.
Which skills belong in files, retrieval, or weights?
Route each skill on four axes: call frequency, executor stability, audit requirement, and portability need. High-frequency, stable-executor, low-audit skills are weight candidates; everything else defaults to text.
| Axis | Prompt files | Retrieval | Weights |
|---|---|---|---|
| Per-call token cost | Full skill text, every call | Retrieved fragment only | Zero |
| Upfront cost | None | Index build and maintenance | Strong-model critic, environments, eval runs |
| Edit cost | One commit | Re-index | Retrain |
| Model upgrade | Carries over untouched | Carries over untouched | Re-extract from scratch |
| Cross-model portability | Any model that reads text | Any harness with retrieval | Locked to one checkpoint |
| Code review | Full diff visibility | Fragment plus index config | Training-run logs |
| Known failure mode | Context crowding, eviction at limit | Retrieval misses (structured backend ranked third on two of five domains in one study) | Multi-skill interference, unquantified |
The portability row deserves emphasis because multi-model harnesses are already production reality, not a hypothetical. Amazon Q runs multiple foundation models on Bedrock and applies logic to route each task to the best-fit model. In that architecture, a prompt-file skill travels with the task to whichever model wins the route. A weight-stored skill cannot follow; it exists in exactly one checkpoint, and the router’s best-fit model is, by definition, sometimes a different one. Every model added to your routing pool is a model your weight-stored skills do not exist in.
Applied to a real library, the routing looks like this. Always-on constraints (house style, security invariants, output format rules) that must port and must be reviewed stay as prompt files. The large, rarely-touched long tail (runbooks, per-vendor API quirks, historical incident procedures) belongs behind retrieval, accepting the miss rate the MemoryArena results document. The narrow, hot, stable skills (a specific test-fixture setup invoked hundreds of times a day against one pinned executor) are the only honest candidates for weights, and only when you control the model, can build verifiable training environments, and can afford a strong-model critic in the loop.
What would change the verdict?
The verdict is a routing default, not a permanent one: keep skills as versioned text (prompt files for always-on constraints, retrieval for the large rarely-used tail) whenever they must port across executors, survive model upgrades, or be reviewed as diffs, and move narrow, high-frequency, executor-stable skills into weights only when you control the model and can afford the language-level RL loop, accepting that provenance moves from a readable file to training-run logs.
Three measurements would sharpen that from structural reasoning to engineering. First, an honest accounting of per-invocation token overhead for context-loaded skill systems in production; right now the “context tax” is derived, not metered, and the crossover point cannot be computed without it. Second, a multi-skill interference study for weight-stored skills: SKILLER’s closed-source parity is single-skill only, and a library is precisely the case it does not measure. Third, transfer results to non-Qwen executors, since every current number sits on two Qwen3.5 checkpoints.
The strongest limitation is worth stating flatly: SKILLER is a non-peer-reviewed preprint with author-reported numbers on two checkpoints and five benchmarks, and no fetched evidence quantifies the context-side costs, the multi-skill weight behavior, or executor transfer. The routing framework above survives those gaps because it rests on structure: who pays per call, who pays per edit, who survives a model swap, and who can show a diff to an auditor. Those four questions have answers independent of any benchmark. The benchmark numbers tell you the weights option is no longer hypothetical. They do not yet tell you where the crossover is, and anyone claiming a specific crossover point today is extrapolating past the evidence.
Frequently Asked Questions
How can a team measure the real per-call token cost of its skill library?
Run an identical task set twice, once with skills loaded and once stripped, and diff the input token counts per call, then multiply by daily invocation volume. No published measurement covers Claude Skills, Cursor rules, or Superpowers-style libraries, so a self-run diff is the only way to turn the crossover math from derived into metered.
What happens to baked skills when the pinned executor checkpoint is retired?
The training spend becomes a sunk cost and the skill has to be re-extracted on the new checkpoint. Keep the canonical text of every baked skill in version control even after baking: model parameters are not a readable backup, and without the source text the next extraction run has no seed to work from.
How can you test for multi-skill interference before moving a library into weights?
Bake one skill at a time and re-run the eval suite for every previously baked skill after each extraction; a score drop on an older skill is the interference signal. SKILLER’s closed-source parity was measured on single-skill tasks only, so until someone publishes a multi-skill study, this regression pass is the only check available.
Is putting skills into weights just fine-tuning on skill demonstrations?
No. SKILLER runs a strong model as both actor and critic, treats the small-model agent system as the environment, and propagates every reinforcement signal as natural language; supervised fine-tuning on skill transcripts has neither critic nor environment feedback. Its 4.3 to 20.4-point range came against skill generation and evolution methods, not against a plain SFT baseline.
Does baking skills into weights remove the closed-model bill?
The closed-model dependency stays; only the payment schedule changes. The extraction loop needs a strong closed model as actor and critic, so closed-model tokens are spent on the initial run and again on every retrain, while the per-invocation spend disappears at runtime. Teams with no closed-model API budget cannot run the loop at all.
