arXiv 2605.12484 1 proposes training LLMs with two distinct update pathways: a fast-learning prompt population that ingests new tasks immediately, and a slow-learning parametric pathway that consolidates retained knowledge. A routing rule decides which pathway absorbs each example. On Qwen3-8B, the authors report up to 3× RL-only sample efficiency with KL divergence roughly 70% lower 1.
The Forgetting Problem in Production LLMs
Fine-tune a model on a new domain and watch its prior capabilities erode. This is catastrophic forgetting, the reason production teams currently choose between LoRA stacks, RAG overlays, and full retraining runs that cost millions. The choice matters because there is no clean way to add knowledge without displacing what is already there. Retrieval-augmented generation sidesteps the problem by keeping the base model frozen, but it adds inference latency and fails when the needed knowledge is procedural rather than declarative. Low-rank adapters are cheaper, yet stacking them creates interaction effects that compound as the stack grows. Full fine-tuning is the most direct route, and the most destructive to everything the model already knew.
Continual learning has been an open problem since the foundation-model era began. Teams running model-versioning pipelines need retention guarantees as much as they need new-task accuracy. What they do not have is a training recipe that treats plasticity and stability as jointly optimizable objectives rather than a trade-off to manage.
What FST Actually Does: Fast vs. Slow Weights
The authors call their method Fast-Slow Training (FST). It is built on two premises. First, that prompt populations can serve as fast weights: soft prompts optimized per-task that change quickly without touching the underlying parameters. Second, that the base model’s weights can serve as slow weights: updated through RL, but constrained to stay close to the initialization.
The fast pathway uses GEPA, a prompt-optimization component that costs roughly $10 per run 2. The slow pathway uses CISPO, a policy-gradient method. The routing rule is the operational heart of the system: for each training example, FST decides whether to update the prompt population, the base parameters, or both. Separating the two channels means new tasks can be absorbed without overwriting consolidated knowledge, and consolidated knowledge can be reinforced without freezing the model’s ability to learn the next thing.
This is not RAG. The prompts are learned, not retrieved. It is not adapter stacking, because the slow weights are the actual model parameters, not low-rank matrices bolted onto them. It is a fourth option, and its feasibility depends on whether the gains hold outside the Qwen3-8B 1 experiments the authors report.
The Numbers: Sample Efficiency, Ceiling, and KL Divergence
The paper benchmarks FST against RL-only baselines on CodeIO, HoVer-hard, and Math (Polaris). The headline figures are task-dependent. On CodeIO and HoVer-hard, FST is up to 3.0× more sample-efficient, measured by steps to reach the RL baseline’s running peak. On Math, the gain is 1.4× 3.
Sample efficiency is only half the claim. At matched reward, FST-trained models remain up to 70% lower 1 in KL divergence from the base policy. That matters for production teams who have seen fine-tuned models drift into unpredictable output distributions. Lower KL means the adapted model stays closer to the behavior of the base it replaced, which reduces the surface area for regression testing.
The experiments run on Qwen3-8B 1 (the SFT variant for Polaris) across 8× H100 nodes, costing 25, 40 GPU-hours per run 1. Those numbers are preprint-grade and specific to this model and hardware stack.
Continual Learning Results: Where RL Stalls and FST Doesn’t
The authors test a three-stage continual-learning protocol: HoVer, then CodeIO, then Physics, 200 steps each 1. On the CodeIO stage, FST reaches 37.7% 1 while RL-only manages 20.7% 1. That is roughly an 8× difference in within-stage acquisition rate 3.
More telling is the plasticity probe. After training on Math, the authors switch to HoVer-hard. Models initialized from RL-only training collapse HoVer-hard learnability to near-zero within 40 steps and stay flat. FST-init models perform close to the base-init reference. The RL pathway has lost the ability to learn something new; the FST pathway has not.
This is the result that should interest teams running sequential training jobs. If a model that has been through one adaptation cycle becomes resistant to the next, the operational cost is not just the next training run. It is the full retraining run you did not plan for.
Practical Implications for Model Versioning Pipelines
For teams currently choosing between LoRA stacks, RAG, and full fine-tuning, FST introduces a fourth design point: jointly optimize prompt populations as fast weights while keeping parametric slow weights closer to the base model. The implication is that you could run sequential adaptations with less catastrophic forgetting and preserved plasticity for the next task.
Whether this fits a given pipeline depends on assumptions that are not yet tested. All results are on Qwen3-8B 1 with CISPO+GEPA. Transfer to other models or RL algorithms is unproven. The routing rule itself is a new hyperparameter surface. Teams will need to validate that their task distribution decomposes cleanly into fast-weight and slow-weight updates.
Limitations and Open Questions
The speedup varies by task: 1.4× on Math versus roughly 3× on CodeIO and HoVer 1. That variance suggests FST’s advantage depends on how well a task maps to prompt-based fast learning. Tasks that resist prompt optimization may see smaller gains or none at all.
The blog post is authored by the research team, so treat performance claims as preprint-grade until independent replication. The paper was posted 2026-05-12 and revised 2026-05-14, which means the community has had less than a week to reproduce the results.
A deeper question is whether the routing rule generalizes. The authors demonstrate it on a specific trio of tasks. Production pipelines see noisier data, overlapping concepts, and tasks that arrive without clean boundaries. FST will need to show it can handle a stream rather than a sequence of staged benchmarks before teams should restructure their versioning workflows around it.
Frequently Asked Questions
Does the author lineup signal any path to production tooling?
Matei Zaharia—Databricks co-founder and Apache Spark creator—is a co-author, alongside Inderjit Dhillon of UT Austin. That affiliation makes Databricks’ hosted fine-tuning platform the most plausible integration vector, though no product roadmap has been announced. Rishabh Tiwari leads the implementation work.
How does FST’s anti-forgetting mechanism differ from sparse-adapter approaches like JumpLoRA?
JumpLoRA projects updates into low-rank subspaces to limit task interference. FST avoids interference structurally: the fast prompt population (GEPA) and slow parametric weights (CISPO) run separate optimization algorithms, so a routing decision—not a rank constraint—governs what gets overwritten. The tradeoff is that FST introduces a learned routing rule as a new failure mode, whereas sparse adapters have no routing step.
Does the 3× sample-efficiency gain translate to wall-time savings?
Not directly. Each FST step runs both prompt optimization and a policy-gradient update, so per-step wall-time exceeds RL-only unless rollout reuse is implemented. The 25–40 GPU-hour figures reflect total compute on 8× H100 nodes, not clock time. Teams whose bottleneck is wall-time rather than sample budget will see smaller practical gains until that reuse optimization is added.
How would FST behave with overlapping or interleaved task streams?
The published protocol uses staged benchmarks with 200 discrete steps per task and clean stage boundaries. Production pipelines see gradual distribution shifts, overlapping concepts, and no stage labels. The routing rule must then classify each example on the fly, introducing a classification-error surface the paper never tests. This is the largest gap between the experimental setup and operational deployment.
What should independent replication efforts test first?
Two open questions matter most: whether FST transfers beyond Qwen3-8B and the CISPO+GEPA pairing to other model families (e.g., Llama, Gemma) and RL algorithms (e.g., GRPO, PPO), and whether the 70% KL-divergence reduction holds when the slow-weight learning rate is increased to match aggressive production fine-tuning schedules. As of 2026-05-17, no independent reproduction results exist.