RL-Found CUDA Kernels Beat cuBLAS: Kernel Tuning Shifts to Reward Design

Reinforcement learning can discover CUDA kernels that beat NVIDIA’s own libraries at matrix multiplication, according to the CUDA-L2 paper revised on arXiv four days ago: +19.2% over cuBLAS in offline benchmarks, +26.0% in a simulated serving mode, and +11.4% to +15.9% over NVIDIA’s built-in autotuner. Every figure is self-reported and independently unreproduced. The durable story is what the method does to the market for kernel-tuning skill.

What did CUDA-L2 actually do?

CUDA-L2 converts half-precision general matrix multiplication (HGEMM) kernel optimization into a reinforcement learning problem: a large language model guides the search, measured CUDA execution speed serves as the reward, and the system works through roughly 1,000 kernel configurations, per the paper’s abstract.

HGEMM is the right target if you want maximum attention. Matrix multiplication dominates LLM training and inference compute, so even single-digit gains on it move real money. It is also the most-profiled kernel in existence, which cuts both ways.

The mechanics are worth understanding because they explain why any headroom exists at all. A CUDA GPU splits work into thread blocks, each block resident on a single streaming multiprocessor with access to high-speed shared memory, and modern parts run 80-plus SMs managing thousands of threads concurrently, according to this CUDA programming overview. A GEMM kernel’s speed is decided by scheduling choices made at that level: how tiles of the matrices partition across SMs, how data stages through shared memory, which operand layouts get used. The paper’s headline comparisons are taken at CUDA-L2’s optimal layout configuration, NN and TN, the standard GEMM transpose flags.

The combinatorics of those choices defeat manual search. Human kernel engineers prune the space with experience and habit. Vendor libraries prune it with heuristics, because one library call has to behave sanely across every shape a customer might send. The authors’ claim is blunt: the abstract states that “even the most performance-critical, heavily-optimized kernels like HGEMM can be improved through LLM-guided RL automation,” arguing that a learned search can explore configuration spaces at volumes impractical for humans. In effect, the model proposes and the GPU scores.

The scale gap is the mechanism. cuBLASLt-AutoTuning, the strongest baseline in the study, already searches up to 100 candidate algorithms per shape. CUDA-L2 searched on the order of 1,000 configurations. Ten times the search width plus a proposer that improves with measurement is essentially the whole trick. There is no new hardware insight in the loop, just more of the space actually explored.

How much faster is it, and against which baseline?

Reported speedups span +11.4% to +28.7%, and which number applies depends entirely on which baseline you accept.

All figures come from the CUDA-L2 abstract. Offline mode executes kernels consecutively; server mode executes them at random intervals intended to simulate real-time inference.

Read the table top to bottom and it forms a gradient of baseline strength. Default cuBLAS is a library call with no search. cuBLASLt-heuristic applies a heuristic pick among algorithms. cuBLASLt-AutoTuning actually measures up to 100 candidates and takes the fastest. The +11.4% offline figure is therefore the honest bar: it is what a 1,000-configuration learned search buys over NVIDIA’s own 100-candidate search, and it is the number a skeptical reader should carry around.

The same numbers yield a useful piece of derived arithmetic. Dividing the reported ratios (1.192 by 1.114) implies cuBLASLt-AutoTuning alone is worth roughly 7% over default cuBLAS offline, and about 8.7% in server mode. NVIDIA’s built-in autotuner, in other words, already captures about a third of the total edge CUDA-L2 reports, at zero engineering cost to the user. The RL system’s contribution is the remaining two-thirds, bought with search infrastructure the library does not ship.

The largest figures in the paper are the least anchored. The +22.0% offline and +28.7% server numbers are reported against a baseline the abstract identifies only by hyperlink, so it cannot be named here. Treat those two numbers as directionally interesting and literally unverifiable until the comparison target is identified.

Why does the skill premium shift from hand-tuning to reward design?

If a learned search over configurations can beat the best-maintained vendor library on the most-profiled kernel in existence, the scarce input to kernel performance stops being the engineer who hand-writes inner loops and becomes the engineer who can define the reward, bound the search space, and audit the result.

Kernel optimization was always a search problem. The industry just paid humans to be the search. The old premium sat on a small pool of people who could reason about SM occupancy, shared-memory staging, and scheduling tradeoffs, then write the loop by hand. That skill is genuinely rare, and vendor libraries like cuBLAS owe much of their durability to having accumulated more of that labor than any single customer could justify funding.

Reward design is a different artifact. Deciding what “fast” means sounds trivial and is not: fast for one shape or for a distribution of shapes, measured with consecutive execution or under server-mode intervals, scored on the kernel alone or on the surrounding fused operation. Each choice changes what the search finds. The CUDA-L2 setup made the easy version of every one of those decisions: a single kernel family, execution speed as reward, 1,000 configurations. Real deployments will require harder versions, and someone has to make them correctly.

The moat implication follows. A vendor library’s edge is accumulated hand-tuning. If search, given enough measurements, rediscovers and then exceeds that tuning, the moat narrows to hardware knowledge and distribution. That erosion is conditional: it requires the CUDA-L2 numbers to reproduce and the method to transfer, neither of which is established yet. But the direction of pressure is clear enough that teams should price it into hiring plans now.

The craft does not die; it reprices. Someone still has to understand why a kernel is slow in order to write a reward function that finds a faster one, and someone has to catch a search that has learned to game the measurement harness instead of the hardware. The labor moves up one level of abstraction, the same migration compilers forced on assembly programmers, with the same result: fewer people doing the low-level work, and those people paid more.

Where is NVIDIA already steering kernel developers?

NVIDIA’s own tooling roadmap has been moving developers up the abstraction ladder for years, and RL-discovered kernels extend that trajectory rather than break it.

The rungs, in order. cuBLAS, proprietary: call the library, accept what you get. CUTLASS, NVIDIA’s template library for custom linear algebra algorithms: instantiate templates, control more of the schedule. Most recently, tile-level programming models: CUDA Tile C++ and cuTile Python, both built on the CUDA Tile IR specification, promoted on NVIDIA’s CUDA toolkit page as the supported way to write tile kernels, an explicit abstraction layer between hand-written CUDA and library calls like cuBLAS. CUDA 13.2 extended CUDA Tile support to Ampere and Ada architectures and added closures and recursion to cuTile Python.

Each rung concedes the same premise: hand-written CUDA is the wrong level for most teams. RL search is the next rung logically, not philosophically. Do not write the kernel, do not even instantiate the template; specify the objective and let measurement drive the search. If the CUDA-L2 result holds, the abstraction ladder NVIDIA has been climbing for years ends at a reward function.

Notice what does not erode: the platform moat. Every kernel CUDA-L2 emits is still CUDA, and CUDA works only on NVIDIA GPUs, as the CUDA overview states plainly. The erosion story operates at the library layer, where “our kernels are uniquely fast” was always a weaker lock-in than “your code runs nowhere else.” If anything, learned kernel search strengthens the platform argument while weakening the library argument: the better the search tooling gets at exploiting NVIDIA silicon, the less reason anyone has to leave it.

When should you reach for each option?

For most teams the decision order is unchanged: autotune first, templates second, tile abstractions third, and RL-style search only when a single kernel family is expensive enough to pay for its own search infrastructure.

1. cuBLASLt-AutoTuning. Zero new infrastructure, already searches up to 100 candidates, and the paper’s own ratios imply it is worth roughly 7 to 9% over default cuBLAS on its own. This should be on by default in any stack that touches GEMM. If it is not on in yours, that finding is worth more to you than anything else in this article.
2. CUTLASS. When you need custom algorithms or fusion patterns the library calls do not expose. Templates demand real CUDA knowledge but stay inside vendor-supported territory.
3. CUDA Tile C++ / cuTile Python. NVIDIA’s supported middle layer, now reaching Ampere and Ada per the 13.2 feature text. The right call when you need scheduling control without abandoning vendor backing, and increasingly the level NVIDIA itself wants you to write at.
4. RL-style kernel search. Justified when profiling shows one kernel family dominating your compute bill, shapes are stable enough to amortize per-shape search cost, you can construct an honest measurement harness to serve as reward, and you can independently verify whatever the search returns. The costs are real: search infrastructure, verification burden, no vendor support, and reproducibility risk if the learned artifacts are shape-fragile.
5. Hand-tuning. A residual but nonzero role, mostly relocated into reward design, search-space pruning, and the long tail of irregular kernels nobody has bothered to search yet.

Does this transfer beyond HGEMM?

Nobody knows, the paper does not close the question, and that open question is the most important caveat in the entire result.

HGEMM is the friendliest possible target for learned search. The computation is dense and regular, the shapes are well-behaved, the reward signal is unambiguous, and decades of profiling mean the measurement methodology is mature. If RL search were going to work anywhere first, it would work here. That makes the result easy to overread in both directions. The bull case: even the most-mined seam in GPU computing still had ore, so less-explored kernels should have more. The bear case: the method has demonstrated itself only on the one kernel where every condition favors it, and the remaining two-thirds of the edge may be specific to matmul’s regularity.

The kernels that actually bottleneck real workloads are frequently the other kind: fused operations with shape-dependent behavior, where the schedule for one component constrains the schedule for the next, and where “execution speed of the kernel” stops being a clean reward because the kernel’s behavior couples to its neighbors. The paper does not cover fused or shape-irregular kernels, and its claims should not be extended to them. Whether the approach transfers is a per-kernel-family empirical question, and each family will need its own reward design, which is precisely the labor-market point from earlier: the scarce skill travels with the reward function, not with any single search result.

What should infrastructure teams do now?

Change your process, not your libraries.

First, turn on cuBLASLt-AutoTuning and re-profile before touching anything exotic; the free third of the edge is sitting there. Second, verify that matmul is actually your bottleneck before acting on a matmul result. Matmul’s dominance of LLM compute is a population-level claim, and individual serving stacks regularly bottleneck on kernels nobody writes papers about. Third, treat every CUDA-L2 figure as unverified. No independent reproduction exists as of 2026-07-17, and capacity plans built on self-reported numbers have a poor historical survival rate.

Fourth, if profiling does show one kernel family genuinely dominating your bill, per-shape search is now a budgeted engineering activity rather than a curiosity. The autotuner is the cheap version; learned search is the expensive version with a higher reported ceiling. Fifth, re-price your kernel engineering accordingly. The skills to hire and retain are reward design, search-space engineering, and measurement discipline, the ability to tell a real 11% from a harness artifact. Finally, one speculation, flagged as such: if these results reproduce, the cheapest path for NVIDIA is to widen the learned search inside cuBLASLt, which is already an autotuner. In that scenario the library gap closes from both directions and the durable advantage belongs to whoever can define objectives well, not whoever searches widest.

What are the asterisks?

Every number in this article carries at least one caveat, and several carry structural ones.

• Self-reported throughout. All CUDA-L2 results are author-reported, with no independent reproduction as of 2026-07-17.
• Baseline selection. The named comparisons are against library defaults and heuristics, not against the best-known tuned kernel for each shape, and the +19.2% offline figure is taken at CUDA-L2’s optimal layout configuration (NN and TN), the comparison’s most favorable axis.
• The unnamed baseline. The largest reported gains, +22.0% offline and +28.7% server, are measured against a baseline the abstract identifies only by hyperlink. Those numbers are unanchored until the comparison target is identified.
• The server-mode direction. Gains expanding under less controlled measurement conditions is atypical and deserves scrutiny before the +26.0% figure gets quoted anywhere persuasive.
• Revision timing. The paper’s submission history shows v1 on 2 December 2025, v2 on 12 December 2025, and v3 on 13 July 2026, with the PDF nearly tripling from 545 KB to 1,494 KB. A revision that large, landing four days before this article, means the current draft contains substantial material the abstract’s numbers may not yet reflect. Revisions that triple a paper’s size usually mean the reviewers had opinions.
• Vendor documentation in motion. NVIDIA’s toolkit page currently mixes CUDA 13.2 feature text with 13.3 links, so version-specific claims about tile programming support are approximate.

Even if independent reproduction shaves the reported gains, the conceptual result stands on its own: kernel optimization is now visibly a search problem, and the scarce input is the reward function. Vendor libraries will respond by searching wider, practitioners should respond by autotuning first, and the engineers who can define what “fast” means, measurably and honestly, just got more expensive.

Frequently Asked Questions

Does RL search work for fused kernels or irregular operations?

The CUDA-L2 paper does not cover fused or shape-dependent kernels. HGEMM is the friendliest possible target: dense computation, regular shapes, clean reward signal. The approach has demonstrated itself only where every condition favors it. Whether it transfers to kernels with coupled components or shape-dependent behavior is a per-kernel-family empirical question the paper leaves open.

What infrastructure is required to run RL-style kernel search?

You need search orchestration around the RL loop, a reproducible test rig for CUDA execution timing, enough GPU hours to explore roughly 1,000 configurations per kernel family, and verification capacity to audit learned artifacts. The paper’s 10x search width over cuBLASLt-AutoTuning requires infrastructure NVIDIA’s library does not ship. Teams also need expertise to design reward functions that reflect production workloads rather than synthetic consecutive-execution benchmarks.

How do I distinguish real speedups from measurement artifacts?

Reproduce with your own shapes and hardware. The server-mode gains expanding under less controlled conditions is atypical and warrants suspicion. Check whether the learned kernel uses shape-specific tricks that fail on different matrix sizes. Verify against cuBLASLt-AutoTuning rather than default cuBLAS, since the free autotuner already captures 7-9% of the total edge on the paper’s own ratios. Any result that cannot survive re-profiling on your actual serving distribution should not enter a capacity plan.

When would CUTLASS or CUDA Tile be better than RL search?

CUTLASS and CUDA Tile remain vendor-supported, while RL search carries no support guarantee and reproduction risk. Use CUTLASS when you need custom algorithms or fusion patterns library calls do not expose. Use CUDA Tile when you need scheduling control without abandoning vendor backing. RL search is justified only when profiling shows one kernel family genuinely dominating your compute bill and shapes are stable enough to amortize the per-shape search cost. NVIDIA can update CUTLASS and Tile; a broken RL search is your problem.

What happens if the learned kernels fail on new GPU architectures?

RL-discovered kernels are CUDA code, but the learned schedules and tiling choices may be architecture-specific. A configuration optimal for one generation could underperform on the next. Unlike cuBLAS, which NVIDIA validates and updates across architectures, RL artifacts require per-architecture re-verification. The search infrastructure and reward function must persist as ongoing engineering costs, not one-time investments. This is where NVIDIA’s advantage consolidates: if learned search works, the cheapest path is widening the search inside cuBLASLt itself.