Can One Attention Kernel Serve Nvidia and AMD GPUs?

One Triton attention kernel, with the same source code, has been reported to match vendor-tuned state-of-the-art libraries on both an Nvidia A100 and an AMD MI250. That is the strongest evidence yet that mixed-fleet serving teams can stop maintaining two attention stacks. The condition attached to that “yes” matters more than the headline, though: every performance number behind it is author-reported and unreplicated, the same-code parity result is cited from the authors’ earlier work rather than measured fresh, the new preprint’s own evaluation runs on an H100 and an MI250 (end-to-end, an H100 and an MI300) with no competitive AMD vendor library on its side of the comparison, and parity was achieved only after substantial per-system auto-tuning. The practical answer is that a single portable kernel is a credible serving default where that tuning has already been done and validated, with vendor kernels kept warm as fallbacks on the Nvidia side, and it is not yet a default you can assume on GPUs the evidence does not cover.

The stakes are real. Teams serving open-weight models on mixed or non-Nvidia fleets currently pick attention backends per vendor: FlashAttention-2 or FlashAttention-3 on CUDA, ROCm ports or Triton on AMD. Each choice drags its own maintenance line, and performance diverges between them in ways that have to be re-measured after every framework upgrade. If one kernel code base can genuinely hold parity on both vendors, the tuning burden moves from hand-porting kernels per vendor to selecting a backend at the scheduler level, and AMD capacity becomes cheaper to run alongside Nvidia. The newest paper inviting exactly this reading, CHARM+ (arXiv:2611.05906), is days old as of this writing and, as we will see, does not actually claim vendor portability. Sorting out what each piece of evidence covers is the point of this article.

What the Triton evidence actually shows

The direct answer to the headline comes from The Anatomy of a Triton Attention Kernel, a November 2025 preprint that reconstructs what it takes to make a generic Triton attention kernel competitive. Three findings carry the weight.

First, portability is real. Describing prior work, the authors write: “There, we compared a Triton implementation of Flash Attention v2 Dao, 2023 with the flash_attn library and the ROCM flash attention implementation and demonstrated that Triton can achieve comparable performance to the vendor-specific SoTA libraries on both an NVIDIA A100 and an AMD MI250, using the same kernel code.” Same kernel code, two vendors, comparable performance to each vendor’s own tuned library. That is precisely the property a mixed-fleet operator wants. Note the framing, though: this is a result the paper cites from the authors’ own earlier work, not a fresh measurement in this preprint, and it does not name the models or sequence lengths in the quoted passage.

Second, parity was not free. The paper’s own contribution is the work required to close the gap. The authors list “the key algorithmic and system-level improvements,” the parameter auto-tuning they describe as required for efficiency, and “the integrations into a popular inference server that are necessary to bring the performance of a generic Triton attention kernel from 19.7% of the state-of-the-art to 105.9%” (source). That progression was measured against FlashAttention3 on an H100: “the baseline implementation achieved only 19.7% of the performance of FlashAttention3,” and the final static-launch-grid kernel “achieves even 98.6% – 105.9% versus FlashAttention3.” The paper’s evaluation hardware is “the NVIDIA H100-80GB and the AMD MI250-128GB,” with end-to-end serving experiments “on both NVIDIA H100 and AMD MI300 GPUs,” and the tuning was per-platform: the limited autotuning approach “further reduces the kernel latency for short (up to 9.8× on H100) and medium prompts (up to 75%) on both platforms.” Read that range carefully. An off-the-shelf Triton attention kernel started at roughly a fifth of vendor performance. Closing the gap required algorithmic changes, system-level work, parameter auto-tuning, and integration into a serving stack. The portable kernel won, but only after someone paid the tuning cost.

Third, the AMD side of the paper’s own comparison has no vendor library in it. “For AMD GPU there is no competitive paged attention implementation besides ours,” the authors write, so their AMD figures pit Triton variants against each other, and same-code parity against each vendor’s tuned library rests entirely on the earlier A100-and-MI250 work. What the AMD side does have is adoption: the kernels have “been adopted as the default attention kernel for AMD GPUs” in vLLM, and the conclusion states they “are the default attention backend in vLLM for AMD deployments.” On vLLM’s AMD path, the portable kernel is not an experiment; per the authors’ reporting, it is what ships.

So the Triton evidence supports a narrower and more useful claim than “one kernel serves both vendors.” It supports: one kernel code base can serve both vendors, provided each target GPU gets its own tuning pass and serving-stack integration. That distinction drives everything below.

Same source is not the same performance

The sharpest counter-evidence comes from outside LLM serving entirely. PortLBM, a lattice Boltzmann simulation code written in SYCL, achieves the portability ideal: “The same PortLBM source compiles and runs unmodified across all three vendors, though achieving consistently high performance still requires the system-specific tuning we study in this work.” Its authors are blunt about the ceiling: “there is no one-size-fits-all configuration for performance-portable LBM code, as architectural designs (in particular, cache hierarchies and memory bandwidth) vary significantly across GPU vendors.”

This is cross-domain evidence, and I want to be honest about that limit. Lattice Boltzmann kernels are not attention kernels, and what cache hierarchy does to one does not transfer mechanically to the other. But the mechanism generalizes. Attention is memory-bandwidth-bound in exactly the regime where PortLBM’s authors found vendor architectures diverging, and the Triton anatomy paper independently confirms the pattern inside LLM serving: its authors present parameter auto-tuning as a requirement for efficiency, not optional polish. Two unrelated lines of work landing on the same conclusion is weaker than a replicated attention benchmark but stronger than either alone.

The operational consequence: “write once, run anywhere” is achievable for correctness today and for performance only after a per-GPU tuning campaign. Budget for that campaign per new GPU SKU, not per vendor. An MI250 and an MI300X are the same vendor and, per everything above, not the same tuning target.

CHARM+ is communication work, not vendor portability

CHARM+ (arXiv:2611.05906) arrived in November 2026 with “Cross-Hardware Attention” in its title, and it will be cited as evidence that one attention implementation now spans vendors. The paper does not support that reading.

The mechanism is a communication reorganization for multi-GPU attention. The authors describe reorganizing “sequential communications into several levels” to achieve “intra-level parallelism and computation overlap,” and decoupling cross-hardware communication from intra-hardware communication to eliminate synchronization overhead. “Cross-hardware” here means across devices and nodes in a parallel execution, not across GPU vendors. The re-merge multistream design is about overlapping communication with computation when attention is sharded across many GPUs.

The evaluation makes the scope concrete: “To evaluate scalability, we conducted experiments on the H20 system with 2 nodes using 8 GPUs and 2 nodes using 16 GPUs.” H20 is an Nvidia part. No AMD GPU appears anywhere in the evaluation, which spans L20, H20 and H100 systems. Nothing in the paper demonstrates a kernel running portably across vendors, so it cannot anchor a cross-vendor claim regardless.

The CHARM+ numbers worth keeping

Scoped correctly, CHARM+ still matters to the serving decision, just on a different axis. If your bottleneck is multi-node attention rather than per-GPU kernel choice, the paper reports “substantial speedups at larger scales” on the 8- and 16-GPU H20 configurations, attributed to overlapping communication with computation.

The memory cost is the number I would carry into a capacity review. CHARM+ logically requires four more buffers than ring-attention, but the authors report that temporal reuse of GPU memory keeps peak buffer usage roughly unchanged, with observed memory overhead of only 3%~5% over ring-attention (source). That figure comes from runs on “L20 with 8 GPUs across varying sequence lengths,” not from the H20 scalability experiments. If it replicates, the re-merge multistream approach buys communication overlap at a memory price most serving stacks can absorb, since attention buffers are usually small next to model weights and KV cache. If it does not replicate, four extra logical buffers is exactly the kind of overhead that surfaces at long context lengths, which is where ring-attention-style schemes get deployed in the first place.

Treat all of this as author-reported, unreplicated, and days old. The decision it informs (ring-attention versus re-merge multistream for multi-node serving) is orthogonal to the vendor question this article’s title asks.

Where AMD tooling actually stands

The portability question only exists because AMD’s software ecosystem has historically trailed Nvidia’s, so the current state of ROCm tooling is a decision input, not background color.

On the positive side, Kerncap, a May 2026 paper on automated kernel extraction for AMD GPUs, anchors a current toolchain: it evaluates across “AMD Instinct™ MI300X (gfx942, CDNA3), AMD Instinct™ MI210 (gfx90a, CDNA2), and AMD Radeon™ PRO W7900 (gfx1100, RDNA3),” with “all systems run[ning] ROCm 7.2.0 on RHEL 9.6.” It reports a 13.6x speedup over the traditional workflow on a llama.cpp case study, compressing kernel isolation from a multi-hour process to a single command. Kernel-level debugging and experimentation on AMD is getting faster, which lowers the per-GPU tuning tax the Triton and PortLBM results say you will keep paying.

On the negative side, the AMDKernelVault authors document a persistent asymmetry in the data that increasingly drives kernel optimization: “the data and tooling behind these systems remain largely CUDA/NVIDIA-centric. In contrast, AMD’s HIP/ROCm ecosystem lacks large-scale, execution-verified kernel data for training and evaluating kernel LLMs, especially across both native HIP and Triton backends.” If agentic kernel tuning becomes the normal way kernels get optimized, a CUDA-centric training corpus means AMD targets start each tuning campaign with weaker priors. That is a structural reason per-vendor maintenance divergence persists even when portable source exists.

One further caveat on fleet planning: the case for AMD hardware in LLM serving, per an abstract-level MI300X analysis, rests on “high HBM capacity, matrix cores, and their proprietary interconnect” against Nvidia’s “mature CUDA software stack”. That is a reasonable framing of the tradeoff, but an abstract is orientation, not measured evidence.

The decision rule

Pulling the threads together, here is the rule I would apply, with its conditions stated rather than implied.

Standardize on one auto-tuned Triton attention kernel as your serving default when all three hold: your GPUs are ones where the Triton path has been tuned and validated (the author-reported evidence covers an H100 and an MI250 in the kernel benchmarks, an H100 and an MI300 end-to-end, and the A100-and-MI250 vendor-parity result in the authors’ earlier work), your models and sequence lengths sit inside the evaluated envelope, and your team can absorb a per-GPU auto-tuning pass when hardware changes. In that regime, the reported 105.9%-of-SoTA result, measured against FlashAttention3 on an H100, says you give up nothing to the vendor library on that silicon, and you retire a whole maintenance track.

Keep vendor-tuned kernels as the production path when any fail: GPUs outside the paper’s evaluated set, such as MI300X or anything newer than an H100, where you have no validated Triton tuning in hand, workloads outside the evaluated models and lengths, or latency targets tight enough that an unreplicated parity claim is not a foundation you can build an SLA on. On CUDA, FlashAttention and its wrappers remain the incumbent default, and the portable kernel is a benchmark you run at upgrade time, not a bet you ship. On AMD the question has partly inverted: the paper reports its kernels as vLLM’s default attention backend for AMD deployments, so the practical work there is validating that default against your own traffic, not routing around it.

Either way, keep the fallback warm. The evidence says portable parity is achievable; it does not say it survives the next ROCm or Triton release without re-validation.

The support-and-performance map:

One row deserves emphasis: CHARM+ does not belong in the portability column at all. It answers a multi-node communication question on Nvidia hardware. Conflating it with cross-vendor kernel portability is the specific mistake this article exists to prevent.

What to verify before production

The evidence has hard edges, and they map directly onto a pre-production checklist:

• Independent replication. Every figure behind the recommendation is author-reported, and nothing cited here shows a third party reproducing any of it. The anatomy paper’s own numbers cover an H100 and an MI250, end-to-end an H100 and an MI300; it names H100, MI250 and MI300, and never MI300X. Get numbers on your silicon.
• Serving-stack defaults beyond vLLM. The anatomy paper documents vLLM’s attention-backend abstraction and reports its kernels as “the default attention backend in vLLM for AMD deployments.” That covers vLLM, as of a November 2025 preprint, on the authors’ reporting. For SGLang or any other serving stack, and for defaults after later releases, check the documentation for your pinned version.
• Current ROCm SKU support. Kerncap’s ROCm 7.2.0 matrix (MI300X, MI210, W7900) is a recent anchor for what runs; confirm your exact cards against AMD’s current support matrix before purchasing.
• Evaluated envelope. The anatomy paper’s end-to-end numbers come from Llama-3.1-8B at batch size 1 with 500-token prompts, and its kernel benchmarks from a Llama3-8B architecture. Whether Triton parity holds beyond those models and sequence lengths is unknown. Reproduce the comparison on your actual workload before promoting it.
• CHARM+ scope. No AMD GPU appears anywhere in its evaluation, which spans L20, H20 and H100 systems; the 3%~5% memory figure comes from 8-GPU L20 runs and the multi-node speedups from H20. Wait for replication before planning around either.

My judgment, bounded by the evidence: on a fleet of the evaluated GPUs, an H100 alongside MI250 or MI300 parts, serving models inside the evaluated envelope, I would run the auto-tuned Triton kernel as the default and measure it against the vendor libraries on my own traffic, because the reported parity plus the retired maintenance track is worth a benchmarking week. On the AMD side of vLLM that is close to what already ships, so the trial is mostly confirmation. On silicon the paper does not name, MI300X included, I would assume nothing and benchmark the Triton path before promoting it, because the portability claim has no measured footing there yet. The question “can one attention kernel serve both vendors” now has a supported answer of “yes, where tuned,” and the work that turns “where tuned” from the authors’ test GPUs into your fleet is verification work nobody has published. Plan accordingly.

Frequently Asked Questions

What hardware was used to evaluate the Triton attention kernel’s performance?

The paper’s evaluation hardware is “the NVIDIA H100-80GB and the AMD MI250-128GB,” with end-to-end serving experiments “on both NVIDIA H100 and AMD MI300 GPUs,” and the tuning was per-platform: the limited autotuning approach “further reduces the kernel latency for short (up to 9.8× on H100) and medium prompts (up to 75%) on both platforms.”

Does the CHARM+ paper demonstrate cross-vendor GPU portability?

No AMD GPU appears anywhere in the evaluation, which spans L20, H20 and H100 systems. Nothing in the paper demonstrates a kernel running portably across vendors, so it cannot anchor a cross-vendor claim regardless.

What is the memory overhead of CHARM+ compared to ring-attention?

CHARM+ logically requires four more buffers than ring-attention, but the authors report that temporal reuse of GPU memory keeps peak buffer usage roughly unchanged, with observed memory overhead of only 3%~5% over ring-attention (source). That figure comes from runs on “L20 with 8 GPUs across varying sequence lengths,” not from the H20 scalability experiments.