HuggingFace’s AMD org page documents a getting-started path built on upstream PyTorch ROCm wheels and the Optimum-AMD acceleration library, and with ROCm 7.2.4 (released 2026-05-29) tightening vLLM profiling stability, the MI300X has become a serviceable self-hosted inference target. The gap that remains is kernel coverage: ROCm still trails CUDA on attention variants and training paths, so parity is selective rather than universal.
What’s in the HuggingFace AMD recipe?
The documented workflow ships as two layered pieces: official PyTorch ROCm wheels and the Optimum-AMD acceleration library. Neither requires maintaining a fork.
PyTorch’s ROCm support is upstreamed into mainline, with wheels built and tested as part of stable and nightly releases. The install command HuggingFace cites is pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2. Note the version pin: the wheel index targets ROCm 6.2, while the current ROCm release is 7.2.4. Practitioners should match the wheel’s ROCm minor to the installed driver rather than assume the latest wheel tracks the latest ROCm.
Optimum-AMD is the interface between Hugging Face libraries and the ROCm software stack, documenting Flash Attention 2, GPTQ quantization, and ONNX Runtime integration, per ROCm’s Hugging Face inference guide. For teams that prefer containers, the AMD page points generically to the ROCm Docker Hub and AMD Infinity Hub rather than a curated serving image, and the inference guide ships a Dockerfile for Flash Attention on ROCm.
Which models run on AMD GPUs today?
HuggingFace states on its AMD page that the 200+ standard transformer models are “regularly tested” on supported AMD hardware, with BERT, BLOOM, LLaMA, OPT, and T5 called out as popular starting points, and that derivatives of those core models should also function.
The newer addition is OpenAI’s GPT OSS family. HuggingFace’s GPT OSS welcome post documents AMD ROCm support for the models, which are both mixture-of-experts. That matters because mixture-of-experts models are exactly the workload where the MI300X’s large HBM capacity pays off, and a dedicated kernel path rather than a generic fallback is the difference between usable and not. A MoE model that routes to an unoptimized expert kernel pays the penalty on every token, not just the warmup.
For practitioners who want to skip PyTorch entirely, llama.cpp’s GGUF build guide documents AMD ROCm support, enabling GGUF model deployment through llama-cli and llama-server. This is the lower-friction path for consumer Radeon cards where a full PyTorch serving stack is overkill.
Where does ROCm still trail CUDA?
ROCm’s kernel coverage lags CUDA on two fronts: newer attention variants and the training toolchain. The HuggingFace recipe documents Flash Attention 2 via Optimum-AMD; Flash Attention 3 and later attention kernels have historically arrived on AMD after CUDA, and the current ROCm inference guide does not document their coverage [unverified].
The training story is blunter. SemiAnalysis, after a five-month benchmarking effort across H100, H200, and MI300X, described the public software stack as “riddled with bugs rendering out of the box training with AMD is impossible,” requiring workarounds on public stable releases, and cited a “challenging out of the box experience.” Their headline conclusion was that the MI300X’s on-paper specs “are not representative of performance that can be expected in a real-world environment.”
A structural reason for the lag: many AMD libraries are forks of NVIDIA’s, translated through the Hipify source-to-source tool, which means AMD builds on top of a competitor’s platform rather than a native one. That lineage surfaces wherever the workload pushes past well-trodden inference paths, because the fork only gets hardened for the cases its maintainers actually run.
How does MI300X perform against H100 for inference?
On memory-bound inference, the MI300X’s hardware advantage holds. A comparison survey reports a 40% latency advantage over the H100 for LLaMA2-70B inference. The caveat is provenance: this figure comes from a secondary comparison rather than a peer-reviewed benchmark, and the research brief flags it as medium-confidence. Treat it as directional, not definitive.
The mechanism is straightforward. Inference on large transformer models is bandwidth-bound: the dominant cost is moving weights through HBM, not compute. The MI300X carries 192 GB of HBM3 at 5.3 TB/s. The H100 carries 80 GB at 3.35 TB/s and the H200 141 GB at 4.8 TB/s; per RunPod’s H200 guide, the H200’s upgrade over the H100 is concentrated in the memory subsystem. More bandwidth per token means lower decode latency; more capacity means a larger model fits on fewer cards. That is the entire case for AMD in inference, and it is a real case when the workload is decode-heavy and memory-bound.
| GPU | HBM | Bandwidth | Marketed TFLOP/s |
|---|---|---|---|
| AMD MI300X | 192 GB HBM3 | 5.3 TB/s | 1,310 |
| NVIDIA H100 | 80 GB HBM3 | 3.35 TB/s | 989 |
| NVIDIA H200 | 141 GB HBM3e | 4.8 TB/s | 989 |
MI300X and H100 specs per the comparison survey; H200 specs per RunPod’s H200 guide.
Where the advantage evaporates is compute-bound paths and anything hitting an unoptimized kernel. A model whose attention layer falls back to a non-accelerated implementation will not see the bandwidth advantage because it stalls on compute.
When does MI300X make sense for on-prem inference?
MI300X earns its place when the workload is memory-bound inference on large models, where 192 GB of HBM3 lets a single card hold a 70B-class model that an 80 GB H100 cannot without tensor parallelism. For a team pricing on-prem inference, that widens the GPU pool they can quote against and lowers the cost of leaving the NVIDIA default. It also changes the topology: fewer cards per replica means less inter-GPU communication, which sidesteps the weakest part of the MI300X platform.
The ROCm 7.2.4 release notes add a few specific fixes that matter for inference operators: improved profiling stability for vLLM workloads traced with PyTorch’s torch.profiler, with substantially reduced idle gaps between GPU kernels in traces; reduced hipGraphLaunch latency for multi-list graph topologies; and a fix for an H2D memory copy latency regression on MI300 Series GPUs in CPX mode. None are headline features, but graph launch and host-to-device copy latency are the small overheads that eat decode throughput on a serving stack, and their absence is the kind of thing that makes a deployment feel sluggish without an obvious culprit.
For consumer Radeon cards, the calculus differs. The same ROCm stack and PyTorch wheels apply, but the use case is single-user inference and experimentation, not serving. llama.cpp’s GGUF path is the more proportionate tool there.
What should you not yet run on AMD?
Avoid newer attention kernels on AMD today. If your model uses Flash Attention 3 or a non-standard attention variant, verify kernel support before committing, because the historical pattern is that these arrive on AMD after CUDA and the current ROCm inference guide does not document them [unverified].
The honest summary: HuggingFace’s recipe makes AMD a credible inference target for the models it explicitly supports, and ROCm 7.2.4’s profiling and latency fixes make the serving stack less fragile. It does not make AMD a credible training target on public releases, and it does not erase the kernel-coverage gap that appears the moment you leave the well-trodden path. Price the inference win, and keep the training plan on NVIDIA until AMD’s public stable releases close the gap SemiAnalysis documented.
Frequently Asked Questions
Can the HuggingFace AMD recipe run on a desktop Radeon GPU, or is it Instinct-only?
It officially covers both Instinct and Radeon on Linux, but the recipe is aimed at serving on Instinct. For a consumer Radeon card, llama.cpp’s GGUF build is the proportionate path. A ghcr.io/huggingface/text-generation-inference:latest-rocm container exists for serving, yet the AMD page points to generic ROCm Docker Hub rather than a curated image, and you still need the ROCm driver installed before the pip wheels will run.
Which ROCm 7.2.4 fixes matter most for vLLM inference?
The release improves profiling stability for vLLM under torch.profiler, lowers hipGraphLaunch latency for multi-list graph topologies, and fixes an H2D copy latency regression on MI300 series GPUs in CPX mode. CPX is a multi-chip partitioning configuration, so the H2D fix matters mainly if you are running partitioned rather than unified. Flash Attention 3 is still not documented, so the update improves serving hygiene rather than attention coverage.
What prerequisite must be in place before pip3 install torch --index-url https://download.pytorch.org/whl/rocm6.2 will work?
The ROCm driver, runtime, and compatible kernel modules must be installed at the OS level first. Inside a container you also need to pass the AMD devices (/dev/kfd and /dev/dri) and use a ROCm-enabled image. The 6.2 wheel index matches a 6.2 driver; running it on a 7.2.4 driver without compatibility can produce runtime errors even after pip reports success.
Why does the MI300X sometimes fall short of its marketed FP16 performance?
SemiAnalysis measured roughly 620 TFLOP/s for BF16 operations on MI300X, well below the advertised 1,310 TFLOP/s. Real inference often hits unoptimized kernels or compute-bound paths, especially when attention falls back from a fused implementation. The reported 40% latency win for LLaMA2-70B reflects HBM bandwidth on a memory-bound model, not a guarantee that every kernel reaches peak FLOP/s.
Which workloads should still stay on NVIDIA for now?
Training and any model that depends on Flash Attention 3 or newer attention variants. ROCm historically gets these kernels after CUDA, and SemiAnalysis found public ROCm stable releases still require workarounds to train at all. OpenAI’s GPT OSS MoE models do run on AMD through a MegaBlocks kernel path, but that is an inference win, not a sign that the training stack is ready.
