Running MiniCPM-V-4.6 on Fermi: What 6 GB of VRAM Forces

A single-author arXiv preprint posted 2026-07-16 runs MiniCPM-V-4.6, a multimodal assistant, end-to-end on a 2011 NVIDIA Tesla C2075 with 6 GB of VRAM and no Tensor Cores, answering image questions in 1.7 seconds according to the paper. That is not a buy-Fermi recommendation. It is a stress test of what software staging recovers when the GPU is held fixed at a 2011 floor: most of a multimodal pipeline, paid for in latency.

What actually fits in 6 GB of 2011 VRAM?

A compact multimodal assistant fits in 6 GB of Fermi VRAM; a 35B mixture-of-experts model at 4-bit does not, and that capacity boundary, not the silicon’s age, decides whether the pipeline stays all-GPU or spills to a hybrid.

The test bed is a Tesla C2075: Fermi architecture, compute capability 2.0, 6 GB of device memory. Fermi’s streaming multiprocessors execute integer and floating-point arithmetic through multiply-add (MAD) units, per background on CUDA’s SM design, and predate the dedicated matrix and tensor units that later Nvidia generations added for fast quantized GEMM. On that card, the author stages MiniCPM-V-4.6 in full: a SigLIP2 vision encoder, a window-attention merger that compresses visual tokens 16-fold, and a compact hybrid gated-delta-net backbone serving as the language model.

The port is validated stage by stage against a locally generated reference forward pass, and the full vision tower reproduces its reference to a max error of 1.4e-5. A number that small is the difference between a faithful port and an approximate one, and it is the kind of evidence that lets a reader trust the downstream latency claims. The port was not without casualties. Position-embedding bucketization failed on exact rational ties, because float tie-breaking in index arithmetic is implementation-defined. The rule the author draws generalizes beyond Fermi: call the reference operator rather than reimplement it, because the failure mode is a silent mis-bucketing that passes a smoke test and breaks on edge inputs.

The capacity boundary is the more important finding. A companion study (arXiv:2606.24031, referenced in the paper) ran a 35B mixture-of-experts model on the same C2075 as a GPU-prefill, CPU-decode hybrid, because the 4-bit model would not fit in 6 GB of device memory. That pair of results brackets the floor precisely. A model engineered to be small runs entirely on device; a large MoE spills its decode to the host CPU. The hardware does not make that call. The ratio of model footprint to 6 GB does.

The practical implication is that “what fits” is the first question to answer on constrained hardware, and it is answered by footprint accounting, not by shopping. Once you know the footprint, you know whether you are writing an all-GPU pipeline or a split one, and the rest of the engineering follows from that.

Why does 4-bit quantization make decode slower on Fermi?

On pre-Turing hardware, lower precision is not monotonically faster: on Fermi, 4-bit decode runs slower than 8-bit, because the architecture uses shift instructions to pull nibbles out of packed weights, and those shifts issue at half the rate of their 8-bit counterparts.

That is the most counterintuitive result in the paper, and it breaks the assumption embedded in most local-LLM guides, which treat stepping down in precision as a free speed win. The mechanism is architectural rather than numerical. Fermi’s shift units move nibbles, the 4-bit chunks inside packed weights, out at half rate, so the dequantization path that is nearly free on Ampere and Hopper becomes a tax on Fermi. The author reports that 8-bit weights, dequantized once and fed to the vendor SGEMM, outperform 4-bit decode on this silicon.

A second precision-related correction is worth keeping, because it is the kind of error that propagates. The author’s best hand-written GEMM reached only 37% of FP32 peak, per the paper, and that figure had been reported as the performance ceiling for this class of kernel on Fermi. It was not. The vendor SGEMM, still shipped in the last Fermi toolchain, hit 64% of FP32 peak, per the paper, a measured refutation of an underperformance-as-ceiling claim. The gap between 37% and 64%, per the paper, is not a rounding error. It is roughly the difference between “this hardware cannot do it” and “this hardware can do it if you stop fighting the vendor library.”

The takeaway for anyone porting to old or constrained silicon is direct. Before writing a custom kernel to work around a reported ceiling, verify the ceiling against the vendor library on the same hardware. The reported ceiling on Fermi was a measurement of one implementation, not of the silicon.

Where does the attention wall hide, and how do you flatten it?

A naive attention kernel’s prefill throughput collapses from 114 tokens per second at 2k context to 21 at 10k (per the paper), an O(N²) falloff that short benchmarks never expose.

The 2k-token benchmark that most local guides run looks fine, because at that length the quadratic cost is still modest. Push the context to 10k and the same kernel loses roughly five-sixths of its throughput. The collapse is structural, not a tuning problem. Attention is quadratic in sequence length, and a kernel that recomputes or re-allocates per length amplifies the penalty rather than absorbing it.

The author’s rewrite is unglamorous and effective. Attention is reformulated as per-head vendor-GEMM calls that write directly into the existing score buffer, which means the rewrite costs zero extra memory. The resulting profile is flat: 408 tok/s at 2k context and 361 tok/s at 10k (per the paper), roughly a 17x improvement at the long end, verified by exact needle-in-a-haystack retrieval from 60% depth (per the paper). The verification step is the part worth pausing on. The author did not just report a speedup and move on; the author confirmed that the fast path still retrieves the correct token at depth, which is the only test that matters for a serving claim.

The same attention rewrite cascades into the vision path. Image encoding drops 6x to 0.93 seconds, and the system answers an image question end-to-end in 1.7 seconds. That 1.7s figure is the practical latency floor for the capability set the author chose to retain, and it is the number to weigh against any claim about Fermi usability. It is fast enough for an interactive single-user assistant and too slow for a fanned-out serving tier.

A related fusion win shows up in the recurrent layers. The gated-delta-net backbone’s sequential scan was replaced with a chunked delta-rule formulation that ran 2.8x faster, again once attribution exposed a single bad kernel, per the paper. The pattern across all three rewrites is the same: none required new hardware. Each required identifying the slow kernel, reusing the vendor library, and fusing where the memory budget allowed. Software staging, not procurement, delivered the gains.

What does Fermi lack that a modern GPU has?

Fermi lacks the Tensor Cores, dedicated matrix units, and low-precision fast paths that make quantized inference fast on current GPUs, which is why the floor is real and the speed is paid in latency.

The architectural gap is concrete. Streaming multiprocessors on Fermi have MAD units for integer and floating-point work, per CUDA background. They do not have the tensor units and int8/int4 matrix paths that later Nvidia generations added for fast quantized GEMM. That is why the paper leans on a vendor SGEMM from a toolchain Nvidia no longer ships. The current CUDA stable release is 13.3.0, dated 2026-05-26, per the CUDA article. The paper itself refers to “the last Fermi toolchain,” and anyone reproducing this work has to locate a toolchain that still compiled for sm_20 and still bundled the vendor SGEMM.

The toolchain lock-in is structural, not incidental. CUDA is proprietary and runs only on Nvidia GPUs, with no first-party emulation or fallback for modern revisions, per the CUDA article. The Fermi result is therefore locked to Nvidia’s own old toolchain, which is a hard constraint on who can reproduce it and a reason the paper reads as a practitioner report rather than a portable recipe. A practitioner with an AMD card or an Intel integrated GPU cannot run this pipeline as described; the software staging lessons transfer, the exact pipeline does not.

Does this reframe what you actually need to buy?

On constrained hardware, the binding constraint on local inference is software staging, not GPU procurement, and most recommended minimums over-provision by roughly an order of magnitude for the workloads people are willing to trim.

The Fermi result holds the GPU fixed at a 2011 floor and asks what software staging can recover, and the answer is enough to run a multimodal assistant. That implies the standard minimums over-provision for the capabilities a user is willing to cut.

The reframing is in the question the reader is left with. “What GPU do I need” assumes the bottleneck is procurement. The evidence in this paper points elsewhere. The bottleneck is how the workload is staged: which ops are fused, which vendor libraries are reused, and which capabilities are triaged away. Once that framing holds, the purchase decision becomes secondary to the engineering one. You are not choosing a card so much as choosing a capability set and then finding the cheapest hardware that can stage it.

This is not an argument for running inference on a Tesla C2075. It is an argument that the floor for a usable local assistant sits lower than the guides imply, and that the gap between the implied floor and the real one is bridged by software work the guides do not price in. The reader who internalizes that gap asks a sharper question before spending: not what GPU runs everything, but what am I willing to give up, and what staging recovers the rest.

What should you trust and not trust in these numbers?

The figures describe a single-author, non-peer-reviewed preprint running one model on one card, classified under a condensed-matter physics category, and should be read as a practitioner report rather than a vetted benchmark.

The provenance matters for how much weight to put on any single number. The preprint (arXiv:2607.14568) is single-author, posted by J. Q. Lu, classified under cond-mat.other and cs.AI rather than a dedicated machine-learning venue, and is 17 KB. Cond-mat.other is where condensed-matter physics submissions land; an LLM inference engine is an unusual tenant there. That classification does not invalidate the work, but it is a flag on both the peer-review status and the audience that has reviewed it, and a reader should calibrate accordingly.

The latency tax is the honest cost of the result. A 1.7-second image question is workable for an interactive assistant and impractical for a serving tier that needs to fan out across many requests. Throughput on Fermi is low by modern standards, and no amount of kernel fusion recovers the units the silicon lacks. The floor is real; it is paid in latency, and the paper does not claim otherwise.

The structural lessons are what survive the specific hardware. Quantization precision interacts with a chip’s shift and unpack units non-monotonically, so “lower is faster” is a hardware-dependent claim rather than a law. Short-benchmark attention profiles hide O(N²) collapse, so a throughput figure measured at 2k context tells you almost nothing about behavior at 10k. And the binding constraint on local inference is software staging, not VRAM headroom. Those three hold for any constrained accelerator, from an old discrete GPU to a mobile NPU to an edge SoC. The Tesla C2075 is simply the device that made them impossible to ignore, because on it the missing units left nowhere to hide the staging debt.

Frequently Asked Questions

Can I run this on an AMD GPU using ROCm or ZLUDA?

No. CUDA is proprietary and runs only on NVIDIA GPUs, with no first-party emulation or fallback for modern revisions. Third-party paths like ZLUDA or HIP are partial substitutes and do not provide the vendor SGEMM or sm_20 toolchain support required for this pipeline.

Does the 4-bit slowdown apply to NVIDIA’s 10-series Pascal cards?

Pascal (sm_60) introduced native int8 and int4 tensor operations, which Fermi lacked. The Fermi slowdown was caused by shift-unit limits on pre-Turing silicon. Pascal and later architectures use dedicated matrix units, so lower precision remains faster there.

What happens if I try to run this on a mobile NPU?

The structural lessons transfer, but the exact pipeline does not. Mobile NPUs trade the CUDA toolchain for vendor-specific runtimes like LiteRT or MLCD. You would need to rewrite the chunked delta-rule and attention rewrites for the NPU’s specific operator set, but the principle of software staging over hardware procurement still applies.

Is this preprint peer-reviewed or part of a major conference?

No. The paper is single-author, non-peer-reviewed, and classified under cond-mat.other (condensed-matter physics) rather than a dedicated machine-learning venue like NeurIPS or ICML. Treat the numbers as a practitioner report, not a vetted benchmark.