How to Run an LLM on an Old Laptop: Matching Model Size to RAM

Pick the model after you pick the memory budget, not before. On old hardware the binding constraint is RAM, not compute, and the arithmetic is simple enough to do on the back of an envelope: usable memory (what is left after the OS, not the sticker number) must cover the model’s weights at your chosen quantization level, plus runtime overhead, with headroom. If it does not, the model does not run slowly. It does not run at all. In a preprint measuring on-device inference on an iPhone 15 Pro, “For larger models with 7B and 8B parameters, the F16 and Q8 configurations exceeded the device’s memory capacity, resulting in mmap failures.” Meanwhile a 1-billion-parameter model on the same device ran CPU-only at 17 tokens per second. The lesson generalizes to the 4–8 GB laptops this article is for: size the weights to the memory first, then argue about quality.

A note on evidence before the numbers. The measured results here come from two preprints, one on CPU versus GPU inference and one comparing 13 llama.cpp quantization levels on a single checkpoint; both are preprint-reported and not yet replicated in the literature cited here. A third preprint examines what quantization damages that benchmark scores miss. The most extreme low-RAM results come from a single self-reported community project with no independent benchmarks anywhere in this research, and every one of its figures is labeled accordingly below. Where I extend the measured results into per-hardware-class recommendations, that is arithmetic with stated assumptions, not measurement.

RAM is the ceiling, and the CPU is usually not the problem

The folk intuition about old machines is that the processor is what makes local AI painful. The available measurements point the other way. In the CPU/GPU on-device inference preprint: “Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration.” Seventeen tokens per second is faster than most people read. That was two threads, at full F16 precision, on a phone chip.

The same study shows where the wall actually is. At 7B and 8B parameters, the F16 and Q8 configurations did not degrade gracefully; they failed to load, because the runtime could not memory-map files that large into the available RAM. This is the asymmetry that should drive your decision. Compute problems produce slow output. Memory problems produce no output. A decade-old CPU with enough RAM will run a small model at usable speed; a fast CPU with insufficient RAM will run nothing.

One caveat on the old-x86 case specifically: this research did not include official llama.cpp documentation, so CPU instruction-set requirements for current builds (such as AVX2 on older Intel and AMD processors) are unverified here. If your laptop predates AVX2 support, check the runtime’s current requirements before assuming any of what follows applies. That check costs minutes; a failed 8 GB download costs an afternoon.

Budget usable RAM, not the sticker number

Installed memory and usable memory are different numbers, and the gap is the first place sizing goes wrong. The clearest illustration comes from the device matrix in the smolmaxxing project, a self-reported community effort to run LLMs on very small devices. Its three showcase devices:

• A Xiaomi TV Box S (3rd Gen), Amlogic S7D with 4 Cortex-A55 cores, 1.9 GB RAM of which the author reports about 1 GB usable, running Android 14 without root.
• An iPad Air from 2013, Apple A7 with 2 cores, 1 GB RAM of which about 700 MB is usable, on iOS 12.5.7, jailbroken.
• A OnePlus Nord CE 2 (2022), MediaTek Dimensity 900, 7.4 GB RAM with about 3 GB free, on Android 13 without root.

Notice the pattern: 1.9 GB installed becomes roughly 1 GB usable. 7.4 GB installed becomes roughly 3 GB free. The operating system, its services, and whatever else you keep open take their cut before your model sees a byte. On a typical laptop the overhead differs by OS and habit, but the direction never changes. A machine advertised as 8 GB might offer 5 to 6 GB to a model on a good day, and less with a browser full of tabs. A 4 GB machine might offer 2.5 to 3 GB. Those usable-RAM figures for laptops are my assumptions for the arithmetic below, stated so you can replace them with your own measurement (free -h on Linux, Activity Monitor on macOS, Task Manager on Windows), not values any cited source measured.

The practical rule: compute your budget from memory actually free on your machine, with your normal workload running, and treat that number as the ceiling.

The sizing arithmetic, assumptions on the table

Model weight memory is, to first order, parameters multiplied by bytes per weight. Bytes per weight is set by the quantization level. At full F16 precision each weight is 16 bits, so 2 bytes: a 1B-parameter model needs about 2 GB, a 7B model about 14 GB. That part is definitional, not a measurement, and it is consistent with the preprint result above: a 1B model at F16 fit and ran, while 7B and 8B at F16 did not.

Quantization reduces the bits per weight. The decision axis is well documented: a unified evaluation preprint tested “13 quantization configurations (Q3_K*, Q4_0/1, Q4_K*, Q5_0/1, Q5_K*, Q6_K, Q8_0) alongside a full-precision FP16 baseline, all derived from the same Llama-3.1-8B-Instruct checkpoint and quantized with the official llama.cpp tooling.” The ladder runs from Q8_0 (8 bits, near-lossless, near-full size) down through the Q4_K and Q3_K levels (roughly 4 and 3 bits, much smaller, progressively rougher).

Nominal bit math says N bits per weight is N/8 bytes: 8-bit quants land near 1 byte per weight, 4-bit near half a byte. Two honest limitations apply. First, actual quantized file sizes run above the nominal figure because formats carry metadata and keep some tensors at higher precision, and this research did not verify the per-level bytes-per-weight constants from official llama.cpp documentation, so treat the nominal figures as optimistic floors and check real file sizes before downloading. Second, weights are not the whole cost: the runtime and the context (the KV cache, which grows with conversation length) add overhead on top, which is why the budget needs headroom rather than an exact fit.

Worked examples, all nominal and all stated-assumption calculations:

• 8B at FP16: 16 GB. Fails on any 8 GB machine, consistent with the measured mmap failures.
• 8B at Q8: about 8 GB. Fails on an 8 GB laptop once the OS takes its share, also consistent with the measured failures.
• 8B at Q4-class: about 4 GB nominal, realistically more. Fits a 5–6 GB usable budget, with context length as the variable to watch.
• 3B at Q4-class: about 1.5 GB nominal. Comfortable on a 4 GB machine.
• 1B at F16: 2 GB. The configuration the preprint measured at 17 tok/s CPU-only.

The procedure, then: measure usable RAM, subtract headroom for the runtime and your intended context length, divide the remainder by the bytes-per-weight of the quant level you are considering, and the result is your parameter ceiling. Walk down the quantization ladder or down in parameter count until model-plus-overhead fits.

A shortlist by hardware class

Applying that procedure with assumed usable-RAM budgets (replace my assumptions with your measurement before acting on any row):

Two traps this table cannot absorb. First, the failure mode is binary: a row that misses does not run slowly, it hits an mmap failure, so when in doubt pick the smaller cell and verify. Second, mixture-of-experts models break the arithmetic. The smolmaxxing project’s biggest reported runs (a 16B model on about 1 GB of RAM, a 35B on about 3 GB free) are MoE models combined with flash streaming, so total-parameter sizing alone mispredicts them. Do not read the 8 GB row as “35B fits.”

What quantization actually costs

Smaller quants are the main lever for fitting old hardware, so the quality cost deserves more than a shrug. The controlled comparison exists: the llama.cpp quantization evaluation holds the checkpoint constant (Llama-3.1-8B-Instruct, official tooling) and varies only the quant level across 13 configurations, which is exactly the experiment you would design to price the ladder. That is one 8B checkpoint, preprint-reported, so treat its rankings as a starting point rather than a universal law of quantization.

The sharper warning comes from a different angle. A preprint analyzing quantization through sparse-autoencoder features found that “task-level metrics can miss feature damage: perplexity can remain stable or improve while many SAE features degrade.” At intermediate bit-widths the damage is smearing rather than destruction: “At intermediate bit-widths, degradation dominates damage (e.g., Gemma INT6: 48.7% degraded, 0% damaged): features are often blurred rather than destroyed.” SAE features are interpretability probes, not end-task accuracy, so this does not tell you your chatbot got worse at summarizing email. It does tell you that “the Q4 scores match FP16 on perplexity” is weaker evidence than it sounds. The model can look intact on the aggregate metric while internal capabilities quietly blur.

The practical consequence is about verification, not avoidance. When you step down the ladder to fit memory, validate on your own workload: a handful of prompts you actually care about, run against the higher-quant and lower-quant versions, will tell you more than any published score. My read of the evidence is that Q4_K-class quants are a reasonable default starting point for fitting 8 GB machines, with the explicit condition that you spot-check quality on your tasks, and Q3-class is where I would start demanding that check before trusting the output.

The exception that is not a rule: streaming models bigger than RAM

Everything above assumes the model must fit in memory. The smolmaxxing project claims otherwise, and the claim is worth understanding precisely because it is the one most likely to be misquoted. In the project’s own words: “The TV box and the iPad run 8B and 16B mixture-of-experts models on about 1 GB of RAM, and the phone runs a 35B one with about 3 GB free. They all keep only part of each model in memory and read the rest from flash as it’s needed.”

The self-reported results, no independent benchmark exists for any of them: the TV box reportedly runs Qwen3.5-2B at 7.7 tok/s as an everyday model and a 16B MoE at 5.1 tok/s as its ceiling; the 2013 iPad reportedly runs a 1-bit Bonsai-1.7B at 24–26 tok/s daily and the 16B at 2.6–4.5 tok/s; the phone reportedly runs the 16B at 11–15 tok/s and a 35B-A3B MoE at 4.9–5.7 tok/s. The author also reports the 1-bit kernel needs only basic NEON instructions and the streaming works on any Linux or Android device with more model than RAM.

Two reasons not to fold this into the sizing rules. It is a single self-reported project on ARM TV-box, tablet and phone hardware, not x86 laptops, with no third-party replication, so every tok/s figure is a claim rather than a measurement. And the mechanism is MoE-specific: streaming works because these models activate only a slice of their parameters per token, which is why total-parameter arithmetic mispredicts them and why the trick does not obviously transfer to the dense models in the shortlist. If you want to test it on your own hardware, treat it as an experiment on work that can tolerate failure, not as capacity you can plan around.

Validate on your own machine

The strongest limitation of everything here is that the per-class shortlist is a stated-assumption calculation. The only independently measured numbers are on one phone: 1B at F16 fits and runs at 17 tok/s CPU-only, 7B/8B at F16 and Q8 fail to load. The quantization ladder is one 8B checkpoint. The feature-damage study measures interpretability features, not task accuracy. The extreme low-RAM results are self-reported and unreplicated. There are no laptop-specific measurements in this research at all.

That is enough to act on, because the procedure degrades safely. Measure your free memory, pick a starting point one cell below your theoretical ceiling, and confirm the model loads before you download anything larger. llama.cpp’s mmap failure is fast and free; discovering it after a 5 GB download is the expensive version. Then run your own prompts against the quant level you chose, because perplexity parity will not warn you about blurred features, and step up or down the ladder based on what you see. The frontier releases will keep arriving at sizes your hardware was never meant to hold. The budget arithmetic outlives all of them.

Frequently Asked Questions

How much usable RAM is typically available on an 8 GB laptop for running an LLM?

A machine advertised as 8 GB might offer 5 to 6 GB to a model on a good day, and less with a browser full of tabs.

What is the recommended starting point for an 8 GB laptop?

7–8B at Q4_K class (~4 GB nominal) with moderate context; smaller models at Q6_K/Q8 for quality