// guide

How Much VRAM Do You Need to Run a Local LLM?

The short answer: enough to hold the model file plus its working memory. A typical 8B model needs about 5 GB at the popular 4-bit setting, about 8.5 GB at 8-bit, and 16 GB unquantized. This guide turns your graphics card — or your system RAM, if you have no card — into a concrete answer for every common model size.

Last updated: November 2026 · Publisher: LibreAI · By the LibreAI team · Figures computed from model parameter counts and published GGUF file sizes — method at the end of this page

Key takeaways

  1. Weights dominate. VRAM needed ≈ the model file's size on disk, plus roughly 10–20% headroom for the KV cache and runtime buffers at chat-length contexts.
  2. An 8B model at Q4_K_M (≈4.9 GB) is the entry point — any 8 GB graphics card runs it fully on the GPU.
  3. A 24 GB card (RTX 3090 / 4090 class) runs a 32B model at Q4_K_M with room to spare; a 70B model needs 48 GB at that setting, or partial offloading to RAM.
  4. No GPU is not a dead end: the same size table reads as system RAM for CPU-only runs — slower, but it works.
  5. gpt-oss-20b was built for 16 GB cards; gpt-oss-120b needs an 80 GB-class GPU or a high-memory Mac.

The short answer

A language model in memory has three parts: the weights (the model file itself), the KV cache (the model's working memory of your conversation, which grows with context length), and runtime buffers. In normal chat use the weights are 80–95% of the total, so the model file's size on disk is the number to start from. Quantization is what makes home hardware viable: storing each weight in about 4 bits instead of 16 shrinks the file roughly fourfold, at a small quality cost. Every figure below is a real GGUF file size, or a direct calculation from one.

VRAM by model size and quantization

Weights only, in GB — the size of the model file you download. The same numbers apply to system RAM on the CPU-only path. Add headroom for the KV cache and buffers (next section) before comparing against your card.

Model sizeFP16Q8_0 (8-bit)Q4_K_M (4-bit)Q2_K (2-bit)
~1B2.01.10.80.6
~3B6.43.42.01.3
~8B168.54.93.0
~14B30169.05.6
~32B65352012
~70B141754326
gpt-oss-20b (MoE)——≈14 (native MXFP4)—
gpt-oss-120b (MoE)——≈65 (native MXFP4)—

Mixture-of-experts models such as gpt-oss store all of their experts, so the full parameter count sets the file size even though only part of the model is active for each token.

The KV cache: why context length eats memory

The KV cache is the model's memory of everything said so far, and it grows linearly with context length. An 8B model of the Llama 3.1 design stores about 0.13 MB of cache per token in full precision, so an 8,000-token conversation adds roughly 1 GB — and a 32,000-token context adds about 4 GB. Larger models store more per token. Most runtimes can quantize the cache too, halving or quartering it at a small quality cost. Practical rule: add 1–2 GB on top of the weights for normal chat, and 4 GB or more if you plan to feed the model long documents.

What fits on your graphics card?

"Fits" here means the whole model stays on the GPU with a normal chat context — the fast configuration. Cards named are examples of each tier (an RTX 3060 has 12 GB, an RTX 4060 has 8 GB, an RTX 4090 and an RX 7900 XTX have 24 GB).

Your VRAMLargest comfortable full-GPU setup
4–6 GBA 3B model at Q4_K_M; an 8B model only with layers offloaded to RAM
8 GBAn 8B model at Q4_K_M — the entry-level sweet spot
12 GBAn 8B model at Q8_0, or a 14B model at Q4_K_M
16 GBA 14B model at Q8_0, or gpt-oss-20b at its native MXFP4
24 GBA 32B model at Q4_K_M, with room for a long context
32 GBA 32B model at Q8_0, or a 70B model at a low 3-bit-class quant
48 GBA 70B model at Q4_K_M, fully on the GPU
80 GBgpt-oss-120b, or a 70B model at Q8_0

Worked example: an 8 GB card, step by step

Take a desktop with an RTX 4060 (8 GB). Download an 8B model at Q4_K_M: the file is 4.9 GB. Load it in llama.cpp or Ollama with an 8,000-token context: the KV cache adds about 1 GB and the buffers a few hundred MB — roughly 6.3 GB in total, leaving comfortable headroom on the card. The same model at Q8_0 (an 8.5 GB file) does not fit fully, so the runtime offloads some layers to system RAM and generation slows noticeably. That is the whole decision this page exists for: pick the quant that fits, not the biggest file.

No graphics card? The CPU-only path

Everything above reads the same with system RAM in place of VRAM: the model file lives in RAM instead. A CPU run of an 8B model at Q4_K_M needs the same ~5 GB of RAM for the weights, plus room for the operating system — 16 GB of total RAM is workable, and 32 GB is comfortable and also covers a 14B model at Q4_K_M. Speed is set by memory bandwidth rather than core count: expect single-digit tokens per second from a typical desktop CPU on an 8B model — fine for summaries, batch jobs and overnight work, sluggish for live chat. Hybrid mode splits the difference: llama.cpp and KoboldCpp can keep part of the model on a small GPU and the rest in RAM.

Apple Silicon: one pool of memory

Macs with M-series chips share a single memory pool between CPU and GPU, so the VRAM column becomes your Mac's total memory minus what macOS and your apps need — leave roughly a quarter free. In practice: a 16 GB Mac runs an 8B model at Q4_K_M well and a 14B at Q4_K_M tightly; 32 GB runs a 32B model at Q4_K_M; 64 GB runs a 70B model at Q4_K_M; and a 128 GB Mac is the configuration that holds gpt-oss-120b. Desktop apps such as Jan and GPT4All make this a two-click setup on macOS.

Which runtime should you use?

Once the size is settled, the runtime decides how the model gets loaded. The catalogue's Local LLM category covers the field; the usual starting points are llama.cpp (the GGUF engine most tools build on, with the finest control over offloading and cache), Ollama (one-command models with an API), Jan and GPT4All (desktop apps, CPU-friendly), KoboldCpp (a single-file runner built for hybrid GPU/RAM setups), text-generation-webui (a full web interface), WebLLM (runs in the browser over WebGPU, within the browser's memory limits), and vLLM (for serving many users at once — a different trade, throughput over home fitting). Each project's page states its own documented requirements.

How much VRAM do I need for an 8B model?

About 5 GB at Q4_K_M, 8.5 GB at Q8_0 and 16 GB at FP16 for the weights; add roughly 1–2 GB for the KV cache and buffers at chat-length contexts. An 8 GB graphics card is the comfortable minimum for the 4-bit version, and the most common home setup.

Is 8 GB of VRAM enough to run local LLMs?

Yes — it is the standard entry point. It runs 7–8B models at Q4_K_M fully on the GPU, and smaller 1–3B models at 8-bit or full precision. It is not enough for a 14B model without offloading some layers to system RAM, which works but runs slower.

What is the biggest model I can run on 24 GB of VRAM?

A 32B model at Q4_K_M — about 20 GB of weights — fits fully, with room left for a long context. A 70B model at Q4_K_M (about 43 GB) does not fit; it needs partial offload to RAM, a lower quant, or a second GPU.

Can I run a local LLM without a GPU?

Yes. On CPU, your system RAM plays the role of VRAM and the same size table applies — an 8B model at Q4_K_M needs about 5 GB of RAM for its weights. Generation is much slower, typically single-digit tokens per second for an 8B model on a desktop CPU, but it works, and batch tasks do not care about the wait.

What does Q4_K_M mean?

It is a GGUF quantization format: weights stored at roughly 4 bits each — about 0.6 bytes per parameter once the scaling factors are counted — with the more sensitive tensors kept at higher precision (the "K" scheme; "M" is the medium size). It is the standard balance of file size and answer quality for home hardware, and the column most readers should shop by.

Does more VRAM make a model faster?

Not directly. VRAM decides what fits; generation speed comes from memory bandwidth and compute. Extra VRAM helps speed only indirectly — by keeping the whole model, and a longer context, on the GPU instead of spilling into system RAM.

How much memory does gpt-oss-120b need?

Its MXFP4 weights are about 65 GB, so it targets 80 GB-class datacenter GPUs, or a Mac with 96 GB or more of unified memory. Its smaller sibling gpt-oss-20b (about 14 GB of weights) was designed to run on a single 16 GB card.

How these figures are calculated

The table's file sizes are the published GGUF sizes of representative models at each size class (the Llama, Qwen, Mistral and Gemma families), and the FP16 column is simply parameter count × 2 bytes. Quantized sizes follow each format's bytes-per-weight (Q8_0 ≈ 1.06, Q4_K_M ≈ 0.6, Q2_K ≈ 0.35 bytes per parameter). The KV-cache figures are computed from the Llama 3.1 8B architecture — 32 layers, 8 KV heads, 128-dimensional heads. No benchmark speeds are quoted on this page, because speed depends on your exact hardware, drivers and runtime settings; each catalogue project states its own requirements on its page. If a figure here disagrees with a project's own documentation, the project's documentation wins — tell us via the contact page and we will fix it.

See also: our Open-Source AI Statistics 2026, counted from the same verified catalogue.

AdvertisementToolNest · Free online tools146 free tools. Zero sign-ups.Calculators, converters, PDF & developer tools — right in your browser.Open ToolNest →