18 minute read

Every week, r/LocalLLaMA gets the same post: “I have X GB of VRAM and want to run Y model. What quantization should I use?” The replies converge on the same approximate answers (Q4_K_M is usually fine, AWQ is better for GPUs, FP8 is for H100s). This guide is the 2026 version of that answer. A few things changed in early 2026. BitNet LoRA fine-tuning now runs on phones. TurboQuant (ICLR 2026) shrinks the KV cache to around 3 bits with no measured accuracy loss. Flash-MoE got a 397B model running on a MacBook. The old advice still mostly holds but the map needs updating.

TL;DR: Pick quantization by hardware tier. Edge and mobile use BitNet or GGUF Q4_K_M. Consumer GPUs (8-24 GB VRAM) get GGUF or AWQ. A100 runs INT8 W8A8 or 4-bit weight-only; Ada, Hopper and Blackwell run FP8. KV cache compression (FP8 KV, or TurboQuant at around 3 bits) is a separate choice that stacks on top of any weight quantization. The decision tree below walks you through the whole thing in under two minutes.

A network switching matrix panel with labeled input ports on the left and output ports on the right, one specific signal path illuminated amber thr...


The three axes that determine your quantization choice

Before reaching for a format name, answer three questions.

First: how much memory do you have? This is a hard constraint, not a preference. GGUF Q4_K_M of a 70B model takes roughly 42 GB (about 4.8 bits per weight); Q8_0 takes about 75 GB (8.5 bits per weight). Add KV cache on top. If it doesn’t fit, nothing else matters.

Second: what’s your latency budget? Interactive chat at less than 200 ms TTFT has different tolerances than overnight batch processing. Lower bit-width usually means faster decode, but can add first-token latency on formats without native hardware support.

Third: what quality floor can you live with? In one public vLLM benchmark (Qwen2.5-32B, JarvisLabs), every 4-bit method stayed within about 6% of FP16 perplexity, but HumanEval dropped from 56.1% to 51.8% for AWQ and Q4_K_M and to about 46% for GPTQ. A large Red Hat study (arXiv:2411.02355) found FP8 W8A8 effectively lossless and well-tuned INT8 W8A8 within 1 to 3%. If you’re doing medical summarization, your floor is higher than it is for a personal coding assistant.

Those three questions determine the answer. The format names come after.


Edge, mobile and Apple Silicon laptops

GGUF Q4_K_M for standard models on devices with 16 GB or less, BitNet TQ1_0 for 1-bit-native models, Flash-MoE 4-bit if you’re running a large MoE on a 48 GB Apple Silicon laptop. Since March 2026 you can also fine-tune a 1B BitNet model on a phone: Tether reports about 78 minutes on a Galaxy S25 with its QVAC Fabric framework, on a small (roughly 300-document) dataset.

GGUF Q4_K_M

GGUF is the llama.cpp format. It runs on CPU, Apple Metal, and Vulkan, so it works on anything with enough RAM. Q4_K_M is the 4-bit K-quant variant with mixed precision on attention and feed-forward layers, and it’s the community default for good reason: 6.74 WikiText-2 perplexity on Qwen2.5-32B versus AWQ’s 6.84 and FP16’s 6.56 in JarvisLabs’ vLLM benchmarks (lower is better, minimal difference). On consumer hardware with < 16 GB, Q4_K_M is still the safe default for models that weren’t trained as BitNet.

BitNet (TQ1_0)

Microsoft’s BitNet stores weights as ternary values (-1, 0, +1) at 1.58 bits per parameter. That’s not a quantization of an existing model: the model gets trained this way from scratch. You can’t take Llama 3 and BitNet-ize it. That’s the constraint most people miss.

What you get in exchange: Tether reports BitNet-1B using 77.8% less VRAM than Gemma-3-1B at 16-bit. On x86 CPUs, Microsoft reports bitnet.cpp speedups of 2.37x to 6.17x with 71.9% to 82.2% less energy. And since March 2026, QVAC Fabric runs LoRA fine-tuning on BitNet models on-device: Tether’s figures are 1 hour 18 minutes for a 1B model on a Samsung S25 and 1 hour 45 minutes on an iPhone 16, on a dataset of about 18,000 tokens.

Flash-MoE (48 GB Apple Silicon)

Flash-MoE shipped in March 2026. It runs Qwen3.5-397B-A17B on a MacBook Pro M3 Max with 48 GB RAM at about 4.4 tokens/second. The 209 GB model doesn’t fit in memory. It streams only 4 active experts per layer (fewer than the model’s default) from an SSD rated at 17.5 GB/s using hand-tuned Metal shaders, with 4-bit quantization on the weights. Details are in the Flash-MoE post. The result works because MoE models only activate a fraction of their parameters per token, so you never need the full model at once.

One important thing: 2-bit quantization pushes this to 5.74 tok/s, but it breaks tool calling with malformed JSON. Stick with 4-bit if you need function calling to work.

Edge format table:

Format Bit-width Memory vs FP16 Quality retention CPU support Best for
GGUF Q4_K_M ~4.8-bit ~30% Small loss Yes Universal edge default
GGUF Q8_0 ~8.5-bit ~53% Near-lossless Yes Quality-critical, larger RAM
BitNet TQ1_0 1.58-bit ~10% Native (trained ternary) Yes BitNet-native models only
Flash-MoE 4-bit 4-bit streaming Production quality Metal only 48 GB MoE inference

Consumer GPU (8-24 GB VRAM)

Most people reading this are in this tier. GGUF Q4_K_M or Q8_0 if you need flexibility across hardware. AWQ if you’re on NVIDIA and serve with vLLM. GPTQ when the checkpoint you want is only published in GPTQ or your stack (ExLlama) is built around it.

GGUF on GPU

GGUF isn’t just for CPU. With llama.cpp GPU offloading, you push layers to VRAM and overflow to CPU RAM. On a 16 GB GPU running a 13B model at Q4_K_M (~8 GB), you can offload all layers and get full GPU inference speed. The advantage: flexibility. One format, any hardware, CPU-GPU split as needed. Throughput won’t match dedicated GPU formats, but the operational simplicity is worth it for personal deployments.

AWQ (Activation-Aware Weight Quantization)

AWQ calibrates quantization by observing actual activation magnitudes and protecting the outlier weights that matter most. In the JarvisLabs Qwen2.5-32B benchmark, AWQ’s 4-bit accuracy held up better than GPTQ’s (HumanEval 51.8% vs 45.7%, FP16 56.1%). With the Marlin kernel in vLLM, AWQ reached 741 tok/s output throughput, about 1.6x the FP16 baseline (461 tok/s) and 10.9x the plain AWQ kernel, slightly ahead of Marlin-GPTQ’s 712 tok/s. If you’re on NVIDIA and quality retention matters, AWQ is the call.

GPTQ

GPTQ uses layer-by-layer Hessian-based weight compensation. It lost more accuracy than AWQ in the benchmark above, but it has broad toolchain support and many published checkpoints. ExLlama v2 and Marlin kernels push GPTQ decode speed well beyond the plain kernel. Throughput rankings between GPTQ and AWQ depend on the kernel, the model and the vLLM version, so measure on your own hardware.

The 24 GB sweet spot

At 24 GB VRAM (RTX 3090, RTX 4090): a 70B model at Q4_K_M needs ~42 GB, still too large for VRAM-only. Use either CPU-GPU split in GGUF or a smaller model (34B at AWQ 4-bit needs ~18 to 20 GB). At 34B and below, AWQ or GPTQ run fully in VRAM. A 70B at INT8 (~70 GB of weights) needs an 80 GB card or two 48 GB cards.

Consumer GPU format comparison:

Format Accuracy at 4-bit Peak throughput Hardware req. Toolchain
GGUF Q4_K_M HumanEval 51.8% 93 tok/s (vLLM) Any llama.cpp, Ollama
GGUF Q8_0 Near-lossless (8-bit) Moderate Any llama.cpp, Ollama
AWQ 4-bit HumanEval 51.8% 741 tok/s (Marlin) NVIDIA preferred vLLM, LMDeploy
GPTQ 4-bit HumanEval 45.7% 712 tok/s (Marlin) NVIDIA preferred vLLM, ExLlama v2
bitsandbytes NF4 HumanEval 51.8% 168 tok/s NVIDIA HuggingFace

Numbers are from JarvisLabs’ Qwen2.5-32B run on one H200 under vLLM (FP16 baseline: HumanEval 56.1%, 461 tok/s). GGUF throughput here is vLLM’s GGUF path, not llama.cpp; Q8_0 was not in that test.


Prosumer GPU (24-80 GB VRAM)

AWQ or GPTQ for 24-48 GB. FP8 W8A8 on Ada Lovelace (L40S at 48 GB, RTX 4090 at 24 GB), which has FP8 tensor cores. INT8 W8A8 or 4-bit weight-only for A100. FP8 W8A8 for H100. This tier is where format choice starts overlapping with serving infrastructure, you’re probably not running llama.cpp anymore.

At 48 GB (dual RTX 3090, A6000, L40S): AWQ 4-bit handles 70B models (~38 GB of weights) with a little room for KV cache. INT8 handles up to roughly 34B with better quality; a 70B at INT8 does not fit. For production multi-user inference, shift to vLLM with AWQ or INT8. vLLM’s continuous batching and paged KV cache handle many concurrent users better than llama.cpp’s server, which is built around a small number of parallel slots.

At 80 GB (A100 80 GB): A single A100 fits 70B at INT8 (~70 GB of weights, leaving little room for KV cache) or 34B at FP16 (~68 GB). This is where you move away from GGUF and toward dedicated serving stacks. A100 has no FP8 tensor cores. vLLM can load FP8 checkpoints on Ampere as weight-only W8A16 through its Marlin kernel, which saves memory but gives no FP8 compute speedup. For W8A8, use INT8, which runs natively on A100 tensor cores; well-tuned INT8 W8A8 loses about 1 to 3% accuracy in Red Hat’s evaluation.


Data center and cloud (A100/H100, cloud API)

A100 gets INT8 via vLLM. H100 gets FP8 (W8A8), which Red Hat’s evaluation found effectively lossless and vLLM’s docs credit with up to 1.6x throughput improvement. For long-context workloads, KV cache compression is a separate, stackable decision, don’t confuse it with weight quantization.

A100: INT8 is the answer

INT8 W8A8 on A100 needs calibration: SmoothQuant-style activation smoothing plus careful weight quantization. Done well, Red Hat’s large study (arXiv:2411.02355) measured 1 to 3% accuracy degradation. The same study found 4-bit weight-only (W4A16) competitive with 8-bit and the most cost-efficient choice for low-concurrency serving, while W8A8 wins under heavy continuous batching. Memory reduction for INT8: a 70B model goes from ~140 GB to ~70 GB, fitting on a single A100 80 GB with little room left for cache.

H100: FP8 is the answer

H100 has native FP8 Tensor Core support. The reason FP8 beats INT8 on quality isn’t mysterious: FP8’s exponential bit spacing handles outlier activations better than INT8’s linear range. NVIDIA’s TensorRT-LLM comparison reported H100 with FP8 at up to 4.6x the max throughput and 4.4x faster first-token latency of A100 at FP16, a figure that mixes the hardware upgrade with the precision change. In vLLM, either load a pre-quantized FP8 checkpoint or quantize at load time with the --quantization flag (fp8 in older releases; recent releases name variants such as fp8_per_tensor). FP8 W8A8 runs on Ada Lovelace (L40S, RTX 4090) as well as Hopper and Blackwell; Ampere only gets the weight-only W8A16 path.

KV cache compression is a separate problem

Weight quantization and KV cache compression solve different problems. Weight quantization reduces how much memory the model itself takes up. KV cache compression reduces the runtime memory consumed during inference as context length grows. They’re independent, and you can run both.

The baseline option is FP8 KV cache (--kv-cache-dtype fp8 in vLLM), which halves KV memory relative to BF16. Google’s TurboQuant (ICLR 2026) goes further: it quantizes the KV cache to around 3 bits with no measured accuracy loss on Google’s long-context benchmarks, and Google reports at least 6x lower KV memory and up to 8x faster attention-logit computation on H100 (that speed figure is for 4-bit keys against 32-bit keys). The approach: randomly rotate the vectors to simplify their geometry, quantize them with PolarQuant using most of the bit budget, then spend 1 bit on a QJL correction of the residual error. vLLM merged a TurboQuant KV cache backend in April 2026 with presets from about 2.6x to 4.9x compression. If you’re serving 32K+ context at any volume, test it on top of whatever weight quantization you chose. The TurboQuant post has the memory math.

MoE models in the data center

For Mixture-of-Experts models at data center scale, every expert stays resident in GPU memory, because any token can route to any expert. Per-token compute follows the active parameters, but memory follows the total. FP8 on the expert weights halves that footprint on Ada, Hopper and Blackwell. SSD expert streaming in the style of Flash-MoE is a single-user laptop technique; it does not deliver data center throughput. Flash-MoE’s own results also show the limit of going lower: at 4-bit tool calling works, at 2-bit it breaks.

Data center format comparison:

Format Hardware Accuracy loss Memory reduction Throughput gain
INT8 W8A8 (calibrated) Turing, Ampere and newer 1-3% ~50% Moderate
FP8 W8A8 Ada, Hopper, Blackwell Effectively lossless ~50% Up to 1.6x (vLLM docs)
FP8 KV cache Any vLLM GPU Small 2x KV cache Depends on backend
TurboQuant KV NVIDIA, vLLM backend No measured loss (Google) 2.6-4.9x KV (vLLM presets) Up to 8x attention logits (Google, 4-bit)
AWQ 4-bit (W4A16) Ampere and newer Small; task-dependent ~70% ~1.6x vs FP16 (Marlin, one benchmark)

Accuracy figures come from arXiv:2411.02355 and the JarvisLabs benchmark cited above; your model and tasks will differ.


The decision tree

Work top to bottom. Stop at the first branch that matches your situation.

START: What's your hardware?
│
├─ Edge / mobile (≤16 GB unified or phone)
│  ├─ Is the model BitNet-native (trained with ternary weights)?
│  │  ├─ Yes → BitNet TQ1_0 via bitnet.cpp or QVAC Fabric
│  │  └─ No  → GGUF Q4_K_M (llama.cpp, Ollama)
│
├─ 48 GB Apple Silicon, running a very large MoE?
│  └─ Yes → Flash-MoE 4-bit (NOT 2-bit, breaks tool calling)
│
├─ Consumer GPU (8-24 GB VRAM, NVIDIA)
│  ├─ Do you need CPU-GPU split or portability?
│  │  └─ Yes → GGUF Q4_K_M or Q8_0
│  ├─ Serving with vLLM on NVIDIA?
│  │  └─ Yes → AWQ 4-bit (Marlin kernel)
│  └─ Model only published as GPTQ, or ExLlama stack?
│     └─ Yes → GPTQ 4-bit (ExLlama v2 or Marlin kernel)
│
├─ Prosumer GPU (24-80 GB VRAM)
│  ├─ Ampere (A6000, A100 80 GB), serving users?
│  │  └─ Yes → INT8 W8A8 via vLLM (calibrated), or AWQ 4-bit
│  ├─ Ada Lovelace (L40S, RTX 4090)?
│  │  └─ Yes → FP8 W8A8
│  └─ Multi-GPU cluster?
│     └─ Yes → See data center tier below
│
├─ Data center (A100 / H100)
│  ├─ A100?
│  │  └─ INT8 W8A8 or 4-bit weight-only (FP8 only as weight-only W8A16)
│  ├─ H100 / H200 / Blackwell?
│  │  └─ FP8 W8A8 via vLLM (--quantization)
│  └─ Long-context serving (32K+ tokens)?
│     └─ + FP8 or TurboQuant KV cache (stacks on any weight quant)
│
└─ Cloud API
   └─ You don't control quantization. The provider does:
      - Open-model hosts often serve reduced precision (FP8 or lower);
        check the model page or ask, and evaluate on your tasks
      - Closed-model APIs: proprietary, no visibility
      See /ai-agents/0047-cost-management-for-agents/ for cost per token analysis

Mermaid flowchart

flowchart TD
    A[What hardware?] --> B[Edge/mobile ≤16GB]
    A --> B4[48GB Apple Silicon + large MoE]
    A --> C[Consumer GPU 8-24GB VRAM]
    A --> D[Data center A100/H100]
    A --> E[Cloud API]

    B --> B1{BitNet-native model?}
    B1 -->|Yes| B2[BitNet TQ1_0\nQVAC Fabric]
    B1 -->|No| B3[GGUF Q4_K_M\nllama.cpp / Ollama]
    B4 --> B5[Flash-MoE 4-bit]

    C --> C1{Need CPU-GPU split?}
    C1 -->|Yes| C2[GGUF Q4_K_M or Q8_0]
    C1 -->|No| C3{Checkpoint format?}
    C3 -->|AWQ available| C4[AWQ 4-bit\nMarlin kernel]
    C3 -->|GPTQ only| C5[GPTQ 4-bit\nExLlama v2 or Marlin]

    D --> D1{GPU type?}
    D1 -->|A100| D2[INT8 W8A8 or AWQ 4-bit\nvLLM]
    D1 -->|H100/H200| D3[FP8 W8A8\nvLLM --quantization]
    D2 --> D4{Long context?}
    D3 --> D4
    D4 -->|Yes 32K+| D5[+ FP8 or TurboQuant KV\n2x to ~5x KV reduction]

    E --> E1[Provider-managed\nSee cost analysis post]

Quick-reference comparison table

Format Bit-width Quality CPU NVIDIA GPU Apple Metal Primary use
GGUF Q4_K_M ~4.8 Small loss ✓ ✓ (offload) ✓ Edge, consumer, universal
GGUF Q8_0 ~8.5 Near-lossless ✓ ✓ (offload) ✓ Quality-critical, ≥16 GB
AWQ 4-bit 4 Small loss n/a ✓ (Marlin) n/a Consumer/prosumer NVIDIA
GPTQ 4-bit 4 Small to moderate loss n/a ✓ (ExLlama, Marlin) n/a NVIDIA, GPTQ checkpoints
BitNet TQ1_0 1.58 Native ✓ ✓ ✓ BitNet-trained models only
INT8 W8A8 8 1-3% loss n/a ✓ Turing+ n/a A100 production serving
FP8 W8A8 8 Effectively lossless n/a ✓ Ada, Hopper+ n/a H100 production serving
TurboQuant KV ~3-4 (KV) No measured loss n/a ✓ (vLLM backend) n/a KV cache at scale (stacks)
Flash-MoE 4-bit 4 (MoE) Production n/a n/a ✓ 48GB Large MoE on Apple Silicon

FAQ

What is the best quantization format for a 16 GB MacBook?

GGUF Q4_K_M for most models. BitNet (TQ1_0) for models trained natively in 1.58-bit. Flash-MoE with 4-bit for MoE models like Qwen3.5-397B if you have 48 GB.

Does AWQ actually outperform GPTQ?

In JarvisLabs’ vLLM benchmark of Qwen2.5-32B on an H200, AWQ scored 51.8% on HumanEval against GPTQ’s 45.7% (FP16 baseline 56.1%), and with Marlin kernels AWQ reached 741 tok/s against GPTQ’s 712 tok/s. Rankings shift with model, kernel and vLLM version, so benchmark your own model before choosing.

Can I use FP8 on my A100?

Not natively. A100 (Ampere) has no FP8 tensor cores. vLLM can still load FP8 checkpoints on Ampere as weight-only W8A16 through its Marlin kernel, which saves memory but does not give FP8 compute speedups. On A100 use INT8 W8A8 or 4-bit weight-only formats; reserve FP8 W8A8 for Ada Lovelace (RTX 4090, L40S), Hopper (H100, H200) and Blackwell.

What is TurboQuant and do I need it?

TurboQuant (Google, ICLR 2026) quantizes the KV cache to around 3 bits with no measured accuracy loss on Google’s long-context benchmarks, and Google reports at least 6x KV memory reduction. It targets inference serving, not weight compression. vLLM added a TurboQuant KV cache backend in April 2026. If you’re serving 32K+ context requests at scale, test it on top of your weight quantization choice.

Is BitNet only for Microsoft’s models?

BitNet b1.58 is a training-time technique: the model must be trained with ternary weights from the start. You can’t post-training-quantize a standard LLM into BitNet. Tether’s QVAC Fabric adds LoRA fine-tuning on-device for adaptation, but the base model still needs to be BitNet-native.