9 minute read

“Matrix multiplication without multiplication: that is what ternary weights buy you, at least for the weight matrices.”

TL;DR

BitNet b1.58 restricts weights to {-1, 0, +1}, so the weight matrix multiplies become additions and subtractions. Microsoft estimates 71.4x lower arithmetic energy for those matmuls on 7nm, and the weights shrink about 10x versus FP16. Released ternary models top out around 2-10B parameters; no trained 100B ternary model exists, so the “100B on a CPU” figure is a speed test, not a usable model. ParoQuant (ICLR 2026) takes a different path: rotation-based INT4 post-training quantization that beats AWQ by 2.4% on reasoning tasks, with MLX support for Apple Silicon. The practical answer: 4-bit remains the production sweet spot today, and ternary weights are a candidate for edge and CPU workloads once larger trained models ship.

A kitchen scale weighing a single feather against a stack of heavy books

What changes when weights have only three values?

Standard LLM inference is dominated by matrix multiplication. Every token generated requires multiplying activation vectors against weight matrices with billions of floating-point entries. This is what GPUs are built for, and why inference is expensive.

BitNet b1.58 restricts every weight to one of three values: -1, 0, or +1. The “b1.58” refers to the information content: log₂(3) ≈ 1.58 bits per weight. This is not standard quantization; the model is trained with the constraint.

When a weight is -1, you subtract the activation. When it is +1, you add it. When it is 0, you skip it entirely. The weight matrix multiplications, which are most of the compute, need no multiplications at all. The rest of the model still does: attention between activations, normalization and the per-tensor scaling factors all multiply, so the forward pass is mostly, not entirely, additions.

The quantization method is absmean: divide weights by their mean absolute value, round to {-1, 0, +1}, and pair with 8-bit activations. Microsoft’s 2024 paper (arXiv:2402.17764) reported that at 3B parameters BitNet b1.58 matches an FP16 LLaMA of the same size in perplexity and end-task accuracy.

graph LR
    A[Standard LLM<br/>FP16 weights] -->|Matrix multiply| B[Multiple GPUs<br/>200 GB for 100B]
    C[BitNet b1.58<br/>Ternary weights] -->|Add/subtract for weight matmuls| D[CPU viable<br/>~20 GB for 100B]
    A -->|Matmul arithmetic energy| E[1x baseline]
    C -->|Matmul arithmetic energy| F[71.4x lower<br/>7nm estimate]

The energy numbers come from the arithmetic itself. Integer addition costs far less energy than a floating-point multiply-accumulate. The paper’s 71.4x figure is an estimate of the arithmetic energy for the matrix multiplications on 7nm chips, not an end-to-end measurement; memory traffic, which dominates real inference, shrinks by roughly the ratio of bits per weight instead.

BitNet.cpp (github.com/microsoft/BitNet) is the inference runtime. It runs ternary models with optimized CPU kernels. The repo reports 1.37-5.07x speedups on ARM and 2.37-6.17x on x86 over full-precision baselines, with energy reductions of 55-70% on ARM and 72-82% on x86.

Where is the 100B model everyone talks about?

No trained 100B ternary model has been released. Microsoft’s own native model, BitNet b1.58 2B4T, has about 2B parameters (listed as 2.4B in the bitnet.cpp table). The largest ternary models bitnet.cpp supports are community conversions around 8-10B parameters, such as Llama3-8B-1.58 and the 1.58-bit Falcon3 10B. Since no trained 100B ternary model exists, the README’s “100B on a single CPU at 5-7 tokens per second” result can only be a speed test with generated weights (the repo ships a script that builds dummy models of a given layout for benchmarking).

The memory math works: a 100B ternary model would need roughly 20 GB for weights (1.58 bits × 100 billion ÷ 8 bits per byte ≈ 19.75 GB). That fits in the RAM of a 32 GB laptop. The same model in FP16 needs 200 GB, which means several 80 GB GPUs.

But ternary does not work as a simple post-training step. A model has to be trained with the ternary constraint, either from scratch or through a long quantization-aware fine-tune: Hugging Face’s Llama3-8B-1.58 was converted from Llama 3 8B with about 100B tokens of further training. Doing that at 70B+ is expensive, and at the time of writing no one has published such a model.

The 2B model demonstrates that the architecture works and that the memory and speed gains are real at that size. It does not show that near-FP16 quality holds at 100B. That is an open question.

If you need to run large models on consumer hardware today, standard 4-bit quantization (GGUF via llama.cpp, or GPTQ/AWQ) applied to existing models is the practical choice; the quantization decision tree covers which method to use on which hardware. BitNet’s advantage materializes if trained ternary models at 70B+ ship.

What does ParoQuant do differently?

ParoQuant (arXiv:2511.10645, ICLR 2026, github.com/z-lab/paroquant) solves a different problem. Instead of training new models with extreme constraints, it takes existing FP16 models and quantizes their weights to INT4 after applying independent Givens rotations plus channel-wise scaling that suppress outlier values before rounding.

Standard quantization struggles with outliers. A few weights with unusually large magnitudes stretch the quantization grid, leaving the remaining weights with poor precision. AWQ protects the most important channels with activation-aware scaling; GPTQ compensates rounding error with second-order updates. Rotation-based methods spread the outliers out before rounding instead.

Givens rotations are orthogonal transforms: each one mixes a pair of dimensions without losing information, and the inverse rotation is folded back in at inference time. Apply enough pairwise rotations and the weight distribution evens out, the outliers shrink, and the tensor quantizes to INT4 with less error. Because the rotations are independent and pairwise, they are cheap to run on hardware.

The reasoning angle is the differentiator. The paper reports an average 2.4% accuracy improvement over AWQ on reasoning tasks, with less than 10% overhead. One plausible reason the gap shows up on reasoning: long chains of thought give small per-token errors more chances to compound.

Deployment support is broad: NVIDIA GPUs via vLLM and Hugging Face Transformers, and Apple Silicon via MLX. The project publishes pre-quantized checkpoints for several families, including Qwen3.5, Qwen3, Gemma 4 and Llama. Unlike BitNet, you can apply ParoQuant to an existing model without retraining.

What degrades first when you quantize?

Not all capabilities are equally robust to precision reduction. The clearest evidence for reasoning models comes from Liu et al., “Quantization Hurts Reasoning?” (2025), which tested DeepSeek-R1-Distill models from 1.5B to 70B and QwQ-32B on AIME, MATH-500, GPQA-Diamond and LiveCodeBench:

Setting Reported effect on reasoning benchmarks
8-bit weights and activations (W8A8) Near-lossless
4-bit weights, 16-bit activations (W4A16) Near-lossless
Lower bit widths Significant accuracy risk
Harder tasks and RL-trained models Larger drops than easier tasks
Aggressive low-bit on small models Longer outputs as well as lower accuracy

The pattern: harder, multi-step tasks degrade first. A model that summarizes well at 3 bits may already be failing at competition math. Choose your quantization method based on your task profile, and evaluate on the tasks you actually serve rather than on perplexity alone.

Emerging research pushes further. LittleBit (Samsung Research, NeurIPS 2025) reports results down to 0.1 bits per weight, compressing Llama2-13B to under 0.9 GB by factorizing weights into low-rank binary factors. AQLM operates at 2-3 bits and reports Pareto-optimal results below 3 bits. QuIP# targets 2 bits with incoherence processing and lattice codebooks.

When does each approach make sense?

The decision depends on what you are optimizing for.

Constraint Best choice Why
Production serving, quality matters 4-bit GPTQ/AWQ Best quality-speed tradeoff today
Apple Silicon, reasoning tasks ParoQuant INT4 + MLX Reasoning-focused INT4 with MLX support
Edge/IoT, no GPU available BitNet (small models today) CPU-only, minimal memory
Cost-sensitive batch inference BitNet on commodity CPUs, once larger models exist Cheap hardware and memory; no trained large model to price yet
Maximum quality, cost no object FP16/BF16 on GPU No quality loss
Research, pushing boundaries AQLM 2-bit or LittleBit sub-1-bit Pareto frontier exploration

The economics are not settled. With no trained ternary model at 70B+, any per-token cost comparison between BitNet on CPUs and 4-bit models on GPUs would be a guess. The direction is clear: ternary weights cut weight memory about 10x versus FP16 and let commodity CPUs hold much larger models, at a lower per-stream speed than a GPU.

The trade-off: latency. Even bitnet.cpp’s best-case 100B figure of 5-7 tokens per second works out to roughly 140-200 ms per token, while a GPU serving a 4-bit model is typically several times faster per stream. For interactive applications, that gap matters. For batch processing, document analysis and offline workloads, it often does not.

Key takeaways

  • Ternary weights remove multiplication from the weight matmuls. BitNet’s {-1, 0, +1} scheme turns those matmuls into additions and subtractions, with an estimated 71.4x lower arithmetic energy. Trained models exist up to roughly 10B; 100B remains a speed test with generated weights.
  • ParoQuant preserves reasoning at INT4. Pairwise Givens rotations smooth outliers before quantization, beating AWQ by 2.4% on reasoning tasks. Works with existing models, no retraining.
  • Hard tasks degrade first. Quantization hits multi-step reasoning before simpler language tasks, and the gap grows at lower bit widths. Evaluate on your own task mix.
  • 4-bit is still the sweet spot. For production serving today, GPTQ/AWQ at 4-bit gives the best quality-speed ratio. BitNet’s economics become compelling if larger ternary models ship.
  • CPU inference is a different product. Not a cheaper GPU but a different deployment model: offline batch processing, edge devices and privacy-sensitive workloads that cannot use cloud GPUs.

Further reading

Related posts