Gemma 4: the three architectural decisions that changed what a small model can do
TL;DR: Gemma 4 (Google DeepMind, April 2026, Apache 2.0) launched in four sizes: E2B and E4B for devices, a 26B MoE and a 31B dense model. Three design choices matter more than the benchmark table: interleaved sliding-window and global attention that keeps most of the KV cache fixed as context grows to 256K, a vision encoder with a per-image token budget you choose at serving time, and a 26B MoE that activates 3.8B parameters per token while staying within 3 points of the 31B dense model on MMLU-Pro, AIME 2026 and LiveCodeBench (Google’s reported numbers). Apache 2.0 removes the custom-license review that earlier Gemma releases needed.

Every major model release produces two kinds of coverage: benchmark tables and architectural analysis. The benchmark tables move fast and forget faster. The architectural analysis is where the durable lessons are.
Gemma 4’s benchmarks are strong for its size. According to Google, the 31B ranked third among open models on the Arena text leaderboard at launch and the 26B MoE sixth. But the benchmarks aren’t why ML systems teams should study this release. Three design decisions tell you something concrete about where open-weight models are heading.
The lineup, from Google’s model card:
| Model | Parameters | Layers | Context | Sliding window | Input |
|---|---|---|---|---|---|
| E2B | 2.3B effective (5.1B with embeddings) | 35 | 128K | 512 | Text, image, audio |
| E4B | 4.5B effective (8B with embeddings) | 42 | 128K | 512 | Text, image, audio |
| 26B A4B (MoE) | 25.2B total, 3.8B active | 30 | 256K | 1,024 | Text, image |
| 31B (dense) | 30.7B | 60 | 256K | 1,024 | Text, image |
A 12B “Unified” model with no separate vision or audio encoder followed in June 2026; this post covers the April lineup.
Decision 1: Hybrid attention for 256K context without a 256K cache in every layer
With modern attention kernels (FlashAttention and its successors), nobody materializes the full attention matrix, so the long-context costs that remain are compute that grows with the square of the sequence length and a KV cache that grows linearly with it, in every layer.
Gemma 4 attacks both with hybrid attention: most layers are local, a minority are global, and the final layer is always global.
Illustrative pattern (the Gemma 4 model card does not publish the exact ratio;
Gemma 3 used five local layers per global layer)
Layer 1: Local attention (1,024-token window)
Layer 2: Local attention (1,024-token window)
...
Layer k: Global attention (full 256K context)
...
Final layer: Global attention
Local layers attend only to a sliding window: 512 tokens on E2B and E4B, 1,024 on the 26B and 31B. Their KV cache never grows beyond the window, and their attention compute grows linearly with sequence length rather than quadratically.
Global layers attend to the full sequence. They’re a minority of layers, but they let information flow across the whole context. Gemma 4 uses standard RoPE on the sliding-window layers and proportional RoPE (p-RoPE) on the global layers, and its global layers use unified keys and values. Some variants also let the last layers reuse K/V tensors from earlier layers of the same type, which shrinks the cache further.
The serving consequence: at long context, only the global layers hold a full-length cache, so the KV memory per request is a fraction of what a full-attention model of the same depth would need. That is what makes 256K context practical on a single GPU or a large-memory workstation rather than a multi-GPU node.
The trade-off is that local layers cannot directly relate tokens that are far apart; that connection must route through the global layers. For tasks where long-range dependencies dominate (cross-document reasoning, very long-context RAG), test against a full-attention model before assuming parity.
Decision 2: Image cost as a serving knob
Vision-language models usually resize or crop images to a fixed resolution, which distorts screenshots and documents and fixes the number of image tokens per request.
Gemma 4’s vision encoder (about 550M parameters on the 26B and 31B, about 150M on E2B and E4B) keeps the original aspect ratio, using learned 2D positions and multidimensional RoPE. More useful for serving, it lets you choose the token budget per image: 70, 140, 280, 560 or 1,120 tokens.
That turns image cost into a per-request decision:
- 70 to 140 tokens per image for high-volume classification, moderation or visual search, where you process thousands of images and only need the gist.
- 560 to 1,120 tokens per image for documents, charts and screenshots, where layout and small text matter.
Since prefill cost and KV cache scale with token count, a 16x range in image tokens is a 16x range in the image share of each request’s cost. Pick the budget per route, not globally.
E2B and E4B add an audio encoder (about 300M parameters, the same base design as Gemma 3n) for speech input, which makes the small models usable for on-device voice features. The June 2026 12B model goes further and drops the separate encoders, projecting image patches and audio directly into the language model.
Google’s model card does not say whether the vision encoder was trained jointly with the language model or separately, so claims that Gemma 4 is “natively” multimodal in a deeper sense than other models are not something the documentation supports. Validate vision quality on your own data.
Decision 3: MoE at 26B with 3.8B active parameters
The 26B A4B is the first mixture-of-experts model in the Gemma family, and it shipped alongside a dense 31B trained for the same release, which makes for an unusually clean comparison.
The architecture: 25.2B total parameters, 128 experts per MoE layer with 8 routed experts plus 1 shared expert active per token, about 3.8B active parameters (15% of the total).
Token arrives at MoE layer
│
▼
┌────────────────────────┐
│ Router │
│ Selects 8 of 128 │
└────────┬───────────────┘
│
┌────┴────┐
▼ ▼
8 routed experts + 1 shared expert
│ │
└────┬────┘
│
▼
Weighted sum of expert outputs
At inference time, only the selected experts run for each token; the rest sit idle. GPU memory requirement: proportional to total parameters (you must store all experts). Compute requirement: roughly proportional to active parameters.
Google’s reported benchmarks for the instruction-tuned models:
| Model | MMLU-Pro | AIME 2026 (no tools) | LiveCodeBench v6 | GPQA Diamond | Active params |
|---|---|---|---|---|---|
| Gemma-4-31B dense | 85.2% | 89.2% | 80.0% | 84.3% | 30.7B |
| Gemma-4-26B A4B (MoE) | 82.6% | 88.3% | 77.1% | 82.3% | 3.8B |
| Gemma 3 27B (reference) | 67.6% | 20.8% | 29.1% | 42.4% | 27B |
The MoE reaches 97% of the dense model’s MMLU-Pro score and 99% of its AIME 2026 score with about 12% of its parameter count active per token. The gap is wider on agentic and competitive coding tasks: Google reports a Codeforces Elo of 1,718 for the MoE versus 2,150 for the dense 31B, and 68.2% versus 76.9% on Tau2.
For production, the 8x difference in active parameters is a ceiling, not a forecast. At small batch sizes, decoding is limited by memory bandwidth and the MoE reads far fewer weight bytes per token, so it is much faster per token. At large batch sizes, the batch as a whole touches most experts, attention and the shared expert don’t shrink, and expert routing adds overhead, so the throughput advantage narrows. Benchmark at your real batch size. And remember the memory side: the MoE needs nearly as much memory for weights as the dense model (25.2B versus 30.7B parameters), and its KV cache does not shrink at all; see KV cache for MoE.
The Apache 2.0 decision
Previous Gemma releases used Google’s Gemma Terms of Use: permissive for most uses, but not OSI-approved, and tied to a prohibited-use policy that legal teams had to review. Gemma 4 ships under Apache 2.0, with standard commercial use, modification and distribution rights.
The practical effect: companies that have already approved Apache 2.0 software can treat Gemma 4 the same way, without a model-specific license review. It also puts Gemma on the same footing as Qwen and many Mistral models, which already use Apache 2.0. Llama still uses Meta’s Llama Community License, which has its own acceptable-use policy and a clause for companies above 700 million monthly active users.
Deployment reality
Gemma 4 was available at launch on Hugging Face, Kaggle, Google AI Studio and Vertex AI, with launch-day support reported for Ollama, vLLM, llama.cpp (GGUF) and MLX.
Weight memory, computed from the parameter counts (KV cache, activations and the vision encoder come on top):
| Model | BF16 | 8-bit | 4-bit |
|---|---|---|---|
| 31B dense (30.7B) | about 61 GB | about 31 GB | about 15.5 GB |
| 26B A4B MoE (25.2B) | about 50 GB | about 25 GB | about 12.6 GB |
| E4B (8B with embeddings) | about 16 GB | about 8 GB | about 4 GB |
Real 4-bit files are a little larger because of quantization scales and layers kept at higher precision (a Q4_K_M GGUF averages close to 5 bits per weight). The 31B fits on one 80 GB H100 in BF16 but leaves under 20 GB for KV cache, so 8-bit is the practical single-GPU choice for long contexts. NVIDIA publishes an NVFP4 checkpoint of the 31B for Blackwell GPUs. For choosing between these formats, see the quantization decision tree.
Where Gemma 4 falls short
Every number above is Google’s own. Independent evaluations are the ones to trust for a production decision, and your own task-specific evals more than either.
Closed frontier models still lead on long multi-step reasoning and agentic work, and the gap shows up even inside the Gemma family: the dense 31B beats the MoE by a wide margin on Codeforces and Tau2, the benchmarks closest to agentic coding and tool use.
Multimodal performance: validate on your own data distribution before committing, especially for documents and charts, and test the token budget you plan to use, since quality at 70 tokens per image is not quality at 1,120.
For the KV cache and memory implications of serving MoE models, see KV cache for MoE.
Key takeaways
- Hybrid attention keeps most layers’ KV cache capped at a 512 or 1,024-token window, so only the global layers pay for the full 256K context.
- The vision encoder keeps aspect ratios and offers 70 to 1,120 tokens per image, which turns image cost into a per-route serving choice.
- The 26B MoE activates 3.8B parameters per token (8 of 128 experts plus a shared one) and scores within 3 points of the 31B dense on MMLU-Pro and AIME 2026, but trails it clearly on Codeforces and Tau2.
- At 4-bit the 31B needs about 15.5 GB for weights alone; the MoE saves compute, not much memory.
- Apache 2.0 removes the model-specific license review that earlier Gemma releases required.
FAQ
What are the Gemma 4 model variants? E2B (2.3B effective), E4B (4.5B effective), 26B A4B (25.2B MoE, 3.8B active) and 31B (30.7B dense), all Apache 2.0 and all multimodal; E2B and E4B also take audio. A 12B encoder-free model followed in June 2026.
What is hybrid attention in Gemma 4? Most layers use local sliding-window attention (512 tokens on E2B and E4B, 1,024 on the larger models), whose KV cache stays fixed as context grows. A minority of layers, including the final one, use global full-context attention. Sliding layers use standard RoPE; global layers use proportional RoPE.
How close is the 26B MoE to the 31B dense? 3.8B of 25.2B parameters are active per token. Google reports 82.6% versus 85.2% on MMLU-Pro and 88.3% versus 89.2% on AIME 2026, but a larger gap on Codeforces (1,718 versus 2,150 Elo) and Tau2 (68.2% versus 76.9%).
How does Gemma 4 handle images? A dedicated vision encoder (about 550M parameters on the larger models) that keeps the original aspect ratio, with a configurable budget of 70, 140, 280, 560 or 1,120 tokens per image.
What does Apache 2.0 licensing change? It replaces the Gemma Terms of Use with a standard OSI-approved license that most legal teams have already approved. Qwen and many Mistral models use Apache 2.0 too; Llama still uses Meta’s Llama Community License.
Further reading
- Gemma 4 model card: architecture details and benchmark tables
- Gemma 4 model page: Google DeepMind’s overview
- Hugging Face Gemma 4 blog post: deployment guide
- KV cache for MoE: what MoE inference costs in memory terms