ML System Design
Whether an AI product is fast and affordable often comes down to a few unglamorous choices: which model, at what precision, on which hardware, and how much memory goes to the KV cache.
At Samsung my team fit speech models onto phones, where every megabyte and millisecond counted, and I still look at large models the same way. Newer posts here cover LLM inference: quantization, KV cache, speculative decoding, serving fleets and running big models on your own hardware. Earlier posts work through classic ML system design: recommendations, ranking, data pipelines, monitoring and MLOps.
Start here
- The build vs buy decision for AI in 2026: a practical framework: when to build custom AI, buy an API or bring in outside help, with a data-moat test for telling them apart.
- The quantization decision tree: which method for which hardware in 2026: GGUF, AWQ, GPTQ, INT8 and FP8 matched to hardware tiers, with sourced accuracy and memory numbers.
- Mistral Small 4: one model, three jobs, and what ‘reasoning effort’ means for serving: what folding reasoning, vision and coding into one MoE model changes for routing, hardware and cost.
- Blink: CPU-free LLM inference via SmartNIC, up to 8.47x lower P99 TTFT: a research system read critically, including where its results stop applying.
- Speculative decoding in 2026: Saguaro, Nightjar, and the universal draft model: three recent approaches compared, with a table for picking one for your serving stack.
- The First Principles of Machine Learning: A Deep Dive into ML/DL Fundamentals: bias-variance, validation and leakage, regularization and evaluation as one reference.
LLM inference and serving
- Batch vs Real-Time Inference: Cost, Latency and Hybrid Serving
- Model Serving Architecture: REST vs gRPC, Batching, Shadow and Canary Deploys
- LLM Serving Infrastructure
- Precision Speed: 14 Techniques for High-Performance ML Inference
- KV cache for MoE: why sparse experts don’t shrink the memory bill
- The residual stream revelation: why KV cache may be theoretically unnecessary
- TurboQuant: 5x KV cache compression without quality loss
- vLLM’s semantic router: smarter inference for multi-model deployments
- The Workload-Router-Pool: how vLLM thinks about fleet inference
- Caching for ML Systems: Tiers, Eviction, Semantic and KV Caches
- CDN Design for ML Systems: Edge Caching, GeoDNS and Edge Inference
Quantization and speculative decoding
- Speculative decoding meets 4-bit quantization: the gains stack, but don’t multiply
- QuantSpec: when speculative decoding meets hierarchical KV cache quantization
- SDSL: speculative decoding finally has scaling laws, and they predict your draft model is too big
- 1-bit LLMs on consumer hardware: what ternary weights actually cost you
- NanoQuant: sub-1-bit quantization shrinks a 70B model from 138GB to 5.75GB
- Model Compression Techniques
Open models and local inference
- The Year of Open Reasoning: A Retrospective of 2025’s Major Model Releases
- Gemma 4: the three architectural decisions that changed what a small model can do
- Flash-MoE on MacBook: running 397B parameters on consumer hardware
LLM applications, NLP and evaluation
- Designing a Global-Scale Conversational AI: The Chatbot System Design
- RAG Systems
- Vector Databases
- Semantic Search Systems
- AI harness engineering: what the top labs get right
- Advanced NLP Pipelines at Scale
- Character-Level Language Models
- Beam Search Decoding
- Tokenization Systems
- Pattern Matching in ML
- Sequence Modeling in ML
Infrastructure, scaling and cost
- ML Capacity Planning and Infrastructure Scaling
- GPU Resource Allocation for ML: Bin-Packing, Fair Share, Preemption and Autoscaling
- Resource Partitioning in ML Clusters
- Distributed ML Systems: Replication, Sharding, Consensus and Fault Tolerance
- Distributed Training: Data, Tensor and Pipeline Parallelism, ZeRO and FSDP
- Model Replication Systems
- Cost Optimization for ML
MLOps, data pipelines and monitoring
- MLOps & LLMOps: The Production Playbook for Global-Scale AI
- Data Preprocessing Pipelines: Validation, Cleaning, Skew and Drift
- Data Validation
- Feature Engineering at Scale: Encodings, Feature Stores and Monitoring
- Feature Stores
- Data Augmentation Pipeline
- Batch Processing Pipelines
- Event Stream Processing
- ML Pipeline Dependencies & Orchestration
- DAG Pipeline Orchestration
- Experiment Tracking Systems
- Model Monitoring Systems
- Model Serialization Systems
- Online Learning Systems: Incremental Updates, Drift Detection and Rollback
Training, evaluation and model development
- Model Evaluation Metrics: Classification, Regression, Ranking and Calibration
- A/B Testing Platforms for ML: Assignment, Power Analysis, CUPED and Sequential Tests
- Model Ensembling
- Neural Architecture Search
- Hyperparameter Optimization
- Model Architecture Design
- Transfer Learning Systems
- Model Interpretability and Explainability (XAI)
- Federated Learning
- AutoML Systems
Recommendations, search and ranking
- Recommendation System: Candidate Retrieval
- Ranking Systems at Scale: Retrieval, Two-Tower Models, LTR and Re-Ranking
- Graph-based Recommendation Systems
- Real-time Personalization
- Trie-Based Search Systems (Typeahead)
- Knowledge Graph Systems
Classification, detection and security
- Production Classification Pipeline: Validation, Serving, Thresholds and Drift
- Hierarchical Classification Systems
- Designing a Malicious URL Classifier: Building the Immune System of the Web
- Anomaly Detection
- Clustering Systems
- Image Boundary Detection: From Canny Edges to U-Net Segmentation
Choosing models, serving stacks or vendors for your own product? Arun works with teams as a contract engineer or fractional CTO, and runs technical due diligence for investors evaluating AI companies.
Recently added
- SDSL: speculative decoding finally has scaling laws, and they predict your draft model is too big
- The build vs buy decision for AI in 2026: a practical framework
- Blink: CPU-free LLM inference via SmartNIC, up to 8.47x lower P99 TTFT
- QuantSpec: when speculative decoding meets hierarchical KV cache quantization
- NanoQuant: sub-1-bit quantization shrinks a 70B model from 138GB to 5.75GB
Posts on new research summarize other people’s papers and link to the originals. Some posts are drafted with AI assistance and edited by Arun. If you spot an error, let me know and it gets fixed.