ML System Design

Whether an AI product is fast and affordable often comes down to a few unglamorous choices: which model, at what precision, on which hardware, and how much memory goes to the KV cache.

At Samsung my team fit speech models onto phones, where every megabyte and millisecond counted, and I still look at large models the same way. Newer posts here cover LLM inference: quantization, KV cache, speculative decoding, serving fleets and running big models on your own hardware. Earlier posts work through classic ML system design: recommendations, ranking, data pipelines, monitoring and MLOps.

Start here

LLM inference and serving

Quantization and speculative decoding

Open models and local inference

LLM applications, NLP and evaluation

Infrastructure, scaling and cost

MLOps, data pipelines and monitoring

Training, evaluation and model development

Recommendations, search and ranking

Classification, detection and security

Choosing models, serving stacks or vendors for your own product? Arun works with teams as a contract engineer or fractional CTO, and runs technical due diligence for investors evaluating AI companies.

Recently added


Posts on new research summarize other people’s papers and link to the originals. Some posts are drafted with AI assistance and edited by Arun. If you spot an error, and it gets fixed.