AI Infra

  1. 1 Prefill, Decode and the KV Cache "Each request has two phases. Prefill runs the whole prompt in one parallel pass; it is compute-bound and sets TTFT. Decode makes one token per step and must read all the weights each step, so it's memory-bandwidth-bound and sets TPOT. The KV cache stores each token's keys and values so decode on... 6 min read
  2. 2 Continuous Batching "Decode is memory-bound, so batching shares each weight read across many requests. Static batching wastes that because outputs have very different lengths: slots go idle and new requests wait. Continuous batching schedules every decode step: it evicts finished requests and admits waiting ones if ... 5 min read
  3. 3 PagedAttention and KV Cache Memory Management "Without paging, the server reserves a contiguous KV cache slab for each request's maximum length, and since most requests stop early, 60–80% of cache memory sits unused. PagedAttention splits the cache into fixed blocks of, say, 16 tokens, allocates them on demand, and gives each request a block... 5 min read
  4. 4 Prefix Caching "Keys and values for a token depend only on the tokens before it, so identical prompt prefixes have identical KV caches. With a paged cache, the server hashes each block chained with its predecessor, reuses the longest matching run of blocks, and prefills only the rest. It cuts prefill compute an... 5 min read
  5. 5 Quantization: INT8, INT4 and FP8 "Quantization stores numbers in fewer bits, a scale times a small integer. For serving, the main win is memory: a 70B model goes from 140 GB in FP16 to 70 GB in FP8 and 35 GB in INT4, so it fits on fewer GPUs, and memory-bound decode reads fewer bytes per token. The hard part is outliers, handled... 6 min read
  6. 6 Speculative Decoding "Decode is memory-bound, so one target pass can check several tokens almost as cheaply as it generates one. A small draft model, extra heads or n-gram lookup propose k tokens; the target verifies them in one pass and keeps the prefix it agrees with, plus one token of its own. With the acceptance ... 5 min read
  7. 7 FlashAttention "Standard attention writes the N × N score and probability matrices to HBM and reads them back, so it's memory-bound and its memory grows with N². FlashAttention fuses the whole computation into one kernel: it keeps a block of queries in on-chip SRAM, streams blocks of keys and values past it, an... 5 min read
  8. 8 Serving LLMs: Metrics, Autoscaling and Disaggregated Prefill "I'd measure TTFT and TPOT at p95, tokens per second per GPU, queue depth, KV cache usage and goodput against the SLO, not GPU utilisation. Replicas take minutes to start because of node provisioning and loading 100+ GB of weights, so I'd autoscale on queue depth or KV usage, keep warm spares, pr... 5 min read
  9. 9 Parallelism for Training: Data, Tensor, Pipeline "Data parallelism copies the model and splits the batch, with a gradient all-reduce each step; it scales throughput but not memory unless it's sharded like ZeRO or FSDP. Tensor parallelism splits each layer's matrices and talks in every layer, so it stays inside an NVLink server. Pipeline paralle... 5 min read
  10. 10 ZeRO and FSDP: Sharding Training State "With Adam and mixed precision, training state is about 16 bytes per parameter, and plain data parallelism duplicates it on every GPU. ZeRO shards it across the data-parallel group: stage 1 the optimizer state, stage 2 the gradients, stage 3 the weights too, so memory per GPU drops to roughly 16P... 5 min read
  11. 11 Training Memory Math: Weights, Gradients, Optimizer, Activations "With mixed precision and Adam, training needs about 16 bytes per parameter: 2 for BF16 weights, 2 for gradients and 12 for the optimizer, meaning an FP32 master copy plus two Adam moments. So a 7B model needs about 110 GB before activations, versus 14 GB to serve it. Activations scale with seque... 5 min read
  12. 12 Mixed Precision Training "Mixed precision runs forward and backward in 16-bit on tensor cores, which is several times faster and halves activation memory, but keeps FP32 master weights and optimizer state, so small updates aren't rounded away, and keeps sensitive ops like softmax and normalization in FP32. FP16 has a sma... 5 min read
  13. 13 Checkpoints and Fault Tolerance in Long Training Runs "In synchronous training one failed GPU stops the job, and at thousands of GPUs you see failures every few hours, so I'd optimise goodput. A checkpoint has to hold weights, optimizer state, scheduler, data loader position and RNG state, sharded and written by every rank in parallel. I'd make save... 5 min read
  14. 14 GPU Basics for AI Engineers "A GPU has 100+ SMs with many simple cores and tensor cores for matrix tiles, so it suits neural networks, which are mostly matrix multiplies. Data moves through a hierarchy: SRAM on each SM, an L2 cache, then HBM at a few TB/s, NVLink between GPUs in a server, and slower PCIe and network beyond.... 5 min read
  15. 15 Compute-Bound vs Memory-Bound: The Roofline View "Arithmetic intensity is FLOPs per byte moved. Attainable speed is the minimum of peak FLOPS and intensity times bandwidth; on an H100-class GPU the ridge is around 300 FLOPs per byte. Decode at batch 1 has intensity about 1, so it's memory-bound and runs at a fraction of a percent of peak by des... 5 min read
  16. 16 GPU Interconnects and Collectives: NVLink, InfiniBand, All-Reduce "There are two networks: NVLink inside a server, about 900 GB/s per GPU, and InfiniBand or RoCE between servers, about 50 GB/s per GPU with RDMA. Training uses collectives (all-reduce for data-parallel gradients and tensor-parallel outputs, reduce-scatter and all-gather for FSDP, all-to-all for M... 5 min read
  17. 17 GPU Job Scheduling and Queues "Distributed jobs need gang scheduling: admit a job only when all its GPUs can be placed, otherwise jobs deadlock on partial allocations. I'd bin-pack small jobs to keep whole nodes free and place large jobs topology-aware, inside one NVLink server or under one switch. Teams get guaranteed quotas... 5 min read
  18. 18 GPU Utilization and Cost "I'd separate allocated, kernel-active, SM-active and useful-work utilisation; nvidia-smi utilisation only says a kernel was running. For serving, cost per token is dollars per GPU-hour divided by tokens per second times busy time, so batch size and occupancy dominate: batch 1 can cost 30× more p... 5 min read
  19. 19 Building a Training Data Pipeline: Dedup, Filtering, Mixing "I'd build it as versioned batch stages: extract clean text, filter by language and quality with heuristics plus a small quality classifier, scrub toxicity and PII, remove exact duplicates by hash and near-duplicates with MinHash and LSH, decontaminate against eval sets, then mix sources by weigh... 5 min read
  20. 20 Document Ingestion for RAG: Parsing, Cleaning, Updating "Ingestion is a pipeline: connectors with change feeds and permissions, layout-aware parsing that keeps reading order, tables and headings, cleaning and de-duplication with one authoritative copy, structure-aware chunks prefixed with their title path and rich metadata, then embedding and indexing... 5 min read
  21. 21 Synthetic Data for LLMs "I treat synthetic data as a pipeline: diverse seeds (real examples, a topic taxonomy, personas, our documents), generation grounded in those sources, then verification (unit tests, exact answers, schema checks, or a calibrated LLM judge), rejection sampling, dedup and diversity checks. I blend i... 5 min read
  22. 22 Labeling and Preference Data "Label quality comes from the program: clear guidelines with edge cases, a calibration round, gold items and overlap in production, adjudication by experts, and versioned labels with rater IDs. For preference data I'd collect pairwise comparisons with randomised order, ties and reasons, measure a... 5 min read
  23. 23 LLM Observability and Tracing "I'd trace every request end to end, with a span per step: guardrails, retrieval, each model call, each tool call. LLM spans record model and prompt versions, tokens, cost, TTFT and finish reason; retrieval and tool spans record what came back. On top I'd track operational metrics, quality signal... 5 min read
  24. 24 Versioning Prompts and Models, and Rolling Them Out "I'd treat everything that shapes behaviour (prompt, pinned model snapshot, parameters, tool schemas, retrieval config, guardrails) as one immutable, versioned release in a registry, with eval results attached and the release ID on every trace. Rollout goes offline evals, shadow, a sticky canary ... 5 min read
  25. 25 Online Evaluation and A/B Tests for LLM Features "Offline evals gate releases; online evaluation measures real impact. I'd combine explicit feedback, implicit behaviour like copying, editing or retrying, a calibrated LLM judge on a sample of traffic, and business outcomes. Changes go through an A/B test randomised by user, with one primary metr... 5 min read
  26. 26 Model Routing and Fallbacks "I'd put routing in an LLM gateway behind the API gateway. Routing uses rules first (feature, tenant, region), then a small classifier or a cascade that tries a cheap model and escalates when a verifier or judge rejects the answer, within budget and latency limits. Fallbacks go primary, one jitte... 5 min read