LLM Foundations

  1. 1 Tokens and Tokenization "The model works on tokens, not words. Most tokenizers are BPE-style: a learned vocabulary of sub-word pieces, so common words are one token and rare ones split. That matters because price, context limit and latency are all counted in tokens, and the ratio changes with language and format. So I'd... 6 min read
  2. 2 How a Transformer Works: Attention Without the Math Fog "Each token makes a query, key and value. Its query is scored against the keys of earlier tokens, softmax turns the scores into weights, and the output is the weighted sum of values. Many heads do this in parallel, followed by an MLP, with residuals and normalisation, stacked N times. A causal ma... 5 min read
  3. 3 How an LLM Picks the Next Token: Temperature, Top-k, Top-p "The model gives a probability for every token; decoding chooses one. Greedy is deterministic but loops on long text. Sampling with temperature divides the logits by T: lower is safer, higher is more varied. Top-k or top-p trims the unlikely tail before sampling; top-p adapts to how confident the... 5 min read
  4. 4 Context Windows and Token Cost "The model is stateless, so each call must carry everything: system prompt, history, retrieved text and the new message, plus room for the answer, inside the context window. Because history is resent, cost per conversation grows faster than linearly. I'd count tokens before each call, cap output,... 5 min read
  5. 5 Embeddings: Turning Text into Vectors "An embedding model maps text to a vector so that similar meanings point in similar directions. It's trained contrastively on matching pairs. For search, I embed document chunks offline, embed the query with the same model online, and find nearest neighbours by cosine similarity using an ANN inde... 5 min read
  6. 6 Positional Encoding and RoPE "Attention is order-blind, so positions must be added. Early transformers added a learned or sinusoidal vector per slot. Most current LLMs use RoPE: they rotate each pair of query and key dimensions by an angle proportional to position, at several frequencies. Because both are rotated, their dot ... 5 min read
  7. 7 How a Chat Model Is Made: Pretraining, SFT and RLHF "Pretraining on trillions of tokens with next-token prediction gives a base model its knowledge. Supervised fine-tuning on instruction-response pairs teaches it to follow instructions and formats. Preference tuning, with RLHF via a reward model or DPO directly on preferred-vs-rejected pairs, shap... 5 min read
  8. 8 Why LLMs Hallucinate "LLMs are trained to produce likely text, not verified text, so when knowledge is missing they fill the gap with something plausible. Causes are the objective itself, knowledge gaps such as cutoff or private data, sampling, and bad or ignored context. I'd ground answers with retrieval, allow abst... 4 min read
  9. 9 Choosing a Model: Size, Open vs Closed, Cost "I'd start from constraints: can data go to a vendor, what latency and cost per request we can afford. Then I'd build a small eval set from real inputs and test a few candidates, picking the cheapest that meets the quality bar. If only part of the traffic is hard, I'd route: small model by defaul... 4 min read
  10. 10 Multimodal Models: Images, Audio and Text Together "Most multimodal LLMs add a vision encoder and a projector that turn image patches into tokens in the LLM's embedding space, so the transformer attends over image and text tokens together; audio works the same way with an audio encoder. Images cost tokens in proportion to resolution, so I'd crop ... 4 min read