AI Engineering Cheat Sheet

Free with account Updated Nov 16, 2026 · 3 min read
On this page

    The night before an AI engineering interview is not the time to learn attention from scratch. It is the time to have the key ideas in one place, each in a sentence you could say out loud, with a link back to the full article if something feels shaky. This page is that list, in the track's order.

    🧭

    The thread through every answer: what it costs (tokens, latency, GPUs), how you'd measure it (evals before and after), how it fails, and what you'd trade off. If an answer has all four, it's a strong answer.

    One request, end to end

    The path of one LLM request and the metric that matters at each step.
    The path of one LLM request and the metric that matters at each step.

    LLM foundations

    • Tokens: models read and bill in tokens, about 4 English characters each; cost and context limits are in tokens. Tokens and Tokenization
    • Attention: each token builds its output from a weighted mix of all earlier tokens; cost grows with the square of the sequence length. How a Transformer Works
    • Sampling: temperature reshapes probabilities; top-k and top-p cut the tail; greedy for extraction, sampling for variety. How an LLM Picks the Next Token
    • Context window: input + output must fit; long prompts cost more and raise latency. Context Windows and Token Cost
    • Embeddings: text → vector; similar meaning → nearby vectors; the basis of semantic search. Embeddings
    • RoPE: positions encoded by rotating query and key pairs, so attention depends on relative distance. Positional Encoding and RoPE
    • Training stages: pretraining gives knowledge; SFT gives instruction following; preference tuning shapes helpfulness and safety. How a Chat Model Is Made
    • Hallucination: the model produces plausible text, not checked facts; ground it, let it abstain, measure it. Why LLMs Hallucinate