How a Transformer Works: Attention Without the Math Fog
On this page
Older language models read a sentence one word at a time and squeezed everything they had read into a single running summary. By the end of a long paragraph, the start was mostly forgotten, and the model could not be trained in parallel because each step waited for the one before. Two problems, then: words need context to mean the right thing, and the model needs to look at all of that context at once.
The transformer solves both with one mechanism, attention. Almost every LLM you will use or be asked about is a decoder-only transformer, so this is the one architecture an AI engineer must be able to explain on a whiteboard.
How it works
The problem attention solves
Take the token "bank". In one sentence it is a riverside, in another a place that charges a fee. Its starting vector (its embedding) is the same in both, so the model needs a way to let "bank" pull in information from the words around it.

One attention step
Every token produces three vectors from its current state, each by a learned matrix multiplication:
- a query: what this token is looking for;
- a key: what this token offers to others;
- a value: the information it hands over if chosen.
For the token "bank", the model compares its query with every earlier token's key (a dot product), scales the scores by the square root of the vector size so they don't explode, and turns them into weights with softmax. The new vector for "bank" is the weighted sum of the values.

In matrix form, for all tokens at once: Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V. Because it is plain matrix multiplication over the whole sequence, a GPU computes it for every position in parallel. That is why transformers train so much faster than the older step-by-step models.
Multi-head attention runs several of these in parallel, each with its own smaller Q, K and V. One head may track grammar, another which noun a pronoun refers to. Their outputs are joined and mixed by one more matrix.
The full model

- Embedding. Each token ID becomes a vector. Position information is added (see Positional Encoding and RoPE), since attention by itself has no idea of word order.
- N identical layers, each with two parts:
- masked self-attention: tokens mix information, and a causal mask stops any token from looking at tokens after it, so the model cannot cheat by reading the answer;
- feed-forward network (MLP): each token's vector is processed on its own. Most of the parameters, and much of the stored "knowledge", live here.
Each part is wrapped in a residual connection (add the input back) and normalisation, which keeps deep stacks trainable.
- Output head. The final vector of the last position is turned into one score per vocabulary token. Softmax gives probabilities, and a sampling rule picks the next token (see How an LLM Picks the Next Token).
Generation is a loop: append the chosen token, run again, pick the next one.
Trade-offs and costs
- Attention cost grows with the square of the sequence length. Every token compares itself with every earlier token, so doubling the context roughly quadruples attention work during prompt processing. This is why long contexts are expensive and why kernels such as FlashAttention matter.
- Generation is sequential. Training sees all positions at once, but generating text produces one token per full pass through the model. Inference engineering is mostly about making that loop cheap.
- Memory for past tokens. To avoid recomputing keys and values for earlier tokens at every step, servers keep them in a KV cache, which grows with context length and batch size and often limits how many users one GPU can serve.
- Parameters versus layers. Wider and deeper models know more but cost more per token; most of the weights sit in the MLPs.
In production
- Latency splits into two phases: processing the prompt (all tokens in parallel, compute-heavy) and generating output (one token at a time, limited by memory bandwidth). Time to first token and tokens per second measure them.
- Cost scales with tokens processed and with model size: roughly two floating-point operations per parameter per token.
- Failure modes: context too long for the window; quality drop on information buried in the middle of a long prompt; out-of-memory errors when the KV cache for many long requests does not fit.
When to reach for this knowledge
You rarely build a transformer at work, but you reason about one all the time: why a long prompt is slow, why output speed is limited, why a smaller model with a better prompt can beat a bigger one, and why ML coding rounds ask you to implement attention.
Example
A request sends a 3,000-token prompt and gets back 200 tokens from a 7B-parameter model. The prompt pass runs all 3,000 tokens through the stack in parallel, then fills the KV cache. Each of the 200 output tokens then runs one position through all layers, reading every weight once per step, so generating 200 tokens takes 200 sequential passes. If the user complains the answer starts slowly, look at the prompt length; if it streams slowly, look at model size, batching and memory bandwidth.
Common mistakes
- Saying attention "remembers" earlier steps. It recomputes relations among all visible tokens on every pass; memory across requests needs the app to resend context.
- Forgetting the causal mask when implementing attention in a coding round, or applying it after the softmax instead of before.
- Forgetting the √d scaling, which makes softmax saturate and training unstable.
- Mixing up encoder and decoder models. BERT-style encoders see both directions and are used for embeddings and classification; GPT-style decoders generate.
- Claiming cost is linear in context. The attention part grows quadratically; the MLP part grows linearly.
Say this in the interview
"Each token makes a query, key and value. Its query is scored against the keys of earlier tokens, softmax turns the scores into weights, and the output is the weighted sum of values. Many heads do this in parallel, followed by an MLP, with residuals and normalisation, stacked N times. A causal mask keeps it left-to-right. Attention is quadratic in sequence length, generation is one token per pass, and the KV cache stores past keys and values so we don't recompute them."
Interview questions on this
- Why divide by √d? Dot products of long vectors have large variance; without scaling, softmax becomes nearly one-hot and gradients vanish.
- What does the causal mask do and where is it applied? It sets scores for future positions to minus infinity before softmax, so their weights are zero.
- Why multiple heads rather than one big head? Different heads can focus on different relations at the same total cost.
- Where are most of the parameters? In the feed-forward layers, typically about two thirds of each block.
- Why is generation slower per token than prompt processing? The prompt is processed in parallel; output tokens are produced one at a time, each pass reading all weights from memory.
Used in
- AI Engineering: Positional Encoding and RoPE; Prefill, Decode and the KV Cache; FlashAttention; Implement Multi-Head Attention (ML coding); Implement a KV Cache.
- AI System Design: Prefill vs decode and KV cache; Design an LLM Inference Platform.