Implement Multi-Head Attention
On this page
Problem statement
Implement the forward pass of causal multi-head self-attention in NumPy. Input
xhas shape (B, T, D): B sequences of T tokens with D features each. Given weight matricesWq,Wk,Wv,Woof shape (D, D) and a number of heads H that divides D, return an output of shape (B, T, D). Each token may attend only to itself and earlier tokens.
The single most commonly reported ML coding task for LLM roles. The concept is short; the interview is about shapes, masking, scaling and testing your own code.
Example

x: (B=2, T=5, D=8), H=2 → head dim = 4
Q, K, V: (2, 5, 8) → split → (2, 2, 5, 4)
scores: (2, 2, 5, 5) weights rows sum to 1, zero above the diagonal
output: (2, 5, 8)
Constraints (assumed)
- D divisible by H; T up to a few thousand in the interview setting.
- NumPy only; no deep-learning library.
- Forward pass only; no dropout, no biases.
What this question tests
- Correct shapes and axis order through reshape and transpose.
- The scaling by √(head dim) and a numerically stable softmax.
- The causal mask applied before softmax.
- Testing: a reference implementation and a causality check.
Questions to ask first
- Causal (decoder) or bidirectional (encoder)? (Causal here.)
- Is there a padding mask for sequences of different lengths? (Not required; a follow-up.)
- Biases in the projections? Dropout? (No.)
- Separate weight matrices or one fused (D, 3D) matrix? (Separate.)
- Should it return attention weights too? (Optional, useful for debugging.)
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.