Implement Multi-Head Attention

Free with account Updated Nov 16, 2026 · 5 min read
On this page

    Problem statement

    Implement the forward pass of causal multi-head self-attention in NumPy. Input x has shape (B, T, D): B sequences of T tokens with D features each. Given weight matrices Wq, Wk, Wv, Wo of shape (D, D) and a number of heads H that divides D, return an output of shape (B, T, D). Each token may attend only to itself and earlier tokens.

    The single most commonly reported ML coding task for LLM roles. The concept is short; the interview is about shapes, masking, scaling and testing your own code.

    Example

    Writing the shape next to each step is the easiest way to avoid bugs, and interviewers like seeing it.
    Writing the shape next to each step is the easiest way to avoid bugs, and interviewers like seeing it.
    x:  (B=2, T=5, D=8),  H=2  →  head dim = 4
    Q, K, V: (2, 5, 8) → split → (2, 2, 5, 4)
    scores:  (2, 2, 5, 5)   weights rows sum to 1, zero above the diagonal
    output:  (2, 5, 8)

    Constraints (assumed)

    • D divisible by H; T up to a few thousand in the interview setting.
    • NumPy only; no deep-learning library.
    • Forward pass only; no dropout, no biases.

    What this question tests

    • Correct shapes and axis order through reshape and transpose.
    • The scaling by √(head dim) and a numerically stable softmax.
    • The causal mask applied before softmax.
    • Testing: a reference implementation and a causality check.

    Questions to ask first

    1. Causal (decoder) or bidirectional (encoder)? (Causal here.)
    2. Is there a padding mask for sequences of different lengths? (Not required; a follow-up.)
    3. Biases in the projections? Dropout? (No.)
    4. Separate weight matrices or one fused (D, 3D) matrix? (Separate.)
    5. Should it return attention weights too? (Optional, useful for debugging.)