Implement Rotary Position Embeddings (RoPE)
On this page
Problem statement
Implement rotary position embeddings (RoPE). Given query and key tensors of shape (T, head_dim) and their token positions, return rotated versions so that the attention score between a query and a key depends on how far apart they are, not on where they sit in the sequence. Use NumPy.
Example

head_dim = 8, base = 10000
inv_freq = [1, 0.1, 0.01, 0.001] # one frequency per pair
q at position 5, k at position 2 -> score -1.209
q at position 105, k at position 102 -> score -1.209 (same distance, same score)
(The scores use random example vectors with seed 0; your numbers depend on the vectors.)
Constraints (assumed)
- head_dim is even; typical values are 64 or 128.
- Positions are non-negative integers; during decoding they continue from the cache length.
- Works on any leading batch and head dimensions: shape (..., T, head_dim).
What this question tests
- Whether you know why position information is needed at all (attention by itself ignores word order).
- Whether you can turn "rotate pairs" into vectorised code without a Python loop over dimensions.
- Whether you know where RoPE goes (Q and K, inside every attention layer) and the practical gotchas: pairing convention and decode positions.
- Whether you can prove your code works with property tests instead of eyeballing numbers.
Questions to ask first
- Which pairing does the model use: adjacent pairs (x0, x1) or the two halves (x0, x4)? Both exist in real checkpoints.
- Is it a fresh sequence, or decoding with a KV cache? That decides which positions to use.
- Is the base fixed at 10000, or is this a long-context model with a scaled base?
- Do we rotate the whole head or only part of it? Some models rotate only a fraction of the dimensions.
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.