Implement Rotary Position Embeddings (RoPE)

Free with account Updated Nov 16, 2026 · 5 min read
On this page

    Problem statement

    Implement rotary position embeddings (RoPE). Given query and key tensors of shape (T, head_dim) and their token positions, return rotated versions so that the attention score between a query and a key depends on how far apart they are, not on where they sit in the sequence. Use NumPy.

    Example

    Each pair of dimensions is rotated by position × its own frequency. Early pairs spin fast, late pairs spin slowly.
    Each pair of dimensions is rotated by position × its own frequency. Early pairs spin fast, late pairs spin slowly.
    head_dim = 8, base = 10000
    inv_freq = [1, 0.1, 0.01, 0.001]      # one frequency per pair
    
    q at position 5,   k at position 2   -> score -1.209
    q at position 105, k at position 102 -> score -1.209   (same distance, same score)

    (The scores use random example vectors with seed 0; your numbers depend on the vectors.)

    Constraints (assumed)

    • head_dim is even; typical values are 64 or 128.
    • Positions are non-negative integers; during decoding they continue from the cache length.
    • Works on any leading batch and head dimensions: shape (..., T, head_dim).

    What this question tests

    • Whether you know why position information is needed at all (attention by itself ignores word order).
    • Whether you can turn "rotate pairs" into vectorised code without a Python loop over dimensions.
    • Whether you know where RoPE goes (Q and K, inside every attention layer) and the practical gotchas: pairing convention and decode positions.
    • Whether you can prove your code works with property tests instead of eyeballing numbers.

    Questions to ask first

    • Which pairing does the model use: adjacent pairs (x0, x1) or the two halves (x0, x4)? Both exist in real checkpoints.
    • Is it a fresh sequence, or decoding with a KV cache? That decides which positions to use.
    • Is the base fixed at 10000, or is this a long-context model with a scaled base?
    • Do we rotate the whole head or only part of it? Some models rotate only a fraction of the dimensions.