Positional Encoding and RoPE

Updated Nov 16, 2026 · 5 min read
On this page

    "Dog bites man" and "man bites dog" contain exactly the same tokens. To attention, which compares every token with every other, the two sentences look identical: the operation treats its input as a set and has no idea which word came first. A language model that can't tell these apart is useless, so word order has to be added from outside. How it is added also decides how well a model copes with prompts longer than the ones it was trained on, which is why this topic shows up in both ML coding rounds and "how would you extend the context window?" questions.

    How it works

    Swap the words and attention, on its own, produces the same output. Position information fixes that.
    Swap the words and attention, on its own, produces the same output. Position information fixes that.

    Absolute positions: add a vector per slot

    The original transformer added a position vector to each token's embedding before the first layer: one vector for slot 0, another for slot 1, and so on.

    Each token's input becomes "what the token is" plus "where it is".
    Each token's input becomes "what the token is" plus "where it is".

    There are two common ways to make these vectors:

    • Learned: a table with one trainable vector per position, up to a maximum length. Simple, but positions beyond the table simply don't exist.
    • Sinusoidal: fixed sine and cosine waves of many frequencies. No training needed and defined for any position, though models still do poorly far past their training length.

    Both inject absolute position. But what usually matters for language is relative position: "the adjective right before this noun", "the question three sentences ago".

    RoPE: rotate instead of add

    Rotary Position Embedding (RoPE), used by most current open LLMs, takes another route. It leaves the embeddings alone and instead rotates the query and key vectors inside every attention layer:

    1. Split each query and key vector into pairs of numbers; treat each pair as a 2-D point.
    2. Rotate the pair by an angle equal to position × frequency. Each pair has its own frequency, from fast-spinning pairs (sensitive to nearby positions) to slow ones (sensitive to long distances).
    3. Compute attention scores as usual.
    Rotating both vectors by their own positions means the angle between them, and so their dot product, depends only on how far apart they are.
    Rotating both vectors by their own positions means the angle between them, and so their dot product, depends only on how far apart they are.

    The useful property: rotating the query by angle m·θ and the key by n·θ changes their dot product only through the difference (m − n)·θ. The attention score depends on relative distance, while each token still carries its absolute position. You can check it in a few lines:

    import math
    
    def rope(vec, pos, base=10000.0):
        """Rotate each pair (vec[2i], vec[2i+1]) by angle pos * base**(-2i/d)."""
        d = len(vec)
        out = []
        for i in range(0, d, 2):
            theta = pos * base ** (-i / d)
            x, y = vec[i], vec[i + 1]
            out += [x * math.cos(theta) - y * math.sin(theta),
                    x * math.sin(theta) + y * math.cos(theta)]
        return out
    
    def dot(a, b):
        return sum(x * y for x, y in zip(a, b))
    
    q, k = [0.3, -1.2, 0.8, 0.5], [1.0, 0.4, -0.6, 0.9]
    print(round(dot(rope(q, 5), rope(k, 3)), 6))      # 1.265584
    print(round(dot(rope(q, 105), rope(k, 103)), 6))  # 1.265584: same distance, same score

    Values (V) are not rotated, and nothing is added to the embeddings, so RoPE costs almost nothing extra to compute.

    Other approaches you may hear about

    • ALiBi: no position vectors at all; it subtracts a penalty from attention scores that grows with distance. Simple and extrapolates reasonably.
    • Relative position biases (as in T5): a learned bias per distance bucket, added to scores.

    Trade-offs and costs

    MethodStrengthWeakness
    Learned absoluteSimpleHard limit at the table size
    SinusoidalDefined for any positionAbsolute only; weak beyond training length
    RoPERelative distances, cheap, widely supportedQuality drops past the trained length without scaling tricks
    ALiBiExtrapolates well, trivialStrong bias toward nearby tokens

    In production

    • Context extension. Models with RoPE can often be stretched to longer contexts by rescaling the rotation frequencies (position interpolation, NTK-aware scaling, YaRN and similar), usually followed by a short fine-tune on long documents. Without that, quality falls apart past the trained length even if memory allows it.
    • Config must match. The RoPE base and scaling settings are part of the model; a serving engine using different values produces fluent-looking nonsense.
    • KV cache. Keys are stored after rotation, so cached keys are tied to their positions. Moving text to a new position (for example reusing a cached document at a different offset) needs care.
    • Failure modes: degraded answers on very long prompts; silent quality loss from a mismatched rope setting after a model or engine upgrade.

    When to reach for it

    You need this when choosing or fine-tuning a model for long documents, when debugging long-context quality, when an ML coding round asks you to implement RoPE, or when explaining why a model's "supported context" and its "good context" differ.

    Example

    A team has a model trained with an 8k-token context and needs 32k for contract review. Simply feeding 32k tokens produces garbled answers, because the slow-spinning RoPE pairs reach angles the model never saw. They apply a frequency-scaling method that maps the 32k range into angles similar to those seen in training, then fine-tune briefly on long contracts. Quality on long inputs recovers to close to the short-input level; they confirm it with a retrieval test that hides facts at different depths of long documents.

    Common mistakes

    • Saying attention "knows" word order by itself. It doesn't; position must be injected.
    • Rotating the value vectors as well as queries and keys.
    • Pairing the wrong dimensions when implementing RoPE (implementations differ: interleaved pairs vs first half with second half); match the model's convention.
    • Assuming a bigger context window in the config means the model works there. Check quality at long lengths.

    Say this in the interview

    "Attention is order-blind, so positions must be added. Early transformers added a learned or sinusoidal vector per slot. Most current LLMs use RoPE: they rotate each pair of query and key dimensions by an angle proportional to position, at several frequencies. Because both are rotated, their dot product depends only on the distance between them, which gives relative-position behaviour for almost no cost. Going beyond the trained length needs frequency scaling plus some long-context fine-tuning."

    Interview questions on this

    1. Why does RoPE give relative positions? Rotating q by mθ and k by nθ makes their dot product depend on (m − n)θ only.
    2. Why rotate q and k but not v? Position should affect who attends to whom (scores), not the content passed along.
    3. Why different frequencies for different pairs? Fast ones resolve nearby order, slow ones encode long distances without wrapping around too soon.
    4. How do you extend a RoPE model's context? Rescale frequencies (interpolation, NTK or YaRN-style) and fine-tune on long data; verify with long-context retrieval tests.
    5. What happens with learned absolute positions past the table size? There's no vector for those positions; the model can't run there without changes.

    Used in

    • AI Engineering: How a Transformer Works; Implement Rotary Position Embeddings (ML coding); Prefill, Decode and the KV Cache; Context Windows and Token Cost.
    • AI System Design: Design Enterprise RAG (long-document handling).