Implement a LoRA Linear Layer

Free with account Updated Nov 16, 2026 · 5 min read
On this page

    Problem statement

    Implement a LoRA (low-rank adaptation) version of a linear layer. The original weight W and bias b are frozen. Add two small trainable matrices so the layer computes y = x·Wᵀ + b + (α/r)·x·Aᵀ·Bᵀ. Write the forward pass, the backward pass for A and B, and a function that merges the adapter back into one weight matrix. Use NumPy.

    Example

    The frozen path and the low-rank path run side by side and are added. For a 4096 × 4096 layer with r = 8, LoRA trains about 0.4% of the parameters.
    The frozen path and the low-rank path run side by side and are added. For a 4096 × 4096 layer with r = 8, LoRA trains about 0.4% of the parameters.
    layer: 4096 -> 4096, rank r = 8
    full fine-tune: 4096 × 4096       = 16,777,216 trainable params
    LoRA:           8 × (4096 + 4096) =     65,536 trainable params  (about 0.4%)
    
    test run (6 × 10 layer, r = 2, learning a rank-2 change):
    step 0    loss 1.45707
    step 1000 loss 0.00019
    step 3000 loss 0.0

    Constraints (assumed)

    • x has shape (N, in); W has shape (out, in), as in PyTorch's Linear.
    • r is small (1 to 64) and much less than min(in, out).
    • α is a fixed hyperparameter; the scale α/r keeps the effective step size similar when you change r.

    What this question tests

    • Whether you know what LoRA freezes and what it trains, and why that saves memory.
    • Whether you can derive the backward pass for a product of matrices without a framework.
    • Whether you know the initialisation trick and why one matrix must start at zero (but not both).
    • Whether you know the deployment options: merge into W, or serve many adapters on one base model.

    Questions to ask first

    • Do you want only the forward pass, or also gradients and a training step?
    • Which layers get adapters? In practice it is usually the attention projections, sometimes the MLP too.
    • Should merging be supported, or will adapters be swapped at serving time?
    • Is dropout on the adapter input needed? Many implementations have it; it is optional here.