Implement a LoRA Linear Layer
On this page
Problem statement
Implement a LoRA (low-rank adaptation) version of a linear layer. The original weight W and bias b are frozen. Add two small trainable matrices so the layer computes y = x·Wᵀ + b + (α/r)·x·Aᵀ·Bᵀ. Write the forward pass, the backward pass for A and B, and a function that merges the adapter back into one weight matrix. Use NumPy.
Example

layer: 4096 -> 4096, rank r = 8
full fine-tune: 4096 × 4096 = 16,777,216 trainable params
LoRA: 8 × (4096 + 4096) = 65,536 trainable params (about 0.4%)
test run (6 × 10 layer, r = 2, learning a rank-2 change):
step 0 loss 1.45707
step 1000 loss 0.00019
step 3000 loss 0.0
Constraints (assumed)
- x has shape (N, in); W has shape (out, in), as in PyTorch's Linear.
- r is small (1 to 64) and much less than min(in, out).
- α is a fixed hyperparameter; the scale α/r keeps the effective step size similar when you change r.
What this question tests
- Whether you know what LoRA freezes and what it trains, and why that saves memory.
- Whether you can derive the backward pass for a product of matrices without a framework.
- Whether you know the initialisation trick and why one matrix must start at zero (but not both).
- Whether you know the deployment options: merge into W, or serve many adapters on one base model.
Questions to ask first
- Do you want only the forward pass, or also gradients and a training step?
- Which layers get adapters? In practice it is usually the attention projections, sometimes the MLP too.
- Should merging be supported, or will adapters be swapped at serving time?
- Is dropout on the adapter input needed? Many implementations have it; it is optional here.
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.