FlashAttention
On this page
A team raises its model's context from 4,000 to 32,000 tokens and training immediately runs out of GPU memory, even though the weights didn't grow. Profiling shows attention taking a large share of step time while the GPU's math units are mostly idle. The culprit is the attention score matrix: N × N numbers per head, written to and read back from GPU main memory several times. FlashAttention computes exactly the same attention without ever storing that matrix, by working in tiles that fit in fast on-chip memory.
How it works
Where standard attention loses time

A GPU has two kinds of memory that matter here:
- HBM, the main memory: tens of GB, around 3 TB/s on a current data-centre GPU.
- SRAM next to the compute units: a few hundred KB per streaming multiprocessor (tens of MB in total), roughly an order of magnitude faster.
Standard attention runs as separate steps: compute S = QKᵀ and write it to HBM, read it back for softmax and write P, read P again to multiply by V. For one head at 8,192 tokens in FP16, S alone is 134 MB; across 32 heads that's about 4.3 GB per layer, and it grows with N². The math is cheap compared with moving all those bytes, so attention is memory-bound (see Compute-Bound vs Memory-Bound).
Tiles in SRAM

FlashAttention fuses the steps into one kernel:
- Load a block of queries (say 128 rows) into SRAM.
- Stream blocks of K and V through SRAM one at a time. For each, compute that tile of scores, update the softmax statistics, and add the tile's contribution to an output accumulator.
- After the last block, write the finished output rows to HBM once.
HBM traffic drops to reading Q, K, V and writing O. Memory use grows linearly with sequence length instead of quadratically. For training, the backward pass needs the probabilities again; FlashAttention recomputes each tile from Q and K (saved per row: just the max and sum) rather than storing them. That is more arithmetic, but recomputing is faster than reading from HBM.
Online softmax

Softmax over a row needs that row's max and sum, which you only know after seeing every tile. The fix is to keep running values: when a new tile raises the max from m to m_new, multiply the sum and the accumulator so far by exp(m − m_new), then add the new tile. At the end, divide by the sum. The result is exact, not an approximation. The included flash_demo.py implements both versions in NumPy; on 1,024 tokens the outputs differ by less than 10⁻¹⁵.
Versions and relatives
- FlashAttention-2 improved how work is split across GPU threads; FlashAttention-3 uses newer GPU features (asynchronous copies, FP8) on Hopper-class hardware.
- Decode is different: one query token against a long KV cache. Kernels such as FlashDecoding split the cache along the sequence across many thread blocks so the whole GPU helps one long request; PagedAttention kernels read the cache through block tables.
Trade-offs and costs
- Hardware-specific kernels. Tile sizes are tuned per GPU generation; new hardware needs new kernel versions.
- Feature coverage. Unusual attention variants (custom masks, some biases) may not have fused kernels and fall back to slower paths.
- It doesn't change complexity of compute. Attention FLOPs are still O(N²); FlashAttention removes the memory bottleneck, not the math. Very long contexts still need other ideas (sliding windows, sparse or linear attention, context parallelism).
- Numerics. Results match up to floating-point rounding, but not bit-for-bit, because the order of additions differs.
In production
- You rarely call it directly: PyTorch's
scaled_dot_product_attention, Hugging Face Transformers and serving engines pick a fused kernel automatically when the inputs allow it. Check that the fast path is actually used (dtype FP16/BF16, supported head size, mask type). - In training, it is a large part of why long-context training is affordable: activation memory for attention no longer scales with N².
- In serving, prefill benefits most; decode relies on decode-specific kernels.
When to reach for it
Whenever attention is a large share of time or memory: long-context training and fine-tuning, long-prompt prefill, and any time someone proposes storing the full attention matrix. If a profile shows attention kernels as the top cost, check whether the fused path is being used before anything else.
Example
Fine-tuning at 16,384 tokens with an older eager attention implementation fails with out-of-memory at batch size 1. Per layer, the score and probability matrices for 32 heads in BF16 would take about 32 × 16,384² × 2 bytes ≈ 17 GB each, before counting the saved copies for backward. Switching to the fused kernel (PyTorch SDPA with the flash backend) removes those matrices; the run fits at batch size 4 on the same GPUs and step time drops by about a third. (Example numbers.)
Common mistakes
- Calling it approximate attention. It is exact; approximation methods are a different family.
- Saying it reduces FLOPs. It reduces memory traffic and memory use; it even adds recomputation.
- Confusing it with PagedAttention, which manages KV cache memory for serving.
- Assuming it's on. A mask type or dtype the kernel doesn't support silently falls back to the slow path.
Say this in the interview
"Standard attention writes the N × N score and probability matrices to HBM and reads them back, so it's memory-bound and its memory grows with N². FlashAttention fuses the whole computation into one kernel: it keeps a block of queries in on-chip SRAM, streams blocks of keys and values past it, and uses an online softmax with a running max and sum to get the exact result tile by tile. HBM traffic drops to reading Q, K, V and writing O, memory becomes linear in N, and the backward pass recomputes tiles instead of storing them. Same math, far fewer bytes moved."
Interview questions on this
- Why is naive attention memory-bound? It moves N² intermediate values through HBM several times while doing little math per byte.
- How can softmax be computed in tiles? Online softmax: keep a running max and sum, and rescale earlier partial results when the max changes.
- Why recompute in the backward pass instead of storing? Recomputing from SRAM-resident tiles costs less time than reading N² values from HBM, and saves memory.
- Is FlashAttention approximate? No; it computes the same result up to floating-point rounding order.
- What does it do for decode in serving? Little directly; decode uses split-KV kernels and paged caches.
- Does it make attention O(N)? Memory yes, compute no.
Used in
- AI Engineering: How a Transformer Works; Implement Multi-Head Attention; Compute-Bound vs Memory-Bound: The Roofline View; GPU Basics for AI Engineers; Training Memory Math; PagedAttention and KV Cache Memory Management.
- AI System Design: Design an Inference Platform.