Implement Softmax and Cross-Entropy (Numerically Stable)
On this page
Problem statement
Implement
softmax(logits)and the mean cross-entropy losscross_entropy(logits, targets)for a batch, using NumPy.logitshas shape (N, C): one row of raw scores per example, one column per class.targetshas shape (N,) with the correct class index for each row. Both functions must stay correct for very large or very small logits. Bonus: return the gradient of the loss with respect to the logits.
A warm-up that appears in ML coding rounds on its own or as the first step of a bigger task (a classifier, a language-model loss, attention). It's short, so interviewers look closely at numerical stability, shapes and the gradient.
Example

logits = [[2.0, 1.0, 0.1]] targets = [0]
softmax → [[0.659, 0.242, 0.099]]
loss → 0.417 (= -log 0.659)
logits = [[1000, 999, 998]] targets = [2]
softmax → [[0.665, 0.245, 0.090]] (naive version returns nan)
loss → 2.4076
Constraints (assumed)
- 1 ≤ N ≤ 10⁵ rows, 2 ≤ C ≤ 10⁵ classes (language-model vocabularies are this large).
- Logits are any finite float32/float64 values, possibly in the thousands.
- Use NumPy; no deep-learning library.
What this question tests
- Knowing why the max shift and log-sum-exp are needed, not just that they exist.
- Correct axis handling and broadcasting for batches.
- Picking the right probabilities with fancy indexing.
- Deriving or at least stating the gradient, and checking it numerically.
Questions to ask first
- Mean or sum over the batch? (Here: mean.)
- Are targets class indices or one-hot vectors? (Here: indices.)
- Should I support label smoothing or class weights? (Not required; mention as a follow-up.)
- Which axis holds the classes? (Here: the last one.)
- Is a gradient needed, or just the forward pass?
Try it yourself first
You have the full problem. Sketch your answer, then sign up free to read the complete walkthrough.
- The complete design, flow by flow, with pictures
- Deep dives into the hard parts
- What mid, senior and staff candidates are expected to say
- Full company question lists
Free forever. No card needed.