Implement Softmax and Cross-Entropy (Numerically Stable)

Free with account Updated Nov 16, 2026 · 5 min read
On this page

    Problem statement

    Implement softmax(logits) and the mean cross-entropy loss cross_entropy(logits, targets) for a batch, using NumPy. logits has shape (N, C): one row of raw scores per example, one column per class. targets has shape (N,) with the correct class index for each row. Both functions must stay correct for very large or very small logits. Bonus: return the gradient of the loss with respect to the logits.

    A warm-up that appears in ML coding rounds on its own or as the first step of a bigger task (a classifier, a language-model loss, attention). It's short, so interviewers look closely at numerical stability, shapes and the gradient.

    Example

    Softmax makes the scores positive and sum to 1; the loss is low when the correct class gets high probability.
    Softmax makes the scores positive and sum to 1; the loss is low when the correct class gets high probability.
    logits  = [[2.0, 1.0, 0.1]]   targets = [0]
    softmax → [[0.659, 0.242, 0.099]]
    loss    → 0.417                 (= -log 0.659)
    
    logits  = [[1000, 999, 998]]   targets = [2]
    softmax → [[0.665, 0.245, 0.090]]   (naive version returns nan)
    loss    → 2.4076

    Constraints (assumed)

    • 1 ≤ N ≤ 10⁵ rows, 2 ≤ C ≤ 10⁵ classes (language-model vocabularies are this large).
    • Logits are any finite float32/float64 values, possibly in the thousands.
    • Use NumPy; no deep-learning library.

    What this question tests

    • Knowing why the max shift and log-sum-exp are needed, not just that they exist.
    • Correct axis handling and broadcasting for batches.
    • Picking the right probabilities with fancy indexing.
    • Deriving or at least stating the gradient, and checking it numerically.

    Questions to ask first

    1. Mean or sum over the batch? (Here: mean.)
    2. Are targets class indices or one-hot vectors? (Here: indices.)
    3. Should I support label smoothing or class weights? (Not required; mention as a follow-up.)
    4. Which axis holds the classes? (Here: the last one.)
    5. Is a gradient needed, or just the forward pass?