Numbers Every AI Engineer Should Know

Free with account Updated Nov 16, 2026 · 4 min read
On this page

    "How much GPU memory does that need?" "How long will the answer take?" "What does a million users cost?" Interviewers ask these to see whether you can reason from a few known numbers instead of guessing. You don't need exact figures. You need the right order of magnitude and the formula behind it, so you can say "about 140 GB, so at least two 80 GB GPUs, realistically four". These are the numbers worth knowing, as of late 2026. Hardware and prices change; the formulas don't.

    📏

    How to use these in an interview: state the assumption, do the arithmetic out loud, round generously, and say how you'd confirm it ("then I'd load-test"). Numbers here are rules of thumb, not vendor specifications.

    Five formulas

    Five formulas cover most sizing questions about memory, compute and speed.
    Five formulas cover most sizing questions about memory, compute and speed.

    Tokens and text

    NumberValueWhat it means
    English text per tokenabout 4 characters, about 0.75 words1,000 tokens ≈ 750 words ≈ 1.5 pages
    Other languagesoften more tokens per word than Englishthe same prompt can cost more in Chinese, Japanese or Hindi; measure with the real tokenizer
    Typical context window128k tokens common; some models offer 1M or morefitting is not the same as using well: long prompts are slower and costlier
    Human reading speedroughly 4–7 tokens per secondstreaming at 20+ tokens/s feels instant to a reader