GPU Utilization and Cost

Updated Nov 16, 2026 · 5 min read
On this page

    A finance review finds the company spent $1.4M on GPUs last quarter. The infrastructure dashboard says average GPU utilisation was 85%, so the GPUs look well used. A closer look tells another story: a third of the reserved GPUs sat in idle notebooks and over-sized reservations, the chatbot ran at small batch sizes overnight, and the "85%" came from a metric that counts a GPU as busy whenever any kernel is running. GPU cost is mostly a utilisation problem, and utilisation has to be measured at the right level.

    How it works

    Four meanings of "utilisation"

    Allocated, kernel-active, SM-active and useful work are very different numbers for the same job.
    Allocated, kernel-active, SM-active and useful work are very different numbers for the same job.
    1. Allocated: a job or person has reserved the GPU. Says nothing about use.
    2. Kernel-active ("GPU-Util" in nvidia-smi): the share of time any kernel was running. A tiny kernel on one SM counts as 100%.
    3. SM and tensor-core activity (DCGM profiling metrics such as SM active, tensor pipe active, DRAM bandwidth): how much of the chip actually worked.
    4. Useful work vs the hardware roof: MFU (model FLOPs utilisation) for training, achieved memory bandwidth or tokens per second against a target for decode (see Compute-Bound vs Memory-Bound).

    Cost decisions should rest on levels 3–4 and on how much allocated time is busy at all.

    Cost per token for serving

    The same GPU and model cost 30× more per token at batch size 1 than at batch 64, and idle hours multiply the cost again.
    The same GPU and model cost 30× more per token at batch size 1 than at batch 64, and idle hours multiply the cost again.

    cost per token = $ per GPU-hour ÷ (tokens per second × 3,600 × fraction of time busy)

    With an example price of $3 per GPU-hour and an 8B model (cost_calc.py):

    • Batch 1 at 120 tokens/s: about $6.94 per million output tokens.
    • Batch 64 at 4,000 tokens/s: about $0.21, because decode is memory-bound and batching is nearly free until the ridge.
    • Batch 64 but busy only 30% of the day: about $0.69: idle hours triple the cost.

    So the cheapest serving is high batch, high occupancy: continuous batching, enough KV cache room, autoscaling to follow demand, and sharing GPUs across models or tenants when traffic is low.

    Cost for training

    Training compute is about 6 × parameters × tokens FLOPs (2 for the forward pass, 4 for backward). GPU-hours = FLOPs ÷ (peak FLOPS × MFU × 3,600). A 7B model on 1 trillion tokens is 4.2 × 10²² FLOPs: about 29,500 GPU-hours on H100-class GPUs at 40% MFU, about 47,000 at 25%. MFU is the multiplier on the whole bill, which is why layout, kernels and failure recovery (see Checkpoints and Fault Tolerance) are cost work, not just performance work.

    Trade-offs and costs

    Price, right-sizing, occupancy, efficiency and doing less work are the five levers on the GPU bill.
    Price, right-sizing, occupancy, efficiency and doing less work are the five levers on the GPU bill.
    • Reserved vs on-demand vs spot: commitments are cheaper per hour but cost money when idle; spot is cheapest but preemptible, fine for checkpointed training, risky for serving.
    • Big vs small GPUs: flagship GPUs win on throughput per dollar for big models and high load; an 8B model with low traffic may be cheaper on smaller or older GPUs, or a slice of one (MIG).
    • Latency vs cost: larger batches lower cost per token but raise TPOT; set the SLO first, then maximise batch within it.
    • Self-host vs API: at low or spiky volume, a hosted API's per-token price often beats paying for idle GPUs; at high steady volume, self-hosting can win. Do the occupancy math honestly.

    In production

    • Dashboards per team and per service: GPU-hours allocated, GPU-hours busy (by SM activity), cost per million tokens or per training run, MFU for training jobs.
    • Policies: reclaim idle notebooks, cap reservation lengths, scale serving replicas down at night, move batch jobs to off-peak or spot.
    • Tag every GPU workload with an owner and purpose so cost is attributed.
    • Review the biggest spenders monthly: often a few jobs or services hold most of the waste.

    When to reach for it

    Budget reviews, choosing between hosting options, capacity planning, "how would you cut LLM cost 10×" questions, and any time someone quotes "GPU utilisation" as evidence.

    Example

    The company from the opening sets three targets: allocated-but-idle under 10%, serving at batch ≥ 32 during the day, and training MFU ≥ 35%. Actions: idle-notebook reclaim after 2 hours and 3-day reservation caps (frees 22 GPUs); continuous batching plus FP8 weights for the chatbot (cost per million tokens from $1.90 to $0.45); scale-to-two-replicas at night; checkpointed training moved to spot capacity at a lower hourly price. Quarterly spend falls to about $0.8M for more tokens served. (Example numbers.)

    Common mistakes

    • Trusting nvidia-smi utilisation as a measure of efficiency.
    • Comparing GPUs by hourly price instead of cost per token or per training run.
    • Sizing serving for peak 24/7.
    • Ignoring MFU in training cost estimates.
    • Reserving capacity "just in case" without tracking how much of it is used.

    Say this in the interview

    "I'd separate allocated, kernel-active, SM-active and useful-work utilisation; nvidia-smi utilisation only says a kernel was running. For serving, cost per token is dollars per GPU-hour divided by tokens per second times busy time, so batch size and occupancy dominate: batch 1 can cost 30× more per token than batch 64. For training, FLOPs are about 6 × parameters × tokens, and MFU is the multiplier on GPU-hours. The levers are price model, right-sized GPUs, keeping them busy, doing more per GPU with batching and quantization, and doing less work with smaller models and caching."

    Interview questions on this

    1. Why can nvidia-smi show 100% while a GPU is mostly idle? It counts any kernel running, regardless of how much of the chip it uses.
    2. Estimate GPU-hours to train a 13B model on 2T tokens at 40% MFU on 989-TFLOPS GPUs. 6 × 13e9 × 2e12 = 1.56e23 FLOPs ÷ (3.96e14 × 3,600) ≈ 110,000 GPU-hours.
    3. Why is batch size the biggest lever on inference cost? Decode is memory-bound; more requests share each weight read.
    4. When is a hosted API cheaper than self-hosting? Low or spiky volume where self-hosted GPUs would sit idle.
    5. How would you find waste in a shared GPU cluster? Compare allocation with SM activity per job and owner; look for idle reservations.
    6. What is MFU and why does it matter for cost? Useful FLOPs / peak; GPU-hours scale with 1/MFU.

    Used in

    • AI Engineering: Compute-Bound vs Memory-Bound: The Roofline View; GPU Job Scheduling and Queues; Continuous Batching; Quantization; Serving LLMs: Metrics, Autoscaling and Disaggregated Prefill; How Would You Cut LLM Cost by 10x?
    • AI System Design: Design an Inference Platform; Design an LLM Gateway.