GPU Utilization and Cost
On this page
A finance review finds the company spent $1.4M on GPUs last quarter. The infrastructure dashboard says average GPU utilisation was 85%, so the GPUs look well used. A closer look tells another story: a third of the reserved GPUs sat in idle notebooks and over-sized reservations, the chatbot ran at small batch sizes overnight, and the "85%" came from a metric that counts a GPU as busy whenever any kernel is running. GPU cost is mostly a utilisation problem, and utilisation has to be measured at the right level.
How it works
Four meanings of "utilisation"

- Allocated: a job or person has reserved the GPU. Says nothing about use.
- Kernel-active ("GPU-Util" in
nvidia-smi): the share of time any kernel was running. A tiny kernel on one SM counts as 100%. - SM and tensor-core activity (DCGM profiling metrics such as SM active, tensor pipe active, DRAM bandwidth): how much of the chip actually worked.
- Useful work vs the hardware roof: MFU (model FLOPs utilisation) for training, achieved memory bandwidth or tokens per second against a target for decode (see Compute-Bound vs Memory-Bound).
Cost decisions should rest on levels 3–4 and on how much allocated time is busy at all.
Cost per token for serving

cost per token = $ per GPU-hour ÷ (tokens per second × 3,600 × fraction of time busy)
With an example price of $3 per GPU-hour and an 8B model (cost_calc.py):
- Batch 1 at 120 tokens/s: about $6.94 per million output tokens.
- Batch 64 at 4,000 tokens/s: about $0.21, because decode is memory-bound and batching is nearly free until the ridge.
- Batch 64 but busy only 30% of the day: about $0.69: idle hours triple the cost.
So the cheapest serving is high batch, high occupancy: continuous batching, enough KV cache room, autoscaling to follow demand, and sharing GPUs across models or tenants when traffic is low.
Cost for training
Training compute is about 6 × parameters × tokens FLOPs (2 for the forward pass, 4 for backward). GPU-hours = FLOPs ÷ (peak FLOPS × MFU × 3,600). A 7B model on 1 trillion tokens is 4.2 × 10²² FLOPs: about 29,500 GPU-hours on H100-class GPUs at 40% MFU, about 47,000 at 25%. MFU is the multiplier on the whole bill, which is why layout, kernels and failure recovery (see Checkpoints and Fault Tolerance) are cost work, not just performance work.
Trade-offs and costs

- Reserved vs on-demand vs spot: commitments are cheaper per hour but cost money when idle; spot is cheapest but preemptible, fine for checkpointed training, risky for serving.
- Big vs small GPUs: flagship GPUs win on throughput per dollar for big models and high load; an 8B model with low traffic may be cheaper on smaller or older GPUs, or a slice of one (MIG).
- Latency vs cost: larger batches lower cost per token but raise TPOT; set the SLO first, then maximise batch within it.
- Self-host vs API: at low or spiky volume, a hosted API's per-token price often beats paying for idle GPUs; at high steady volume, self-hosting can win. Do the occupancy math honestly.
In production
- Dashboards per team and per service: GPU-hours allocated, GPU-hours busy (by SM activity), cost per million tokens or per training run, MFU for training jobs.
- Policies: reclaim idle notebooks, cap reservation lengths, scale serving replicas down at night, move batch jobs to off-peak or spot.
- Tag every GPU workload with an owner and purpose so cost is attributed.
- Review the biggest spenders monthly: often a few jobs or services hold most of the waste.
When to reach for it
Budget reviews, choosing between hosting options, capacity planning, "how would you cut LLM cost 10×" questions, and any time someone quotes "GPU utilisation" as evidence.
Example
The company from the opening sets three targets: allocated-but-idle under 10%, serving at batch ≥ 32 during the day, and training MFU ≥ 35%. Actions: idle-notebook reclaim after 2 hours and 3-day reservation caps (frees 22 GPUs); continuous batching plus FP8 weights for the chatbot (cost per million tokens from $1.90 to $0.45); scale-to-two-replicas at night; checkpointed training moved to spot capacity at a lower hourly price. Quarterly spend falls to about $0.8M for more tokens served. (Example numbers.)
Common mistakes
- Trusting
nvidia-smiutilisation as a measure of efficiency. - Comparing GPUs by hourly price instead of cost per token or per training run.
- Sizing serving for peak 24/7.
- Ignoring MFU in training cost estimates.
- Reserving capacity "just in case" without tracking how much of it is used.
Say this in the interview
"I'd separate allocated, kernel-active, SM-active and useful-work utilisation;
nvidia-smiutilisation only says a kernel was running. For serving, cost per token is dollars per GPU-hour divided by tokens per second times busy time, so batch size and occupancy dominate: batch 1 can cost 30× more per token than batch 64. For training, FLOPs are about 6 × parameters × tokens, and MFU is the multiplier on GPU-hours. The levers are price model, right-sized GPUs, keeping them busy, doing more per GPU with batching and quantization, and doing less work with smaller models and caching."
Interview questions on this
- Why can
nvidia-smishow 100% while a GPU is mostly idle? It counts any kernel running, regardless of how much of the chip it uses. - Estimate GPU-hours to train a 13B model on 2T tokens at 40% MFU on 989-TFLOPS GPUs. 6 × 13e9 × 2e12 = 1.56e23 FLOPs ÷ (3.96e14 × 3,600) ≈ 110,000 GPU-hours.
- Why is batch size the biggest lever on inference cost? Decode is memory-bound; more requests share each weight read.
- When is a hosted API cheaper than self-hosting? Low or spiky volume where self-hosted GPUs would sit idle.
- How would you find waste in a shared GPU cluster? Compare allocation with SM activity per job and owner; look for idle reservations.
- What is MFU and why does it matter for cost? Useful FLOPs / peak; GPU-hours scale with 1/MFU.
Used in
- AI Engineering: Compute-Bound vs Memory-Bound: The Roofline View; GPU Job Scheduling and Queues; Continuous Batching; Quantization; Serving LLMs: Metrics, Autoscaling and Disaggregated Prefill; How Would You Cut LLM Cost by 10x?
- AI System Design: Design an Inference Platform; Design an LLM Gateway.