GPU Job Scheduling and Queues

Updated Nov 16, 2026 · 5 min read
On this page

    A company buys 64 GPUs and shares them between eight teams on a first-come, first-served basis. Within a month: one research job holds 32 GPUs for three weeks; the production inference service can't scale up during a launch because the cluster is full of experiments; two 16-GPU training jobs each grab half of what they need and sit deadlocked; and the dashboard shows 30 free GPUs while an 8-GPU job waits for hours. GPUs are too expensive to leave idle and too contested to hand out freely, so a GPU cluster needs a scheduler that understands gangs, quotas, priorities and topology.

    How it works

    Gang scheduling

    Without all-or-nothing admission, two jobs can each hold part of what they need and both wait forever.
    Without all-or-nothing admission, two jobs can each hold part of what they need and both wait forever.

    A distributed training job with 16 GPUs can't start with 12: every rank must join the first collective. A plain Kubernetes scheduler places pods one at a time, so two such jobs can each take part of the free GPUs and deadlock. Gang scheduling admits a job only when all its pods can be placed at once. On Kubernetes this comes from a batch layer (Kueue, Volcano, or commercial schedulers); HPC clusters use Slurm, which has always allocated nodes per job.

    Placement: packing and topology

    The same 32 busy GPUs either scattered over all nodes or packed into four, leaving four whole nodes free.
    The same 32 busy GPUs either scattered over all nodes or packed into four, leaving four whole nodes free.
    • Bin-packing: place small jobs on already-busy nodes first, so whole nodes stay free for large jobs. Spreading small jobs across nodes creates fragmentation: lots of free GPUs, none usable by an 8-GPU job.
    • Topology awareness: an 8-GPU job should land on one NVLink server; a 64-GPU job on servers under the same network switches. Bad placement can cost more throughput than a slower GPU would (see GPU Interconnects and Collectives).
    • Whole GPUs vs slices: Kubernetes exposes GPUs as whole units by default. For small workloads, NVIDIA's MIG splits one data-centre GPU into isolated slices (up to 7 on A100/H100-class cards), and time-slicing or MPS shares one GPU between processes with weaker isolation.

    Quotas, borrowing and preemption

    Each team has a guaranteed quota; idle quota can be borrowed, and borrowed GPUs are taken back when the owner needs them.
    Each team has a guaranteed quota; idle quota can be borrowed, and borrowed GPUs are taken back when the owner needs them.
    • Guaranteed quota per team or project (for example 24 GPUs for production inference).
    • Borrowing: idle quota is lent to other teams' queued jobs, so the cluster stays busy.
    • Preemption: when the owner needs its GPUs back, the scheduler evicts borrowed jobs, lowest priority first. Preemptible jobs must checkpoint and resume automatically (see Checkpoints and Fault Tolerance).
    • Priority classes: production inference > interactive notebooks > scheduled training > experiments. Within a class, fair share (recent usage counts against you) keeps one user from monopolising the queue.
    • Limits: maximum GPUs per job and maximum runtime, so nothing holds 32 GPUs for three weeks unannounced.

    Queues

    Jobs wait in queues per team or per class. A good queue shows its position, an estimated start time, and why a job is waiting (quota, capacity, topology). Small jobs can backfill gaps in front of a large job if they finish before the large job's resources free up.

    Trade-offs and costs

    • Utilisation vs latency to start: packing tightly and lending quota keeps GPUs busy, but owners wait for preemptions to finish when they come back.
    • Preemption wastes work since the last checkpoint; frequent preemption needs cheap, frequent checkpoints.
    • Inference and training on one pool improves utilisation but risks production latency; many teams give inference a reserved pool and let training borrow only its spare.
    • GPU slicing helps notebooks and small models but complicates capacity planning and isolation.
    • Complexity: quotas and priorities become organisational policy; write them down.

    In production

    • Common stacks: Kubernetes + NVIDIA GPU Operator + a batch layer (Kueue or Volcano) for training; Slurm in classic HPC clusters; hosted platforms and commercial GPU schedulers in between.
    • Track per team: GPUs allocated vs actually busy (a job can hold 8 GPUs and use one), queue wait time by job size, preemptions, and idle reserved capacity.
    • Reclaim idle allocations automatically: notebooks idle for N hours, jobs with near-zero GPU activity.
    • Separate failure handling: health checks drain bad nodes so the scheduler stops placing jobs there.

    When to reach for it

    Any shared GPU cluster with more than one team or more than one kind of workload, and "design a GPU platform / training platform" interviews. A single team on a few GPUs can live without it.

    Example

    The 64-GPU cluster from the opening, redesigned: 24 GPUs reserved for production inference (it can burst into the shared pool with top priority); 40 GPUs shared by research and fine-tuning with quotas of 24 and 16; borrowing allowed, borrowed GPUs preemptible after a 10-minute grace period to checkpoint; gang scheduling and bin-packing on; a 7-day runtime cap and idle-notebook reclaim after 4 hours. Reported results after a month: allocated-but-idle GPUs fall from 35% to 9%, median wait for 8-GPU jobs falls from 5 hours to 40 minutes, and inference scaled through a launch without paging anyone. (Example numbers.)

    Common mistakes

    • Pod-by-pod scheduling for distributed jobs (no gang scheduling), causing partial allocations and deadlocks.
    • Spreading small jobs, fragmenting nodes.
    • Ignoring topology, so jobs span distant switches.
    • Preempting jobs that can't resume.
    • Measuring allocation instead of use: a reserved GPU isn't a busy GPU.

    Say this in the interview

    "Distributed jobs need gang scheduling: admit a job only when all its GPUs can be placed, otherwise jobs deadlock on partial allocations. I'd bin-pack small jobs to keep whole nodes free and place large jobs topology-aware, inside one NVLink server or under one switch. Teams get guaranteed quotas, can borrow idle capacity, and borrowed GPUs are preemptible with checkpointing; priority classes put production inference first. On Kubernetes that's the GPU operator plus a batch layer like Kueue or Volcano; I'd track queue wait, allocated-versus-used GPUs and preemptions."

    Interview questions on this

    1. Why can't the default Kubernetes scheduler run multi-node training well? It schedules pods individually; there's no all-or-nothing admission.
    2. What is GPU fragmentation and how do you reduce it? Free GPUs scattered across nodes; bin-pack small jobs and reserve whole nodes for large ones.
    3. How do you share GPUs between inference and training safely? Reserved inference capacity with top priority; training borrows spare capacity and is preemptible.
    4. What do jobs need to tolerate preemption? Frequent checkpoints and automatic resume.
    5. When would you use MIG? Many small workloads (notebooks, small models) that need isolation on big GPUs.
    6. How do you find wasted GPUs? Compare allocation with measured GPU activity; reclaim idle allocations.

    Used in

    • AI Engineering: GPU Utilization and Cost; GPU Interconnects and Collectives; Checkpoints and Fault Tolerance in Long Training Runs; Serving LLMs: Metrics, Autoscaling and Disaggregated Prefill; Parallelism for Training.
    • AI System Design: Design a Fine-tuning Platform; Design an Inference Platform.