GPU scheduling for ML teams: NVIDIA KAI Scheduler, Kueue or Slurm, and a 16-GPU design

KAI Scheduler vs Kueue vs Slurm: which fits which company

A company with two to four GPU servers and several ML teams needs three things from a scheduler: a fixed share of cards per team, a way to lend idle cards and take them back, and jobs that start only when all their GPUs are free. On Kubernetes, NVIDIA's KAI Scheduler does all three as a replacement scheduler, Kueue does them as a quota layer above the standard scheduler, and outside Kubernetes Slurm does them for batch jobs. For a cluster used only for GPU work, with notebooks that should share cards, KAI is the closer fit; for a cluster that mixes card types or job frameworks and keeps the default scheduler, Kueue.

KAI Scheduler README, queue and fairness documentation on GitHub; Kueue overview (modified 9 October 2026) and installation guide (1 October 2026); all read on 25 October 2026. Kueue quota fields and Slurm options as documented in our guides to shared GPU platforms and university lab servers.

The card-types row matters for mixed fleets. KAI's queue documentation lists quota fields for cpu, memory and gpu, with no field per GPU model, so a team that should hold H200 NVL cards and RTX PRO 6000 Server Edition cards as separate figures is easier to express with Kueue's ResourceFlavors. A cluster built from one card type, such as four inference nodes, does not have this problem.

NVIDIA KAI Scheduler in brief

KAI Scheduler started inside Run:ai. NVIDIA's developer blog of 1 April 2025 announced that the scheduler, "Originally developed within the Run:ai platform", was available under the Apache 2.0 licence and would continue to ship as part of NVIDIA Run:ai. The project's README calls it a CNCF sandbox project built on kube-batch. Its repository now sits in the kai-scheduler organisation on GitHub, and on 25 October 2026 the latest release was v0.18.5, which added Helm chart OCI tags without the "v" prefix so that semver ranges can be used.

The README lists the features: gang scheduling ("Ensure all pods in a group are scheduled simultaneously or not at all"), bin packing or spreading, workload priority, hierarchical queues, per-queue quotas and weights with Dominant Resource Fairness, time-based fairshare that counts past usage with time decay, elastic workloads, DRA support through ResourceClaims, GPU sharing, topology-aware scheduling and consolidation of running workloads to reduce fragmentation. It names Ray, through KubeRay, as a native integration; NVIDIA's announcement also listed Kubeflow, Argo and the Training Operator.

Installation is one Helm release from the OCI chart at ghcr.io into the namespace kai-scheduler, with the version taken from the releases page. The prerequisites are a running Kubernetes cluster, Helm and, for GPU workloads, the NVIDIA GPU Operator; the README states no minimum Kubernetes version. A pod opts in with schedulerName: kai-scheduler and names its queue with the label kai.scheduler/queue. A fresh install creates default-parent-queue and the leaf default-queue, so a first pod can run before any team queues exist.

GPU quotas and fair sharing in KAI: queues and reclaim

Queues are a custom resource, apiVersion scheduling.run.ai/v2, kind Queue. Each has an optional parentQueue, a priority and, for each of cpu, memory and gpu, three values: quota, the share reserved for the queue; limit, the hard cap; and overQuotaWeight, the share of spare resources within a priority level. "Only leaf queues (queues with no children) can be used for scheduling jobs"; parent queues divide resources among their children.

Two defaults deserve attention. A quota of 0 or unset gives the queue no reserved resources, and -1 means unlimited. A limit of 0 or unset means, in the documentation's words, "No additional resources allowed", and -1 means no limit. A team queue that should borrow idle GPUs therefore needs an explicit limit of -1 or a number above its quota.

The fairness documentation sets the order. "Deserved quota for a queue will always be allocated to it", spare capacity is then shared "based on priority and weight attributes", and workloads from queues below their fair share can evict workloads from queues above it. Quota reclaim only targets resources above quota. Reclaiming in-quota resources from a queue of lower priority needs the option queuePriorityInQuotaReclaim, off by default and marked alpha. With the defaults, a team that stays within its quota keeps its GPUs.

Kueue and Slurm for comparison

Kueue describes itself as "a kubernetes-native system that manages quotas and how jobs consume them": it decides when a job waits, is admitted or is preempted, and "does not replace any existing Kubernetes components". Its overview lists built-in support for batch Jobs, Kubeflow training jobs, RayJob, RayCluster, JobSet, AppWrappers and plain Pods, a timeout-based all-or-nothing mode, partial admission with reduced parallelism, topology-aware scheduling and MultiKueue, which dispatches jobs to other clusters. The installation guide, read on 25 October 2026, installs v0.20.3 and recommends Kubernetes 1.34 or newer. How ClusterQueues, cohorts, borrowing and lending limits work for departments is explained in our guide to a shared GPU platform with quotas and chargeback, so this article does not repeat it.

Slurm fits a different user: researchers who submit batch jobs with sbatch, MPI codes and servers without Kubernetes. It schedules GPUs as generic resources, supports MIG devices created beforehand, and counts GPUs in fair-share only once TRESBillingWeights gives them a weight, as our guide to GPU servers for university research labs explains. A company whose ML teams deploy containers, notebooks and model services has more to gain from Kubernetes with KAI or Kueue.

Our AI servers serve as Kubernetes GPU nodes with the GPU Operator's prerequisites in place. Tell us how many teams and GPUs the cluster will have, and our team proposes the nodes and the card layout.

A queue design for three teams on 16 GPUs

Take four servers with four RTX PRO 6000 Server Edition cards each, the inference node on our AI servers page, so 16 cards of 96 GB on one Kubernetes cluster. Three teams share them: a product team serving models to staff in continuous operation, an applied ML team fine-tuning models in multi-GPU jobs, and a data science team working in notebooks.

Our example design, written against KAI's Queue fields (quota, limit, overQuotaWeight, priority, parentQueue) as documented on 25 October 2026. Quotas add up to the 16 cards.

The serving queue holds six cards and never more, enough for three tensor-parallel pairs of gpt-oss-120b, each pair carrying 93 conversations of 32K tokens with a 16-bit KV cache by our users-per-card rule as in our four-card server guide. Because it stays within its quota, KAI's default reclaim never takes those cards away. Model services should still run as several replicas on different nodes, so that a node failure costs capacity and not the service.

The training queue holds six cards and may borrow all idle ones. A four-GPU fine-tuning job in one pod lands on one node; a job of several pods across nodes, for example a PyTorch or Ray job with two workers of four GPUs, uses gang scheduling so that it starts only when all eight cards are free. When the notebook team returns in the morning, KAI's fair-share reclaim evicts the borrowed training pods, so training jobs need checkpoints. The weight of 2 gives training twice the share of spare cards that the notebooks get.

The notebook queue holds four cards and a limit of eight. With GPU sharing switched on at install (global.gpuSharing=true), a notebook pod can ask for half a card with the annotation gpu-fraction set to 0.5, or for a memory amount with gpu-memory. KAI's documentation is explicit that "By default, KAI Scheduler does not enforce memory allocation limits or perform memory isolation between processes", and that pods sharing a GPU "may consume more memory than requested"; it points to HAMi for enforcement. If one notebook must not be able to exhaust another's memory, MIG instances of 24 GB with their own memory, as listed in our RTX PRO 6000 MIG profiles, are the stronger choice. KAI's GPU-sharing documentation does not mention MIG, so test MIG resources with KAI on one node first.

The same design in Kueue

In Kueue the three teams become three ClusterQueues in one cohort, with nominalQuota 6, 6 and 4 for nvidia.com/gpu, a LocalQueue in each team namespace, and a lendingLimit of zero on the serving queue so its cards never enter the pool. Training gets a borrowingLimit, and reclaimWithinCohort on the lending queues lets the notebook team take its cards back. Placement stays with the default scheduler. Kueue's overview lists no GPU fractions of its own; whole cards or MIG profiles exposed by the GPU Operator become the unit of quota, as our guide to GPU sharing in Kubernetes shows.

In neither project's documentation that we read did we find the two schedulers combined on the same pods. Pick one per cluster, or at least per workload.

Run:ai, now open source, and NVIDIA AI Enterprise

NVIDIA's Run:ai product page, published on 13 October 2026, calls Run:ai "the open source AI workload and GPU orchestration platform for Kubernetes" and links its source repository, which carries the Apache License 2.0 with trademarks excluded. That repository describes a web UI, CLI and REST API, departments and projects with quotas, role-based access and the KAI Scheduler underneath. The same page offers Run:ai as part of NVIDIA AI Enterprise, the commercial suite with support. For a company starting with KAI, this gives a path to a user interface and department reporting on the same scheduler, either self-supported from the open-source code or supported under an AI Enterprise licence. Our guide to NVIDIA AI Enterprise licensing explains the per-GPU entitlement.

Before you choose

A few facts decide most cases:

1. If the cluster mixes card types and each team's share must be counted per type, start with Kueue and its ResourceFlavors.
2. If notebooks should share cards and the teams accept shared memory, KAI's fractions are the simplest tool; for isolation, use MIG.
3. If multi-pod training jobs must start complete, both offer it; KAI builds it in, Kueue uses a timeout-based mode.
4. If users need a UI, departments and reports, Run:ai on top of KAI gives them, open source or with AI Enterprise support.
5. If the work is MPI batch jobs on bare metal, Slurm remains the tool.

Kubernetes 1.34 reaches end of life on 27 October 2026, so a new GPU cluster starts on a newer release; our guide to Kubernetes DRA for NVIDIA GPUs covers what changes when pods claim GPUs by attribute.

What we supply

We build AI servers to order as Kubernetes GPU nodes: an entry server with two RTX PRO 6000 Server Edition cards, an inference node with four, and a training node with eight H200 NVL, each with two AMD EPYC processors, assembled and tested before delivery and delivered across the EU on one contract and invoice. We also supply NVIDIA AI Enterprise licences, which include Run:ai, with the hardware. Send us your teams and their workloads, and we reply with a node configuration within one business day.

FAQ