GPU server for a university or research lab: Slurm, MIG and fair sharing of the cards

What a GPU server for a university or research lab needs

Besides the cards, a GPU server that a research lab or a course shares needs a scheduler that queues jobs and hands out GPUs, partitioning so that small jobs run without holding a whole card, and quotas with fair-share so that one group cannot occupy the machine for weeks. On bare metal the usual scheduler is Slurm, which handles GPUs as generic resources (GRES) and accounts for them per user and account. On a Kubernetes cluster, Kueue adds quotas and queueing next to the normal scheduler. NVIDIA’s Multi-Instance GPU (MIG) splits an H200 NVL into up to seven instances and an RTX PRO 6000 into up to four, each with its own memory. Training and double-precision simulation lead to the H200 NVL, inference and teaching to the RTX PRO 6000 Server Edition.

Research use comes in bursts. A doctoral student fine-tunes for two days and then pauses for a week, and a course opens thirty notebooks in the same hour. Without a scheduler, whoever logs in first takes the cards, and an idle notebook keeps a 96 GB card to itself.

Slurm and GPUs: GRES configuration, job requests and accounting

Slurm’s GRES documentation (version 26.05 as of October 2026) says “You must explicitly specify which GRES are to be managed” in slurm.conf. GresTypes=gpu names the resource type, and each node line declares its cards, for example Gres=gpu:4. The device files and the CPU cores close to each card go into gres.conf, written by hand or filled in with AutoDetect=nvml, which in Slurm’s words supplies the details “for any system-detected GPU”. The count in slurm.conf stays required, so that the controller knows how many cards to expect.

Users ask for cards with --gpus or with --gres, whose form is name, optional type and count, as in the documentation’s --gres=gpu:kepler:2. Type names are set by the administrator, which lets a lab keep H200 NVL and RTX PRO 6000 nodes in one cluster and let jobs ask for one or the other.

With AccountingStorageTRES=gres/gpu set, Slurm also gathers GPU memory and utilisation per job, as gres/gpumem and gres/gpuutil, for NVIDIA GPUs found with AutoDetect=nvml. The documentation adds that NVML does not support utilisation metrics for MIG instances, so Slurm records neither gpumem nor gpuutil for them. Card-level health, memory errors and XID events come from NVIDIA’s DCGM, which our GPU server monitoring guide covers.

MIG in Slurm: splitting an H200 NVL or an RTX PRO 6000

Slurm has supported MIG devices since version 21.08. They are detected with AutoDetect=nvml and listed in slurm.conf like ordinary GPUs, with an optional type that “must be a substring of the ‘MIG Profile’ string” the node reports; the documentation’s example uses a100_3g.20gb. The same page states that “Slurm expects MIG devices to already be partitioned, and does not support dynamic MIG partitioning.” The MIG layout is therefore a property of the node. Changing it means draining the node, rebuilding the instances and adjusting the configuration, so plan layout changes like any other maintenance. NVIDIA’s MIG guide also states that “NCCL is currently not supported with MIG”, so an instance runs single-GPU jobs, and multi-card training takes whole cards.

NVIDIA’s MIG guide describes instances with “separate and isolated paths through the entire memory system” and states that “MIG ensures one client cannot impact the work or scheduling of other clients”. A student’s notebook that crashes then stops on its own, and a colleague’s job keeps running.

NVIDIA MIG user guide (updated 11 September 2026), which lists the H200 NVL with 7 instances and gives the profiles in one table for the H200 141GB; NVIDIA’s H200 page gives the H200 NVL “Up to 7 MIGs @16.5GB each”. H200 and RTX PRO 6000 Server Edition, Workstation and Max-Q product pages, read 10 October 2026; the 1/64 FP64 rate from NVIDIA’s RTX Blackwell PRO whitepaper v1.1; RTX PRO 6000 FP64 values are our arithmetic.

On the Workstation and Max-Q editions MIG needs a switch of the display mode first, which our MIG runbook for RTX PRO Blackwell describes together with a second point that matters for Slurm. On Hopper and later GPUs, MIG mode and the instances are gone after a reboot. NVIDIA suggests a systemd service with its MIG Partition Editor (nvidia-mig-parted) to recreate them at start-up; on a Slurm node it has to run before slurmd starts, since AutoDetect reads the devices the node has. The L40S, L4, RTX PRO 4000 and RTX PRO 2000 are not in NVIDIA’s MIG guide, so they serve whole jobs or are shared without hardware isolation.

We build GPU servers to order with H200 NVL or RTX PRO 6000 Server Edition cards, with drivers and CUDA installed on request. Tell us how many people share the server and what they run.

Sharing without MIG: Slurm shards and MPS

Slurm also offers shards, a “generic mechanism where GPUs can be shared by multiple jobs”, configured as a count per node such as shard:64 beside the GPUs. The documentation states that sharding “does not fence the processes running on the GPU” and that it “works best with homogeneous workflows”. A GPU is allocated either as a gpu or as shards, not both. Shards suit a course in which every student runs the same small exercise, if the lab accepts that one job using more than its share of memory or compute can slow or block the others.

CUDA MPS is the other option, and Slurm’s documentation warns that “NVIDIA MPS has a built-in limitation regarding GPU sharing among different users”. On a server where every researcher has an account of their own, MIG is the method that separates users in hardware.

Fair-share and GPU quotas in Slurm

Slurm’s multifactor priority plugin weighs nine factors, among them age, fair-share, partition and QOS. It defines fair-share as “the difference between the portion of the computing resource that has been promised and the amount of resources that has been consumed”, and since release 19.05 the Fair Tree algorithm is the default. PriorityDecayHalfLife, 7 days by default, sets how fast past usage fades.

For a GPU server one default needs changing. “By default, the computing resource is the computing cycles delivered by a machine in the units of allocated_cpus*seconds.” A job with two CPU cores and four GPUs would then count as small. The same page continues: “Other resources can be taken into account by configuring a partition’s TRESBillingWeights option.” Giving GPUs a weight in that option makes fair-share charge for cards, and the MAX_TRES_GRES priority flag bills the largest of a node’s CPU and memory shares plus the GPUs.

Hard limits sit beside fair-share. With AccountingStorageEnforce=limits, Slurm enforces limits set on associations, that is on the cluster, an account or a user. GrpTRES caps the TRES that an association and its children use at one time, and the documentation’s example sets gres/gpu=50 for one user with sacctmgr. In a QOS, MaxTRESPerUser caps what one user holds, MaxJobsPerUser the jobs a user runs at once and MaxWallDurationPerJob the run time of each job. The documentation warns that a typed limit such as gres/gpu:tesla=1 is not enforced when a job asks for GPUs without a type, a “design limitation”, and suggests a job submit plugin that rejects requests without a type.

A typical set-up for a lab cluster runs in this order:

1. Partition the MIG cards, then declare GPUs and MIG instances in slurm.conf and gres.conf with AutoDetect=nvml.
2. Add gres/gpu to AccountingStorageTRES, so the accounting database records GPU use per job.
3. Create an account per research group and the users under it, with the shares each group is promised.
4. Set TRESBillingWeights on the GPU partitions, so fair-share counts cards and not only cores.
5. Set AccountingStorageEnforce=limits, a GPU cap per group with GrpTRES, a per-user cap and a maximum run time in a QOS.

Kubernetes labs: Kueue quotas and MIG resources

The NVIDIA GPU Operator and device plugin expose whole cards or MIG instances as resources, and with the mixed strategy each MIG profile gets its own resource name; our Kubernetes GPU sharing guide explains the strategies. As that guide notes, we found no RTX PRO 6000 Workstation or Max-Q entry on the GPU Operator’s support list as of October 2026, so plan a Kubernetes cluster on cards from the list, such as the RTX PRO 6000 Server Edition and the H200 NVL.

Kueue, in its own words “a kubernetes-native system that manages quotas and how jobs consume them”, decides when a job waits, starts or is preempted, and “does not replace any existing Kubernetes components”. Each research group gets a ClusterQueue with a nominalQuota of GPUs. Queues in the same cohort “can borrow unused quota from each other”, up to a borrowingLimit, and a lendingLimit holds back some of a queue’s own quota. With reclaimWithinCohort set to LowerPriority or Any (the default is Never), a pending job can preempt borrowed work in other queues of the cohort that exceed their nominal quota.

Batch jobs, MPI codes and users who know sbatch from a computing centre fit Slurm. Notebooks, model services and training pipelines built as containers fit Kubernetes with Kueue.

Choosing cards for training, inference, teaching and FP64 codes

Double precision separates the two main cards. The H200 NVL delivers 30 TFLOPS of FP64 and 60 on its Tensor Cores. NVIDIA’s RTX PRO Blackwell whitepaper states that “The FP64 TFLOP rate is 1/64th the TFLOP rate of FP32 operations” and that the FP64 cores are there “to ensure any programs with FP64 code operate correctly”. Simulation, quantum chemistry and other codes that compute in FP64 belong on the H200 NVL; our comparison of GPUs for CFD and simulation covers solvers in detail. For machine learning, our H200 NVL and RTX PRO 6000 comparison for fine-tuning sets memory, tensor rates and NVLink against each other.

Our reading of NVIDIA’s MIG guide and product pages (table above) and of the reference configurations on our AI servers page, read 10 October 2026; card counts are examples, not sizing.

Our AI servers page lists two reference nodes at this scale: four RTX PRO 6000 Server Edition cards for inference, and eight H200 NVL cards on NVLink bridges with 400G networking for training. A department cluster can run both under one Slurm controller, in separate partitions.

Describe your lab in the form below, with its teams, workloads and the jobs that run at the same time, and we reply with a configuration and quote within one business day.

Storage and network for a shared GPU server

Home directories, datasets and checkpoints should have the same paths on every node, so that a job runs wherever the scheduler places it. Local NVMe drives serve as scratch space during a run, with a separate model store for what the lab keeps. For a single server, 25 GbE uplinks carry logins and data copies; several nodes training one model need a cluster fabric at 200 or 400G, which the H200 NVL training node on our AI servers page plans for.

What we supply

We build AI servers to order for shared use, with the H200 NVL and NVLink bridges, the RTX PRO 6000 Server Edition, the L40S or the L4, assembled and burn-in tested, with manufacturer warranty on every component and on one EU contract and invoice. Single cards come through our professional GPU range, including the RTX PRO 6000 Workstation and Max-Q editions for desk-side machines. We check the rack, power and airflow before we quote, and install the operating system, drivers, CUDA and a container runtime on request. If the lab wants a Kubernetes platform with Kubeflow built and supported on top, that is our Private AI/ML service, with engineering by our partner Vixen.UNO.

FAQ