Windows Server or Linux for a GPU server: what runs on each, and which to choose

Which operating system for a GPU server

For a GPU server that runs language models, Linux is the operating system to choose: vLLM states "OS: Linux" in its installation guide, NVIDIA NIM recommends Ubuntu 22.04 LTS or later, and NVIDIA AI Enterprise and the NVIDIA Container Toolkit list only Linux for the host. Windows Server is the right choice when the server runs Windows CUDA applications or simulation codes, or when it is a Hyper-V host that shares NVIDIA RTX PRO Blackwell Server Edition, L40S or L4 cards with virtual desktops. Many Windows-centred IT departments end up with both: a Linux inference server next to their Hyper-V hosts.

We wrote this for an IT department that runs Windows Server and Active Directory today and is planning its first GPU server. All facts are from NVIDIA's and the software makers' documentation as read on 25 October 2026; versions change, so check them again before you order.

The decision in one table

vLLM GPU installation guide (latest), NVIDIA NIM for LLMs 2.0.13 prerequisites, NVIDIA Container Toolkit supported platforms (18 Sep 2026), NVIDIA AI Enterprise 8.2 support matrix (2 Sep 2026), CUDA 13.4 installation guides for Windows and Linux (12 Oct 2026), Ollama Windows documentation and our engine comparison for Ollama on Linux, NVIDIA vGPU R595 documentation (29 Sep 2026); MIG prerequisites as in our MIG articles; all read 25 October 2026.

What needs Linux

vLLM. The engine most of our LLM articles size for lists its requirements as "OS: Linux", Python 3.11 to 3.14 and a GPU of compute capability 7.5 or higher. Its guide states: "vLLM does not support Windows natively. To run vLLM on Windows, you can use the Windows Subsystem for Linux (WSL) with a compatible Linux distribution", and mentions community-maintained forks. A fork is not the version that vLLM's security notes and release notes describe, so for a server that serves a whole company we would not plan on one. The engine comparison in our guide to vLLM, SGLang, TensorRT-LLM and Ollama assumes Linux for the same reason.

NVIDIA NIM. NIM for large language models 2.0.13 recommends "Ubuntu 22.04 LTS or later"; other Linux distributions may work but are not validated. It needs driver 580 or later, Docker 24.0 or later and the NVIDIA Container Toolkit 1.14.0 or later, and the prerequisites page does not mention Windows. In production NIM is covered by an NVIDIA AI Enterprise licence, and the AI Enterprise 8.2 support matrix lists Ubuntu 20.04, 22.04 and 24.04, Red Hat Enterprise Linux and SUSE Linux Enterprise Server for bare metal. Windows appears in that matrix only as a guest operating system in virtual machines on vSphere and on Red Hat Enterprise Linux with KVM.

GPU containers. The NVIDIA Container Toolkit, which lets Docker or Podman give containers a GPU, is "tested and expected to work" on a list of Linux distributions: Ubuntu 22.04, 24.04 and 26.04, RHEL 8, 9 and 10, Rocky Linux 9.7, SUSE Linux Enterprise Server 15, Debian 11 and Amazon Linux. Windows is not on it. Most model servers, retrieval stacks and the GPU Operator for Kubernetes are delivered as containers, so this list decides more than any single engine does. Our guide to Docker GPU assignment without Kubernetes shows how the containers get their cards.

MIG. Splitting any RTX PRO 6000 edition into up to four isolated 24 GB instances needs Linux and a driver from 575.51.03, per NVIDIA's MIG prerequisites as our MIG profile guide sets out. On Windows Server the RTX PRO 6000 Server Edition runs without MIG.

Which Linux to choose is a separate question with its own support lists; our guide to Linux for GPU servers: Ubuntu, RHEL or Rocky answers it: Ubuntu 24.04 LTS, RHEL 9 and RHEL 10 are on every NVIDIA list today.

What runs well on Windows Server

CUDA applications. The CUDA 13.4 installation guide for Windows, updated 12 October 2026, lists Windows Server 2022 and Windows Server 2025 among the supported operating systems. A CUDA program or a simulation solver whose vendor ships a Windows build runs on a Windows Server GPU host; the solver vendor's own GPU list decides which cards and driver versions it supports.

The driver model. On Windows, an NVIDIA GPU runs under a driver model that decides whether it acts as a display device or a pure compute device. The CUDA guide says "the WDDM driver model is used for display devices" and that "TCC is enabled by default on most recent NVIDIA Tesla GPUs". NVIDIA's vGPU documentation states that the RTX PRO 6000 Server Edition ships in display-off mode, and which mode a given card and driver allow is the subject of our guide to WDDM, TCC and MCDM. Check it before you plan a workload that needs a particular mode, such as WSL 2 (below) or a solver that asks for TCC.

Ollama. Ollama's Windows documentation says it "runs as a native Windows application", requires "Windows 10 22H2 or newer, Home or Pro" and NVIDIA driver 551.61 or newer, and offers a standalone zip with only the command-line tool and the NVIDIA GPU libraries that can run ollama serve as a system service with a tool such as NSSM. Windows Server is not mentioned. That makes it a good tool for a developer's workstation and a weak base for a shared server, as our article on moving from Ollama on a PC to a company AI server explains.

A Hyper-V host for VDI. For virtual desktops and CAD workstations, Windows Server 2025 is a supported NVIDIA vGPU host. NVIDIA's R595 documentation supports vGPU through GPU partitioning only on Windows Server 2025 and lists the RTX PRO 6000 and RTX PRO 4500 Blackwell Server Editions, the L40S and the L4 for it; Windows Server 2022 gets whole-card DDA only. Our guide to NVIDIA vGPU on Hyper-V with Windows Server 2025 covers the setup, licences and live migration.

We build the server to order and install the operating system you choose, with the NVIDIA driver, CUDA and a container runtime on request; tell us which applications and models the server must run, and our team replies with the configuration and the operating system that supports them.

WSL 2 and Docker Desktop are workstation routes

Two routes bring Linux tools onto a Windows machine, and both are aimed at workstations.

CUDA on WSL 2. NVIDIA's CUDA on WSL user guide (CUDA 13.4, updated 12 October 2026) names Windows 11 and Windows 10; Windows Server is not mentioned. It supports GPUs in WDDM mode and states that "Quadro GPUs in TCC mode and Tesla GPUs are not yet supported". Inside WSL "users must not install any NVIDIA GPU Linux driver", because the Windows driver provides the GPU. The guide also lists limits that matter on a server: NVML, the library behind nvidia-smi, "does not support all the queries yet", and setting ECC, compute mode or persistence mode is not supported; pinned memory is limited, so some deep-learning workloads may fail. For a workstation with an RTX PRO 5000 or an RTX PRO 6000 Workstation Edition, WSL 2 is a practical way to run Linux tools, as our guide to CUDA on WSL2 for RTX PRO workstations shows.

Docker Model Runner with vLLM. Docker announced on 11 December 2025 that its Model Runner "now supports vLLM on Docker Desktop for Windows with WSL2 and NVIDIA GPUs", from Docker Desktop 4.54 and for GPUs of compute capability 8.0 or higher. Docker Desktop is the desktop product, and the post does not mention Windows Server. It runs vLLM inside the WSL 2 Linux environment, so it inherits the WSL limits above.

Neither route turns Windows Server into a supported vLLM or NIM host. For a server that a whole company depends on, the supported combinations in the table above are the safer base.

Running an LLM on Windows Server: a Linux VM on a Hyper-V host

A Windows IT department can also run Linux for the models inside a VM on a Hyper-V host. NVIDIA supports two ways to give that VM a GPU on Windows Server 2025, each with limits.

• Whole card by DDA. For Ubuntu 24.04 and 22.04 and for RHEL 9.x guests, NVIDIA's Windows Server matrix states "GPU pass through is not supported". On a 2025 host, the Linux guests for DDA are Ubuntu 20.04, RHEL 8.10 and SUSE Linux Enterprise Server.
• A full-card vGPU by GPU partitioning. The DC-96Q type gives one Ubuntu 24.04 or RHEL 9 VM the whole RTX PRO 6000 Server Edition under an RTX vWS licence. Compute (C-series) vGPU types "are not available" on Windows Server, MIG-backed vGPUs are not supported for the RTX PRO Server Editions there, and unified memory, which is on by default for DC-96Q in Linux guests, disables live migration.

A VM can hold up to 16 vGPUs on Windows Server 2025, but NVIDIA's pages make no statement about peer-to-peer traffic between them, so we would not split one model across several vGPUs with tensor parallelism. For one model on one card, the VM route works; for a model across two or four cards, a Linux host is the simpler and better-documented choice.

Running a Linux GPU server in a Windows IT department

A Linux inference server changes less for the Windows clients than it may seem. vLLM serves an OpenAI-compatible API over HTTP, as the Gemma 4 model card's example vllm serve command notes, so Windows desktops, scripts and line-of-business applications call it like any web service. Nothing Linux-specific runs on the clients.

The work sits on the server side, and it is limited:

1. Access. Put an LLM gateway with keys and a query log in front of the engine, and let the gateway authenticate staff against your directory. vLLM's --api-key protects only paths under /v1, /v2, /inference and /cohere (vLLM 0.32.0 security guide), so the engine itself stays on an internal network.
2. Patching. Kernel and NVIDIA driver updates go together; pin the driver branch and update both in a maintenance window, as our article on NVIDIA driver branches and CUDA versions describes.
3. Monitoring. GPU temperature, power, memory errors and XID events can feed the monitoring you already run; our guide to GPU monitoring with Zabbix and Checkmk lists the metrics.
4. Setup. The steps from firmware to the first model are in our guide to a new GPU server setup.

Three configurations

Reference configuration on our AI servers page (two RTX PRO 6000 Server Edition, dual AMD EPYC, 384 GB DDR5 ECC, two NVMe); desktop counts from our vGPU profile guides (16 per RTX PRO 4500 Server Edition at 2 GB, four DC-24Q per RTX PRO 6000 Server Edition); power by our estimate with the cards at 600 W, two EPYC 9135 processors and 15 per cent for the rest. We give no estimate for the host with four RTX PRO 4500 Server Edition cards, whose processor and memory depend on the desktop count.

The Linux inference server. The entry configuration on our AI servers page, with two RTX PRO 6000 Server Edition cards, two AMD EPYC processors, 384 GB of DDR5 ECC memory and two NVMe drives, runs gpt-oss-120b on one card with about 19 conversations of 32K tokens by our users-per-card rule, and the second card for retrieval models, speech-to-text or a second model, split by MIG if needed. It draws about 1.8 kW at the wall by our estimate and fits one 230 V 16 A feed, which carries 2.94 kW at 80 per cent.

The Hyper-V host. For desktops, RTX PRO 4500 Server Edition cards give 16 office desktops each on 2 GB vPC profiles; for CAD, RTX PRO 6000 Server Edition cards give four 24 GB seats each. Windows Server 2025 allows one vGPU series and size per card, so a host that serves both puts them on separate cards. Hyper-V hosts with GPUs fit into an existing Windows cluster, with the live-migration rules our Hyper-V guide lists.

Our page on buying a two-card Server Edition server lists the parts of the Linux server. If you want both, describe the AI workload and the desktop count, and we size the inference server and the Hyper-V hosts as one quote.

Recommendation per use case

• LLM inference, retrieval, an internal assistant: Linux on the host (Ubuntu 24.04 LTS, RHEL 9 or RHEL 10), vLLM or NIM in containers. Windows clients use the API.
• Windows CUDA applications and simulation solvers: Windows Server 2022 or 2025, if the solver vendor supports it for your card; check the driver mode first.
• Virtual desktops and CAD seats with shared GPUs: a Hyper-V host on Windows Server 2025 with RTX PRO 4500 or RTX PRO 6000 Server Edition, L40S or L4 cards, or a Linux KVM host such as Proxmox VE.
• Compute VMs with C-series vGPUs or MIG-backed vGPUs: a Linux KVM hypervisor or vSphere; neither type is available on Hyper-V.
• A developer workstation: Windows 11 with WSL 2 is fine; a shared server should not depend on it.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build GPU servers to order with RTX PRO 6000 Server Edition, RTX PRO 4500 Server Edition, L40S, L4 or H200 NVL cards, assembled and tested before delivery, with the operating system, drivers, CUDA and a container runtime installed on request, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. vGPU and NVIDIA AI Enterprise licences go on the same quote, as our licensing page explains.

FAQ