AI server for teaching: LLM access for thousands of students, sized by peak concurrency

What an AI server for teaching has to deliver

A university or college that gives thousands of students access to a language model sizes the AI server by the number of requests running at the busiest moment. A server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards covers a faculty or a pilot, four cards cover a university-wide chat service for one large model, and eight cards, or two four-card servers, carry exam periods and several models at once. A gateway in front of the servers sets keys per course and limits per student, and a chat front end with single sign-on gives every student an account.

For the rector's office or the IT director, the decision rests on three numbers: how many students use the service in the busiest hour, how long their conversations are, and which model the teaching staff want. The rest of this article turns those numbers into a configuration. A research lab that shares GPUs for its own jobs has a different problem, which our guide to a GPU server for a university research lab solves with Slurm, MIG and fair-share.

How many students use the service at the same moment

Enrolment is a poor guide. What loads the GPUs is the number of requests the model is working on at one moment, the requests in flight. That number follows from two values the gateway log shows once the service runs: how many requests arrive per second, and how long each one takes to answer. Multiplied, they give the average number in flight.

An example with assumed values, by our arithmetic: on a weekday evening before exams, 1,000 students are active, each sends a prompt every two minutes, and each answer streams for about 15 seconds. That is 1,000 ÷ 120 = 8.3 requests per second, times 15 seconds, about 125 requests in flight. The same campus at noon in term time might have 300 active students and 40 in flight.

Teaching adds a load that companies rarely see: the burst. A lecturer asks 400 students in a hall to try a prompt now, and 400 requests arrive within a minute. A server that holds 241 conversations answers the first 241 at once and queues the rest until earlier answers finish, in the example after about 15 seconds; for a classroom exercise that is acceptable if the front end shows that the request is waiting. A per-student request limit in the gateway keeps one script or one impatient user from multiplying the burst.

Our example values, not measurements: one prompt every two minutes per active student and 15 seconds per answer (8.3 requests per second × 15 s ≈ 125). Replace them with the arrival rate and answer time from your gateway log.

Which model, and how many conversations each configuration holds

Teaching staff usually want one capable general model for everyone and sometimes a smaller, faster one for programming exercises or bulk use. OpenAI's gpt-oss-120b and gpt-oss-20b are open-weight models whose MXFP4 checkpoints take 60.8 GiB and 12.8 GiB. Our users-per-card rule takes 0.9 × the 95.6 GiB the driver reports per RTX PRO 6000 Server Edition, minus 3 GiB per card, minus the weights, and divides the rest by the KV cache per conversation: 1.125 GiB per 32K conversation for gpt-oss-120b and 0.75 GiB for gpt-oss-20b, in 16-bit.

Our estimate by the users-per-card rule: concurrent conversations at full context length, 16-bit KV cache; gpt-oss-120b 65.3 GB checkpoint (60.8 GiB) and gpt-oss-20b 13.76 GB (12.8 GiB) on Hugging Face; gpt-oss-20b cache from its config.json (12 full-attention layers, 8 KV heads of dimension 64).

These are memory limits. Students who chat in a few quick exchanges rarely fill 32K, so the 8K column is closer to a teaching service, but at several hundred conversations per card each one gets fewer tokens per second. The serving engine's own limit on concurrent sequences has to be set to match. Our guide to how many users one RTX PRO 6000 serves explains the rule and its limits.

Published throughput for eight RTX PRO 6000 Server Edition cards

Memory says how many conversations fit; throughput says how fast they are answered. MLCommons publishes verified results for servers with eight RTX PRO 6000 Server Edition cards, among them the two models closest to a teaching service.

MLPerf® Inference: Datacenter v6.1, closed division, gpt-oss-120b and Llama 3.1 8B, server and interactive scenarios, tokens per second, entries 6.1-0050 and 6.1-0082, each with eight RTX PRO 6000 Blackwell Server Edition, retrieved from mlcommons.org on 24 October 2026. Results verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The server scenario sends queries at random intervals and holds latency limits, which is close to a chat service. If an average answer has 500 tokens, 14,581.7 tokens per second are about 29 answers per second, or about 1,750 per minute, by our arithmetic; a small model of the Llama 3.1 8B class answers about 99 per second. Against the exam-evening example of 8.3 requests per second, an eight-card server running gpt-oss-120b has room to spare. No verified four-card result for gpt-oss-120b is cited here; by our estimate a four-card server, at roughly half the eight-card rate, would still carry the example with less headroom. Our overview of MLPerf Inference v6.1 for PCIe GPU servers lists the other entries.

We build these servers with two, four or eight cards; send us your student numbers, the models you plan and a busy-hour estimate, and our team replies with a configuration and the conversations it holds.

Gateway, keys and quotas for courses and students

A model server knows nothing about students. vLLM checks static API keys but has no user accounts, so a gateway in front of it has to know which course and which student sent each request. Our guide to an LLM gateway with keys, budgets and a query log covers the options; for teaching, LiteLLM's proxy shows the pattern.

• Keys per course or faculty. Each course or faculty gets a team with its own key, allowed models and limits, so a programming course can call the smaller model through the API while the chat front end uses the large one.
• Limits per student. Requests per minute (rpm_limit) and tokens per minute (tpm_limit) per key or team stop scripts and loops from taking the server.
• Budgets per period. A max_budget with a budget_duration such as 7d resets every week; for a private model the budget counts an internal rate per token, not money.
• Batch below interactive. Bulk jobs such as grading support or document processing run under keys whose token limits leave the chat model room for its peak.

The gateway's database then shows tokens per faculty and course, which is the basis for internal cost allocation, as our guide to GPU chargeback and quotas describes.

The chat front end for thousands of accounts

Most students reach the model through a browser. Open WebUI is a common front end, and its scaling guide changes several defaults for a large deployment: PostgreSQL instead of SQLite, Redis for websockets and shared state, several replicas behind a load balancer with one worker each, and an external vector database for document uploads. Its quick reference marks these as required for multiple instances and lists deployments of 1,000 or more users with the same requirements.

Sign-in should come from the university's identity provider over OIDC, so accounts end when students leave and role or programme claims decide which models a student sees. Our guide to Open WebUI for 1,000 users and more sets out the replicas, Redis settings, storage and upgrade order. Courses that teach machine learning itself need GPUs for notebooks as well; MIG splits each RTX PRO 6000 Server Edition into up to four instances of 24 GB, so four cards set aside for courses give 16 isolated notebooks, as our guide to RTX PRO 6000 MIG profiles shows. Cards in MIG mode are not available to a tensor-parallel copy of the chat model, because NVIDIA's MIG guide states that NCCL is not supported with MIG, so plan the notebook cards in addition to the serving cards.

Three configurations: two, four or eight cards

These are the reference configurations on our AI servers page, with memory and processor cores following NVIDIA's configuration guide for NVIDIA-Certified Systems: at least twice the GPU memory in system memory, at least six cores per GPU and two sockets.

Our AI servers page and the NVIDIA-Certified Systems configuration guide (30 September 2026). Wall power is our estimate: cards at 600 W, two EPYC 9135 (two cards), 9355 (four) or 9555 (eight), plus 15 per cent for the rest.

Entry, two cards. A faculty starts here, or a whole university runs a pilot for one semester before committing to a campus service. The pair holds gpt-oss-120b for about 90 long or several hundred short conversations, and fits a 230 V 16 A feed. Our page on the two-card RTX PRO 6000 Server Edition server has the details.

Campus service, four cards. One copy of gpt-oss-120b over four cards holds 241 conversations at 32K, about twice the exam-evening example. Two four-card servers instead of one eight-card server keep the service running during updates and failures, which matters when an exam week depends on it; our comparison of one 8-GPU server or two 4-GPU servers weighs the two.

Eight cards. Two copies of gpt-oss-120b, or one large model plus a small one for programming courses, with the published throughput above. Our guide to what eight RTX PRO 6000 Server Edition cards run covers larger models.

The H200 NVL suits a university that also trains models on the same platform; for serving many students, the RTX PRO 6000 Server Edition gives more cards and conversations per server.

Power, rack and the room

By our estimate, with cards at 600 W, a two-card server with two EPYC 9135 draws about 1.8 kW at the wall, a four-card server with two EPYC 9355 about 3.4 kW and an eight-card server with two EPYC 9555 about 6.3 kW. We plan a 230 V 16 A feed at 80 per cent, 2.94 kW: the two-card server fits it, the four-card server needs 32 A single-phase or three-phase power, and the eight-card server needs three-phase power or several feeds. Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, so the feed is planned on the chassis maker's value. A teaching service idles at night and in the holidays, but the feed is sized for the exam week. Our guide to GPU server power on 16 A, 32 A and three-phase covers redundant supplies and cords.

Student data, logs and retention

Prompts can contain names, matriculation numbers and coursework, so the query log holds personal data. Article 5(1)(e) of the GDPR requires such data to be kept no longer than is necessary for its purpose. What the log stores, who may read it and for how long is a decision for the university's data protection officer, made before the first student signs in; the gateway's settings for log content and retention then follow that decision. This is general information, not a legal assessment.

Ordering the teaching server from us

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. Sizing is free. We assemble the servers, test them before delivery and deliver them anywhere in the EU on one contract and invoice, with manufacturer warranty on every component and the assembly covered by us. On request we install the operating system, NVIDIA driver, CUDA and a container runtime, and commission the servers on site. The gateway, the chat front end and sign-in are part of our Private AI/ML service, built by our engineering partner Vixen.UNO. Quantities and lead time are confirmed in writing with the quote, and tender requirements can be stated in the request.

For an exact configuration we need the number of students and staff, an estimate of the busiest hour, the models the teaching staff want, typical conversation length, whether courses need notebooks or API access, and the rack and power feed. Describe these in the form below, and we reply with a configuration and a written quote.

FAQ