AI server for a mid-sized company: what it does for 200 to 2,000 staff and how to start

What your own AI server does for a mid-sized company

Your own AI server gives a company of 200 to 2,000 staff a private assistant, answers from its own documents and automated document work, all on hardware in its own network or in an EU data centre it chooses. For most companies of that size, a server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards is the entry point and carries the assistant for up to about 500 staff; four cards carry it for 2,000.

Compared with a public chatbot service, prompts, documents and logs stay where you decide, and the cost follows the hardware you buy, while a cloud API charges per request.

This article is written for a managing director or a head of IT who has to decide whether such a server makes sense and what has to come with it. The detailed sizing by headcount, with the arithmetic, is in our guide to a private ChatGPT server by company size; here the questions are what the server does, which of three configurations fits and how to start.

Tasks it takes over in a company of 200 to 2,000

The same server can run several tasks side by side, each as its own model or on the same model.

• A company assistant. Drafting, rewording, summaries and questions in every department, through a web front end that works like a public chatbot, with access controlled by your directory service. Our guide to a private ChatGPT alternative lists the components.
• Answers from your documents. Policies, contracts, manuals and project files become searchable in plain language, with the source cited and each user's access rights respected, as our article on RAG on company data explains.
• Document processing. Invoices, delivery notes or contracts are read and classified overnight, in batches, as in our guide to batch LLM inference on-premise.
• Transcription and minutes. Meetings and calls are transcribed and summarised; OpenAI gives about 10 GB of GPU memory for Whisper large, a fraction of one card, as our Whisper GPU guide shows.
• Translation and coding help. Internal documents are translated, and developers get code suggestions from a model that never sends source code outside, as in our guides to machine translation on-premise and a private coding assistant.

Most companies start with one of these, usually the assistant or answers from documents, and add the others once the first one works.

Three configurations and what each serves

The three servers below follow the reference configurations on our AI servers page and NVIDIA's rules for certified systems: two processors, at least six cores per GPU and system memory of twice the GPU memory. The staff figures use the example values of our company-size guide, about 4 requests in flight at the busy-hour peak per 100 employees, and gpt-oss-120b, an open model under Apache 2.0 that fits one 96 GB card.

Reference configurations on our AI servers page; staff by the example values of our company-size guide (about 4 requests in flight per 100 employees); users by our users-per-card rule; power by our estimate with the cards at 600 W, two AMD EPYC processors (9135, 9355 or 9555 by card count) and 15 per cent for the rest of the server.

Two cards, 384 GB of system memory. By our users-per-card rule, one RTX PRO 6000 Server Edition holds gpt-oss-120b and about 19 conversations of 32K tokens at once, which covers the example peak of about 475 staff. The second card is split by MIG into up to four isolated instances of 24 GB for the embedding model and the reranker that RAG needs, for speech-to-text and for a small model that writes chat titles. This server is also the right pilot machine, because it uses the same card as the larger builds.

Four cards, 768 GB. Two cards run the assistant as one copy split over both, which holds about 93 conversations of 32K, more than the example peak of 80 for 2,000 staff. The third card runs a second model, for example Qwen3.8-27B in FP8 with about 25 conversations of 32K with a 16-bit cache, or about 47 with an FP8 cache, and the fourth carries the search and speech models. This is our inference node for production use by several departments.

Eight cards. Eight RTX PRO 6000 Server Edition hold models that do not fit four cards, such as Llama 4 Maverick at about 417 GB in FP8, or many models side by side. If the company wants to fine-tune its own models, our training node with eight H200 NVL on NVLink bridges is built for it, as our fine-tuning server guide explains. Where the assistant has become part of daily work, two four-card servers that each carry the whole peak are often better than one eight-card server, because one can fail or be updated while the other runs.

The RTX PRO 6000 also comes as the Max-Q, a 300 W workstation card; for a server in a rack that runs continuously we recommend the Server Edition. We supply all three configurations and size them from your staff numbers and tasks; tell us how many employees would use the server and for which tasks, and our team replies with a configuration within one business day.

What else you need besides the server

The hardware is the smaller part of the project. Three things decide whether the server is used.

People

Our summary of the roles in our guides to rolling out a private AI assistant and to operating an LLM platform.

Running an assistant on open models needs no ML engineers in the team; the work is Linux administration, a serving engine and a web front end. Our guide to operating a private LLM platform lists the daily tasks, and managed private AI shows what can be handed over.

Software

The server arrives on request with the operating system, the NVIDIA driver, CUDA and a container runtime. On top run a serving engine such as vLLM, which needs no licence, or NVIDIA NIM, which needs NVIDIA AI Enterprise per GPU; an LLM gateway for keys, budgets and a query log; a front end such as Open WebUI, which our guide to Open WebUI scaling sizes for 500 to 2,000 users; and for RAG an embedding model, a reranker and a vector database. Single sign-on through your directory service decides who sees which documents.

Data

An assistant that answers from documents is only as good as the documents and their permissions. Before a pilot, the data owners decide which collections go in, who may see them, which version is current and who keeps them up to date. Our article on choosing an open LLM for a company assistant covers the model side, including licences and languages.

Own server, EU data centre or cloud

An in-house server suits steady daily use and data that should not leave the company. A cloud API costs per request and needs no hardware, which suits low or uncertain volumes; our guide to private LLM or cloud API shows how to find the break-even with your own token volumes. Between the two, our Private AI/ML service also runs the models on dedicated hardware in a Tier-3 data centre in Lithuania, so data stays in the EU without a server room of your own, and a hybrid setup sends single tasks to public APIs only by your explicit decision, visible in the query log. Our comparison of private LLM hosting in the EU sets out the options.

Rules that apply from day one

This is general information, not legal advice; national practice may differ. Under the GDPR, the assistant processes personal data in prompts, documents and logs, so it needs a legal basis, rules for retention and often a data protection impact assessment, as our DPIA guide describes. Under Article 4 of the EU AI Act, applicable since 2 February 2025 and amended by Regulation (EU) 2026/1744, every company that uses AI systems takes measures to support the development of AI literacy of its staff. Since 2 August 2026 the transparency duties of Article 50 apply where the use fits them, for example when people interact with an AI system. Our articles on the EU AI Act for companies using LLMs and AI literacy training under Article 4 explain both. A written AI usage policy helps, because staff who already use public chatbots need to know what moves to the new server.

Starting with a pilot

A pilot answers two questions at once: does the assistant help with a real task, and how large does the production server have to be. These steps keep it measurable.

1. Choose one process with a result you can count, such as answered support questions, contracts checked or minutes written.
2. Shortlist two or three models whose licences allow company use and whose languages match your staff.
3. Build a test set of real questions with correct answers, and have the users who know the subject grade the answers.
4. Run the pilot on the entry server with two RTX PRO 6000 Server Edition. Latency carries over from a pilot only on the same GPU type, engine and settings, as our guide to what a private AI pilot should measure explains.
5. Log every request at the gateway: arrivals in the busiest hour, prompt and answer lengths, time to first token.
6. Decide with these figures: add cards or a second server, change the model, or stop.

The entry server stays useful after the pilot, as the first production server, a test system for new models or the second server for failover. Our guide to rolling out a private AI assistant to 1,000 employees covers the waves after the pilot. Our Private AI/ML service starts with a free first call and a pilot on one process with clear metrics; describe the process you would start with, and we propose a pilot and the server for it.

Room, power and operation

By our estimate the two-card server draws about 1.8 kW at the wall with two EPYC 9135 processors and fits one 230 V 16 A feed, which carries 2.94 kW at 80 per cent; the four-card server draws about 3.4 kW with the cards at 600 W and needs 32 A or three-phase, or about 2.7 kW with the cards capped at 450 W. Eight cards draw about 6.3 kW by our estimate; Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, and feeds should be planned on the maker's figure. All of them belong in a server room or data centre, for noise and heat, as our guide to a GPU server in an office explains. Once the platform runs, our Private AI/ML service trains your team to run it, and support under an agreed SLA is available.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build the servers to order with two, four or eight RTX PRO 6000 Server Edition or H200 NVL cards, assembled and tested before delivery, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. Models, RAG, the platform and the pilot are our Private AI/ML service, built by our engineering partner Vixen.UNO, and our guide to buying a GPU server for LLMs lists every configuration.

FAQ