A local AI server for a small business of 10 to 100 people: what it does and needs

Does a local AI server make sense for a small business

Running its own AI server makes sense for a company of 10 to 100 people when it has texts and documents that should not go to a public AI service and when staff across the firm would use the assistant every day. The smallest sensible setup is one machine with two NVIDIA RTX PRO 6000 Blackwell cards, which runs a company assistant, answers from your own documents, translation and meeting minutes for the whole firm with capacity to spare.

That machine is either a workstation with two Max-Q cards for an office side room or a rack server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards for a server room. This guide is written for the owner or managing director of a small company, often without IT staff of its own. Three terms come up throughout. A GPU, or graphics card, is the part of the machine that does the computing for AI; its memory decides which models fit. A model is the software that writes the answers; gpt-oss-120b from OpenAI, for example, is published under the Apache 2.0 licence and can run on your own hardware. A request is one question or task sent to the model, and a conversation is the series of requests in one chat. The arithmetic behind the numbers for larger firms is in our guide to a private ChatGPT server by company size, and firms of 200 to 2,000 staff have their own guide for mid-sized companies.

How many people use it at the same time

Headcount does not size the server. What counts is how many requests are being answered at the same moment. Our company-size guide works with example values, because we found no published ratio between employees and users at the same time: 40 per cent of staff use the assistant in the busiest hour, each sends 6 requests in that hour, an answer takes 30 seconds, and the busiest minutes bring twice the hourly average.

Example values of our company-size guide (40 per cent busy-hour users, 6 requests an hour, 30 seconds per request, peak factor 2); requests at once = requests per hour × 30 s ÷ 3,600 s × 2, our arithmetic.

One RTX PRO 6000 has 96 GB of memory, of which the driver reports 95.6 GiB. Our users-per-card rule takes 90 per cent of that, less 3 GiB per card, less the model itself. gpt-oss-120b takes about 60.8 GiB, which leaves 22.2 GiB for the memory each open conversation keeps on the card. If each conversation may grow to 32K tokens (a token is a word or part of a word), one conversation needs 1.125 GiB with the 16-bit cache our company-size guide uses, so one card holds about 19 at once. Even at 100 employees the peak uses 4 of them.

At this company size the tasks decide the hardware: which models you want, whether you want spoken meetings turned into minutes, how many documents the assistant searches, and whether it should keep working when a part fails. A pilot with your own staff replaces the example values with real figures, as our guide to what a private AI pilot should measure explains.

Why two cards are the sensible start

Every configuration we propose for a company starts at two RTX PRO 6000 cards. One card would carry the chat model for 100 people, but it leaves no room for the rest.

• The helper models need a place. Answers from your own documents use two small extra models: one that turns texts into searchable form (an embedding model) and one that ranks the passages it finds (a reranker). Speech-to-text needs one more; OpenAI gives about 10 GB of GPU memory for Whisper large. NVIDIA's MIG feature splits one RTX PRO 6000 into up to four isolated parts of 24 GB each, so the second card takes all of these side by side.
• A reserve when one card fails. gpt-oss-120b fits one card, so if a card fails, the chat model can run on the other one until the replacement arrives, with the helper models paused, by our estimate. Who switches it over is agreed with whoever supports the machine.
• Room for a larger model or a second one. Two cards can hold one copy of a model split across both, which leaves more memory for conversations: about 93 of 32K for gpt-oss-120b by our rule. They can also run a second model next to the first, for example one that is stronger in a language your staff need, as our guide to open LLMs for European languages shows.

Many firms start with a model on a single office PC. That is a good way to try the idea, and our article on moving from Ollama on a PC to a shared company AI server covers the step to a machine that the whole firm can use.

What one server does for a small firm, and what it does not

In a firm of 10 to 100 people one machine serves every department, each with its own task on the same model or on a model of its own.

• Office and sales. Drafting offers and letters, rewording, summarising long email threads, replies in another language.
• Answers from your documents. Manuals, procedures, contracts, specifications and price conditions become searchable in plain language, with the source document named and each user's access rights respected. Our article on RAG on company data explains how this works and where it fails.
• Meeting minutes. Recorded meetings are transcribed and summarised; our guide to meeting minutes with a local AI covers the steps.
• Translation. Internal documents and customer texts are translated without leaving the company network, as in our guide to machine translation on-premise.
• Quality, service and production. Reports, complaints and service notes drafted from your own templates and past cases.

Many of these office tasks would need only a small model. Summarising an email does not call for a 117B-parameter model. The point of one company server is that the same machine carries the small tasks of every department next to the larger ones, such as searching all manuals at once, so the firm buys and looks after one machine instead of one tool per department.

Some things it does not do. A customer chatbot on your website is a different product, and we do not build one. The model does not learn from what staff type: it stays as installed, and answers from documents come from searching them at the moment of the question, which our comparison of fine-tuning and RAG explains. Answers can be wrong while sounding right, so a person still checks anything that goes to a customer or into a contract.

Three hardware setups for a small company

This is the hardware section, for you and for whoever handles your IT. All three setups use the same card family, so a pilot on one carries over to the others.

Power at the wall by our estimate: cards (300 W Max-Q, 600 W Server Edition) plus processors plus 15 per cent for the rest, as in our guides to the two-card Max-Q workstation, a GPU server in an office and the mid-sized company server; a 16 A circuit at 230 V carries 2.94 kW at 80 per cent.

Workstation with two RTX PRO 6000 Max-Q. The Max-Q is NVIDIA's workstation edition of the card: 300 W, with a blower fan that pushes the heat out of the case, and up to four per workstation according to NVIDIA. We build it on an AMD Threadripper PRO or Intel Xeon W platform with 384 GB of system memory, twice the GPU memory as NVIDIA's rules for certified systems ask of servers, and two NVMe drives. It runs on an ordinary 230 V socket. Ask for a platform with out-of-band management (a BMC, a small separate computer on the board that lets a support provider check and restart the machine remotely), since not every workstation board has one. If the case and power supply are chosen for four cards from the start, adding two more later is simple, as our page on buying a two-card Max-Q workstation describes.

Rack server with two RTX PRO 6000 Server Edition. This is the entry configuration on our AI servers page: two Server Edition cards, two AMD EPYC processors, 384 GB of DDR5 ECC memory and two NVMe drives, with a BMC for remote management and redundant power supplies. The Server Edition has no fan of its own and is cooled by the server's fans, which makes the machine loud. For a machine that runs continuously in a rack we recommend this edition over the Max-Q. Our page on buying a two-card Server Edition server lists the parts.

Four cards. Four cards are for models that do not fit 192 GB, or for several larger models at once; at 10 to 100 staff the number of users alone does not call for them. A four-card Max-Q workstation stays on one 16 A circuit of its own; a four-card Server Edition server needs 32 A or three-phase power.

We supply the two-card workstation and the two-card server, both built to order and assembled and tested before delivery; tell us how many people would use it and for which tasks, and our team replies with a configuration within one business day.

Where the machine stands

Practically all the power a machine draws becomes heat in the room, so a workstation at 1.1 kW heats the room like a 1.1 kW heater for as long as it works. The room needs cooling that runs whenever the machine runs, at night, at weekends and in winter, and our guide to a GPU server in an office shows how to size it. ASHRAE recommends 18 to 27 °C at the air intake of the machine.

Noise decides between the office and a separate room. As of October 2026 we found no maker's noise declaration for a tower with RTX PRO 6000 cards under load, so a workstation belongs in a side room with a door, not next to desks. Rack servers are louder: the same guide cites a declared sound power of 8.5 B for a 2U server with four 400 W GPUs under load, the level of a machine for a server room.

Power and network are simpler. A two-card machine fits one 230 V 16 A circuit if printers or kettles do not share the breaker; ask the electrician which outlets do. It also needs a wired connection to the office network. The full list of building checks, and who does each one with the electrician and the IT service provider, is in our article on an AI server without an IT department; our checklist for preparing a server room covers rack depth, power outlets and access. If there is no suitable room at all, the machine can run in a data centre instead (see below).

Who looks after it

A small firm can run its own AI server without an IT department. It does need one person who owns it: someone who decides which documents go in, who may use the assistant and what to do with feedback. Installation, updates and repairs can sit with a supplier. Our article on an AI server without an IT department splits the tasks between that person, an existing IT service provider and a support provider, and explains what happens when a part fails.

On request the machine arrives with the operating system, the NVIDIA driver, CUDA and a container runtime installed. The models, document search, the web interface and user accounts are our Private AI/ML service, with our engineering partner Vixen.UNO; afterwards your own staff take over or our support works under an agreed SLA. The steps from the first conversation to staff using the assistant are in our guide to the AI server project from request to use.

Your own server, an EU data centre or a cloud subscription

A machine of your own suits a firm that uses the assistant every day and wants its documents to stay in its own network. A per-user subscription to a cloud AI service needs no hardware and suits firms that would rather not run anything; our comparison of ChatGPT Enterprise and your own AI server sets out what each provider's terms state. Between the two, our Private AI/ML service also runs the models on dedicated hardware in a Tier-3 data centre in Lithuania, so the data stays in the EU without a server room of your own. A hybrid setup sends single tasks to a public AI service only where you explicitly enable it, and the query log shows what went out.

For the running costs, our article on AI server electricity per year gives the kilowatt-hours. What own hosting changes under the GDPR, and what stays your duty, is in our guide to local AI and the GDPR; your data-protection officer or legal counsel confirms the assessment for your firm.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build workstations with two or four RTX PRO 6000 Max-Q and rack servers with two or four RTX PRO 6000 Server Edition to order, assembled and tested before delivery, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. Sizing is free of charge. Models, document search and the platform are our Private AI/ML service.

FAQ