AI server for consulting firms: research, proposals, client documents on your own hardware

What your own AI server does for a consulting firm

An AI server for a consulting firm runs a language model and its search index on hardware the firm controls, so consultants can search the firm's past engagements, draft proposals and analyse client documents without sending client data to a public AI service. For a firm of up to about 200 consultants, a server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards carries the assistant and the search models; four cards carry about 750 consultants and leave a card for converting and indexing data rooms. The figures in this article are our estimates with example values, labelled as such, and a pilot replaces them with your own.

We wrote this for a managing partner, a COO or a head of IT who has to decide whether such a server makes sense, and for the IT team that will run it. The first half covers the tasks and the configurations; the notes for the IT team come after.

The work in plain words

Consulting produces and consumes documents. Four tasks account for most of what a language model does in a firm.

• Research across the firm's own knowledge. Which earlier projects dealt with a similar question, what the team found, which benchmarks and interview notes exist. The model answers from the documents it retrieves and cites them, as our article on RAG on company data explains.
• Proposals. A first draft from the client's request, earlier proposals, team profiles and references, which a partner then edits. Drafting is the quick part for the model; the value lies in finding the right earlier material.
• Analysis of client documents. Annual reports, contracts, process descriptions, survey results or the contents of a data room in a due diligence: summarised, compared and turned into tables.
• Interviews and workshops. Recordings transcribed, summarised and translated, in the language the client team works in.

The first two use the firm's own documents, the last two the client's. That difference shapes the access rules more than the hardware.

Client data, contracts and where the model runs

This is general information, not legal advice; your legal team assesses your own contracts. Engagement letters and non-disclosure agreements set who may receive a client's information, and data-room terms often add their own conditions. Whether a public AI service counts as a recipient under those terms is a question for each contract. A model on the firm's own server keeps documents, prompts and answers on hardware under the firm's access rules, with no external service unless the firm switches one on. Our article on the CLOUD Act and EU data residency covers what location, operator and contract change when a provider is involved.

The GDPR applies to personal data in client files, interview notes and logs wherever the model runs. Article 32(1) names measures such as the ongoing confidentiality of processing systems and the ability to restore access to data after an incident, and Article 28 governs any provider that processes the data on the firm's behalf. Under the AI Act, a firm that uses an AI system is a deployer; Article 4, applicable since 2 February 2025 and amended by Regulation (EU) 2026/1744, asks it to take measures to support the development of AI literacy of its staff. Our articles on the EU AI Act for companies using LLMs and the DPIA for an internal LLM cover both.

Consulting firms often serve competitors in the same market, so the platform needs barriers between engagements. In the retrieval index every chunk carries the engagement and the access list of its source document, and retrieval filters before it ranks. Access is enforced by that filter, because an instruction in the prompt cannot control it. Our guide to a private LLM for law firms and tax advisers sets out the same design per matter, and it carries over per engagement.

Three configurations and what each serves

The options follow the reference configurations on our AI servers page and NVIDIA's guide for certified systems: two processors, at least six cores per GPU and system memory of twice the GPU memory. The language model in the examples is gpt-oss-120b, released under Apache 2.0, a mixture-of-experts model whose checkpoint of 60.8 GiB fits one 96 GB card.

Consultants use an assistant more than the average employee, so we double the example value of our company-size guide: 8 requests in flight at the busy-hour peak per 100 consultants, instead of 4 per 100 employees. Replace it with your own gateway log after a pilot.

Reference configurations on our AI servers page; chats and reviews by our users-per-card rule with gpt-oss-120b and a 16-bit cache; consultants at 8 requests in flight per 100 (example value); the 235B row with Qwen3-235B-A22B in FP8 and an FP8 cache as in our long-context agent guide; power by our estimate with the cards at 600 W, processors and 15 per cent for the rest of the server.

Two cards. The entry server has two RTX PRO 6000 Server Edition, two AMD EPYC processors, 384 GB of DDR5 ECC and two NVMe drives. Our users-per-card rule takes 90 per cent of the 95.6 GiB the driver reports, less 3 GiB per card, less the weights: 83.0 GiB usable, 22.2 GiB of cache beside gpt-oss-120b. Its config.json gives 8 key/value heads of dimension 64 in 18 full-attention layers, which by our arithmetic makes 1.125 GiB of 16-bit cache per 32K conversation, so 19 fit, or 4 documents at 128K. That covers the example peak of about 235 consultants. MIG splits the second card into four isolated 24 GB instances for the embedding model and reranker, an OCR model and speech-to-text; OpenAI gives about 10 GB of GPU memory for Whisper large.

Four cards. The inference node with 768 GB and four NVMe drives runs the assistant as one copy split over two cards with tensor parallelism. The pair stores the weights once and keeps 2 × 83.0 − 60.8 = 105.3 GiB for the cache: 93 conversations at 32K, or a mix such as 60 chats and 8 document analyses at 128K, which takes 103.5 GiB. That serves about 750 consultants with room for long documents at the peak. The third card converts scanned data rooms with an OCR model, the fourth carries search and speech in MIG instances.

Eight cards, or two four-card servers. Where partners rely on the assistant every day, two four-card servers that each carry the peak keep it running while one is serviced, as our guide to LLM high availability with two GPU nodes describes. Eight cards in one server add a larger model for deep analysis: Qwen3-235B-A22B in FP8 at tensor parallelism of four holds 9 analyses at 131,072 tokens with an FP8 cache by our rule, as our guide to a server for AI agents with long context works out.

The RTX PRO 6000 Max-Q is the workstation card, a 300 W blower card; in a rack it belongs only in a chassis whose maker lists it, and for continuous operation we recommend the Server Edition. We build all three configurations to order; tell us how many consultants would use the server and which document volumes you handle, and our team replies with a configuration.

Data rooms and long documents

A due diligence or a strategy review starts with a pile of documents, often scanned. The work runs in three stages, each with its own load.

OCR by the method of our document AI guide: Datalab's 1.44 pages per second for its 5B chandra-ocr-2 on one H100 80GB, scaled to the RTX PRO 6000 Server Edition by memory bandwidth, about 0.69 pages per second per card. Embedding: NVIDIA lists 144.3 texts of 512 tokens per second on an L40S for its 1B NIM embedding model. Output: StorageReview's gpt-oss-120b figure on four Server Edition cards. Our arithmetic, not measurements.

The OCR stage carries the page volume. By the method of our guide to vision-language models for document AI, one RTX PRO 6000 Server Edition handles about 0.69 pages a second with a 5B OCR model, so 30,000 pages take about 12 hours on one card and about 3 hours on four running one copy each. OCR licences differ: GLM-OCR is MIT and DeepSeek-OCR-2 Apache 2.0, while chandra-ocr-2 carries a modified OpenRAIL-M licence with a revenue threshold, so check the licence before a model goes into client work.

Embedding takes little time compared with OCR. At two chunks of 512 tokens per page, 30,000 pages make 60,000 chunks, about 7 minutes at NVIDIA's L40S rate; we found no published rate for an RTX PRO 6000 MIG instance, so measure it on the target card. The index of a data room is small and belongs to the engagement: delete it with the engagement's other copies when the data-room terms require it.

For summaries and comparison tables, StorageReview measured gpt-oss-120b in NVFP4 on four RTX PRO 6000 Server Edition cards at 47.54 tokens per second per user at batch 32, about 1,520 output tokens per second together. Four thousand documents with a 300-token summary each are 1.2 million output tokens, about 13 minutes of generation by our arithmetic. The prompts are the larger part: a 20-page document is several thousand tokens of input per request, and we found no published prefill rate for these cards, so run a test with your own documents. A single long document, such as an annual report read in one request, takes 4.5 GiB of cache at 128K, and prefix caching reuses it when several consultants ask about the same report. Our guide to batch LLM inference on-premise explains how to queue such runs overnight.

Research and proposals from the firm's own knowledge

The research assistant is a RAG system over the firm's knowledge base: project reports, final presentations, methods, benchmarks and expert interviews. Two choices decide its quality. The first is what goes in: the data owners decide which collections enter the index, which versions are current and which client material may never be reused. The second is the model's languages. The Mistral Small 3.1 card names 25 languages, German, Polish and Romanian among them, Qwen3's launch post lists 119, and EuroLLM covers all 24 official EU languages under Apache 2.0. Our comparison of open LLMs for European languages has the details.

For proposals the model drafts from the client's request, the firm's earlier proposals for similar work, team profiles and approved references. Retrieval must respect the same barriers as research: a reference from one client's project appears only where the firm has permission to use it. The partner edits and signs off the draft. Translation of proposals and reports follows the sizing of our guide to on-premise machine translation, which puts about one million words a day on one 48 GB card with a 9B to 12B translation LLM.

Our engineering partner Vixen.UNO builds this platform in our Private AI/ML service, with RAG that respects each user's access rights and cites the source; describe your knowledge base and how engagement access is granted today, and we propose a pilot and the server for it.

Notes for the IT team

1. Engine and API. vLLM serves an OpenAI-compatible API for the language model and has pooling endpoints for embeddings. Put an LLM gateway with keys, budgets and a query log in front of it, so every request carries the user and the engagement.
2. Access. Single sign-on through your directory service; the access list of each source document travels into the index with every chunk, and changes in the document system re-sync the index.
3. Network. vLLM's --api-key protects only paths under /v1, /v2, /inference and /cohere, while routes such as /invocations and /metrics answer without the key (vLLM 0.32.0 security guide), so keep the engine on an internal network behind the gateway. Run vLLM 0.31.0 or later, keep the NVIDIA Container Toolkit at 1.17.8 or later and the driver at a release NVIDIA lists as fixed in its bulletin of 30 September 2026.
4. Logs and retention. The query log holds client data. Set its retention before go-live; Article 30(1)(f) of the GDPR asks the record of processing to state, where possible, the time limits for erasure. Delete data-room indexes and test copies with the engagement.
5. Software on delivery. The server arrives on request with the operating system, the NVIDIA driver, CUDA and a container runtime. Our guide to RTX PRO Blackwell software support lists the engine versions for these cards.
6. Pilot metrics. Requests in flight at the peak, prompt lengths, time to first token and the share of long documents, as our guide to what a private AI pilot should measure describes.

Power and room

By our estimate, the two-card server draws about 1.8 kW at the wall with two EPYC 9135 processors and fits one 230 V 16 A feed, which carries 2.94 kW at 80 per cent. The four-card server draws about 3.4 kW with the cards at 600 W and needs 32 A or three-phase, or about 2.7 kW with the cards capped at 450 W. Eight cards draw about 6.3 kW by our estimate; Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, and feeds should be planned on the maker's figure. All of them belong in a server room or a data centre, as our guide to GPU server power on 16 A, 32 A and three-phase feeds explains. A firm without a server room can run the same platform on dedicated hardware in a Tier-3 data centre in Lithuania through our Private AI/ML service, so client data stays in the EU.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build the servers to order with two, four or eight RTX PRO 6000 Server Edition cards, assembled and tested before delivery, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. Models, RAG with access per engagement, the platform and the pilot are our Private AI/ML service, with engineering by our partner Vixen.UNO, and our guide to buying a GPU server for LLMs lists every configuration.

FAQ