Customer service chatbot on your own server: model, RAG, concurrent chats, 2 to 8 GPUs

What a customer service chatbot needs from its server

A customer service chatbot on your own server needs GPU memory for one language model and a KV cache for every chat waiting for an answer at the peak, a small card or MIG instance for the retrieval models that find the right passage in your documents, and a second server if the service must stay up during a fault. By our sizing rule, two NVIDIA RTX PRO 6000 Blackwell Server Edition cards running gpt-oss-120b hold 187 chats in flight, enough for several hundred open chats in a mid-sized service; four cards hold twice that. The figures in this article are our estimates for one example, not measurements.

The chatbot here answers customers on a website or in an app, in their language, from your product documentation, terms and help articles, looks up an order or a contract through your systems and hands over to a human agent when it cannot help. Our article on contact centre AI on-premise covers the voice side, call transcription and summaries, and our private ChatGPT server by company size the internal assistant for staff. What we deliver is the server underneath, with the language model and the retrieval set up on it; a simple website chatbot is not what we do.

The task in plain words

For a director, three numbers describe the job. The first is how many customers chat at the same time in the busiest hour, which the current chat or ticket system already records. The second is how many languages the bot has to answer in, which decides the model shortlist. The third is how much of your documentation it has to know, which decides the size of the search index behind it.

The server does two things for every customer message. It searches your documents for the passages that answer the question, and it runs the language model, which writes the reply from those passages and the conversation so far. The model needs most of the hardware. Each chat in progress keeps a working memory on the GPU, the KV cache, and the number of chats that can wait for an answer at once is set by how much memory is left beside the model.

A customer reads and types far longer than the server needs to answer. That is why a server that holds 187 chats in flight serves several hundred open chats: most of them wait for the customer to read and type.

Model, languages and RAG

Three open-weight models show the range. gpt-oss-120b is a mixture-of-experts model with 117B parameters, 5.1B of them active per token, a checkpoint of 65.3 GB under Apache 2.0 with a usage policy, and it fits one 96 GB card. Llama 3.3 70B is a dense model; NVIDIA's FP8 checkpoint takes 67.7 GiB. Mistral Small 3.2, 24B parameters under Apache 2.0, takes 44.7 GiB in BF16 and leaves 38.3 GiB of cache on one RTX PRO 6000 by our rule.

Languages decide between them. The Mistral Small 3.1 card names 25 languages, German, Polish and Romanian among them, and Qwen3's launch post lists 119. Llama 3.3's card names eight languages, German among them, but not Polish or Romanian, and gpt-oss has no language list; OpenAI reports German among the languages of its MMMLU test. Our comparison of open LLMs for European languages has the details, and choosing an open LLM for a company assistant the licence terms. A test set with real customer questions in each language you serve decides the model.

RAG, retrieval-augmented generation, adds an embedding model that turns documents and questions into vectors, a vector index, and usually a reranker that sorts the retrieved passages before the model sees them. These models are small: Qwen3-Embedding-0.6B takes about 1.2 GB in BF16. We size the chat model for a declared context of 16,384 tokens, room for the system prompt, five to ten retrieved passages, the conversation history and the answer. Our guide to RAG on company data explains where retrieval fails first.

From open chats to requests in flight

The memory has to hold the chats whose answer is being generated, the requests in flight. By Little's law from queueing theory, their average number equals the rate of incoming messages times the time each answer takes. With one message per open chat per minute and 20 seconds per answer, 300 open chats send five messages a second and keep 100 answers in flight: a third of the open chats.

These are example values, and your own logs replace them. Twenty seconds covers an answer of about 150 tokens at the latency target below, plus retrieval and the wait for the first token, so a faster server keeps fewer answers in flight. Use the busiest quarter of an hour from your current chat system, and if the bot will also take over channels that humans handle today, add that traffic.

Configuration options by card count

Our users-per-card rule takes 90 per cent of the 95.6 GiB the driver reports on a 96 GB RTX PRO 6000, subtracts 3 GiB per card and subtracts the weights; the rest holds the KV cache. At 16K, gpt-oss-120b takes 0.5625 GiB of 16-bit cache per chat and Llama 3.3 70B 2.5 GiB of FP8 cache.

Our estimates by the users-per-card rule at a declared context of 16,384 tokens; gpt-oss-120b 60.8 GiB with a 16-bit cache, Llama 3.3 70B 67.7 GiB with an FP8 cache; 123.4 GiB usable per H200 NVL. Open chats with the example values above, three per chat in flight.

The worked case for the second row: two cards give 2 × 83.04 = 166.1 GiB, less 60.8 GiB of weights leaves 105.3 GiB, and 105.3 ÷ 0.5625 is 187 chats. gpt-oss-120b has eight KV heads, so the cache splits evenly over two cards. Two copies on two cards hold fewer chats, 78, because each copy stores its own weights, but either card can fail without stopping the bot.

Two cards. The entry server from our AI servers page, two RTX PRO 6000 Server Edition with two AMD EPYC processors, 384 GB of DDR5 ECC and two NVMe drives, carries a bot for a few hundred open chats. Retrieval runs on a small extra card such as an L4, if the chassis has a free slot.

Four cards. The inference node, four cards with 768 GB, carries about a thousand open chats on gpt-oss-120b, or a dense 70B model for about 230. It also leaves room to give one card to MIG instances for the retrieval models and an internal assistant, at the cost of a pair.

Eight cards or H200 NVL. Eight Server Edition cards in one server, or two four-card servers, cover a large service; the second layout is the safer one for customers, as the section on availability explains. Four H200 NVL suit a dense model, with 4.8 TB/s per card for faster streams.

The RTX PRO 6000 Max-Q is the workstation card, a 300 W blower card; in a rack it belongs only in a chassis whose maker lists it, and for continuous operation we recommend the Server Edition. We build each of these servers to order; send us your peak concurrent chats, languages and the model you have in mind, and our team replies with a configuration and the chats it holds.

Retrieval and reranking at chatbot volume

The embedding of one customer question is a light load: NVIDIA's NIM tables list 13 ms for a query of 20 tokens on an L4. The reranker is heavier, because it scores every retrieved passage against the question, so its load is messages per second times passages per message.

NVIDIA reranking NIM performance page (updated 17 August 2026), 40 passages of 512 tokens at three requests in parallel, as quoted in our embedding and reranker guide; messages per second our arithmetic at 20 passages per message.

With the example values, 560 open chats send about 9 messages a second, the limit of one L40S with the 1B reranker, and 1,120 open chats about 19, the limit of the 500m model on the same card. Fewer passages per message raise these figures, at the risk of missing the passage that holds the answer, so test both on your own questions. Our guide to embedding and reranker servers for RAG covers models, engines and MIG placement.

Latency targets a customer notices

A customer sees two delays: the wait until the first words appear and the speed of the text after that. Our article on LLM latency targets sets example targets at the 95th percentile of 2.5 s to the first token and 100 ms per token, 10 tokens per second, for RAG answers with prompts of several thousand tokens. A reply of 150 tokens at exactly that target takes about 17 seconds; at 47 tokens per second the text itself streams in about 3.

The model type matters more than the card count here. StorageReview measured four RTX PRO 6000 Server Edition cards in an HPE ProLiant Compute DL380a Gen12 with vLLM. At batch 32, gpt-oss-120b in NVFP4 at TP=4 gave each user 47.54 tokens per second, while Llama 2 70B Chat gave 8.00 (layout not stated), below a 10-token target. A dense 70B model reads all its weights for every token, which caps one stream at about 23 tokens per second on one Server Edition card by our bandwidth arithmetic; gpt-oss-120b reads about 5 GB per token. For a dense model at many concurrent chats, the H200 NVL with 4.8 TB/s per card is the better fit.

Time to the first token grows with the prompt. Every chat starts with the same system prompt and instructions, and every new message repeats the conversation so far, so vLLM's prefix caching (--enable-prefix-caching) saves recomputing the shared part. Load-test with your own prompts and retrieved passages before you sign off a configuration, as our guide to what a private AI pilot should measure describes.

Notes for the IT team

The model server is one component behind the chat front end, and these points decide how it fits into your systems.

1. API. vLLM serves an OpenAI-compatible API, so most chatbot platforms and frameworks can call it with a changed base URL. Put an LLM gateway with keys, budgets and a query log between the platform and the engine.
2. Tool calls. Looking up an order or opening a ticket is a tool call. vLLM's documentation makes --enable-auto-tool-choice mandatory and pairs it with a parser per model family: openai for gpt-oss, llama3_json for Llama 3.x, which does not support parallel tool calls, hermes for Qwen and mistral for Mistral's format.
3. Authentication. The backend identifies the customer and passes only that customer's data to the tools; the model never sees credentials. Our guide to prompt injection and LLM security explains why a tool must not trust what the model asks for.
4. Network. vLLM's --api-key protects only paths under /v1, /v2, /inference and /cohere, while routes such as /invocations and /metrics answer without the key (vLLM 0.32.0 security guide), so keep the engine on an internal network and expose only the gateway. Keep the NVIDIA Container Toolkit at 1.17.8 or later and the driver at a release NVIDIA lists as fixed in its bulletin of 30 September 2026.
5. Handover. The chatbot platform passes a chat to a human agent; the model can write a short summary of the case for the agent.
6. Data and law. Chat logs hold personal data, so set retention and access rules, and check whether a data protection impact assessment is needed, as our DPIA guide describes. Article 50 of the AI Act has applied since 2 August 2026: a system that interacts with people must inform them that they are interacting with an AI system unless that is obvious. Our article on the EU AI Act gives general information; national practice may differ and your legal department decides.
7. Monitoring and updates. Watch vllm:time_to_first_token_seconds, vllm:kv_cache_usage_perc and vllm:num_preemptions, as our guide to LLM serving metrics shows, and rerun your multilingual test set before every model change, as in LLM evaluation before model upgrades.

Two servers for a service customers depend on

A bot on the website is a public service, and one server is a single point of failure for it: a driver update, a failed card or a power supply swap stops every chat. Two servers that each run the model and carry the whole peak, behind a load balancer that sends requests only to healthy replicas, keep the bot answering. The layout is covered in our guide to LLM high availability with two GPU nodes.

For the example values, two four-card servers with gpt-oss-120b as two pairs each hold 374 chats in flight per server, about 1,120 open chats with one server down. Session affinity at the load balancer returns each chat to the server that holds its cache, which keeps prefix caching effective. By our estimate each four-card server draws about 3.4 kW at the wall with the cards at 600 W and two AMD EPYC 9355 processors, and a two-card server with two EPYC 9135 about 1.8 kW, cards and processors plus 15 per cent for the rest of the server. The configurations are listed in our guide to buying a GPU server for LLMs.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build the GPU servers for the chat model and the retrieval models to order, with RTX PRO 6000 Server Edition, H200 NVL, L40S or L4 cards, assembled and tested before delivery, on one EU contract and invoice, with manufacturer warranty on every component. Our Private AI/ML service, with engineering by our partner Vixen.UNO, deploys the model and RAG on your documents behind your chatbot; the chat widget on your website and its connection to your ticket system stay with your team or your chatbot platform. Describe your chat volume, languages and systems, and we size the servers for them.

FAQ