Voice agent server on-premise: ASR, LLM and TTS on 2, 4 or 8 RTX PRO 6000 Server Edition

A voice agent on one server: what it runs and how many calls

An on-premise voice agent server runs speech recognition, a language model and speech synthesis on the same machine, so that a caller hears an answer less than about a second after they stop speaking and no audio leaves the company network. NVIDIA's voice agent blueprint shows the layout: one GPU for streaming recognition, one for synthesis and two for the LLM, measured on four B200 GPUs at 0.86 to 0.93 s end to end from 1 to 64 parallel streams. On NVIDIA RTX PRO 6000 Blackwell Server Edition cards we plan two cards for about 30 concurrent calls and four cards for up to 64, by our estimate, and eight cards or two four-card servers for more calls with failover.

The figures here are estimates for a request we often see: a company wants a voice agent for a hotline, an appointment line or an internal service desk, in English and one or two European languages, on hardware it controls. Our guide to contact centre AI on-premise covers transcription and summaries of calls between people, and our Whisper and Parakeet sizing guide the recognition models in depth.

The pipeline in plain words

A caller speaks. A turn detector decides when the caller has finished, a streaming recogniser turns the audio into text while it arrives, the language model writes the answer, and a synthesiser speaks it back while the model is still writing. All four steps run for every turn of every call, and the caller notices any one of them that is slow.

This chain is called a cascaded pipeline, and NVIDIA's blueprint is built this way. An orchestration layer holds the call together: NVIDIA's blueprint uses Pipecat and names LiveKit Agents as an alternative, and its repository is published under the BSD-2-Clause licence. NVIDIA's evaluation of the blueprint includes user interruptions, the caller cutting in while the agent speaks, which has to stop the synthesis and start a new turn.

For the hardware, three things follow. Recognition needs one stream per call for as long as the call lasts. The language model needs a quick first token for each turn, more than raw throughput. Synthesis needs a quick first audio chunk for every agent that speaks at that moment. Each of the three slows down as more calls share its GPU, and NVIDIA's measured layout gives each its own.

The latency budget from end of speech to first audio

NVIDIA's blueprint card on build.nvidia.com publishes its measurement on four B200 GPUs with Nemotron Streaming ASR 0.6B, Magpie TTS Multilingual and the Nemotron 3.5 Lightning LLM. The table gives the values without units; they read as seconds.

NVIDIA Nemotron voice agent blueprint card on build.nvidia.com (page updated 3 March 2026), read on 24 October 2026; four B200 GPUs: one for ASR, one for TTS, two for the LLM. B200 systems are not part of our range; the figures show the shape of the budget.

Half of the time goes to deciding that the caller has finished and producing the final transcript. The stages in the table add up to 0.72 s at 64 streams, so by our subtraction about 0.21 s is spent outside them, in the LLM writing the first phrase the synthesiser needs and in moving the audio. NVIDIA also evaluated two turn detectors, Pipecat Smart Turn with its default 1 s timeout and Silero VAD at 500 ms, on Full-Duplex-Bench 1.0. The turn detector's timeout has more effect on the budget than any other single setting: with a shorter wait the agent answers sooner and cuts in more often on callers who pause.

For our configurations we plan with an example budget of one second at the busy hour: 0.5 s for turn detection and the final transcript, 0.2 s to the LLM's first token, 0.12 s to the first synthesised audio and 0.18 s for the first phrase and transport. A pilot replaces each share with a measured value.

Synthesis gets slower as more voices speak at once. NVIDIA's blog on Magpie TTS of 10 August 2026, citing its TTS NIM documentation, gives a time to first audio of 47 ms for one stream and 275 ms for 64 on an H100, and 32 and 239 ms on a B200. A synthesiser that serves 64 speaking agents uses more than our 0.12 s on those GPUs, so the share of agents speaking at the same moment matters as much as the number of calls.

Models for each stage

Recognition for live calls needs a streaming model. Whisper works on 30-second windows and NVIDIA's ASR NIM runs it offline only, as our speech-to-text guide explains, so a voice agent uses Nemotron Speech Streaming 0.6B in English, with chunks of 80 to 1,120 ms, or its multilingual extension Nemotron 3.5 ASR Streaming 0.6B, published on 4 June 2026 for 40 locales, German, Polish, Czech, Slovak, Hungarian and Romanian among them; NVIDIA's ASR NIM support matrix 26.05.0 lists German as transcription-ready and the other five as broad-coverage locales. The blueprint also lists Parakeet CTC 1.1B and the Parakeet 1.1B RNNT multilingual model.

The language model in NVIDIA's blueprint is a 30B-class mixture-of-experts model with 3B active parameters, Nemotron 3.5 Lightning or Nemotron 3 Nano, with Nemotron 3 Super 120B-A12B as the larger option. Few active parameters keep the first token quick. gpt-oss-20b, at 13.8 GB, and gpt-oss-120b, at 65.3 GB with 5.1B active parameters, are open alternatives under Apache 2.0 that fit one RTX PRO 6000. A voice turn is short, so the cache is small: a conversation of 4,096 tokens takes 144 MiB on gpt-oss-120b by our arithmetic from its config.json.

NVIDIA's Magpie TTS blog on Hugging Face (10 August 2026; size given as 364M in the text, 357M on the model listing); Resemble AI's Chatterbox card on Hugging Face; NVIDIA TTS NIM support matrix 26.05.0 (updated 25 June 2026); all read on 24 October 2026. Magpie size and languages as in the blog; the support matrix lists nine Magpie locales.

Magpie's twelve languages are English, Spanish, French, German, Italian, Vietnamese, Mandarin, Hindi, Japanese, Arabic, Korean and Brazilian Portuguese. Chatterbox adds Polish, Dutch, Danish, Swedish, Norwegian, Finnish and Greek, among others; NVIDIA's support matrix calls it a community model and notes that its English quality is strongest. None of the four TTS models in that matrix lists Czech, Romanian or Hungarian, so for those languages the synthesis model is the first thing to test. Every audio file Chatterbox generates carries Resemble AI's Perth watermark, according to its card.

GPU memory per stage

NVIDIA's speech NIM containers reserve memory by batch size. The TTS NIM requires compute capability 8.0 and at least 16 GB of VRAM, and lists "Blackwell RTX 60xx" among its GPUs. Magpie TTS Multilingual takes 10.87 GB at batch 8, 31.55 GB at batch 32 and 60.22 GB at batch 64. The ASR NIM support matrix 26.05.0 (updated 25 June 2026) lists Nemotron ASR Streaming at 6 GB of GPU memory with an ASR batch of 32 and 14.6 GB with an ASR batch of 64.

By our sizing rule, 90 per cent of the memory the driver reports less 3 GiB per card, one RTX PRO 6000 has 83.0 GiB to work with. The streaming ASR at batch 64 and Magpie at batch 32 together take 46.15 GB, 43.0 GiB, so recognition and synthesis share one card with room to spare. Magpie at batch 64 alone takes 56.1 GiB and is better placed on its own card. An LLM card with gpt-oss-120b keeps 22.2 GiB beside the weights, enough cache for well over a hundred short conversations by our arithmetic.

We supply these servers with two, four or eight RTX PRO 6000 Server Edition; send us the languages, the expected concurrent calls and your latency target, and our team replies with a configuration and the call count we plan for.

Concurrent calls per configuration

Count recognition streams as calls, one per caller channel. Count synthesis streams as the agents that may speak at the same moment, which in the worst case is every call. By our estimate the LLM stage is not the limit: in MLPerf® Inference v6.1, eight RTX PRO 6000 Server Edition in an HPE ProLiant Compute DL380a Gen12 produced 14,581.7 tokens per second on gpt-oss-120b in the server scenario and 39,986.3 on Llama 3.1 8B in the interactive scenario, about 1,823 and 4,998 per card by our division, not an MLPerf metric. If each call needs a 60-token answer every 10 seconds, our assumption, 64 calls ask for about 384 tokens per second. MLPerf® Inference v6.1, data centre, closed division, entry 6.1-0050: gpt-oss-120b, server scenario; Llama 3.1 8B, interactive scenario; retrieved from mlcommons.org on 24 October 2026, results verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Our estimates. Two cards: TTS at the batch-32 profile, every caller may hear the agent at once. Four cards: NVIDIA's blueprint layout, with 64 streams as the point NVIDIA measured on B200 GPUs; on the RTX PRO 6000 the count is a target for a load test. Wall power as in our GPU server power guide.

NVIDIA publishes no streaming ASR or TTS figures for the RTX PRO 6000 that we could find. Its ASR NIM page gives one L40S up to 190 live English streams with Parakeet 0.6B CTC at 160 ms chunks, which by our reading makes recognition for 64 calls the lighter stage and synthesis at 64 concurrent voices the tight one. A load test with recorded calls at the busy-hour count is the step before the order.

Two cards, four cards or two servers

The entry server on our AI servers page carries two RTX PRO 6000 Server Edition cards with two AMD EPYC processors, 384 GB of DDR5 ECC memory and two NVMe drives. For a voice agent, one card runs the ASR and TTS containers, the other the LLM. It matches the smaller layout in the blueprint's GitHub repository, which asks for one GPU of at least 72 GB or two of at least 40 GB each with FP8 or NVFP4 weights, and it runs on one 230 V 16 A feed at about 1.8 kW by our estimate.

The inference node carries four cards with 768 GB of DDR5 ECC and four NVMe drives and repeats NVIDIA's measured layout: recognition, synthesis and two LLM copies, each on its own card. The two LLM copies keep the first token quick at the busy hour and let one copy restart while the other serves.

A voice agent that answers customers becomes part of the service, so plan for a server failure. Two four-card servers, each running the whole pipeline behind the telephony layer, keep half the capacity while one is serviced; our guide to high-availability LLM inference on two nodes covers the routing. An eight-card server holds two sets in one chassis, for more calls on one rack position at about 6.3 kW.

Network, telephony and data protection

The audio itself is light. ITU-T G.711 sets 8,000 samples per second at eight bits, 64 kbit/s per channel, so 64 calls take about 4.1 Mbit/s of audio payload in each direction by our arithmetic; the telephone platform or SIP gateway connects to the orchestration layer on the server.

NVIDIA's speech NIM containers need an NVIDIA AI Enterprise licence for production use, according to NVIDIA's NIM FAQ as cited in our speech guide, while open weights such as Magpie TTS on Hugging Face can also run outside NIM. We supply NVIDIA AI Enterprise licences on the same invoice as the server.

Recorded or transcribed calls in which a caller can be identified are personal data under the GDPR; our contact centre guide covers retention. Under the AI Act, Article 50 has applied since 2 August 2026: providers design systems that interact directly with people so that those people are informed they are interacting with an AI system, unless this is obvious, as our guide to the EU AI Act for companies using LLMs explains. In practice, a voice agent tells the caller at the start of the call that it is an AI system. This is general information, not a legal assessment of a specific system.

What to test in a pilot

1. End-to-end latency from the end of the caller's speech to the first audio, at the 95th percentile, at the planned busy-hour call count.
2. The turn detector's timeout against how often the agent interrupts callers who pause.
3. Time to first audio of the TTS model at the number of agents speaking at once.
4. Recognition accuracy on recorded calls in each language, with telephone audio.
5. Synthesis quality in each language with native speakers, above all outside English.
6. GPU memory and utilisation per stage, to see which card runs out first.

Our engineering partner Vixen.UNO deploys open speech models and LLMs on-premise as part of our Private AI/ML service; describe the calls your agent should handle and the languages it needs, and we set up the pilot and size the server from its results.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight RTX PRO 6000 Server Edition, the passive card for rack servers in continuous operation, assembled and tested before delivery, and NVIDIA AI Enterprise licences on the same quote. Operating system, drivers, CUDA and a container runtime are installed on request. Every component carries the manufacturer warranty, and the order runs on one EU contract and invoice.

FAQ