Speaker diarization on-premise: models, GPU time and sizing next to speech recognition

What diarisation adds to a transcript

Speaker diarization on-premise means running a small model on your own server that marks who spoke when, next to the speech recognition that writes down what was said. Two open models cover most company use in October 2026: NVIDIA's Nemotron 3 Diarization, about 100M parameters, up to eight speakers, licensed for commercial use, and pyannote community-1 under CC BY 4.0. The diariser itself needs little GPU time; recognition and the language model that writes the minutes decide which card you buy.

A transcript without speaker labels is hard to use for minutes, call reviews or court records. Diarisation produces anonymous labels, speaker 1, speaker 2 and so on, with start and end times, and the transcript is cut along those labels. Naming the speakers is a separate step. Our guide to Whisper and Parakeet on a speech-to-text server covers recognition models and their memory; this article covers the diarisation stage, its accuracy, its cost in GPU time and the limits to test before you size a server.

Two situations come up in practice. Recorded meetings, interviews and calls are processed after the fact, in batches, and throughput per hour of audio counts. Live meetings, contact-centre calls and subtitles need labels within a second or so, and the number of streams at the same moment counts. A contact centre that records agent and customer on separate channels needs no diarisation at all, as our guide to contact centre AI on-premise explains.

Open diarisation models as of October 2026

Hugging Face cards nvidia/Nemotron-3-Diarization (release 23 September 2026; metadata 99.2M parameters; model.safetensors 397 MB, a q8_0 GGUF of 107 MB) and pyannote/speaker-diarization-community-1, read on 24 October 2026.

Nemotron 3 Diarization is a 31-layer Transformer encoder that reads 16 kHz mono audio and outputs, every 10 ms, a probability of activity for each of eight speaker channels, ordered by when each speaker first appears. For streaming it keeps an arrival-order speaker cache and a FIFO queue, so a speaker who returns after a pause keeps the label. Its card names NeMo Framework 3.0 on Linux and lists Ampere, Ada Lovelace, Blackwell and Hopper GPUs, including RTX PRO. NVIDIA states the model is "ready for commercial or non-commercial use".

pyannote community-1 is a pipeline for pyannote.audio 4.x. Its card states that pipelines run on the CPU by default and move to a GPU with pipeline.to(torch.device("cuda")). You can pass the exact number of speakers, or a minimum and maximum. It also returns an exclusive diarisation, with one speaker per moment, which the card describes as easier to align with transcription timestamps. The files download only after you accept its conditions on Hugging Face; a local clone then loads without an internet connection. Its stronger sibling, precision-2, is a commercial model that, by the card's own code comment, "runs on pyannoteAI servers", so it is not an on-premise option.

NVIDIA's ASR NIM can also add speaker labels inside the recognition container. Its performance page lists profiles named "Silero VAD + Sortformer"; Sortformer is also the NeMo model class, SortformerEncLabelModel, that loads Nemotron 3 Diarization.

Accuracy: diarisation error rate on meetings and calls

Diarisation error rate (DER) adds up missed speech, false alarms and speech given to the wrong speaker, as a percentage of speech time. Lower is better. Both cards publish DER on public meeting and call sets with overlapping speech included.

Model cards of Nemotron 3 Diarization (offline profile 30.4 s, ultra-low-latency profile 0.32 s) and pyannote community-1 (fully automatic, no collar, overlap not skipped), read on 24 October 2026. Nemotron uses a 0.25 s collar on CALLHOME and forced-alignment reference labels for AMI and NOTSOFAR1, which its card calls "not directly comparable" with other labels.

The table shows the scale of the gap, but the two vendors scored with different references, so take it as a reason to test both on your own audio, not as a ranking. Baseten, which serves the model, wrote on 23 September 2026 that Nemotron 3 Diarization at ultra-low latency trailed pyannote community-1 only on AMI among the sets it compared. NVIDIA's card shows the price of low latency within its own figures: on the single-channel NOTSOFAR1 recordings, DER rises from 11.00 to 14.53 between the offline and the 0.32 s profile, and the share of recordings where the speaker count is right falls from 78.12 to 55.00 per cent.

GPU time: what diarisation costs next to recognition

NVIDIA publishes the speed of Nemotron 3 Diarization as RTFx, the inverse real-time factor: seconds of audio processed per second of computing, measured in BF16 on an RTX PRO 5000 Blackwell with PyTorch through NeMo.

Nemotron 3 Diarization model card, RTFx on an NVIDIA RTX PRO 5000 Blackwell, BF16, eager and with torch.compile(), read on 24 October 2026.

At the offline profile and batch 32, the diariser processes 1,000 hours of audio, 3.6 million seconds, in about 238 seconds of GPU time by our arithmetic, or 295 seconds without compilation. Even one file at a time, at an RTFx of 1,340, the 1,000 hours take about 45 minutes. Recognition costs more: on the A100 of the Open ASR Leaderboard paper, cited in our Whisper guide, Parakeet TDT 0.6B v3 reached an RTFx of 3,333 and Whisper large-v3 145.5, which turns the same 1,000 hours into about 18 minutes and 6.9 hours.

The wiring changes the result more than the model. NVIDIA's ASR NIM performance page, updated on 7 October 2026, measures English Parakeet 0.6B CTC offline with 32 parallel requests. With diarisation switched on, the L40S drops from an RTFx of 3,638 to 101.5 and the H100 from 3,707.4 to 170; the 1,000 hours then take about 9.9 hours on the L40S. The Vietnamese profile with Silero VAD and Sortformer loses far less, from 336.4 to 260.5 on the L40S. The page does not say which diariser the English profile uses. Before you size a server from a pipeline, measure the pipeline you will run, with diarisation on.

Live meetings and calls: streaming profiles

For live audio, Nemotron 3 Diarization has three streaming profiles with input buffer latencies of 1.04, 0.64 and 0.32 s; the card states that the checkpoint supports buffers down to 80 ms, but recommends 0.32 s as the lowest. Baseten describes speaker labels that update as often as every 320 ms. At 0.32 s and batch 32, the compiled model processes 292 seconds of audio per second on the RTX PRO 5000. Since each live stream delivers one second of audio per second, that is, by our reading, an upper bound of about 290 streams at that profile, before recognition shares the card.

Baseten reported on 23 September 2026 that one RTX PRO 6000, edition not stated, carried "over 500 concurrent hour-long diarization streams" at its low-latency setting and 190 when transcription was added, using NVIDIA's 600M-parameter English Parakeet model, with load generated by k6 inside the cluster. Diarisation alone is rarely the limit; the recogniser running next to it is.

NVIDIA's integration guide on the model page pairs the diariser with two streaming recognisers: Multitalker Parakeet Streaming 0.6B for English, and Nemotron 3.5 ASR Streaming 0.6B, which supports 32 locales out of the box. The ASR stage keeps one transcription stream per detected speaker, so memory and compute grow with the number of people talking. Each live connection needs its own speaker cache, decoder state and buffers, while the weights are shared, and the guide warns that the chunk sizes of the two models must stay aligned.

We supply the RTX PRO 4500 Server Edition, the RTX PRO 5000 and the RTX PRO 6000 Server Edition for speech servers; send us your hours of audio per day, the number of live streams and the languages, and our team replies with a card and server configuration.

Sizing per 100 and 1,000 recorded hours a day

For recorded audio, add the GPU time of each stage and compare the sum with the hours the server is available, for example a night of eight hours. The figures below come from three different GPUs, so they show the proportions, not the time on your card.

Our arithmetic: hours × 3,600 ÷ RTFx. Rates from the Nemotron 3 Diarization card (offline profile, batch 32 compiled and batch 1 eager), the Open ASR Leaderboard paper as cited in our Whisper guide, and NVIDIA's ASR NIM performance page of 7 October 2026.

With Parakeet and the Nemotron diariser as separate stages, 1,000 hours a day are well under an hour of GPU time on paper; the LLM that writes summaries then becomes the largest load. With Whisper large-v3, or a pipeline that behaves like the diarised NIM profile, the same volume fills a card for most of a night.

Three layouts cover most requests. For an existing server, one RTX PRO 4500 Server Edition, 32 GB, 165 W, passive and single-slot, carries diarisation and recognition; by our users-per-card rule it holds 25.7 GiB of weights and cache, and with MIG it splits into two isolated 16 GB halves, one per stage. NVIDIA publishes no diarisation figure for this card, and its memory bandwidth is 800 GB/s against 1,344 GB/s on the RTX PRO 5000 that NVIDIA measured, so plan with margin and confirm on a test. For transcripts with minutes, add a second card for the LLM: gpt-oss-20b at 13.8 GB fits one RTX PRO 4500 Server Edition. For a contact centre or a broadcaster with hundreds of live streams and a 120B-class model for summaries, the entry server on our AI servers page, two RTX PRO 6000 Server Edition cards with 96 GB each, splits speech and language onto one card each; our guide to an AI server for media uses that layout.

Eight speakers, overlap and language: what to test

Nemotron 3 Diarization labels at most eight speakers. A board meeting, a call or an interview fits; a council session or a panel with a dozen voices does not, and pyannote community-1, where you set a minimum and maximum, is the model to test there. NVIDIA's integration guide adds that the configured maximum is "an upper bound, not a promise that every configured speaker will appear".

Overlapping speech is included in both cards' scores, but the recogniser has to handle it too. The guide notes that conventional recognition run on diarised segments is weaker on overlap, because "an extracted time range still contains every voice that overlaps it"; the multitalker recogniser exists for that case.

Languages need their own check. Nemotron 3 Diarization was trained on about 10,000 hours of real conversations and about 82,611 hours of simulated mixtures; its card names English, Mandarin, Hindi, Kannada, Telugu and Bengali and gives no complete list, and pyannote's card names none. Test German, French or Polish recordings from your own rooms, with your microphones, before relying on a published DER.

A useful pilot takes 10 to 20 hours of your recordings, labelled by hand with who spoke when, and measures DER, the share of recordings with the right speaker count, and GPU time per hour of audio, for each model and profile. Our guide to measuring a private AI pilot describes the set-up. Our engineering partner Vixen.UNO sets up speech pipelines as part of our Private AI/ML service; describe your recordings and the output you need, and we size the pilot and the server from its results.

Voice data, speaker names and the GDPR

Recordings in which people can be identified are personal data. Diarisation labels are anonymous and, by NVIDIA's guide, "do not identify real-world names"; mapping speaker 2 to a named colleague needs an enrolment step with stored voice samples. The GDPR defines biometric data in Article 4(14) as personal data from specific technical processing of physical, physiological or behavioural characteristics "which allow or confirm the unique identification" of a person, and Article 9(1) prohibits processing biometric data "for the purpose of uniquely identifying a natural person" unless an exception applies. Whether a speaker-naming step falls under Article 9 is a question for your data-protection officer; our guide to a DPIA for an internal LLM describes the assessment. This is general information as of October 2026, not legal advice.

On the technical side, keep the hosted options out of the pipeline. The pyannote card offers precision-2 and optional hosting of community-1 on pyannoteAI's cloud; the on-premise set-up loads the pipeline from a local copy. Restrict the server's API and harden it as in our guide to securing a GPU server.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the RTX PRO 4500 Server Edition, the RTX PRO 5000 and the RTX PRO 6000 Server Edition as GPUs for servers you already run, or in AI servers built to order, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Operating system, drivers, CUDA and a container runtime are installed on request; the speech pipeline and the LLM for minutes are part of our Private AI/ML service.

FAQ