LLM guardrail models on-premise: which GPU runs the safety checks beside your main model

Where a guardrail model sits and what it needs

A guardrail model, the safety filter that checks prompts and answers, needs its own GPU memory beside the main LLM: a 24 GB device such as an NVIDIA L4, an RTX PRO 4000 Blackwell or a 24 GB MIG instance of an RTX PRO 6000 Blackwell Server Edition holds one of the open safety models of 4B to 21B parameters with room for several checks at once. It should not share the main model's cache budget, because every user request waits for its verdict.

In an on-premise assistant, an input check classifies the user's prompt before the main model receives it, and an output check classifies the answer before it reaches the user. Each turn therefore adds one or two calls to a second model, and a streamed answer checked in chunks adds more. NVIDIA's NeMo Guardrails, an open-source toolkit under Apache 2.0, at release 0.24.1 when we read its repository on 24 October 2026, organises these as input, dialog, retrieval, execution and output rails and includes "NVIDIA safety models (content safety, topic safety)". The threats the checks address are covered in our article on prompt injection and LLM security, and the gateway that routes and logs requests in our guide to an LLM gateway with keys, budgets and logging. This page sizes the hardware for the second model.

Open guard models in October 2026

Hugging Face model cards, file lists and config.json files, and the build.nvidia.com card of the Safety Guard 8B v3, read on 24 October 2026; weights are our sums of the safetensors files; context of gpt-oss-safeguard-20b from max_position_embeddings in its config.json.

Nemotron Content Safety Reasoning 4B, released in November 2025, is "a finetuned version of Google's Gemma-3-4B-it model". It classifies prompt and response as harmful or unharmful against 22 categories, or against a policy you supply, and has two modes: reasoning off, "A low-latency mode for standard, fast classification", and reasoning on, which writes a short trace first. Its card names H100 and A100 as test hardware and shows vllm serve with --max-model-len 8192.

Llama-3.1-Nemotron-Safety-Guard-8B-v3, on Hugging Face since 1 August 2025, is "a multilingual content safety model" that supports 23 categories in English, Spanish, Mandarin, German, French, Hindi, Japanese, Arabic and Thai, with a context of up to 8K and a JSON answer with user safety, response safety and the categories. Its card on build.nvidia.com lists "Test Hardware L40sx1" and vLLM as the engine.

gpt-oss-safeguard-20b and -120b are "safety reasoning models built-upon gpt-oss" that interpret a written policy you provide, with reasoning effort set to low, medium or high, and "should only be used with the harmony format". The 20b card states that it "fits into GPUs with 16GB of VRAM". The models are for safety use; OpenAI's card recommends the general gpt-oss models for other work. Our gpt-oss-120b hardware guide sizes the larger architecture.

Llama Guard 4 12B handles "mixed text-and-image prompts" and comes under the Llama 4 Community License; the licence section below explains why that matters for companies in the EU.

Memory per check

The guard model holds a KV cache for every check in flight, like any LLM. Its size follows from each config.json; the general method is in our guide to how much VRAM an LLM needs.

• Safety Guard 8B v3 has 32 layers with 8 KV heads of 128 dimensions: 32 × 8 × 128 × 2 × 2 bytes = 128 KiB per token in 16-bit, so a check of 4,096 tokens takes 0.5 GiB and one at its 8K maximum 1.0 GiB.
• gpt-oss-safeguard-20b keeps the full context in 12 of its 24 layers, with 8 KV heads of 64 dimensions: 24 KiB per token in 16-bit, plus a window of 128 tokens in the other 12 layers, so a check of 4,096 tokens takes about 0.1 GiB. Its reasoning trace counts towards that length.
• For the Content Safety Reasoning 4B we could not read the config.json on 24 October 2026, so we give its free memory only.

A check carries the guard's instructions or your policy, the user prompt and, for an output check, the full answer. A policy of a few pages plus a long answer reaches 4K tokens quickly, so measure the prompt tokens of real checks before you size.

Our users-per-card rule takes 90 per cent of the memory the device reports and subtracts 3 GiB for CUDA graphs and activations. That leaves 17.2 GiB on an L4 with ECC on, as our eight-L4 server page derives, about 18.5 GiB on an RTX PRO 4000 by our scaling from the 96 GB card, and 18.3 GiB in a 1g.24gb MIG instance of an RTX PRO 6000, which reported 23.62 GiB in a published nvidia-smi output, as our RTX PRO 6000 MIG profiles guide shows.

Our estimate by the users-per-card rule: free memory after the weights and checks held at once, 14.96 GiB of weights (8B), 12.82 GiB (20b) and 9.26 GiB (4B); each check of 4,096 tokens with a 16-bit KV cache. RTX PRO 4000 figure scaled from the 96 GB card's 95.6 GiB, an assumption; check nvidia-smi on the delivered card.

The 8B guard fits a 24 GB card with little room: four checks of 4K on an L4. That is enough for a team pilot with a handful of requests at a time, and short for a service with dozens of requests in flight, where a 2g.48gb instance or a whole card fits better. gpt-oss-safeguard-20b leaves more room because its 4-bit experts take less memory and its cache per token is a fifth of the 8B guard's.

Latency: every check is on the critical path

An input check runs before the main model starts, so its full time adds to the time to first token; an output check delays the moment the user sees the answer, or each chunk of it. Two numbers decide that time: the prompt the guard reads and the tokens it writes.

Writing is limited by memory bandwidth. A dense 8B model in BF16 reads its 16.1 GB of weights for each generated token, so by our estimate one check writes at most about 19 tokens per second on an L4 (300 GB/s), about 42 on an RTX PRO 4000 (672 GB/s) and about 99 on a whole RTX PRO 6000 Server Edition (1,597 GB/s). These are upper bounds; real rates are lower. On an L4, a verdict of 20 tokens therefore takes at least a second. A guard with a short answer, reasoning switched off, or a reasoning effort of low keeps that time down; a guard that reasons before every verdict writes hundreds of tokens and suits offline review better than an inline check.

Reading the prompt is limited by compute, and a long policy plus a long answer costs prefill time on every check. We found no published latency or checks-per-second figures for these models on the L4, the RTX PRO 4000 or a MIG instance as of 24 October 2026, so measure with your policy, your prompt lengths and your concurrency in a pilot. We supply the cards for this second model and fit them into servers you already run; send us your main model, peak requests and the checks you plan, and our team replies with a placement and the memory figures.

Why the guard needs its own memory

vLLM reserves the share of GPU memory set by --gpu-memory-utilization when it starts, 0.92 by default since release 0.20.0, and fills what the weights leave with KV cache. A second engine on the same card needs both shares set lower so that together they stay below the whole card, and the two engines then compete for the same compute. When the main model runs a large batch, the guard's checks wait, and every user request waits with them.

MIG removes that contention. Each instance gets its own memory, L2 cache and streaming multiprocessors, so the guard in a 1g.24gb instance keeps its own memory and compute whatever the other instances run. MIG is set per card, and NVIDIA's MIG guide states that "NCCL is currently not supported with MIG", so a card that carries part of a tensor-parallel main model cannot also be split. In a server with four RTX PRO 6000 Server Edition, the main model runs on whole cards and one card in MIG mode carries the guard, the embedding model and the reranker in separate instances; our four-card Server Edition server describes that layout. Our step-by-step guide to MIG on RTX PRO Blackwell sets it up, and our article on GPU sharing in Kubernetes with MIG, time-slicing and MPS covers the scheduling side.

Three placements

An L4 in a free slot. The L4 is a 72 W, single-slot, low-profile card with 24 GB of GDDR6, so it fits many existing servers beside the GPUs of the main model. It holds the 4B or 8B Nemotron guard for a pilot or a small team, and the Safety Guard 8B v3 card names an Ada card, the L40S, as its test hardware. vLLM's gpt-oss recipe, dated 15 October 2026, documents Hopper, the B200 and Ampere and says that work to make the models run on Ada Lovelace continues, so we do not plan gpt-oss-safeguard on an L4 without a test. Our comparison of the L4 and L40S for inference covers the card.

An RTX PRO 4000 Blackwell. The same 24 GB at 672 GB/s and 145 W in one slot, with FP4 Tensor Cores and more than twice the L4's bandwidth, so a check finishes sooner. It suits a server where the guard sees steady traffic. It is a workstation card, so in a rack server it belongs in a chassis whose maker lists it. The vLLM recipe does not name the RTX PRO cards either, so test gpt-oss-safeguard on the delivered card. Our RTX PRO 4000 Blackwell specs list the card's figures.

A MIG instance of an RTX PRO 6000 Server Edition. In a server with several Server Edition cards, one card split into four 1g.24gb instances carries the guard, an embedding model, a reranker and a PII redaction model, each isolated from the others. A 2g.48gb instance holds the 8B guard with room for 49 checks of 4K tokens by our estimate. For larger services with many models, our guide to serving several models on one GPU server sets out the options.

Licences of the guard models

• NVIDIA Open Model License (both Nemotron guards): "Models are commercially usable." It ends your rights automatically if you bypass, disable or circumvent a safety guardrail of the model without a substantially similar one, and on patent or copyright litigation over the model. The 8B guard adds the Llama 3.1 Community License, the 4B guard the Gemma Terms of Use and the Gemma Prohibited Use Policy, as their cards state.
• Apache 2.0 (gpt-oss-safeguard): a permissive licence; the repository includes the LICENSE file and a short USAGE_POLICY that asks for safe and responsible use and states that using the models means complying with the laws that apply to you.
• Llama 4 Community License (Llama Guard 4): the Llama 4 Acceptable Use Policy states that for multimodal models "the rights granted under Section 1(a) of the Llama 4 Community License Agreement are not being granted to you" if you are "a company with a principal place of business in, the European Union", and that the restriction "does not apply to end users of a product or service that incorporates any such multimodal models". Llama Guard 4 takes images, so for a company based in the EU the other three families are the simpler choice. Our Llama 4 hardware guide covers the clause for the main models.

Whether a clause applies to your company is for your legal counsel to confirm, and our overview of open LLM licences for company use compares the main model families. A guard model is also one part of the controls a company documents for its AI use; our article on the EU AI Act for companies using LLMs gives the general picture.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the L4, the RTX PRO 4000 and the RTX PRO 6000 Server Edition as cards for servers you run, after a compatibility check, or in AI servers built to order, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Deploying the main model, the guard model and the serving layer is our Private AI/ML service, delivered with our engineering partner Vixen.UNO; describe your assistant, its users and its policies, and we propose where each model runs.

FAQ