AI server for a security operations centre: alert triage and log analysis on your own GPUs

What an AI server does in a SOC

An AI server for a security operations centre runs a local LLM next to the SIEM, so alerts, logs and incident notes are triaged and summarised on hardware the SOC controls instead of being sent to a cloud model. For one company's SOC, a server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards carries the triage queue and the analysts' questions; a managed security service provider (MSSP) with many tenants starts at four cards.

The model reads an alert together with the events around it, writes a verdict with its reasons for an analyst, answers questions about a host or a user over the logs the SIEM returns, drafts detection rules and turns an investigation into an incident report. The figures below are published ratings and specifications with their source, or our estimates, labelled as such. We wrote this for the head of the SOC or the MSSP's director who decides whether a local model is worth it, and for the platform team that connects it to the SIEM. Our comparison of EDR, XDR, SIEM and MDR explains the tool categories; this page covers the GPUs behind the language model.

Where the tokens come from

A SOC model reads far more than it writes. The SIEM correlates the raw events and raises an alert; an enrichment step pulls the related events, the asset and user records and threat intelligence; the model receives all of it and returns a structured verdict. Nobody sends a day of raw logs to a language model: the SIEM's queries decide what the model sees.

Our summary; token counts are example values, to be replaced by the averages of your own alerts.

The ratio sets two limits. The cache per request is sized for the long prompt, so a 32K context is a sensible ceiling for triage with room for the answer. And the time per alert depends on reading the prompt (prefill) as much as on writing the verdict; prefill is compute-bound and no figure is published for this model on these cards, so it is measured in a pilot with vllm bench throughput at your input and output lengths, the method our guide to batch LLM inference on-premise describes.

Choosing a model for security work

Security analysis is a narrow task, and general rankings say little about it. Elastic publishes an LLM performance matrix for the AI features of Elastic Security 9.0 and later; we read it on 24 October 2026, and its open-source section is marked as last measured in September 2026. It rates each model from 1 to 10 per task and treats 5 or below as not recommended.

Elastic, Large language model performance matrix, Elastic Security 9.0+, open-source models, "last measured in September 2026", read 24 October 2026; excerpt of four rows and three of its Agent Builder task columns. Ratings are Elastic's, for its own features and prompts.

The matrix rates models inside one vendor's features, so it is a starting point, not a verdict for your SIEM. It shows that a model that serves a general assistant well, gpt-oss-120b, can fail at alert analysis in a security tool, and that mid-size dense models can do better. The same matrix gives Gemma 4 31B IT 4.3 for Elastic's Attack Discovery, below its own threshold. Build a test set from a few hundred of your own closed alerts with the analysts' verdicts and score each candidate on it before you size the server.

Our Private AI/ML service sets up candidate models on a pilot server and scores them on your closed alerts; describe your SIEM and how alerts reach your analysts, and we propose the pilot and the hardware for it.

Google's model card for gemma-4-31B-it, read on 24 October 2026, lists 30.7B parameters, a context of 256K tokens, native support for structured tool use and the Apache 2.0 licence. The files page lists two safetensors files of 49.8 GB and 12.8 GB, 62.6 GB in BF16. Our guide to Gemma hardware requirements covers the other sizes.

Elastic documents two ways to connect its AI Assistant to a self-hosted model through its OpenAI connector set to "Other (OpenAI Compatible Service)": one with LM Studio, and one with vLLM for air-gapped environments, which Elastic says "does not require any outbound network access". The same pages note that this connector "is deprecated and is being progressively removed from the create connector UI" in favour of Elasticsearch inference endpoints, so check the integration path for your version. Elastic's vLLM guide was run on a cloud server with two A100 80GB and the vLLM 0.9.1 image; on RTX PRO Blackwell cards use a current vLLM release.

Sizing by alerts per day

By our users-per-card rule, which takes 90 per cent of the 95.6 GiB the driver reports, less 3 GiB per card and less the weights, Gemma 4 31B in BF16 (58.3 GiB) leaves 24.7 GiB of cache on one card. Its ten full-attention layers have four KV heads of dimension 512 and its fifty sliding-window layers attend over 1,024 tokens, so a 32K request takes up to about 3.3 GiB of 16-bit cache, as our Gemma guide counts it. One card holds 7 requests at 32K and a tensor-parallel pair, with 107.8 GiB of cache, 32. vLLM's recipe for the 31B says an FP8 cache "saves ~50% KV memory", which roughly doubles these counts.

Writing speed has a ceiling set by memory bandwidth: each step reads the weights and the cache of every request in flight. With 1,597 GB/s per card and seven full 32K requests, one card reads about 81 GiB per step, about 18 steps a second, so at most about 128 output tokens a second for the seven together. A pair splits the weights and holds 32 requests, for at most about 580 a second. These are upper bounds by our estimate: they leave out prefill and the exchange between two cards over PCIe. We plan with half of them, about 64 and about 290 tokens a second.

Our estimates for Gemma 4 31B in BF16 with a 16-bit cache at 32K and 500 output tokens per alert; ceilings from 1,597 GB/s per RTX PRO 6000 Server Edition (NVIDIA), weights from the Hugging Face files page and cache from config.json as counted in our Gemma guide; alert volumes are example values. Prefill of the 12,000-token prompts is not included and is measured in a pilot.

A single SOC with 2,000 alerts a day that reach the model fits one card with room for analysts' questions, but an alert storm queues for about an hour on one card and clears in about a quarter of an hour on a pair. Plan for the storm: it decides whether the model gets one card or a pair. The MSSP example needs about 14 hours on one pair, 7 hours on two pairs and 3.6 on four, before prefill, which is why an MSSP plans its cards as tensor-parallel pairs.

Configuration options by card count

The options follow the reference configurations on our AI servers page and NVIDIA's guide for certified systems: two processors, at least six cores per GPU and system memory of twice the GPU memory.

Reference configurations on our AI servers page; requests and tokens per second by our estimate for Gemma 4 31B in BF16, 32K, half the bandwidth bound; power by our estimate with the cards at 600 W, processors and 15 per cent for the rest of the server.

Two cards. The entry server has two RTX PRO 6000 Server Edition, two AMD EPYC processors, 384 GB of DDR5 ECC and two NVMe drives. Run the model as one copy over both cards: the pair holds more than four times the requests of one card and clears an alert storm more than four times faster by our estimate. Alternatively, card 1 runs the model for triage and card 2 is split by MIG into four 24 GB instances for an embedding model over past incidents and playbooks, a reranker and a test instance; this suits a SOC with a steady, low alert volume.

Four cards. The inference node with 768 GB and four NVMe drives runs two pairs. A SOC gives one pair to the triage queue and one to the analysts, so a storm never blocks a question during an incident. An MSSP gives the pairs to different tenant pools or keeps both behind one queue. This is the configuration we would size first for an MSSP.

Two four-card servers or eight cards. Triage that runs every hour of every day needs a second server, so one can fail or be patched while the other works the queue, as our guide to LLM high availability with two GPU nodes describes. Eight cards in one server give four pairs at about 6.3 kW by our estimate, without failover.

The RTX PRO 6000 Max-Q is the workstation card, a 300 W blower card; in a rack it belongs only in a chassis whose maker lists it, and for continuous operation we recommend the Server Edition. We build these servers and set up the model and the connection to your SIEM through our Private AI/ML service; send us your alerts per day, your peak hour and the SIEM you run, and our team replies with a configuration.

Logs are untrusted input

Much of what a SOC model reads was written by the attacker: a user-agent string, a URL path, a file name, an email subject, a command line. Any of them can carry text aimed at the model, such as an instruction to call the alert benign. OWASP lists prompt injection first, as LLM01, in its Top 10 for LLM Applications 2026, and our guide to prompt injection and LLM security explains why no filter removes it reliably. The design therefore limits what an injected instruction can do.

1. The model advises, people act. It writes verdicts and drafts; closing an alert, isolating a host or blocking an account stays with an analyst or with a playbook whose rules are outside the model.
2. Read-only tools. If the model may query the SIEM, give it a read-only account limited to the tenant and the time window of the alert.
3. Mark the evidence. Put log content in a delimited block of the prompt, cut overlong fields and tell the analyst in the output which fields came from the event.
4. Measure misses. Add alerts with injected text to the test set and check that the verdict does not follow them.

Tenant separation for an MSSP

An MSSP processes each client's logs on the client's behalf, so Article 28 of the GDPR asks for a contract with each, and clients often ask where their logs go and who else shares the hardware. There are three levels of separation.

• A card or a pair per tenant. Physical separation for the largest clients or the strictest contracts. Gemma 4 31B needs a whole card.
• A MIG instance per tenant. MIG splits an RTX PRO 6000 into up to four 24 GB instances with their own memory and fault isolation. An instance holds an embedding model or a model of about 8B parameters in BF16, about 16 GB of weights at 2 bytes per parameter by our arithmetic, not the 31B; Elastic's matrix rates the small models it lists low for alert analysis, so test before you offer it.
• A shared model with logical separation. One model serves all tenants; the gateway gives each tenant its own keys, index, log stream and retention. vLLM's prefix cache reuses computed blocks between requests with the same prefix, and its design documentation for version 0.32.0 describes per-request salting that "prevents timing-based attacks where an adversary could infer cached content by observing latency differences": with a cache_salt per tenant, "only requests with the same salt can reuse cached KV blocks". Set it at the gateway, not in the client.

Our guide to self-hosted LLMs for SaaS products covers multi-tenant serving in more detail.

Notes for the IT team

1. Integration. vLLM serves an OpenAI-compatible API, which the SIEM's connector or the SOAR playbook calls. Put an LLM gateway with keys and a query log in front of it and log tenant, alert ID, model version and output.
2. Structured verdicts. vLLM's structured outputs make the answer follow a JSON schema with fields such as verdict, confidence and cited event IDs, so the SOAR platform can route it.
3. Network. vLLM's --api-key protects only paths under /v1, /v2, /inference and /cohere, while routes such as /invocations and /metrics answer without the key (vLLM 0.32.0 security guide), so keep the engines on an internal management network behind the gateway. Run vLLM 0.31.0 or later, the NVIDIA Container Toolkit at 1.17.8 or later and a driver NVIDIA lists as fixed in its bulletin of 30 September 2026. Our guide to securing a GPU server covers the host.
4. Data. Logs hold user names, IP addresses and email addresses, which are personal data; set retention for prompts and outputs in line with the SIEM's own. For entities under NIS2, our article on NIS2 logging and monitoring requirements covers what the rules ask of logs.
5. Model updates. Pin the model and run each new version through your alert test set before it replaces the old one, as our guide to LLM evaluation before model upgrades describes.

Power and room

By our estimate, the two-card server draws about 1.8 kW at the wall with two EPYC 9135 processors and fits one 230 V 16 A feed, which carries 2.94 kW at 80 per cent. The four-card server draws about 3.4 kW with the cards at 600 W and needs 32 A or three-phase, or about 2.7 kW with the cards capped at 450 W. Eight cards draw about 6.3 kW by our estimate; Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, and feeds should be planned on the maker's figure. A measurement on the delivered system confirms the draw. Our guide to GPU server power on 16 A, 32 A and three-phase feeds works through the feeds. Without a server room of your own, our Private AI/ML service runs the platform on dedicated hardware in a Tier-3 data centre in Lithuania, inside the EU.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build the servers to order with two, four or eight RTX PRO 6000 Server Edition cards, assembled and tested before delivery, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. Our Private AI/ML service, with engineering by our partner Vixen.UNO, deploys the model, the gateway with per-tenant keys and logs, and the connection to your SIEM. Our guide to buying a GPU server for LLMs lists every configuration.

FAQ