In-house AI security: ten questions management should put to IT or the provider

When a local AI server is secure enough

An in-house AI is as secure as the server it runs on, the way people sign in and the documents it can read. It is secure enough when staff sign in with their company accounts, the server cannot be reached from the internet, a few named people hold administrator rights, updates follow the vendors' security bulletins, backups are tested by restoring them, and the assistant can only read what the person asking may open. Ten questions below test each point; for every question we give what a good answer contains and the warning signs.

Running AI on an own server keeps prompts and documents inside the company, which is often the reason to buy it. It also adds a new system with its own software, its own administrators and its own weak points. This guide is written for the owner or managing director who has to sign off that system without being an IT specialist, and who wants to know what to ask the IT department or the service provider. Technical terms are explained where they first appear, and each section links the technical guide that your IT team can work from.

Ten questions and what a good answer contains

Our summary; each row is explained in the sections below.

Who can reach the AI, and how staff sign in

Question 1: From where can the AI be reached? An AI server runs an inference engine, the program that holds the model and answers requests, and usually a chat front end that staff open in the browser. Several engines answer anyone who can reach them over the network. The documentation of vLLM, a common engine, says its API key protects only "endpoints under the /v1, /v2, /inference, and /cohere path prefixes", warns against relying on the key alone and recommends a proxy that "explicitly allowlists only the endpoints you want to expose". Ollama, another engine, binds to the server itself (127.0.0.1) by default, but the OLLAMA_HOST setting changes that, and our guide to securing a GPU server notes that its official container image listens on all addresses. A good answer: the engines are reachable only from inside the server or a closed internal network, staff reach the AI through one gateway, and nothing is open to the internet. Remote staff come in through the company VPN like for any internal system.

Question 2: How do staff sign in? The answer you want is "with their normal company account", through the company directory or single sign-on, with two-factor sign-in where the company uses it elsewhere. Separate passwords for the AI multiply accounts that nobody manages. Open WebUI, a widely used chat front end, has self sign-up on by default (ENABLE_SIGNUP is True) and gives new accounts the role "pending" until an administrator approves them, according to its configuration documentation. That is a reasonable default for a test, and a provider should be able to say whether sign-up is switched off and sign-in bound to your directory. Our guide to Open WebUI security lists the settings.

Question 3: What happens when someone leaves or changes department? If access comes from the directory, disabling the account ends access to the AI as well, and a move to another department changes what the assistant can show that person. A list kept by hand in the AI tool goes out of date.

Administrators and the remote management port

Question 4: Who holds administrator rights, on which parts? An AI server has several administrator levels: the operating system, the AI software, the chat front end and the BMC. The BMC, the baseboard management controller, is a small separate computer on the server's main board that lets an administrator switch the server on and off, watch it and reinstall it over the network, even when the server itself is shut down. Whoever controls the BMC controls the server.

That is why BMCs belong on a separate management network. An example shows what is at stake. CVE-2024-54085, a flaw in AMI's MegaRAC SPx BMC firmware, lets an attacker "bypass authentication remotely"; AMI scores it 10.0, the highest value, and the US agency CISA added it to its catalogue of known exploited vulnerabilities on 25 June 2025. A good answer names the people with administrator rights, keeps the BMC off the office network and the internet, uses long individual passwords and two-factor sign-in for administrators, and includes BMC firmware in the update routine. Our guides to privileged access management and firmware updates on a GPU server give the detail.

Updates: who watches for security bulletins

Question 5: Who checks for security updates, and how often? AI software changes fast, and so do its security fixes. NVIDIA's security bulletin of 30 September 2026 lists fixed Linux drivers 580.178.04 and 595.91.07. The first page of Open WebUI's advisory list showed ten advisories from September and October 2026, and vLLM published about thirty in the same two months, as our articles on Open WebUI security and vLLM security advisories record. A server installed once and left alone soon runs versions with known flaws.

A good answer names the person responsible, a fixed routine such as a monthly check of the bulletins of NVIDIA, the engine and the front end, a maintenance window for updates and a record of the versions installed. Ask for that record: a provider that can show it on request has a routine.

What the assistant may read, and prompt injection

Question 6: What may the assistant read, and for whom? An assistant that answers from company documents searches an index of those documents before it answers; this is called retrieval, or RAG. If the search returns a document the user may not open, the model can quote it, whatever its instructions say. The OWASP project, which publishes a widely used list of risks for AI applications, therefore asks to enforce document- and chunk-level authorisation "inside the index query, not at the application layer after retrieval", as our guide to prompt injection and LLM security quotes. A good answer: the search checks each user's rights from the directory, and payroll or board documents are not visible to people who cannot open them in the file share. Our guide to RAG on company data explains how.

Question 7: What can the assistant do on its own? Prompt injection is text that changes what a language model does: a sentence hidden in a supplier's PDF, an email or a web page, telling the assistant to do something its user did not ask for. OWASP puts it first in its Top 10 for LLM applications 2026. The UK's National Cyber Security Centre explained the cause on 8 December 2025: inside the model "there's no distinction made between 'data' or 'instructions'", and it is "very possible that prompt injection attacks may never be totally mitigated". Its advice is to reduce the risk and the impact, for example with an approach in which a model that processes information from a party has no more privileges than that party.

For management this turns into a simple rule. An assistant that reads outside content, such as incoming email, supplier documents or the web, gets no tools that can send, pay, delete or change records, and every action with an effect outside the chat waits for a person to approve it. A good answer lists the tools the assistant may call and who approves which action.

Our Private AI/ML service covers protection against prompt injection, permissions on documents and logging of queries and answers; name the systems your assistant should read and act on, and our team replies with how we would set it up.

Logs, backups and the model files

Question 8: What is logged, who can read it and for how long? A query log records who asked what, when, and what the answer was. It is how you find out what happened after a complaint or an incident, and it often contains personal data, because staff type names, customer details and HR questions into an assistant. A good answer: the log exists, a few named people can read it, and the retention period has been agreed with the data protection officer, as part of a data protection impact assessment if one is needed. Our guide to an LLM gateway with keys and a query log shows where the log is kept.

Question 9: How is the AI backed up, and when was a restore last tested? The model files can be downloaded again; the configuration, the document index, the chat history and any fine-tuned additions cannot. Article 32(1) of the GDPR lists the ability to restore the availability of personal data in a timely manner after an incident and regular testing of the measures, as our article on backup retention quotes. A good answer gives the date of the last restore test. Our guide to backing up an AI server lists what to include.

Question 10: Where do the models come from? Open models are downloaded from public repositories, and a model file can contain code. Hugging Face, the largest of these repositories, warns: "There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file." A good answer: models come from known publishers, in the safetensors format where available, at a fixed version, through one controlled download path; the server itself needs no internet access to run them.

If your company falls under NIS2

For companies that are essential or important entities under the NIS2 Directive, these questions are a duty of the management itself. Under Article 20(1) the management bodies approve the cybersecurity risk-management measures and oversee their implementation, and they can be held liable for infringements of Article 21; Article 20(2) requires their members to follow training, as our guide to NIS2 management training sets out. National transposition may differ in detail, and whether your company is in scope is a question for your legal counsel. Outside NIS2 the same ten questions are good practice, and customers ask them too, as our guide to AI questions in customer audits shows.

AI server security in the building: room, power and users

Security also depends on where the server stands. The table shows our reference configurations and what they need in the building; a locked server room or a rack in a data centre keeps the hardware, the drives and the management port away from casual access.

Chats by our users-per-card rule (0.9 × driver-visible memory, less 3 GiB per card and less the weights) with gpt-oss-120b at 16K tokens; power by our estimate (cards, processors and 15 per cent for the rest), confirmed by a measurement on the delivered system.

A chat in flight is a request being answered at the same moment; far more staff can use the assistant, because most of them are reading or typing. The entry server has two NVIDIA RTX PRO 6000 Blackwell Server Edition cards, two AMD EPYC processors, 384 GB of DDR5 ECC and two NVMe drives, and our page on private ChatGPT servers by company size relates chats to headcount. Our servers come with out-of-band management through the BMC, and its port goes on the separate management network from question 4. Without a server room, our Private AI/ML service runs the platform on dedicated hardware in a Tier-3 data centre in Lithuania, inside the EU. If nobody in the company runs servers, our guide to an AI server without an IT department sets out who does what.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight cards, assembled and tested before delivery, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us, and we install the operating system, drivers, CUDA and a container runtime on request; quantities and lead time are confirmed in writing with the quote. Our Private AI/ML service, with our engineering partner Vixen.UNO, sets up the models, document search with each user's access rights, protection against prompt injection and the query log, with ongoing support under an agreed SLA. For the network around the server, Vixen.UNO's Cyber Resilience work covers segmentation, zero-trust access with multi-factor authentication and scheduled test restores. Send us the answers you received to these ten questions, and we tell you where we see gaps.

FAQ