Hybrid AI: confidential data on your own server, the rest in a cloud assistant

Hybrid AI: which data goes where

Hybrid AI means that a company uses local and cloud AI together: it keeps a cloud assistant such as ChatGPT for public and general internal work and runs a language model on its own server for everything confidential. A short rule with four data classes tells staff which tool to use for which material, and the software enforces it: the chat interface offers each person only the models they may use, and a gateway between interface and models decides where a request may go. The own server then needs to carry only the confidential share of the work, which keeps it smaller than a server for the whole company.

We write for the managing director or owner who has to set the rule, and in the second half for the IT team or service provider that configures it. The rule comes first because it decides the technology. Our article on shadow AI and an AI usage policy covers the wider policy and how to find unapproved tools; this one sets out the split between local and cloud and how to make it work in daily use.

Data classification for AI tools: four classes from manufacturing

The classes below are an example for a manufacturing company. Your own classes may differ, and your data-protection officer and legal counsel confirm the personal-data rows and any customer contracts.

Example classification for a manufacturer; the decision for each class is yours. Health data in class 4 is a special category under Article 9(1) GDPR.

Class 3 is where most of a manufacturer's value sits. A drawing, a recipe or a calculation that leaves the company in a prompt is outside its control, and a customer's NDA may forbid handing its specifications to a third party at all. Our guide to trade secrets, NDAs and AI tools explains why companies keep such material in-house.

Class 4 needs a legal basis whatever tool is used. Article 4(1) of the GDPR defines personal data as "any information relating to an identified or identifiable natural person", and Article 9(1) prohibits processing special categories, among them "data concerning health", unless one of the exceptions in Article 9(2) applies. Running the model on your own server keeps the data under your control; it does not replace the legal basis, which your data-protection officer confirms, as our article on local AI and the GDPR explains. Passwords, access keys and certificates belong in no prompt of any class.

A rule staff can follow, and what they see

A rule only works if staff can apply it in a second. One sentence is enough: "Anything you would not send to a stranger by email goes into the company AI, never into the cloud assistant." The four classes stand behind it for the cases where someone is unsure.

The chat interface then does most of the work. Staff open one browser page and choose from a short model list with names that carry the rule, for example "Company AI (all data)" and "Cloud assistant (public and internal only)". The document search over drawings, quality records or HR files is connected only to the local model, so a question about a confidential document cannot reach the cloud even by mistake. Engineering sees the engineering collections, HR sees HR, and a user team that has no business with the cloud assistant does not see it in the list at all.

Training is short and practical: what the four classes are, which name in the list belongs to which, and who to ask. The query log shows later whether the rule holds. Our Private AI/ML service keeps such a log: nothing goes to public APIs until you enable hybrid mode for a specific task, and what goes there is visible in the log.

What the cloud side offers, as the provider states it

The cloud half of a hybrid set-up can be a business plan that staff open in the browser, or a cloud model reached through the company's gateway, which is what hybrid mode in our Private AI/ML service does. Either way, what the provider promises comes from its own pages. OpenAI's business data page, read on 25 October 2026 and showing no date of its own, states: "We don't train our models on your organization's data by default". It names ChatGPT Enterprise, ChatGPT Business, ChatGPT Edu and the API platform among the products covered. "Qualifying organizations" can set how long OpenAI keeps business data, and API customers can choose zero data retention. Storage at rest in Europe is offered to "eligible ChatGPT Enterprise, ChatGPT Edu, ChatGPT for Healthcare, and API platform customers"; the page does not list ChatGPT Business for it. OpenAI's announcement of 5 February 2025 introduced European data residency for new Enterprise and Edu workspaces and new API projects, and its update of 16 January 2026 added in-region GPU inference in Europe for eligible Enterprise, Edu and Healthcare customers.

Our checklist; the OpenAI statements above from openai.com/business-data and its post introducing data residency in Europe, read on 25 October 2026.

These are the provider's statements about its own service. Under Article 28(1) of the GDPR a controller may use only processors that can implement appropriate technical and organisational measures, so your data-protection officer reads the data processing agreement of the plan you buy; our guide to GDPR cloud hosting, the DPA and subprocessors lists what to check. A data region fixes where data is stored, not the jurisdiction the provider is subject to, as our comparison of private LLMs and cloud APIs notes. A per-seat business plan against an own server on its own is compared in ChatGPT Enterprise or own AI server.

We set up the local half and the rule between the two; send us your data classes and the teams that work with each, and our team replies with a configuration for the own server.

Sizing the local side for the confidential share

The own server does not have to carry the whole company, only the people and tasks in classes 3 and 4. Our users-per-card rule takes 90 per cent of the 95.6 GiB the driver reports on a 96 GB RTX PRO 6000, subtracts 3 GiB per card and the model's weights, and divides the rest by the memory one conversation needs. With gpt-oss-120b at a 32K context and a 16-bit cache that is about 19 conversations in flight on one card, as derived in our guide on how many users one RTX PRO 6000 serves. With the example values of our company-size guide, about four requests are in flight per 100 staff at the busiest moment, so one card covers the peak of about 475 people who use the company AI.

Our estimates: conversations by our users-per-card rule with a 16-bit cache; users by the example values of our company-size guide; wall power from cards, processors and 15 per cent for the rest, as in our two- and four-card guides. A measurement on the delivered system confirms the figure.

In a hybrid set-up the local model is the one staff use for confidential work, so its answer quality decides whether they follow the rule. If it is clearly weaker on their tasks, people drift back to the cloud assistant. Test the local model on real questions from engineering, quality and HR before the rollout, as our comparison of local LLM and ChatGPT answer quality suggests. Where the own server would stand and how much noise and heat it makes is in our guide to a GPU server in an office.

How the routing works: gateway, keys and user teams

For the IT team, the set-up has three layers: sign-in, the chat interface and the gateway in front of the models. Users sign in through the company directory, the interface shows each user team its models, and the gateway checks every request against the key or team it carries.

In LiteLLM's proxy, documented on its model access page read on 25 October 2026, a virtual key created with /key/generate takes a models list, and "The models list both hides a model from GET /v1/models and blocks calls to it." A team created with /team/new takes a models list too, and keys issued for that team are bound to that list; a call outside the list returns an error that names the valid models for the team. Models can be bundled with access_groups under model_info in config.yaml and granted to a key or team by that name; LiteLLM's documentation marks only access on wildcard models as requiring an enterprise licence. A practical layout gives the cloud model its own name, which only the keys of approved teams may call, so every request to it carries a team and a log entry.

LiteLLM pages on model access and access_groups, Open WebUI documentation on teams and model access, read on 25 October 2026; layer layout is our recommendation.

In Open WebUI, per its documentation read on 25 October 2026, an administrator sets a model's visibility to "Private or Restricted" and gives chosen user teams or individual users read access, which lets them see and use it, or write access, which lets them change it. Teams can be kept in step with the identity provider when ENABLE_OAUTH_GROUP_MANAGEMENT is enabled, and by default such a team grants no permissions beyond the global defaults, per the same page. Our guide to Open WebUI security hardening covers the related security settings.

Two details protect the rule. Keep fallbacks among private models: a fallback list that names the cloud model sends confidential prompts out whenever the local model is down, as our guide to an LLM gateway with keys, budgets and logging explains. And do not rely on automatic detection alone to catch confidential content in prompts to the cloud model: our guide to PII redaction before an LLM reports that the best of five PII models reached an average F1 of 0.471 on one published benchmark, so detection is a second line behind the team-based rule.

Cost drivers on both sides

We give no prices here; both sides change, and the comparison belongs in a spreadsheet with your own quotes. The main drivers are listed below.

Our summary; NVIDIA AI Enterprise licensing as in our NVIDIA AI Enterprise guide; wall power as in our two-card guide.

A hybrid set-up can lower the cloud side, because fewer seats need the plan with the strictest data terms, and it keeps the own server small, because it carries only the confidential share. Our guides to AI server electricity per year and to the break-even of a private LLM against a cloud API turn these drivers into figures.

Introducing the rule step by step

1. Write the four classes with examples from your own departments, and have the data-protection officer and legal counsel confirm classes 3 and 4.
2. Decide which teams may use the cloud assistant and for which classes.
3. Set up the own server, the local model and the document search, and connect only the local model to confidential collections.
4. Configure sign-in, the model list per team and the gateway keys, without fallbacks to the cloud.
5. Train staff on the one-sentence rule and the names in the model list.
6. Review the query log after the first weeks and adjust the classes where the rule was unclear.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight RTX PRO 6000 Server Edition cards, assembled and tested before delivery, on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. The local models, document search with access rights, the query log and hybrid mode for chosen tasks are our Private AI/ML service, with our engineering partner Vixen.UNO.

FAQ