AnythingLLM hardware requirements: desktop app for one person or Docker server for a team

AnythingLLM desktop app or Docker server for your company

Take the AnythingLLM desktop app when one person chats with their own PDFs and other documents on their own computer, and the Docker server when several colleagues share workspaces, need separate accounts or use a chat widget on the intranet. The hardware depends on the model you run: a workstation with one 24 GB RTX PRO 4000 Blackwell runs a 20B-class model well for one person or a small team, and a 96 GB RTX PRO 6000 Workstation Edition runs gpt-oss-120b for a department.

AnythingLLM is an open-source application for chatting with your own documents through a language model, published by Mintplex Labs under the MIT licence. The desktop app is the exception: its own terms of use, version of 14 July 2025, grant a non-transferable licence for personal or internal business use and forbid distributing, selling or otherwise transferring it to a third party, so each user downloads it from Mintplex Labs. As of 30 October 2026 the project is actively developed; its latest release on GitHub dates from 27 August 2026. Its installation overview puts the choice in plain terms: the desktop app is for when "you do not need multi-user support" and "everything needs to stay only on your device"; the Docker version is for an instance "that many people can use at the same time" with "admin and rule-based access for workspaces and documents".

This article covers that choice, the defaults that matter for German documents, what the app sends out, and the graphics card per tier. For the concept of document chat see our guide to RAG on company data; the step from a PC experiment to a shared company server is in our guide on moving from Ollama on a PC to a company AI server.

What the desktop app does on one computer

The desktop app installs like any other program on Windows, macOS or Linux and calls itself a "single-player" application. It brings a built-in model engine, which the README describes as powered by Lemonade and llama.cpp, with CUDA on NVIDIA cards, so a downloaded model runs without a separate server. It can also use Ollama or LM Studio on the same machine.

The documentation's system requirements for the desktop are modest for the app itself: 16 GB of RAM and an "8-core CPU (any)". The model decides the rest. The page says "the LLM is often the bottleneck to a decent experience" and recommends a GPU for local models. It also suggests a pattern that suits a small firm: if a computer has no suitable GPU, run the model on another machine that has one and connect the app to it over the network. Several colleagues can then each keep their own desktop app and documents on their laptop, while one GPU workstation serves the model to all of them.

Multi-user Docker server with roles and chat widgets

The Docker version runs as a server that colleagues open in the browser. Its access page lists the roles of multi-user mode:

AnythingLLM documentation, Security and Access, read 30 October 2026.

The page states that these settings "are only available in the Docker version", and that multi-user mode cannot be switched back to single-user mode once it is on. Without multi-user mode, one password protects the whole instance, and anyone with it can see all chats. For single sign-on, AnythingLLM's configuration page documents a "Simple SSO" passthrough (SIMPLE_SSO_ENABLED): a system you run requests a single-use login link for an existing user through the AnythingLLM API, and the link expires after 10 minutes. It is not OpenID Connect or SAML. A firm that wants staff to sign in directly through its directory takes a front end that documents it, such as the Open WebUI stack in our guide to a private ChatGPT alternative.

The Docker version also publishes chat widgets, small chat windows for a web page that answer from one workspace. The installation overview names "you will publish chat widgets to the public internet" as a reason for Docker. For an intranet page the widget stays inside; anything on the public internet needs the same care as any public web service. Under Article 50(1) of the AI Act, applicable since 2 August 2026, people who chat with an AI system have to be told so unless it is obvious; our guide to AI Act obligations explains the timeline, and your legal counsel confirms what applies to your widget.

Attach, embed or pin documents in a workspace

A workspace is a collection of documents and chats for one topic or team. AnythingLLM handles documents in three ways, as its documentation on chatting with documents describes:

• Attach in a chat: by default the app inserts the full text of the document into the conversation, scoped to that thread; if the document exceeds the model's context window, the app asks whether to embed it instead.
• Embed into the workspace: the document is split into chunks, turned into vectors by the embedder and stored in the vector database, LanceDB by default; each question retrieves the most relevant chunks, and the documentation advises about 4 to 6 snippets for most models.
• Pin: the full text of an embedded document goes into every answer of the workspace, which the documentation calls a last resort for documents that fit in the context window.

The default embedder matters for German files. The built-in embedder is all-MiniLM-L6-v2, about 25 MB, run on the processor, and the documentation says it is "primarily trained on English documents". Ollama is one of the local embedder providers listed, so a multilingual model such as bge-m3, a 1.2 GB download in Ollama that its page describes as supporting "more than 100 working languages", can run on the GPU instead. Decide before you load documents: the embedder page warns that changing it means you "will have to delete your uploaded documents and re-embed them".

Attaching full documents has a hardware cost. A long contract attached in full makes a long conversation, and the memory each conversation holds on the GPU grows with its length, as the next section shows.

You can order the workstation alone, the setup of the AnythingLLM Docker version, Ollama or vLLM and the models alone, or both together: the workstation delivered with everything installed and configured; write to us and we suggest the best solution for your tasks and budget.

Built-in engine, Ollama or vLLM as the model backend

For one person, the built-in engine or Ollama is enough. For a team the backend decides how many colleagues get an answer at the same time. Ollama's FAQ states that it processes one request per model at a time by default (OLLAMA_NUM_PARALLEL is 1) and queues the rest; the setting can be raised, and each extra slot needs memory for its context. A serving engine such as vLLM batches many conversations on one card and offers an OpenAI-compatible interface, which AnythingLLM reaches through its generic OpenAI provider. Our comparison of vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama covers the engines for the IT team.

Which GPU for AnythingLLM, and for how many users

We size with our users-per-card rule from the guide to how many users one RTX PRO 6000 serves: 90 per cent of the memory the driver reports, less 3 GiB for the runtime, minus the model weights, leaves the memory for conversations. The counts below are conversations of 32,768 tokens in flight at the same moment with a 16-bit cache, which is not the same as headcount.

Our estimates by the users-per-card rule, not measurements; weights and cache per conversation (0.75 GiB for gpt-oss-20b, 1.125 GiB for gpt-oss-120b at 32K) from our gpt-oss-120b guide.

The entry row: an RTX PRO 4000 holds 18.5 GiB by our rule, gpt-oss-20b leaves 5.7 GiB, and 5.7 ÷ 0.75 gives 7 conversations, as in our article on 30B MoE models on 24 GB GPUs. The 48 GB RTX PRO 5000 holds 40.0 GiB, which leaves 27.2 GiB and about 36 conversations; a 32 GB RTX PRO 4500 lands at about 17. The larger gpt-oss-120b needs the 96 GB card for a team: on the 72 GB RTX PRO 5000 it loads with about 0.7 GiB to spare, as our RTX PRO 5000 48 GB or 72 GB comparison shows, which leaves no room for conversations. On one RTX PRO 6000 it leaves 22.2 GiB, about 19 conversations; two cards as separate copies hold about 38, by our gpt-oss-120b guide.

Two things move these counts. Long attached documents lengthen conversations: at the full 131,072 tokens, gpt-oss-120b needs 4.5 GiB per conversation and one RTX PRO 6000 holds four. And people do not all ask at the same second. Our guide to a private ChatGPT server by company size works through an example in which 500 staff produce about 20 requests in flight at the busiest moment; with your own usage the figure differs, and a pilot measures it.

The cards differ in speed as well as memory. A single conversation is limited by how fast the card reads the weights, and the bandwidth is 672 GB/s on the RTX PRO 4000, 1,344 GB/s on the RTX PRO 5000 and 1,792 GB/s on the RTX PRO 6000 Workstation Edition. We found no published AnythingLLM measurement on these cards; the speed depends on the model, the engine, the context length and how many people ask at once. Two RTX PRO 6000 Workstation Edition cards at 600 W each are the realistic maximum for that edition in a workstation, and three or four Max-Q cards belong only in a chassis built for them with strong directed airflow, as our page on the four-card Max-Q workstation explains. For larger teams, a rack server with RTX PRO 6000 Server Edition cards is the next step.

What AnythingLLM sends out

The README describes telemetry as "anonymous usage information" sent to PostHog: the installation type, events when documents are added or removed without document information, the vector database type, the model provider and model tag, and an event when a chat is sent, without its content. It states: "No IP or other identifying information is collected." The README's opt-out instructions imply that it is on by default; it is turned off with DISABLE_TELEMETRY set to "true" in the server's environment file, or in the app under Privacy. The README also lists other outbound connections: cdn.anythingllm.com for model downloads and GitHub domains for files that cache context-window sizes. A firewall rule that blocks outbound traffic from the machine catches what a setting misses, as our article on telemetry in self-hosted AI tools explains for other components.

With a local model and a local embedder, the documents, the vectors and the chats stay on your hardware. If someone switches the workspace to a cloud model provider, which AnythingLLM supports, the text goes there. In the Docker version only the admin role can change the model and embedder settings, which is one reason to keep few admins.

AnythingLLM vs Open WebUI for the whole company

AnythingLLM suits a firm or department that thinks in workspaces: a set of documents per client, project or topic, with the chat beside them and agents in the same window. Open WebUI is the front end of the stack in our private ChatGPT alternative guide, with sign-in through OpenID Connect documented there and guides on scaling it to 1,000 users. Both talk to the same vLLM server, so the GPU decision does not depend on the front end. Start with the one that matches how your people work, on the hardware in the table above.

The same GPU can serve other document tools beside AnythingLLM: Paperless-ngx with local AI files and tags incoming mail, Onyx searches across SharePoint and Confluence, Open Notebook and other local NotebookLM alternatives turn sources into cited answers and audio summaries, and the ONLYOFFICE AI tools work inside the editor.

What the AnythingLLM setup contains

When we set up AnythingLLM, the workstation or server has the Docker version installed, the model engine on the GPU with a model sized to the card, a multilingual embedder for German and English documents, telemetry switched off and the first workspaces created. Colleagues who prefer the desktop app download it themselves under Mintplex Labs' terms of use and connect it to the same model engine. For a team, the accounts and roles are set up and the server is reachable only inside your network. Our guide to preparing company documents for an AI assistant helps you choose the first documents.

We deliver workstations with one RTX PRO 4000, RTX PRO 5000 or RTX PRO 6000 Workstation Edition and servers with RTX PRO 6000 Server Edition for AnythingLLM; tell us how many colleagues will use it and which documents, and we propose the tier that fits.

FAQ