Dify with a local LLM: vLLM and Ollama as model providers, knowledge base and licence

Connecting Dify to a model on your own GPU server

Dify connects to a local LLM through a model provider plugin: the OpenAI-API-compatible provider for vLLM or another server with an OpenAI-style API, and the Ollama provider for Ollama. You enter the server's base URL, the model name and its limits, and Dify then offers the model to every app in the workspace, for chat, workflows, agents and the knowledge base. Dify itself runs on CPUs; the GPU server holds the models.

Teams choose Dify as an internal builder for chat assistants, workflows and retrieval over documents. This article covers what changes when the models run on your own hardware: the provider fields that decide whether long contexts and tool calls work, the models the knowledge base needs, where Dify runs, how to size the GPU server and what Dify's licence says. Names and defaults are those of the documentation and plugin schemas read on 24 October 2026, when v1.17.1 was the latest release. The automation counterpart is our guide to n8n with a local LLM, and a chat front end for staff is covered in our private ChatGPT alternative.

vLLM through the OpenAI-API-compatible provider

vLLM serves an OpenAI-compatible API under /v1. In Dify, install the OpenAI-API-compatible provider from the Marketplace (plugin version 0.0.42 on 24 October 2026), add a model of type LLM and fill the fields below. The provider's schema lists LLM, text embedding, rerank, speech-to-text and text-to-speech models, so one plugin covers the whole stack on a vLLM server.

Schema openai_api_compatible.yaml of Dify's official plugins and the plugin's Marketplace page (0.0.42, dated 26 March 2026), read on 24 October 2026; the right-hand column is our recommendation.

Two defaults cause most of the first support tickets. A context size of 4096 makes Dify plan prompts as if the model held 4,096 tokens, although vLLM may serve 32K or more; enter the value of --max-model-len, which also decides how many full-length conversations fit on the card. With Function Call Type left at Not Support, the model is declared as one without tool calling, and by our reading of the agent documentation its agents then use the ReAct strategy described below.

Tool calls need both sides. vLLM's documentation marks --enable-auto-tool-choice as mandatory for automatic tool choice, together with the --tool-call-parser of the model family, for example openai for gpt-oss and hermes for Qwen3. Our article on vLLM tool calling parsers lists the parsers and their limits per model.

Dify's agent settings show which strategy applies: "Function Calling" for models with native tool support and "ReAct" for others, where Dify guides the model through a prompting strategy. Dify recommends "models that are strong at reasoning and natively support tool calling". Maximum Iterations caps the think-act-observe cycles per request, and each cycle is a model call; for GPU sizing, one agent request can therefore be several requests in flight one after the other.

Ollama as a provider

The Ollama provider (plugin 0.1.3 on 24 October 2026) asks for the model name as Ollama lists it, the base URL, the model type, the context length, the maximum token limit and whether the model supports vision. Dify's Ollama guide gives the base URL as the Ollama host's address with port 11434 and, when Dify runs in Docker, a local network address or http://host.docker.internal:11434, since localhost inside a container is the container itself. Its example context length is 4096; set it to what the model and Ollama are configured for.

Ollama binds 127.0.0.1 port 11434 by default, and OLLAMA_HOST changes that; Ollama's container image sets OLLAMA_HOST=0.0.0.0, so in a container it listens on all interfaces. Ollama's local API has no authentication, so if it listens on the network, allow port 11434 only from the Dify host, as our guide to securing a GPU server explains. For embeddings, change the model type to Text Embedding. Rerank models are not supported by Ollama, and the plugin page points to vLLM, llama.cpp, Text Embeddings Inference or Xinference for them. Our comparison of serving engines covers the choice between vLLM and Ollama.

We supply the GPU server and set up vLLM with the models and parsers your Dify apps need; send us the apps you plan, their users and the models you have in mind, and our team replies with a configuration.

Embedding and rerank models for the knowledge base

Dify's knowledge base indexes documents in one of two ways. High Quality "uses an embedding model to convert content chunks into vector representations" and offers vector, full-text and hybrid search. Economical uses ten keywords per chunk and no embedding model, "at the expense of reduced retrieval accuracy". A knowledge base created as High Quality cannot be switched to Economical later; Economical can be upgraded.

Hybrid search combines vector and full-text results either with weights, without a rerank model, or with a rerank model that sorts the passages. Rerank is off by default; Top K defaults to 3 and the score threshold to 0.5. The workspace's default models, set under Default Models, include a system reasoning model, an embedding model and a rerank model.

Dify knowledge base and environment documentation, read on 24 October 2026; vLLM endpoints as in our guide to embedding and reranker servers.

vLLM runs embedding and scoring models in its pooling runner with /v1/embeddings, /v1/score and /v1/rerank. Its --api-key covers only paths under /v1, /v2, /inference and /cohere, not the routes /rerank, /score or /pooling outside them, so keep that server reachable only from the Dify host. Choosing and sizing the two models is the subject of our guide to embedding and reranker GPU requirements; the small models fit in about 15 GB, so one 24 GB card such as the L4 holds them. VECTOR_STORE defaults to Weaviate, and Dify lists about three dozen alternatives, among which our vector database comparison helps choose. Why retrieval rather than the model decides answer quality is explained in RAG on company data.

Running Dify next to the GPU server

Dify's Docker Compose guide asks for at least 2 CPU cores and 4 GiB of RAM, Docker 19.03 or later and Docker Compose 2.24.0 or later on Linux. The stack starts the API, worker, web frontend, plugin daemon and agent backend with PostgreSQL, Redis, Weaviate, nginx, two SSRF proxies and two sandboxes. Before the first start, the guide requires a generated token in DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN; the first administrator is created at /install.

Run Dify on a virtual machine or a small server of its own and leave the GPU server to the models. That keeps upgrades of the two apart and lets the firewall allow only Dify's host to reach the model ports. Before going live, change the defaults the environment reference flags:

• SECRET_KEY "must be replaced for production"; it signs sessions and tokens and encrypts stored OAuth credentials, and changing it later logs out all users
• INIT_PASSWORD protects the /install page until the first administrator exists
• CHECK_UPDATE_URL, by default https://updates.dify.ai, can be left empty in air-gapped environments, as our article on telemetry in self-hosted AI tools explains
• UPLOAD_FILE_SIZE_LIMIT is 15 MB for documents by default

Dify routes its outbound HTTP requests, from HTTP nodes and image downloads among others, through an SSRF proxy that blocks internal and private IP ranges. A workflow that should call an internal system therefore needs a deliberate decision about the proxy, made with your security team. The workspace roles are owner, admin, editor and member; owners and admins manage model providers, and the documentation warns that provider API keys "grant workspace-wide model access".

Sizing the GPU server for Dify apps

Size the GPU server for the requests in flight at the busiest moment, counted at the model server. A chat app produces one model request per message; a workflow one per LLM node; an agent up to Maximum Iterations per question; and each knowledge base query adds an embedding call and, with rerank, a scoring call.

NVIDIA RTX PRO 6000 Blackwell Server Edition. Our estimate by the users-per-card rule (0.9 × 95.6 GiB driver-visible − 3 GiB per card − 60.8 GiB weights), 1.125 GiB of 16-bit KV cache per 32,768-token conversation, as in our four-card server guide.

The table assumes the NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB per card. Our users-per-card rule takes 90 per cent of the memory the driver reports, subtracts 3 GiB per card and the weights, and divides the rest by the KV cache per conversation; the derivation is in how many users one RTX PRO 6000 can serve. These are concurrent full-length conversations, far fewer than the staff with access; how that ratio works out for a large staff is shown in our guide to Open WebUI for 1,000 users. An entry server with two RTX PRO 6000 Server Edition cards runs a pilot with several Dify apps; a four-card node, described in our article on the four-card RTX PRO 6000 Server Edition server, adds room for more apps, an embedding model and a reranker next to the main model.

Dify's licence: what the conditions say

Dify's LICENSE file is titled "Open Source License" and begins: "Dify is licensed under a modified version of the Apache License 2.0, with the following additional conditions:". The conditions, quoted from the file on GitHub as read on 24 October 2026:

• "1. Dify may be utilized commercially, including as a backend service for other applications or as an application development platform for enterprises. Should the conditions below be met, a commercial license must be obtained from the producer:"
• "a. Multi-tenant service: Unless explicitly authorized by Dify in writing, you may not use the Dify source code to operate a multi-tenant environment."
• "Tenant Definition: Within the context of Dify, one tenant corresponds to one workspace. The workspace provides a separated area for each tenant's data and configurations."
• "b. LOGO and copyright information: In the process of using Dify's frontend, you may not remove or modify the LOGO or copyright information in the Dify console or applications. This restriction is inapplicable to uses of Dify that do not involve its frontend."
• The frontend "includes all components located in the web/ directory when running Dify from the raw source code, or the "web" image when running Dify with Docker."
• "2. As a contributor, you should agree that: a. The producer can adjust the open-source agreement to be more strict or relaxed as deemed necessary. b. Your contributed code may be used for commercial purposes, including but not limited to its cloud business operations."
• "Apart from the specific conditions mentioned above, all other rights and restrictions follow the Apache License 2.0."

The file also states "The interactive design of this product is protected by appearance patent." and carries "© 2025 LangGenius, Inc."

The conditions turn on workspaces and on the frontend. Dify's workspace page states that "One workspace is created during installation" and that multiple workspaces for complete isolation are a Dify Enterprise feature. Whether a planned use counts as operating a multi-tenant environment, for example affiliated companies or a service for clients, each with its own workspace, and whether a rebranded frontend is planned, are questions for your legal counsel before rollout; we give no legal assessment. Our overview of open LLM licences for company use covers the licences of the models themselves.

Upgrades and security releases

The latest release on 24 October 2026 was v1.17.1 of 10 September. Its notes require self-hosted installations with the bundled Weaviate to upgrade it in stages from 1.27.0 to 1.39.2, because "skipping minors is unsupported". v1.17.0 listed security enhancements, among them sign-in protection, resource scoping and SSRF and outbound hardening, and renamed environment variables. v1.16.1 put the agent's local sandbox behind a forward proxy and introduced DIFY_AGENT_API_TOKEN, whose default must be replaced in production.

The Compose guide says "Upgrade steps may vary between releases": read the notes of each release, compare .env.example with your .env, and reapply your own changes to the new docker-compose.yaml rather than keeping the old one. Back up PostgreSQL and the vector store before each upgrade.

What we supply

You can buy the GPU server for your Dify apps from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two to eight cards, assembled and tested before delivery, with the operating system, drivers, CUDA and a container runtime installed on request. Setting up vLLM with your models, the embedding and rerank services and the connection to Dify is part of our Private AI/ML service. Write to us with your Dify version and the models you use.

FAQ