Langfuse self-hosted: LLM observability and tracing on-premise, next to your GPU server

Self-hosted LLM tracing with Langfuse, and how it differs from vLLM metrics

Langfuse is an open-source platform for tracing LLM applications, and it can be self-hosted inside your own network: its documentation states that "Langfuse does not require internet access". A self-hosted Langfuse stores every request of a RAG assistant or an agent as a trace, a tree of steps with their inputs, outputs, timings, token counts and costs.

A question to an internal assistant becomes one trace with a span for the embedding call, one for the vector search, one for the reranker, one per tool call and one for the model's answer. When an answer is wrong or slow, the trace shows which step caused it.

That is a different job from the serving metrics of the inference engine. Our guide to vLLM metrics with Prometheus and Grafana covers queue length, time to first token and cache usage: counters and histograms across all requests, with no request text in them. They tell you that the 95th-percentile latency rose at 10:40. A Langfuse trace of a slow request from 10:40 tells you that the retrieval step returned 40 chunks instead of 8 and the prompt grew to 30,000 tokens. The two complement each other and serve different people: the platform team watches the metrics, while the team that builds the assistant reads the traces.

Langfuse also covers the work around the traces: prompt management with versions and labels, datasets of test questions, scores attached to traces by people or by an LLM judge, and dashboards of cost and latency. The LLM gateway in front of your models keeps the query log of who asked what; Langfuse sees the application's steps inside each request. Where the gateway already logs every call, decide which of the two holds prompt text, so the same personal data does not sit in two stores with two retention rules.

Components of a self-hosted Langfuse v4: PostgreSQL, ClickHouse, Redis, S3

The self-hosting overview lists six parts. Only the web container has to be reachable by users and SDKs; the others stay on an internal network.

Langfuse self-hosting documentation (overview, infrastructure pages for PostgreSQL, ClickHouse, cache and blob storage, scaling guide "minimum infrastructure requirements"), read 25 October 2026; latest release from the GitHub releases page, 25 October 2026.

Added up, the minimums come to 11 CPU cores and 25.5 GiB of memory by our addition, with MinIO run in-house. None of the services uses a GPU. Three settings need attention on a new installation. ClickHouse "must run with its timezone set to UTC", or queries return wrong or empty results. Redis needs maxmemory-policy=noeviction, so that queued jobs are not evicted. And the S3 bucket is mandatory: Langfuse writes raw event data to it "on every ingestion request" and reads it back to retry failed ingestion jobs, so the bucket grows until a lifecycle rule expires old objects (Langfuse Cloud uses 30 days, according to the same page).

Langfuse v3 installations should note the version floors: v3 ran on ClickHouse 24.3 or later, while v4 requires 25.12. Plan the database upgrade before the Langfuse upgrade, and read the release notes: v4.20.0, for example, replaced the LANGFUSE_AWS_BEDROCK_* variables and shortened the default session lifetime to 14 days.

Docker Compose or Kubernetes

The Docker Compose deployment is a clone of the repository and docker compose up on one machine. Langfuse's guide uses a VM with at least 4 cores and 16 GiB of memory and an example disk of 100 GiB, exposes the UI on port 3000 and MinIO on port 9090, keeps the worker, the MinIO console and the databases on 127.0.0.1, and marks every secret to replace with # CHANGEME. It also states the limits plainly: "The docker compose setup lacks high-availability, scaling capabilities, and backup functionality." That makes Compose the right choice for a pilot with one team, and a poor one once several applications depend on the traces.

For production the documentation points to Kubernetes with the Helm chart. Its ClickHouse page recommends at least three ClickHouse replicas with three Keeper replicas, gives a production example of 2 CPU cores and 8 to 16 GiB of memory with 100 GiB of storage per replica, and warns that the replica count cannot be raised at runtime without manual work or downtime. Langfuse does not support multi-shard ClickHouse clusters. The ClickHouse bundled in the Helm chart and the single-container Docker setup are community-supported; official support covers ClickHouse Cloud, BYOC and the Kubernetes operator. A company that runs everything in-house therefore operates ClickHouse itself, and should plan for that skill.

We would put Langfuse on a VM or a Kubernetes cluster beside the GPU server, not on it. The CPU cores and memory of a GPU server are sized for the cards (at least six cores per GPU and system memory of twice the GPU memory in our reference configurations, following NVIDIA's guide for certified systems), and a driver update or a reboot of the GPU server should not take the trace store down with it. Two system tables also need attention on any self-run ClickHouse: the scaling guide notes that trace_log, text_log and other system log tables have no TTL by default, Langfuse does not read them, and they can dominate disk usage.

Sending traces: SDKs, LiteLLM and OpenTelemetry

There are three common ways in, and they can be mixed.

1. SDK in the application. For Python code that calls an OpenAI-compatible API, Langfuse documents a drop-in import, from langfuse.openai import openai, configured with LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_BASE_URL. It records prompts and completions, latencies, API errors and token usage. For streamed answers, usage arrives only with stream_options={"include_usage": True}. The integration page names only OpenAI's hosted APIs; it does not state support for other OpenAI-compatible servers such as vLLM, so test it against yours.
2. Through the gateway. LiteLLM's proxy sends every call it routes to Langfuse with callbacks: ["langfuse_otel"] in its configuration and the Langfuse keys and host in environment variables. This captures model calls from every application behind the gateway, but not the retrieval and tool steps inside an application.
3. OpenTelemetry. Langfuse accepts OTLP at /api/public/otel (from v3.22.0), authenticated with HTTP Basic auth built from the public and secret key. It accepts HTTP/JSON and HTTP/protobuf; "gRPC is not supported yet", so an OpenTelemetry Collector or exporter in front must use HTTP. The header x-langfuse-ingestion-version=4 makes spans appear in real time; without it, the documentation says data can take up to ten minutes to show. Langfuse maps the OpenTelemetry GenAI attributes such as gen_ai.request.model and gen_ai.usage.* and notes that these conventions are still evolving.

For RAG and agents, the application-level route gives the most useful traces, because only the application knows which documents it retrieved and which tool it called. Langfuse recommends propagating the user ID, session ID, release and tags to every span, not only the root span, so that filters work. It also warns against putting passwords, API keys or personal information into OpenTelemetry baggage, which travels to downstream services.

We supply the GPU servers that run the models you trace and set up the models and retrieval through our Private AI/ML service; send us your models, applications and request volume, and our team replies with a server configuration.

Prompts in traces: masking, access, retention and telemetry

A trace of an internal assistant holds the user's question, the passages retrieved from your documents and the model's answer. That is personal data in most companies, and sometimes trade secrets. The licence decides how much control you get without paying. The LICENSE file puts everything outside the ee directories under the MIT Expat licence, and Langfuse states that all core features and APIs are available in the open-source version "without any limits". Its licence-key page lists the features that need an Enterprise key: project-level RBAC roles, protected prompt labels, data retention policies, audit logs, server-side data masking, UI customisation, organisation creators, the organisation management API and SCIM, the instance management API and FIPS mode.

Langfuse licence-key, data-masking, authentication and telemetry pages, read 25 October 2026. The authentication page does not mark SSO as an Enterprise feature.

Mask in the application first. Client-side masking removes data "before data is sent to Langfuse", which keeps it out of the S3 bucket and ClickHouse alike. The server-side callback of the Enterprise edition applies only to events received through the OpenTelemetry endpoint and fails open by default: if the callback fails, the event is stored unmasked, unless the fail-closed option (..._CALLBACK_FAIL_CLOSED=true) drops it instead. Langfuse itself calls it "a centralized safety net, not a replacement for client-side masking."

Sign-in works through Keycloak, Authentik, Okta, GitLab and other providers, or a custom OAuth provider configured with AUTH_CUSTOM_CLIENT_ID, AUTH_CUSTOM_CLIENT_SECRET and AUTH_CUSTOM_ISSUER. Set AUTH_DISABLE_SIGNUP=true so nobody creates an account outside your identity provider, and AUTH_DISABLE_USERNAME_PASSWORD=true once SSO works. Without the Enterprise roles, everyone with access to a project sees all traces in it, so separate sensitive applications into their own projects.

Without the retention feature, deletion is your job: a scheduled deletion through the API or a TTL that your database team sets, plus the S3 lifecycle rule. How long prompt text may be kept is a decision for your data-protection officer, and our guide to a DPIA for an internal LLM covers the assessment.

The open-source web container sends one aggregated telemetry event to eu.posthog.com at most every 12 hours: the version, up to 30 email domains with user counts and counts of traces, scores and other objects. Langfuse states that it "does not send raw traces, prompts, observations, scores, or dataset contents". The update check to langfuse.com is not configurable and fails gracefully without internet access. Our article on telemetry in self-hosted AI tools shows how to block the rest with an egress filter.

Transport and storage encryption are yours to set up. Langfuse does not terminate HTTPS itself and recommends a load balancer or a service mesh for it; ENCRYPTION_KEY protects stored LLM API credentials, not the traces. For the disks underneath, see our guide to encrypting a GPU server with LUKS and TPM, which applies to a trace server just as well.

Storage and sizing: an example

Langfuse publishes no size per trace, because it depends on the prompts. An example with assumed values shows the order of magnitude. An internal assistant for 300 people, each sending 20 questions a day, writes 6,000 traces a day. If a trace with its retrieved passages, answer and metadata takes 40 KB uncompressed, that is 240 MB a day and about 87.6 GB a year in ClickHouse before compression, plus raw events in the S3 bucket: 7.2 GB with a 30-day lifecycle rule. These are our example values and arithmetic; ClickHouse compresses the data, so measure the real size per trace during the pilot and extrapolate from that.

Langfuse Docker Compose guide, scaling guide and ClickHouse page, read 25 October 2026; totals by our addition.

The scaling guide gives the signal for growth: a worker load above 50 per cent on a 2-CPU container means it is saturated, and the queue depth metric langfuse.queue.ingestion.depth is the main scaling metric. Under heavy ingestion, the ingestion paths can be split from the UI into a second web deployment.

Evaluations with your own models

Langfuse's LLM playground and LLM-as-a-judge evaluations call an LLM API. The networking page says the endpoint can run in the same network, so the judge can be a model on your own GPU server through its OpenAI-compatible API. That keeps evaluation prompts, which contain your traces, inside the company. A judge model takes GPU capacity like any other user: running a judge over every production trace doubles the model calls, so score a sample, or run evaluations at night against datasets. Our guide to LLM evaluation before model upgrades covers test sets, judges and their published biases; Langfuse's datasets and scores are one place to store them.

For a two-card server with NVIDIA RTX PRO 6000 Blackwell Server Edition cards, one layout puts the production model on one card and a judge in a 24 GB MIG instance of the other; our MIG profile guide lists the instance sizes. A judge model large enough to grade answers well is a choice to test, since a small judge can agree with wrong answers.

What we supply

You can buy the GPU servers from us: our team sizes the configuration to your workload and budget and replies within one business day. We build them to order with two, four or eight RTX PRO 6000 Server Edition cards or H200 NVL cards, assembled and tested before delivery, on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. Our Private AI/ML service, with our engineering partner Vixen.UNO, deploys the models and retrieval on-premise and builds the platform around them, including logging of queries and answers. Describe the applications you want to trace, and we propose where the models and the trace store run.

FAQ