AI server for logistics companies: freight documents, multilingual chat and route planning

What an AI server does for a logistics company

An AI server for a logistics company runs the language, document and optimisation models of a carrier or forwarder on hardware the company controls. It reads delivery notes, consignment notes and invoices into the transport management system (TMS), answers customers and drivers in their own languages, sorts and drafts replies to the email traffic of dispatch, and computes routes. For a mid-sized forwarder, a server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards carries the assistant and the document pipeline; four cards add a whole card for document volume and a tensor-parallel pair for several hundred open chats.

We wrote this for the managing director or the head of operations who has to judge what such a server does, and for the IT team that connects it to the TMS. The configurations come first and the technical notes after them. The figures are published measurements with their source, or our estimates, labelled as such.

Three workloads with three different loads

Our summary; page rates by our estimate from the method of our document AI guide; chats by our users-per-card rule; cuOpt as described in its GitHub README, read 24 October 2026.

The three do not compete much for the same resource. The document pipeline is a queue that runs through the shift and catches up at night. Chat is a peak load in office hours, sized by the memory that each waiting conversation keeps on the GPU. Route optimisation runs in bursts: a plan for the next day, then re-plans when orders change. A company can start with one of them and add the others on the same server if the cards are planned for it.

Freight documents into the TMS

Delivery notes, CMR consignment notes, proofs of delivery, invoices from subcontractors and customs papers arrive as scans, photos and PDFs. A common design runs two stages, as our guide to vision-language models for document AI explains. An OCR model turns every page into text with its tables, and a text model then extracts the fields into a fixed schema; vLLM's structured outputs make the answer "follow the JSON schema", so the TMS receives a record it can check. Pages the OCR stage cannot read with confidence, such as stamps, handwriting on a proof of delivery or a damaged photo, go to a general vision-language model that reads the image directly.

The OCR stage carries the volume. Datalab reports 1.44 pages a second for its 5B chandra-ocr-2 on one H100 80GB with vLLM; scaled to the RTX PRO 6000 Server Edition by memory bandwidth, that is about 0.69 pages a second per card by the method of our document AI guide, 19,900 pages in an 8-hour shift.

About 0.69 pages per second per RTX PRO 6000 Server Edition with a 5B OCR model, our estimate scaled from Datalab's chandra-ocr-2 figure on one H100 80GB (model card, undated) by the method of our document AI guide; shipments and pages per shipment are example values. A MIG instance gets a fraction of a card's memory bandwidth, so its rate is lower; no published figure.

The licence comes before the speed. GLM-OCR is published under MIT and DeepSeek-OCR-2 under Apache 2.0, while chandra-ocr-2 carries a modified OpenRAIL-M licence with a revenue threshold. The text model that extracts the fields reads a few thousand tokens per document instead of the page image. Test each document type: a model that reads typed invoices well can fail on a handwritten note on a consignment note. Our guide to batch LLM inference on-premise covers the queue behind such a pipeline.

Chat and email with customers, drivers and partners

Three kinds of conversation reach a logistics company: customers asking where their goods are, drivers and subcontractors reporting delays or asking for instructions, and the email traffic of dispatch and customer service. The model answers status questions through tool calls to the TMS, so the data comes from the system at answer time rather than from an index. vLLM's documentation makes --enable-auto-tool-choice mandatory for that and pairs it with a parser per model family, openai for gpt-oss.

Languages decide the model. The Mistral Small 3.1 card names 25 languages, German, Polish and Romanian among them, and Qwen3's launch post lists 119 languages and dialects (the Qwen3-14B card says 100+), while Llama 3.3's card names eight languages, German among them but not Polish or Romanian, and gpt-oss has no language list on its card. EuroLLM covers all 24 official EU languages under Apache 2.0. A test set with real driver and customer messages in each language you deal with decides the shortlist, as our comparison of open LLMs for European languages suggests.

The memory per conversation sets the capacity. By our users-per-card rule, which takes 90 per cent of the 95.6 GiB the driver reports, less 3 GiB per card and less the weights, gpt-oss-120b leaves 22.2 GiB of cache on one card. At a declared context of 16,384 tokens a chat takes 0.5625 GiB of 16-bit cache, so one card holds 39 chats in flight and a tensor-parallel pair 187. Our guide to a server for a customer service chatbot turns that into open chats with example values of one customer message a minute and 20 seconds per answer: a third of the open chats are in flight, so about 117 open chats per card and 560 per pair.

Email triage is a batch load in the same engine: classify each message, extract the order or shipment number, and draft a reply for a person to approve. Calls from drivers can be transcribed with Whisper, for which OpenAI gives about 10 GB of GPU memory, a fraction of one card; our guide to a voice agent server on-premise covers spoken conversations.

We build servers with two, four or eight RTX PRO 6000 Server Edition cards for this mix; send us your pages per day, your peak chats and the languages you deal with, and our team replies with a configuration.

Route optimisation on the GPU

Routing is an optimisation problem, and solvers are built for it. NVIDIA's cuOpt, published on GitHub under Apache 2.0, solves vehicle routing, the travelling salesman problem and pickup-and-delivery problems, as well as linear and quadratic programs; its mixed-integer solver is marked beta. Its README asks for a GPU of compute capability 7.0 or higher, CUDA 12.0 or 13.0 and later, and a driver from 525.60.13, on Linux on x86_64 or aarch64. It installs with pip, conda or a container image, and has a server mode for routing, linear and mixed-integer problems, which a TMS can call.

The RTX PRO Blackwell cards are compute capability 12.0, inside that rule, although the README does not name them; the driver and CUDA release on the server must also support these cards, as our guide to RTX PRO Blackwell software support explains. We found no published cuOpt benchmark on these cards, and the GPU memory a routing problem needs depends on the number of stops, vehicles and constraints. Run a test with a real day of orders, your fleet and your time windows, and compare the plan and its run time with your current planning tool before you size a card for it. On a four-card server the routing runs can share the card that carries search and speech; a planning team that re-plans every few minutes for a large fleet may justify a card of its own.

Language models do not replace the solver. They can turn a dispatcher's free-text note into a structured order and explain a plan in words, but the routes come from the optimiser.

Configuration options by card count

The options follow the reference configurations on our AI servers page and NVIDIA's guide for certified systems: two processors, at least six cores per GPU and system memory of twice the GPU memory.

Reference configurations on our AI servers page; chats by our users-per-card rule at 16K with gpt-oss-120b; pages by our estimate of 0.69 per second per card; power by our estimate (cards at 600 W, processors, plus 15 per cent for the rest of the server).

Two cards. The entry server has two RTX PRO 6000 Server Edition, two AMD EPYC processors, 384 GB of DDR5 ECC and two NVMe drives. One card runs gpt-oss-120b for chat, email drafts and an internal assistant for dispatch; the other is split by MIG into four isolated 24 GB instances for the OCR model, the embedding model and reranker, speech-to-text and a test instance for routing. It suits a company that starts with one workload and a few thousand pages a day.

Four cards. The inference node with 768 GB and four NVMe drives gives each workload its own resources: a tensor-parallel pair for about 560 open chats, a whole card for about 20,000 pages a shift, and a card for search, speech and routing runs. This is the configuration we would size first for a forwarder that wants all three workloads.

Two four-card servers or eight cards. If customers and drivers depend on the chat, plan two servers that each carry the chat peak, so one can fail or be updated while the other answers, as our guide to LLM high availability with two GPU nodes describes. Eight cards in one server give the same capacity in one chassis, at about 6.3 kW and without failover.

The RTX PRO 6000 Max-Q is the workstation card, a 300 W blower card; in a rack it belongs only in a chassis whose maker lists it, and for continuous operation we recommend the Server Edition. We set up the models on these servers through our Private AI/ML service; describe which of the three workloads you would start with, and we propose a pilot and the server for it.

Notes for the IT team

1. Integration. vLLM serves an OpenAI-compatible API and has pooling endpoints for embeddings, so the TMS, the email system and the chat platform call one interface. Put an LLM gateway with keys and a query log in front of it. cuOpt has its own server API.
2. Tool calls. Status queries and order lookups are tool calls into the TMS. The backend identifies the customer or driver and passes only their data to the tool; the model never sees credentials, as our guide to prompt injection and LLM security explains.
3. Network. vLLM's --api-key protects only paths under /v1, /v2, /inference and /cohere, while routes such as /invocations and /metrics answer without the key (vLLM 0.32.0 security guide), so keep the engines on an internal network behind the gateway. Run vLLM 0.31.0 or later, keep the NVIDIA Container Toolkit at 1.17.8 or later and the driver at a release NVIDIA lists as fixed in its bulletin of 30 September 2026.
4. Software on delivery. The server arrives on request with the operating system, the NVIDIA driver, CUDA and a container runtime; cuOpt publishes packages for CUDA 12 and 13. The software support guide linked above lists the engine versions for these cards.
5. Data and law. Driver messages, call recordings and delivery photos hold personal data, and telematics positions can too, so set retention and access rules and check whether a data protection impact assessment is needed. Since 2 August 2026, Article 50 of the AI Act requires that people are told they are interacting with an AI system unless that is obvious. Our article on the EU AI Act gives general information; national practice may differ and your legal department decides.
6. Chat front end. Our Private AI/ML service deploys the model, the retrieval and the tool connections; the chat widget in your customer portal or driver app and its connection to your platform stay with your team or your platform provider.

Power and room

By our estimate, the two-card server draws about 1.8 kW at the wall with two EPYC 9135 processors and fits one 230 V 16 A feed, which carries 2.94 kW at 80 per cent. The four-card server draws about 3.4 kW with the cards at 600 W and needs 32 A or three-phase, or about 2.7 kW with the cards capped at 450 W. Eight cards draw about 6.3 kW by our estimate; Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, and feeds should be planned on the maker's figure. A depot office is no place for these machines: they need a server room or a data centre, as our guide to GPU server power on 16 A, 32 A and three-phase feeds works through. Without a server room of your own, our Private AI/ML service runs the platform on dedicated hardware in a Tier-3 data centre in Lithuania, inside the EU.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build the servers to order with two, four or eight RTX PRO 6000 Server Edition cards, assembled and tested before delivery, delivered anywhere in the EU on one EU contract and invoice, with manufacturer warranty on every component handled through us; quantities and lead time are confirmed in writing with the quote. Our Private AI/ML service, built by our engineering partner Vixen.UNO, deploys the language and document models, retrieval and the connections to your systems. Our guide to buying a GPU server for LLMs lists every configuration.

FAQ