A private coding assistant on your own hardware: which models, how to serve them, which GPU for which team

Three workloads, three sizing rules

Completion fills in code as you type. The IDE extension sends the code before and after the cursor, the model writes the middle (fill-in-the-middle, FIM), and the suggestion only helps if it arrives before the next keystroke. That calls for a small model trained for the task. The Continue extension says the models it suggests “are trained with a highly specific prompt format” and that “Most of the state-of-the-art autocomplete models are no more than 10b parameters”; for local use it points to ollama run qwen2.5-coder:1.5b, a 986 MB download of a 1.54-billion-parameter Apache 2.0 model with a 32K context. Qwen states that FIM “is supported in every version of Qwen3-Coder” as well.

Chat is a developer asking about code they paste in: prompts of a few thousand tokens and answers read at human speed. On one DGX Spark, the llama.cpp maintainers’ figures for Qwen3-Coder-30B-A3B at 8-bit with 4,096-token prompts work out to 15 tokens per second per request with 8 requests at once and 10 with 16.

Agents edit files and run commands in steps. Each step is a model call that carries the task so far: instructions, the files read, command output and the earlier steps. Prompts grow into tens of thousands of tokens, and Ollama’s documentation asks for at least 64,000 tokens of context for agents and coding tools.

Why agents change the sizing

Reading a prompt is compute work; generating tokens is limited by memory bandwidth, and every step also reads the session’s cache. NVIDIA’s own agent test on DGX Spark, published in March 2026, ran Qwen3 Coder Next in FP8 on vLLM: one 128K-token prompt took 54 seconds to read, and the full answer was complete after 89 seconds. With 32K-token prompts and 1K-token answers, one task took 35 seconds, two at once 54 seconds and four at once 91 seconds. The first token came after a median 9, 12 and 15 seconds, as vLLM read four prompts together about 2.4 times as fast as one, by our arithmetic; generating the answers took the rest, longer than reading the prompts. In the llama.cpp maintainers’ figures prompts do queue: Qwen3-Coder-30B-A3B reads about 2,700 to 2,900 prompt tokens per second on a Spark whether one request waits or thirty-two, so eight 8,192-token prompts take 24 seconds before the last one starts to answer.

NVIDIA’s TensorRT-LLM performance tables, measured at maximum throughput, give Qwen3-30B-A3B, which has the same layer, head and expert counts as the Coder model, in FP4 on one RTX PRO 6000 Server Edition: 9,938 output tokens per second with 1,000-token prompts and answers, then 1,914 and 374 with prompts of 8,192 and 32,768 tokens and 1,024-token answers. By our arithmetic, 374 tokens per second at 1,024 tokens per answer is about 22 completed 32K-token calls per minute for the whole card, shared by every agent on it.

Prefix caching softens this. An agent’s next call usually begins with the same tokens as its last one, and vLLM (prefix caching on by default), SGLang (RadixAttention) and TensorRT-LLM (KV block reuse) can reuse the cached part while it is still in memory. None of the tests above reports such reuse, and keeping the cache resident takes memory. Our article on one DGX Spark for a team has the full concurrency figures for one unit.

Open coding models and their licences

Parameters and licences from the Hugging Face model cards and API, September 2026; weights are file sizes for gpt-oss and tensor bytes for the others. Cache per 64K-token session in BF16, by our arithmetic from each config.json; an FP8 cache halves it.

All six allow commercial use under Apache 2.0. Three models that come up in the same searches carry conditions: Devstral 2 (123B), whose modified MIT licence excludes companies above a monthly revenue threshold; Codestral 22B, whose Mistral AI Non-Production License limits use to “testing, research, Personal, or evaluation purposes in Non-Production Environments”; and Qwen2.5-Coder-3B, whose Qwen Research licence is “FOR NON-COMMERCIAL PURPOSES ONLY”, although its 1.5B and 7B siblings are Apache 2.0. The Qwen3-Coder cards state that the models support only non-thinking mode.

Memory: the weights plus one cache per session

Every open session holds its own key/value cache. Per token it is 2 × layers × key/value heads × head dimension × bytes per value, with the numbers from each model’s config.json. Qwen3-Coder-30B-A3B has 48 layers, 4 key/value heads and a head dimension of 128: 96 KiB per token in BF16, 6 GiB for a 64K-token session and 24 GiB at its native 256K. Devstral Small 2 has 40 layers with 8 key/value heads of dimension 128: 160 KiB per token and 10 GiB per 64K session.

Models that attend to the full context in only some layers need far less. In gpt-oss-120b, 18 of 36 layers use a 128-token sliding window, which vLLM’s cache manager keeps only for “the most recent sliding_window_size tokens”, leaving 36 KiB per token. Qwen3-Coder-Next keeps a per-token cache in 12 of its 48 layers, 24 KiB per token; the other 36 use linear attention with a fixed state per request, which we estimate at under 80 MiB. An FP8 cache (--kv-cache-dtype fp8 in vLLM) halves every per-token figure, not the linear-attention state, and without calibrated scales can cost accuracy.

Weights and cache must fit in the memory the engine may use. We keep a tenth in reserve: 64.8 GiB on the 72 GB RTX PRO 5000, taking 72 GB as 72 GiB since we found no driver figure for it, 86.0 GiB of the 95.6 GiB a 96 GB card shows, and 126.4 GiB on an H200 NVL. vLLM prints the real figure at startup as a maximum concurrency for the context you set, as our article on users per RTX PRO 6000 explains.

Hardware tiers and what the numbers support

Size by simultaneous sessions, not headcount: a developer whose agent is working holds one long context, while completion and chat requests are short. The table counts 64K-token sessions that fit beside the weights, with the nearest published measurement per tier.

Sessions of 64K tokens beside Qwen3-Coder-30B-A3B or Qwen3-Coder-Next in FP8, BF16 cache (FP8 cache in brackets), by our arithmetic; Coder-Next counts include up to 80 MiB of linear-attention state each. DGX Spark working set from our 128 GB article. NVIDIA data: DGX Spark blog, March 2026; TensorRT-LLM performance tables (docs 1.3.0rc28), output tokens per second per GPU at maximum throughput, answers of 1,024 tokens after 32K prompts and 1,000 after 1K.

DGX Spark suits a pilot or a small team: speed limits it before memory does, and NVIDIA’s agent table stops at four parallel tasks, 91 seconds for all four. On the RTX PRO 5000 72 GB, gpt-oss-120b loads with about 4 GiB to spare, which is one session. Two RTX PRO 6000 give the most throughput as two copies of a model that fits one card: in NVIDIA’s table one copy across two cards produced 8,409 output tokens per second per card with 1K prompts, against 9,938 on one. One copy across both holds more 64K sessions, 23 against 18 with a BF16 cache by our arithmetic, and a larger model can still be split across both, over PCIe.

The H200 NVL runs FP8 but, as a Hopper card, not FP4 arithmetic. NVIDIA’s nearest figure is for the H200 SXM, with the same 141 GB and 4.8 TB/s; the NVL has about 16 per cent less FP8 compute in NVIDIA’s specifications. Qwen3-Coder-480B-A35B needs a multi-GPU server: four H200 NVL leave about 56 GiB for cache after its 482.1 GB of FP8 weights, seven 64K sessions with an FP8 cache.

Serving it and connecting the IDE

For a team, serve the main model with vLLM or SGLang, which batch continuously and reuse prefixes; our engine comparison has the details. Agents need tool calling: vLLM takes --enable-auto-tool-choice plus the model’s parser, SGLang the parser alone. The Qwen3-Coder-Next card uses --tool-call-parser qwen3_coder for both, the Devstral Small 2 card --tool-call-parser mistral for vLLM. Set the context you plan for, not the model’s maximum: --max-model-len in vLLM, --context-length in SGLang.

Run the completion model as its own small instance, on its own GPU if agents keep the main one busy: vLLM serves one model per server, and its memory share is set per instance. llama-server has an /infill endpoint for FIM, and Ollama accepts a suffix on its OpenAI-compatible completions endpoint. NVIDIA’s own DGX Spark playbook pairs Continue with Ollama and gpt-oss:120b, which suits a pilot; for more than one user, raise OLLAMA_NUM_PARALLEL and set the context, for example OLLAMA_CONTEXT_LENGTH=64000 ollama serve.

Two Apache 2.0 extensions cover the three workloads. Continue, available as IDE extensions and a CLI, connects to any OpenAI-compatible server with provider: openai and an apiBase, lists vLLM and TensorRT-LLM among compatible servers, and gives a model the autocomplete role for completion. Cline, an agent for the IDE and the terminal that edits files and runs commands with the user’s approval, takes a base URL, key, model ID and context window under its OpenAI Compatible provider and recommends its “Use Compact Prompt” setting for local models.

Governance: the code stays inside

Keep every endpoint on the internal network behind an authenticating proxy. Ollama listens on 127.0.0.1 by default and ignores API keys (“required but ignored” in its examples). Ollama’s own container image, however, sets OLLAMA_HOST=0.0.0.0:11434, and the documented docker run with -p 11434:11434 publishes the port on all host addresses, so a server set up that way exposes an API without authentication. vLLM listens on all interfaces unless --host is set. vLLM and llama-server check an --api-key but manage no user accounts, and vLLM’s documentation warns that its key leaves routes such as /invocations open. Log at the proxy who called which model, when and with how many tokens; prompts contain source code, so decide whether to store them at all. Point the extensions only at your endpoint and check their telemetry: Cline documents a telemetry switch and says its telemetry excludes code, file contents and conversation content.

What we supply

Eurokommerz supplies DGX Spark, the RTX PRO 5000 72 GB, the RTX PRO 6000 Workstation and Max-Q, and the H200 NVL across the EU with manufacturer warranty, as individual units or in AI servers built to order. We configure the machine for the model, context and engine you plan, with NVIDIA AI Enterprise where it is needed; the RTX PRO 6000 page covers the 96 GB card.

FAQ