Private coding assistant on your own GPUs: which model, which GPU for your team

Completion, chat and agents: three sizing rules

A private coding assistant is sized by the agent sessions that run at once. By our users-per-card rule, one RTX PRO 6000 holds Qwen3-Coder-30B-A3B in FP8 with 8 sessions of 64K tokens (17 with an FP8 cache) and an H200 NVL holds 15, while a DGX Spark suits a pilot or a small team. We supply all three and build GPU servers to order with them. Write to us about your developers’ agents and our team will help.

Completion fills in code as you type. The IDE extension sends the code before and after the cursor, the model writes the middle (fill-in-the-middle, FIM), and the suggestion only helps if it arrives before the next keystroke. That calls for a small model trained for the task. The Continue extension, read-only since its final release as described below, says the models it suggests “are trained with a highly specific prompt format” and that “Most of the state-of-the-art autocomplete models are no more than 10b parameters”; for local use it points to ollama run qwen2.5-coder:1.5b, a 986 MB download of a 1.54-billion-parameter Apache 2.0 model with a 32K context. Qwen states that FIM “is supported in every version of Qwen3-Coder” as well.

Chat is a developer asking about code they paste in: prompts of a few thousand tokens and answers read at human speed. On one DGX Spark, the llama.cpp maintainers’ figures for Qwen3-Coder-30B-A3B at 8-bit with 4,096-token prompts work out to 15 tokens per second per request with 8 requests at once and 10 with 16.

Agents edit files and run commands in steps. Each step is a model call that carries the task so far: instructions, the files read, command output and the earlier steps. Prompts grow into tens of thousands of tokens, and Ollama’s documentation asks for at least 64,000 tokens of context for agents and coding tools.

Why agents change the sizing

Reading a prompt is compute work; generating tokens is limited by memory bandwidth, and every step also reads the session’s cache. NVIDIA’s own agent test on DGX Spark, published in March 2026, ran Qwen3 Coder Next in FP8 on vLLM: one 128K-token prompt took 54 seconds to read, and the full answer was complete after 89 seconds. With 32K-token prompts and 1K-token answers, one task took 35 seconds, two at once 54 seconds and four at once 91 seconds. The median first token came after 9, 12 and 15 seconds, and NVIDIA reports that prompt processing throughput rose about 3× from one task to four; generating the answers took longer than reading the prompts. On llama.cpp, the maintainers’ figures show prompts queuing instead. Qwen3-Coder-30B-A3B reads about 2,700 to 2,900 prompt tokens per second on a Spark whether one request waits or thirty-two, so eight 8,192-token prompts take 24 seconds before the last one starts to answer.

NVIDIA’s TensorRT-LLM performance tables, measured at maximum throughput, include Qwen3-30B-A3B, which has the same layer, head and expert counts as the Coder model. In FP4 on one RTX PRO 6000 Server Edition it produces 9,938 output tokens per second with 1,000-token prompts and answers, 1,914 with 8,192-token prompts and 374 with 32,768-token prompts, the last two with 1,024-token answers. By our arithmetic, 374 tokens per second at 1,024 tokens per answer is about 22 completed 32K-token calls per minute for the whole card, shared by every agent on it.

Prefix caching reduces this cost. An agent’s next call usually begins with the same tokens as its last one, and vLLM (prefix caching on by default), SGLang (RadixAttention) and TensorRT-LLM (KV block reuse) can reuse the cached part while it is still in memory. None of the tests above reports such reuse, and keeping the cache resident takes memory. Our article on one DGX Spark for a team has the full concurrency figures for one Spark.

Open coding models and their licences

Parameters and licences from the Hugging Face model cards and API, September 2026; weights are file sizes for gpt-oss and tensor bytes for the others. Cache per 64K-token session in BF16, by our arithmetic from each config.json; an FP8 cache halves it.

All six allow commercial use under Apache 2.0. Three models that come up in the same searches carry conditions: Devstral 2 (123B), whose modified MIT licence excludes companies above a monthly revenue threshold; Codestral 22B, whose Mistral AI Non-Production License limits use to “testing, research, Personal, or evaluation purposes in Non-Production Environments”; and Qwen2.5-Coder-3B, whose Qwen Research licence is “FOR NON-COMMERCIAL PURPOSES ONLY”, although its 1.5B and 7B siblings are Apache 2.0. The Qwen3-Coder cards state that the models support only non-thinking mode. As of 16 October 2026 Mistral’s documentation lists Devstral 2 and Devstral Small 2 under “Deprecated & retired models”; that list concerns Mistral’s API, and the weights of both remain published on Hugging Face. Newer open models need far less cache per session: by the same rule, one RTX PRO 6000 holds Qwen3.6-35B-A3B (Apache 2.0) in FP8 with 18 sessions of 128K tokens, as our guide to Claude Code with a local model shows.

Memory: the weights plus one cache per session

Every open session holds its own key/value cache. Per token it is 2 × layers × key/value heads × head dimension × bytes per value, with the numbers from each model’s config.json. Qwen3-Coder-30B-A3B has 48 layers, 4 key/value heads and a head dimension of 128: 96 KiB per token in BF16, 6 GiB for a 64K-token session and 24 GiB at its native 256K. Devstral Small 2 has 40 layers with 8 key/value heads of dimension 128: 160 KiB per token and 10 GiB per 64K session.

Models that attend to the full context in only some layers need far less. In gpt-oss-120b, 18 of 36 layers use a 128-token sliding window, which vLLM’s cache manager keeps only for “the most recent sliding_window_size tokens”, leaving 36 KiB per token. Qwen3-Coder-Next keeps a per-token cache in 12 of its 48 layers, 24 KiB per token; the other 36 use linear attention with a fixed state per request, which we estimate at under 80 MiB. An FP8 cache (--kv-cache-dtype fp8 in vLLM) halves every per-token figure, not the linear-attention state, and without calibrated scales can cost accuracy.

Weights and cache must fit in the memory the engine may use. By our users-per-card rule, as in our article on users per RTX PRO 6000, that is 90 per cent of the memory the driver reports, less about 3 GiB per card for activations and CUDA graphs, so the counts below are ceilings. That leaves 61.5 GiB on the 72 GB RTX PRO 5000 (we found no driver figure for it and scale the 96 GB card’s 95.6 GiB to 71.7 GiB, as our other RTX PRO 5000 articles do), 83.0 GiB of the 95.6 GiB a 96 GB card shows, and 123.4 GiB on an H200 NVL. vLLM prints the real figure at startup as a maximum concurrency for the context you set, as that article explains.

Hardware tiers and what the numbers support

Size the hardware by simultaneous sessions rather than by the number of developers. A developer whose agent is working holds one long context, while completion and chat requests are short. The table counts 64K-token sessions that fit beside the weights, with the nearest published measurement per tier.

Sessions of 64K tokens beside Qwen3-Coder-30B-A3B or Qwen3-Coder-Next in FP8, BF16 cache (FP8 cache in brackets), by our arithmetic; Coder-Next counts include up to 80 MiB of linear-attention state each. DGX Spark working set from our 128 GB article (0.8 and 0.9 of the pool); at the 0.7 of NVIDIA’s single-Spark vLLM launch script, about 90 GB, Coder 30B-A3B holds about 9 (18) sessions. NVIDIA data: DGX Spark blog, March 2026; TensorRT-LLM performance tables (docs 1.3.0rc29, read 5 October 2026), output tokens per second per GPU at maximum throughput, answers of 1,024 tokens after 32K prompts and 1,000 after 1K.

DGX Spark suits a pilot or a small team: speed limits it before memory does, and NVIDIA’s agent table stops at four parallel tasks, 91 seconds for all four. On the RTX PRO 5000 72 GB, gpt-oss-120b leaves about 0.7 GiB by this rule, short of one 64K session. Two RTX PRO 6000 give the most throughput as two copies of a model that fits one card: in NVIDIA’s table one copy split across two cards by expert parallelism produced 8,409 output tokens per second per card with 1K prompts, against 9,938 on one. One copy across both holds more 64K sessions, 22 against 16 with a BF16 cache by our arithmetic, and a larger model can still be split across both, over PCIe. Four cards hold Qwen3-Coder-Next best as two pairs: with 2 key/value heads, a four-way split would duplicate its cache, and vLLM disables its custom all-reduce for more than two GPUs without NVLink. Each pair exchanges data over PCIe, and no published measurement covers this layout, so plan a load test in your pilot. Our guide to a server for coding agents sizes two, four and eight Server Edition cards for 20 to 100 developers.

The H200 NVL runs FP8 but, as a Hopper card, not FP4 arithmetic. NVIDIA’s nearest figure is for the H200 SXM, with the same 141 GB and 4.8 TB/s; the NVL has about 16 per cent less FP8 compute in NVIDIA’s specifications. Qwen3-Coder-480B-A35B needs a multi-GPU server: four H200 NVL leave about 44 GiB for cache after its 482.1 GB of FP8 weights, five 64K sessions with an FP8 cache.

You can order the server alone, the setup of vLLM and Qwen3-Coder-30B-A3B for Cline or Kilo Code alone, or both together: the server delivered with everything installed and configured; tell us how many agent sessions run at once, and we suggest the best solution for your tasks and budget.

Serving it and connecting the IDE

For a team, serve the main model with vLLM or SGLang, which batch continuously and reuse prefixes; our engine comparison has the details. Agents need tool calling, which vLLM enables with --enable-auto-tool-choice plus the model’s parser, and SGLang with the parser alone. The Qwen3-Coder-Next card uses --tool-call-parser qwen3_coder for both, the Devstral Small 2 card --tool-call-parser mistral for vLLM. Set the context you plan for, not the model’s maximum: --max-model-len in vLLM, --context-length in SGLang.

Run the completion model as its own small instance, on its own GPU if agents keep the main one busy: vLLM serves one model per server, and its memory share is set per instance. llama-server has an /infill endpoint for FIM, and Ollama accepts a suffix on its OpenAI-compatible completions endpoint. NVIDIA’s own DGX Spark playbook pairs Continue with Ollama and gpt-oss:120b, which suits a pilot once the proxy described below sits in front, because the playbook opens Ollama to the network (OLLAMA_HOST=0.0.0.0:11434, OLLAMA_ORIGINS=*). Continue no longer receives updates, so a pilot that should grow is better run with Cline or Kilo Code against the same endpoint. Use Ollama 0.35.0 or later: earlier versions are affected by CVE-2026-103663, published on 8 October 2026, through which an unauthenticated request to /api/pull can write files outside the model store and, where the Ollama service can write to /usr/lib/ollama, lead to code execution as root after a restart. For vLLM, use 0.31.0 or later, which fixes the advisory GHSA-h3rc-6mm3-gc2m, preferably the current release, 0.33.0 of 2 November 2026, whose CUDA 13.0 builds need an R580 or newer driver (as of November 2026). For more than one user, raise OLLAMA_NUM_PARALLEL and set the context, for example OLLAMA_CONTEXT_LENGTH=64000 ollama serve.

IDE extensions: Cline, Kilo Code and Continue

As of October 2026, two maintained open-source extensions connect the IDE to your own model, Cline and Kilo Code, while Continue, which also covered completion, is read-only. Continue, available as IDE extensions and a CLI, connects to any OpenAI-compatible server with provider: openai and an apiBase, lists vLLM and TensorRT-LLM among compatible servers, and gives a model the autocomplete role for completion. As of October 2026, its GitHub repository states that it “is no longer actively maintained and is read-only for all users”, after a final 2.0.0 release of its IDE extensions and CLI. Cline, an agent for the IDE and the terminal that edits files and runs commands with the user’s approval, takes a base URL, key, model ID and context window under its OpenAI Compatible provider and recommends its “Use Compact Prompt” setting for local models; it is under Apache 2.0. Kilo Code, under the MIT licence, takes a base URL, an optional key and model IDs as an OpenAI Compatible provider and documents Ollama and LM Studio for local models; by its documentation, its autocomplete requests are routed through the Kilo Gateway (read 31 October 2026), so for inline completion against your own model see our guide to a code completion server. Roo Code, a third agent extension, was shut down on 15 May 2026 and its repository archived. Which tools a team can still roll out, with their team features and the GPU each needs, is compared in our guide to self-hosted Copilot alternatives.

Governance: keeping source code in your network

Keep every endpoint on the internal network behind an authenticating proxy. Ollama listens on 127.0.0.1 by default and ignores API keys (“required but ignored” in its examples). Ollama’s own container image, however, sets OLLAMA_HOST=0.0.0.0:11434, and the documented docker run with -p 11434:11434 publishes the port on all host addresses, so a server set up that way exposes an API without authentication. vLLM listens on all interfaces unless --host is set. vLLM and llama-server check an --api-key but manage no user accounts; vLLM’s key protects only paths under /v1, /v2, /inference and /cohere, and its security page lists routes the key leaves open, such as /invocations, /pause and /update_weights. Log at the proxy who called which model, when and with how many tokens; prompts contain source code, so decide whether to store them at all. Logs per developer are personal data under the GDPR; your data protection officer confirms their purpose and retention period before the rollout. Point the extensions only at your endpoint and check their telemetry: Cline documents a telemetry switch and says its telemetry excludes code, file contents and conversation content.

What we supply

We supply the hardware for each tier across the EU, on one contract and invoice with manufacturer warranty: the DGX Spark Founders Edition (128 GB), the RTX PRO 5000 72 GB, the RTX PRO 6000 in Workstation, Max-Q and Server Edition, and the H200 NVL. The GPUs come as cards or in AI servers built to order, with NVIDIA AI Enterprise on the same quote where it is needed. If you also want the model deployed on-premise, with access control and logging of queries and answers, that is our Private AI/ML service.

FAQ