Self-hosted code completion server: small FIM models, latency and GPUs for many developers

Completion is its own workload

A code completion server for 100 to 500 developers runs a small fill-in-the-middle (FIM) model of 1.5B to 7B parameters on a few small GPUs, separate from the chat and agent models. By our estimate a 2U server with four NVIDIA RTX PRO 4500 Blackwell Server Edition cards covers the busy hour of a few hundred developers with a 1.5B-class model and short prompts.

The server is sized by how many requests per second it answers within a latency target; memory is rarely the limit. By our arithmetic from published measurements, one RTX PRO 4500 Server Edition starts at most about 10 requests per second for a 4B model with 1,024-token prompts.

Completion fills in code as the developer types. The IDE extension sends the code before and after the cursor, the model writes the part in between, and the suggestion is useful only if it appears before the developer types on. Our guide to a private coding assistant separates the three coding workloads and covers chat and agents; our page on a server for coding agents sizes agents, whose prompts run to tens of thousands of tokens. This page covers the third one.

Our summary of the workloads as described in our coding assistant and coding agent guides; prompt and answer lengths are typical orders of magnitude, not measurements.

Models trained for fill-in-the-middle

A completion model has to be trained for the FIM prompt format: prefix, suffix and a marker for the middle. The Continue extension, quoted in our coding assistant guide, states that "Most of the state-of-the-art autocomplete models are no more than 10b parameters" and suggests Qwen2.5-Coder 1.5B for local use.

Hugging Face model cards, config.json files and file lists (sum of the safetensors files), read on 24 October 2026. Qwen2.5-Coder-7B: full 131,072 tokens with YaRN, 32,768 in the default configuration.

JetBrains describes Mellum-4b-base as "tailored for code completion", with FIM prompts built from <fim_suffix>, <fim_prefix> and <fim_middle> markers; its card notes that it is a base model that supports further fine-tuning, and a Python SFT variant is published. The Qwen2.5-Coder base cards recommend the models for "fill in the middle tasks" rather than conversation. Qwen states that FIM "is supported in every version of Qwen3-Coder", as our coding assistant guide quotes, so the 30B-A3B model can serve chat and completion from one copy on a larger card.

JetBrains' newest open model, Mellum2.1-12B-A2.5B-Thinking, is a different kind of model. Its card calls it "a thinking model" for agentic work in repositories and for hard coding, maths and reasoning problems, and it does not mention fill-in-the-middle. Treat it as a candidate for chat or agents, not for inline completion. Two models that come up in the same searches have licence limits: Qwen2.5-Coder-3B is licensed for non-commercial use and Codestral 22B for non-production use, as our coding assistant guide sets out.

How fast one suggestion has to be

The time a developer waits is the time to the first token (TTFT) plus the generation of the rest of the suggestion. TTFT includes queueing and reading the prompt, which our guide to LLM latency targets defines with NVIDIA's terms. For completion we use these example targets at the 95th percentile, our values for a sizing exercise, not a standard: first token within 200 ms and a complete suggestion of about 32 tokens within 600 ms, measured at the server.

StorageReview published single-user figures for a small dense model on both small data-centre cards on 24 September 2026: Qwen3.5-4B in BF16 with vLLM 0.24.0 in an HPE ProLiant DL145 Gen11 with one AMD EPYC 8534P.

StorageReview, RTX PRO 4500 edge review in the HPE ProLiant DL145 Gen11, 24 September 2026, read on 24 October 2026; TTFT from the shortest to the longest request shape (up to 1,024 prompt tokens), time per token at 256 tokens in and out. Last column: our arithmetic, longest TTFT plus 31 further tokens.

At one user, the RTX PRO 4500 Server Edition meets the example target with a 4B model and the L4 does not. A smaller model shortens both parts: Qwen2.5-Coder-1.5B has 3.09 GB of weights against 8.04 GB for Mellum-4b-base, and every generated token reads the weights once.

Our arithmetic: memory bandwidth (300, 672 and 800 GB/s by NVIDIA's specifications) ÷ weight bytes read per token, one request. Upper bounds by our estimate; the measured 12.4 ms per token of the 4B model on the 4500 Server Edition is 81 tokens per second.

By these ceilings a 7B model in BF16 needs at least 0.6 s for 31 tokens on an RTX PRO 4500 Server Edition and 1.6 s on an L4, on top of reading the prompt. For an L4 fleet, the 1.5B class is the realistic choice; on the RTX PRO 4500 Server Edition, models up to about 4B meet the example target, and a 7B model fits where suggestions are shorter or a slower answer is acceptable.

Requests per second: reading the prompt sets the limit

Under load, the card spends most of its time reading prompts. StorageReview's test includes a shape close to completion, 1,024 prompt tokens and 32 output tokens. At 32 concurrent requests the RTX PRO 4500 Server Edition delivered 11 output tokens per second per user on it, against 44 to 53 on the other shapes, and the L4 4.6, against 10 to 18.

From the single-user figures, the RTX PRO 4500 Server Edition read 1,024 prompt tokens in 99 ms, about 10,000 tokens per second, and the L4 in 219 ms, about 4,700. By that arithmetic one card can start at most about 10 completion requests of 1,024 tokens per second and an L4 about 4.6. The loaded test agrees: 32 requests at 11 tokens per second for 32 tokens is about 11 requests per second on the 4500, and 4.6 on the L4. Each of those requests then took about 3 seconds on the 4500 and 7 on the L4, far outside any completion target, so the usable rate within a latency target is lower. These are our derivations from a 4B general model; a 1.5B model reads a prompt with roughly 40 per cent of the arithmetic, by parameter count, which a test has to confirm.

Three levers move the rate more than the card does:

1. Prompt length. The extension decides how much code around the cursor it sends. Half the prompt is about half the reading work.
2. Prefix caching. Consecutive requests from the same file start with nearly the same tokens, and vLLM's prefix caching, on by default, reuses the blocks it has already computed while they stay in memory. Routing each developer to the same copy keeps those blocks on one card. Measure the cache hit rate in a pilot.
3. Model size. Moving from a 7B to a 1.5B model cuts reading and generating work by about four to five times by parameter count.

To estimate your load, count requests per second. As example values to replace with your own gateway logs: if 20 per cent of 500 developers type at the busiest moment and each produces one request every two seconds, that is 50 requests per second. At the planning figure of about half the 10-request ceiling for a 4B model, that is ten RTX PRO 4500 Server Edition cards, or fewer with a 1.5B model, shorter prompts and a good prefix hit rate. We size completion servers from your request logs and target latency; send us your developer count, IDE extension and current request rate, and our team replies with a card count and a layout.

Memory per card and MIG halves

The KV cache per completion request is small. With the layers, KV heads and head size in each config.json, a 2,048-token request takes 56 MiB on Qwen2.5-Coder-1.5B and 112 MiB on Qwen2.5-Coder-7B in 16-bit. Mellum-4b-base uses full multi-head attention, 24 KV heads in each of 30 layers, so the same request takes 720 MiB.

Our estimates by the users-per-card rule, not measurements: requests of 2,048 tokens in flight with a 16-bit KV cache, usable memory 17.2 GiB (L4, ECC on), 18.5 GiB (RTX PRO 4000) and 25.7 GiB (RTX PRO 4500 Server Edition) per card, as on our configuration pages.

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for the runtime; the rest holds weights and cache, as our guide to how many users one RTX PRO 6000 can serve derives. For the 1.5B and 7B models, memory holds far more requests than the card can process in time. Mellum-4b-base is the exception: with an 8K context and a full cache, an FP8 cache doubles its count, to about 51 requests per RTX PRO 4500 Server Edition.

The RTX PRO 4500 Server Edition splits into two MIG instances of 16 GB, each with half the memory and half the streaming multiprocessors, as our page on four RTX PRO 4500 Server Edition cards in 2U describes. A 1.5B model fits a half easily, so one card can run two independent engines, for example completion for two teams or a production and a test copy. The L4 and the RTX PRO 4000 have no MIG.

Servers for 100 to 500 developers

Completion scales by copies: one copy per card or per MIG half, with a load balancer or the gateway spreading requests. No copy is split over several cards, so nothing crosses PCIe between them.

Our estimates as on our configuration pages for four and eight RTX PRO 4500 Server Edition, eight L4 and four RTX PRO 4000: board power plus processors plus 15 per cent for the rest. A measurement on the delivered system confirms them.

By the derivation above, four RTX PRO 4500 Server Edition cards start up to about 40 requests per second with a 4B model at 1,024-token prompts, eight L4 about 37. The L4 server reaches that figure with twice the cards and at 1.3 seconds or more per 32-token suggestion, so for completion we size L4 servers with 1.5B-class models. The RTX PRO 4500 Server Edition is passive and runs only in a server whose maker lists it; our 8× RTX PRO 4500 Server Edition server page lists chassis and power, and our RTX PRO 4500 benchmarks collect the published results.

Where agents run beside completion, give completion its own cards. An agent step with a 32K-token prompt occupies a card for seconds, and a completion request queued behind it misses its target. A server with four RTX PRO 6000 Server Edition for agents and a separate four-card RTX PRO 4500 Server Edition server for completion keeps the two apart, and our guide to serving several models on one GPU server covers engines that share a machine.

Engines, endpoints and measuring

vLLM serves one model per instance and exposes an OpenAI-compatible completions endpoint; the extension builds the FIM prompt in the model's own format. llama-server has an /infill endpoint for FIM, and Ollama accepts a suffix on its completions endpoint, as our coding assistant guide notes with the minimum versions that fix current security advisories. Put a gateway with keys and logging in front, as in our guide to an LLM gateway with keys, budgets and logging; its logs give the request rate and prompt length you need for sizing.

Speculative decoding shortens the generation part for one user and helps least when a card is saturated by prompt reading; our article on speculative decoding on premise covers when it pays. To measure your own server:

1. Record a day of completion requests from the gateway: prompt length, answer length and timestamps.
2. Replay them with vllm bench serve at rising request rates.
3. Read TTFT and end-to-end latency at the 95th percentile and the prefix cache hit rate.
4. Take the highest rate that meets your target as the capacity per copy.

We set up the completion model, the gateway and the measurement as part of our Private AI/ML service; describe your developer tools and the latency you want, and we propose a pilot on one card.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the RTX PRO 4500 Blackwell Server Edition, the L4 and the RTX PRO 4000, and build AI servers to order with four or eight cards, assembled and tested before delivery, with the operating system, NVIDIA driver, CUDA and a container runtime installed on request. Every component carries the manufacturer warranty, handled through us, on one EU contract and invoice; quantities and lead time are confirmed in writing with the quote. The four-card configuration and its ordering steps are on our page to buy a 4× RTX PRO 4500 Server Edition server.

FAQ