MoE expert offload with KTransformers: Kimi K2.6 on four GPUs and host DDR5 memory

What expert offload does, and where it stops

Kimi K2.6 does not fit four GPUs: its 595 GB checkpoint is more than four RTX PRO 6000 Blackwell Server Edition or four H200 NVL can hold. Expert offload keeps the attention layers, the shared expert and part of the routed experts on the GPUs and computes the remaining experts on the server's processors, from host DDR5 memory. The model then runs on a four-card server for one or a few users at a time, at a speed set by host memory bandwidth; a service used by many people at once still needs eight cards, as our Kimi K2 hardware guide sizes it.

The GPUs process the prompts and run the attention layers; host memory only extends the space for experts, so this remains GPU inference with a memory extension. The approach works because a mixture-of-experts (MoE) model uses a small part of its weights per token. Kimi K2.6 has 1T parameters and 32B active, 61 layers, 384 routed experts of which 8 are selected per token, one shared expert and one dense first layer, by its model card. Our article on MoE total and active parameters explains why all experts still have to sit somewhere in memory.

The Kimi-K2.6 card lists three engines as recommended: vLLM, SGLang and KTransformers. Its deployment guide runs vLLM and SGLang on one H200 node with TP=8 and KTransformers together with SGLang "for CPU+GPU heterogeneous inference".

Where the 595 GB go

The K2.6 configuration quantises the routed experts to 4-bit integers in blocks of 32 and keeps attention, the shared experts, the dense layer, the output head and the vision encoder in 16-bit. In each of the 60 MoE layers, with a hidden size of 7,168 and an expert width of 2,048, each expert has three matrices of 7,168 × 2,048, 44.0 million weights.

Kimi-K2.6 config.json and file list on Hugging Face (595 GB repository), read on 24 October 2026; our arithmetic at 4 bits plus one 16-bit scale per block of 32, which is our assumption for the stored format; the second row is the remainder.

The roughly 24 GB on the GPUs are read for every token; the experts are read only when a token is routed to them. Offload puts the large, rarely used part in host memory and keeps the small, always used part on the cards. The KV cache stays on the GPUs: Kimi K2 uses multi-head latent attention with 68.6 KiB per token in 16-bit, 2.14 GiB per conversation of 32K tokens, which vLLM keeps whole on every card under tensor parallelism, as our Kimi guide computes.

Four RTX PRO 6000 Server Edition or four H200 NVL with host memory

How much of the expert weight stays on the GPUs depends on the cards. We reserve on each server the 24 GB of shared weights, the cache for four conversations of 32K on every card, and, as our assumption, about 9 GB per card for long prompts: KTransformers' kt-kernel documentation states that its layer-by-layer GPU prefill "requires one MoE layer extra VRAM (e.g., ~9GB+ for Kimi-K2-Thinking)".

Our estimates: usable memory by our users-per-card rule, less 22.4 GiB of shared weights, 4 × 4 × 2.14 GiB of cache and 4 × 8.4 GiB of prefill reserve. Compute capability from NVIDIA; wheel list from the kt-kernel README on GitHub, read on 24 October 2026.

For the RTX PRO 6000 that is 332.2 − 22.4 − 34.2 − 33.5 = 242.1 GiB, about 260 GB of expert weights on the cards. The H200 NVL server keeps three quarters of the experts on the GPUs, so far fewer tokens wait for the processors; its NVLink bridge joins the four cards for the tensor-parallel part. Our pages on the 4× RTX PRO 6000 Server Edition server and the 4× H200 NVL server describe both machines.

KTransformers decides which experts sit on the GPUs at start-up. Its kt-kernel documentation lists the placement strategies uniform, the default, which "distributes GPU experts evenly across all MoE layers", and frequency, which "places the most frequently activated experts on GPU" from a statistics file. With real prompts, routing is uneven, so a frequency placement from your own traffic serves more tokens from GPU memory than the share in the table suggests.

We supply both four-card servers and size host memory and processors for offload; send us the model, the number of concurrent users and your context length, and our team replies with the split between GPU and host memory and an estimate of the speed.

Speed: host memory bandwidth is the ceiling

Every expert computed on the processors is read from DDR5. AMD gives the EPYC 9355 a memory bandwidth of 614 GB/s per socket with 12 channels of DDR5-6400, so two sockets with every channel filled offer 1,229 GB/s. One token with all routed experts in host memory reads about 11.9 GB, which gives an upper bound of about 100 tokens per second for one user. With 45 per cent of the experts on four RTX PRO 6000 Server Edition, the processors read about half as much per token.

More users at once do not scale that figure. Each conversation routes its token to its own 8 experts per layer, so a batch reads the union of the experts its tokens need. With uniform routing as our assumption, the union grows quickly:

Our arithmetic, upper bounds by our estimate: all routed experts in host memory, 1,229 GB/s of theoretical DDR5 bandwidth fully used, uniform routing (384 × (1 − (376/384)^users) experts per layer), processor arithmetic and GPU time not counted. Real systems reach a fraction of each figure.

The table gives limits from arithmetic; a server delivers less. Achievable bandwidth on DDR5 is below the theoretical figure, each socket reads fastest from its own memory, and the processors also have to compute the experts. It shows the shape: beyond a handful of concurrent users, each user gets a few tokens per second, while eight GPUs that hold all experts read them from HBM or GDDR7 at 1,597 to 4,800 GB/s per card.

Published results with KTransformers

The deployment guide on Moonshot AI's Kimi-K2.6 card reports for KTransformers with SGLang: "Achieves 640.12 tokens/s Prefill and 24.51 tokens/s Decode (48-way concurrency) on 8× NVIDIA L20 + 2× Intel 6454S." The guide does not say whether 24.51 tokens per second is the total over 48 requests or the rate per request, and the two readings differ by a factor of 48, so the figure cannot size a server on its own. Its launch command also sets --tensor-parallel-size 4 beside the eight-card hardware line.

The KTransformers README gives a second figure: DeepSeek-R1-0528 in FP8 on eight L20 with a Xeon Gold 6454S reached 227.85 tokens per second of total throughput and 87.58 output tokens per second at eight concurrent requests. Both results come from the KTransformers project and Moonshot AI. We found no published Kimi K2.6 offload result on the RTX PRO 6000 or the H200 NVL, so a test on your prompts and concurrency is the only firm figure.

Software: KTransformers, SGLang and llama.cpp

KTransformers is an Apache 2.0 research project of the kvcache-ai organisation for heterogeneous CPU-GPU inference and fine-tuning; its README lists release v0.7.0 of 17 August 2026. It names Kimi-K2, K2-0905, K2-Thinking and K2.5 among supported models, not K2.6, while the K2.6 card states that the model "has the same architecture as Kimi-K2.5, and the deployment method can be directly reused". The K2.5 tutorial asks for at least 48 GB of GPU memory in total, "x86 CPU with AVX512F support", "At least 600GB system memory" and about 600 GB of storage for the weights, one folder for CPU and GPU.

KTransformers kt-kernel README on GitHub, read on 24 October 2026. The README warns that AMXINT4 "May cause huge accuracy drop for some models".

AMD states that 5th Gen EPYC processors "implement the full set of AVX-512 instructions" (AMD blog, 22 January 2025), so the RAWINT4 path, which the K2.6 guide uses, runs on EPYC 9005; the AMX methods need Intel Xeon. The README asks to set --kt-cpuinfer to the physical core count, not hyperthreads, and --kt-threadpool-count to the number of NUMA nodes.

The GPU side is the open point for the RTX PRO 6000. The kt-kernel README lists GPU acceleration for SM 80, 86, 89 and 90 (Ampere, Ada, Hopper) and does not mention Blackwell. The H200 NVL, at compute capability 9.0, is inside that list. For the RTX PRO 6000 at 12.0, plan a source build and a test run before you commit to the layout; our page on Blackwell RTX software support covers SGLang and vLLM on that architecture.

llama.cpp offers a simpler form of offload: its server keeps "all Mixture of Experts (MoE) weights in the CPU" with --cpu-moe, or those of the first N layers with --n-cpu-moe. It needs a GGUF conversion and is not among the engines Moonshot AI recommends for K2.6. vLLM's offload option in the K2.6 recipe works differently: it extends the prefix cache into host memory, as our Kimi guide and our article on KV cache offloading to CPU memory and NVMe explain, and does not move experts.

Processors and memory for an offload server

For offload, the host memory is part of the inference path, and its bandwidth matters as much as its size.

• Every channel filled. Two EPYC 9005 have 24 channels; one module per channel at DDR5-6400 keeps the full 614 GB/s per socket. Our guide to how much RAM a GPU server needs lists the steps: 24 modules of 32, 64 or 96 GB give 768, 1,536 or 2,304 GB.
• Capacity. The K2.5 tutorial asks for at least 600 GB. The 768 GB of our four-card inference node on AI servers meet that with little room for the page cache and the operating system; 1,536 GB with 64 GB modules leave headroom for a second model or a larger cache.
• Cores. KTransformers runs its expert threads on physical cores, and the K2.6 example sets 96 of them. More cores per socket raise the arithmetic the processors can do per step; our guide to the CPU for a GPU server compares EPYC and Xeon.
• Power. Four RTX PRO 6000 Server Edition at 600 W with two EPYC 9355 come to about 3.4 kW at the wall by our estimate; processors with more cores add their TDP, and a measurement on the delivered system confirms the figure.

When offload fits, and when eight cards do

Expert offload fits these cases:

• Evaluating Kimi K2.6 on a four-card server you already have, before buying for production.
• A few users at a time: one team's agent, a batch job that runs overnight, a research team.
• Development work where the quality of the 1T model matters more than the speed of the answer.

It does not fit a company assistant with dozens of conversations at once. By our estimate above, 32 concurrent users would each get about 4 tokens per second at best. Eight H200 NVL hold the INT4 weights in GPU memory, and vLLM's Kimi-K2-Thinking recipe on eight H200 reports cache for about 21 conversations of 32K with tensor parallelism and about 174 with decode context parallelism, as our Kimi guide shows; eight RTX PRO 6000 Server Edition also hold the weights, and our page on eight RTX PRO 6000 Server Edition for LLMs covers that server. Two four-card servers joined by pipeline parallelism are a third route, set out in a 600 GB model across two 4-GPU servers. And a smaller MoE model that fits four cards whole, such as DeepSeek-V4-Flash-0731 at 167 GB or gpt-oss-120b, runs faster on the same hardware.

Moonshot AI publishes K2.6 under a Modified MIT License. Its only modification applies to commercial products or services with "more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in monthly revenue", which must "prominently display "Kimi K2.6"" on their user interface. Whether that applies to your company is for your legal department to confirm. We set up the model, the engine and the measurement as part of our Private AI/ML service; describe the users and tasks you plan for Kimi K2.6, and we propose offload for a pilot or an eight-card server for production.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with four or eight RTX PRO 6000 Server Edition or H200 NVL cards, two AMD EPYC or Intel Xeon processors and DDR5 ECC memory in every channel, assembled and tested before delivery, with the operating system, NVIDIA driver, CUDA and a container runtime installed on request. Every component carries the manufacturer warranty, handled through us, on one EU contract and invoice; quantities and lead time are confirmed in writing with the quote.

FAQ