Speculative decoding on your own GPUs: EAGLE-3, MTP and draft models in vLLM, SGLang, TensorRT-LLM

Speculative decoding: how a draft makes an LLM faster

Speculative decoding lets a small, fast draft propose several tokens and the large target model check them in one forward pass. The draft can be a separate small model, an EAGLE head trained on the target model’s hidden states, or the multi-token prediction (MTP) layers that some models ship in their own checkpoint. With the standard acceptance rule the output follows the large model’s distribution, and the time per output token falls for a single user or a small batch. The gain shrinks as concurrency rises, because a busy GPU has little idle capacity for the extra verification.

Leviathan, Kalman and Matias (arXiv 2211.17192, version of 18 May 2023) sample from the large model “without any changes to the outputs” and report on T5-XXL “a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs”. Chen and colleagues at DeepMind (arXiv 2302.01318, 2 February 2023) call it speculative sampling and use a “modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics”. On Chinchilla, a 70B model, they report “a 2-2.5x decoding speedup in a distributed setup”.

Why the gain is largest for one user and small batches

Generating one token for one user reads every weight from GPU memory, so decode is limited by memory bandwidth rather than compute. Our guide to LLM latency targets puts the single-stream ceiling for Llama 3.3 70B in FP8 at about 23 tokens per second on an RTX PRO 6000 Server Edition and about 68 on an H200 NVL. Verifying five drafted tokens also reads the weights once, so each accepted draft token adds output at little extra cost while the GPU is waiting on memory. In a large batch the weights are already read once for all users, the Tensor Cores fill up, and every extra position to verify costs compute time.

vLLM describes speculative decoding as a way “to reduce inter-token latency under medium-to-low QPS”, and the documentation of TensorRT-LLM’s pre-release 1.4.0rc0 calls it “a technique for accelerating LLM inference at low batch sizes”. The documentation of both the stable 1.2.1 and 1.4.0rc0 states: “There is currently no way to dynamically disable speculation, thus speed ups are only observable at low batch sizes.”

vLLM’s documentation for release 0.31.0 states that “Speculative decoding sampling is theoretically lossless up to the precision limits of hardware numerics.” It adds that log probabilities are not yet stable with speculation and that batch size can cause small variations, so check quality on your own test set.

EAGLE-3, MTP, draft models and n-gram in vLLM, SGLang and TensorRT-LLM

A separate draft model is a smaller model of the same family, such as Qwen3-0.6B for Qwen3-8B in vLLM’s example; by default vLLM requires both to share one vocabulary. EAGLE-3 (Li and colleagues, arXiv 2503.01840, version of 23 April 2025) predicts tokens directly instead of the features earlier EAGLE heads predict, and fuses features from several layers of the target. N-gram speculation, also called prompt lookup, needs no model and proposes tokens by matching n-grams in the prompt.

vLLM documentation of release 0.31.0 (5 October 2026), SGLang speculative decoding guide (no version on the page; current release 0.5.22 of 16 October 2026) and TensorRT-LLM documentation of the stable 1.2.1 and the pre-release 1.4.0rc0, all read on 16 October 2026. Memory column from the SGLang guide, the model cards below and our reading.

vLLM takes every method as one JSON value of --speculative-config, with keys such as method, model, num_speculative_tokens and draft_tensor_parallel_size. SGLang takes --speculative-algorithm, --speculative-draft-model-path and three tuning flags, --speculative-num-steps, --speculative-eagle-topk and --speculative-num-draft-tokens, to be left unset for auto-tuning or set together. TensorRT-LLM reads decoding_type and max_draft_len from the YAML file passed to trtllm-serve with --config.

vLLM’s comparison table names EAGLE, MTP, draft models, parallel draft models (PARD) and MLP speculators as the methods with the largest latency reduction and rates n-gram and suffix decoding at “Low to medium gain” at low QPS. Pages marked “latest” on vLLM’s site describe the main branch, so check names against the documentation of your release.

Open models that ship MTP layers or EAGLE-3 heads

DeepSeek-V3’s model card gives a total of 685B parameters on Hugging Face, “671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights”, and says the module “can also be used for speculative decoding for inference acceleration”. TensorRT-LLM 1.2.1 states that “MTP is currently only supported by Deepseek”; its 1.4.0rc0 documentation adds Qwen3.8 MoE, Qwen3.5 MoE and Step-3.x. vLLM’s MTP page lists the Gemma 4 assistant checkpoints for E2B, E4B, 12B, 26B-A4B and 31B, published by Google as separate repositories.

For models without MTP layers, EAGLE-3 heads are published per target model. Red Hat’s Llama-3.1-8B-Instruct-speculator.eagle3, released on 27 July 2025, lists 1.0B parameters and a vLLM command with the method eagle3 and three speculative tokens. NVIDIA’s gpt-oss-120b-Eagle3-short-context, released on 6 October 2025 with 0.8B parameters on Hugging Face, “is typically better for use cases of less than 8k context length”; NVIDIA released long-context and throughput heads for the same model on 20 August 2025. Each head is trained for the one target model its card names.

Extra GPU memory for the draft

Every method except n-gram adds weights and a KV cache. A head of 1B parameters takes about 2 GB in BF16 at two bytes per parameter, and NVIDIA’s 0.8B head for gpt-oss-120b about 1.6 GB beside the 65.3 GB checkpoint, whose expert weights are in MXFP4. SGLang’s guide states that speculative decoding “may increase GPU memory usage because the draft tree, CUDA graphs, and verification-related buffers consume additional VRAM”. Higher values of --speculative-eagle-topk or --speculative-num-draft-tokens raise it, and a lower --mem-fraction-static leaves room for the buffers.

The memory comes out of the KV cache. By the rule in our server sizing guides, one 96 GB RTX PRO 6000 holds 12 conversations of Llama 3.3 70B in FP8 at a full 8K context with an FP8 cache, 1.25 GiB each, and an H200 NVL holds 44. By our arithmetic a 1B head in BF16, 1.9 GiB, takes the room of one to two of them before its own cache.

Acceptance rate and what lowers it

The speed-up depends on how many drafted tokens the target model accepts. vLLM defines the per-request draft acceptance rate as accepted draft tokens divided by drafted tokens, and the mean acceptance length as the tokens emitted per verification step including the bonus token, between 1.0 and the number of speculative tokens plus one. Its /metrics endpoint exports spec_decode_num_draft_tokens_total and spec_decode_num_accepted_tokens_total, with the vllm: prefix.

Red Hat’s Llama 3.1 8B head card gives mean acceptance lengths of 3.02 for coding (HumanEval), 2.83 for maths (GSM8K) and 2.50 for summarisation (CNN/DailyMail) with three speculative tokens; with seven tokens the figures rise only to 3.89, 3.83 and 2.99. NVIDIA’s card for the gpt-oss-120b head, on MT-Bench with TensorRT-LLM 1.2.0rc0 on B200 and a draft length of 3, gives 2.75 tokens per step for maths, 2.67 for coding and 2.10 for writing. Open-ended text is harder to predict than code or arithmetic, and longer drafts add verification work for few extra tokens.

Context length also matters, and NVIDIA’s card recommends its short-context head for use cases under 8K tokens. SGLang’s guide warns that a deeper draft “risks rejection cascades”, and vLLM’s MTP page calls one speculative token “a good default to start with”.

Speed-up and concurrency: 20 to 50 users or 500

vLLM’s blog of 26 May 2026 benchmarked an EAGLE 3.1 head on Kimi-K2.6-NVFP4, with tensor parallelism over four GB200 GPUs, on the SPEED-Bench coding set. Against a run without speculation it reports 2.03 times the per-user output throughput at concurrency 1, 1.71 times at 4 and 1.66 times at 16. The EAGLE-3 paper reports “a speedup ratio up to 6.5x” and, in SGLang, “a 1.38x throughput improvement at a batch size of 64”.

vLLM’s older blog of 17 October 2024, which links the release 0.6.0 documentation, reports up to 1.5 times faster generation at QPS 1 for Llama 3 70B with a draft model on four H100. At high QPS the same set-up showed slowdowns of 1.4 and 1.8 times, as speculative decoding “can sometimes slow down the system when it is already compute-bound”.

Effects are qualitative except where a figure is given with its source; the H100, B200 and GB200 appear only as the test systems of those sources.

The comparison table in vLLM’s 0.31.0 documentation rates EAGLE and MTP at “Medium to high gain” at high QPS and a separate draft model at “Medium gain”, so treat the 2024 slowdowns as a reason to test.

Run as four replicas, one per card, four RTX PRO 6000 Server Edition cards hold 48 conversations of Llama 3.3 70B at 8K by our sizing rule, or about 40 with a 1B head on each card. For 20 to 50 users each card runs a modest batch and, by our estimate, stays largely memory-bound, the case in which a draft helps most. For 500 users across several servers each card runs large batches and the gain per user is smaller, so more cards or replicas add capacity. NVIDIA’s throughput head for gpt-oss-120b is, by its card, “useful for high-concurrency inference where a single speculated token is the optimal configuration”. Overnight batch jobs are sized by throughput, so test a draft on the chat endpoint first.

We build AI servers with RTX PRO 6000 Server Edition or H200 NVL cards, sized by model and concurrent users. Tell us the model, the peak requests in flight and the speed per user, and the configuration and quote follow within one business day.

How to enable and test speculative decoding

The gain depends on model, task, load and engine version, so test on the delivered server with your prompts.

1. Check the model card for MTP layers or a published EAGLE-3 head trained for exactly your target model and release.
2. Start with one or two speculative tokens, for example method set to mtp in vLLM’s --speculative-config, or --speculative-algorithm EAGLE3 with the head’s path in SGLang.
3. Run vllm bench serve at fixed concurrency, 1, 2, 4, 8 and up to your peak, with and without the draft, and compare TPOT and total output throughput; a streamed chunk can hold several tokens, so ITL gaps do not compare.
4. Read the acceptance counters on /metrics and the KV cache size vLLM reports at startup.
5. Keep the draft at the concurrency where TPOT improves and total throughput does not fall, and record the engine version and settings.

For a private coding assistant, whose edits repeat large parts of the prompt, add an n-gram run to the comparison. Our engine comparison covers releases and defaults.

A pilot plan with metrics comes out of the technical assessment in our Private AI/ML service, whose price is fixed before work begins. Describe the model and the load you would test with and without a draft in the form below.

What we supply

We build AI servers to order for inference and RAG, with 2 to 8 GPUs per node, burn-in tested and with manufacturer warranty on every component. For latency-bound chat the cards are the H200 NVL with 141 GB of HBM3e at 4.8 TB/s and the RTX PRO 6000 Blackwell Server Edition with 96 GB; our GPU range adds the L40S, the L4 and the RTX PRO 6000 Workstation and Max-Q editions. NVIDIA AI Enterprise licences come on the same EU contract and invoice as the hardware. Deploying the model with a draft, measuring it and running it in production is our Private AI/ML service, with engineering by our partner Vixen.UNO.

FAQ