NVIDIA DGX Spark benchmarks: tokens per second, memory bandwidth and TDP

DGX Spark tokens per second by model

With a single request, DGX Spark generates between 2.7 tokens per second on a dense 70B model and about 60 on gpt-oss-120b, depending on the model, the engine and its build; its 273 GB/s memory bandwidth sets that pace, and its GB10 chip has a 140 W TDP within a 240 W external power supply. With many concurrent requests, aggregate throughput on gpt-oss-120b reaches 862.8 tokens per second. The table collects the published figures this article discusses; the sections below explain the spread. Memory, bandwidth, TDP and power figures are in our DGX Spark specs.

Generation speed in tokens per second. 1 request: one user; peak, aggregate: all concurrent requests together, at the number of requests shown. Figures as published by LMSYS and NVIDIA (October 2025), ServeTheHome (launch review) and Dendro Logic (vLLM 26.03, April 2026; vLLM built from source on a partner GB10 system, May 2026). Each run used its own engine, build and quantisation format, so rows from different engines do not compare like for like.

What is inside

DGX Spark is built on the GB10 Grace Blackwell chip, which combines twenty Arm cores (10× Cortex-X925 + 10× Cortex-A725) with a Blackwell GPU that has 6,144 CUDA cores and fifth-generation Tensor Cores. CPU and GPU share coherent memory over NVLink-C2C, so there is no separate VRAM and no copy over a PCIe bus between CPU and GPU memory.

On the 128 GB configuration the system shows about 119 to 122 GiB. The firmware and GPU driver hold back the rest, including a display reserve of 2 GB by default, adjustable to 4 GB since July 2026; since NVIDIA’s October 2026 release, a Spark without a connected display can lower it automatically to the minimum of 512 MB. In September 2026 DevelopersIO and Unsloth reported about 121 GiB per Spark.

Memory bandwidth limits generation speed

An LLM works in two phases: it reads your prompt (prefill), then emits the answer token by token (decode). Prefill is compute-bound. Decode is memory-bound, because every new token requires one pass over the weights it uses: the whole of a dense model, but only the shared layers and active experts of a mixture of experts.

Spark has compute to spare, but its memory delivers 273 GB/s, while an H200 NVL delivers 4.8 TB/s, 17.6 times as much, and that gap explains nearly all reports of slow generation. The spread on gpt-oss-120b looks like this:

The results for the same model differ by a factor of 3.8, because the runs used different engines, builds, quantisation formats and test dates. In May 2026 Dendro Logic reported 60.4 tok/s for one gpt-oss-120b request with vLLM built from source on a partner GB10 system, so newer engine builds have raised the single-request figure since launch. Prefill at 1,725 tok/s is fast, and generation at 55 tok/s is well above reading speed. That is enough for chat but leaves little headroom for batch document pipelines.

Single-user speed with a 70B model

Most buyers want the 128 GB configuration for 70B-class models that do not fit on an ordinary GPU. LMSYS measured Llama 3.1 70B in FP8 at 2.7 tokens per second for one user, so a 300-word answer takes about two and a half minutes to generate. The model fits and runs, and that speed is acceptable for overnight batch work but too slow for interactive use.

A dense 70B in FP8 weighs ~70 GB, and every token requires reading all of it. At 273 GB/s that puts the ceiling below four tokens per second, and the measured 2.7 is within it.

For a 70B model at reading speed, the usual next step is a GPU server with RTX PRO 6000 Server Edition cards; we supply both. Tell us the model and whether it is for interactive use or overnight batches, and we say which fits.

Throughput with many concurrent requests

All of the above assumes one user. With many concurrent requests, the weights are read once per batch, so aggregate throughput rises sharply. Dendro Logic’s vLLM measurements (vLLM 26.03, April 2026):

tokens per second, aggregate

On Nemotron Super 49B, aggregate throughput rises 120-fold, from 5.8 tokens per second with one request to 695.1 with 256. Spark is far better at serving many requests at once than at serving one user who wants a large dense model at interactive speed. By the llama.cpp maintainers’ results, one 128 GB Spark keeps about 16 simultaneous gpt-oss-120b requests with 4,096-token prompts at 10 tokens per second or more each, enough for a team that chats with an assistant (how to size a team on one Spark), while coding agents with long prompts are limited to a handful of parallel tasks.

Send us the number of developers and whether they chat with an assistant or run coding agents, and we will tell you whether one Spark is enough.

Power draw, TDP and the 1 PFLOP figure

NVIDIA’s hardware documentation (updated 21 October 2026) states that the “GB10 SOC Thermal Design Power (TDP) is 140W”, a budget shared by the 20 Arm cores and the GPU on the chip, and that “100W is available for other system components (ConnectX-7, Wi-Fi, SSD, USB-C, etc.)”; together they make up the 240 W of the external supply. In a forum thread titled “Only getting half the advertised performance and capping at 100W?”, NVIDIA staff put the GPU part of the chip at about 120 W at most and said the 240 W supply is rated for peak draw. NVIDIA declares 38.0 W idle and 233.2 W maximum at 230 V. The advertised 1 PFLOP is an FP4 figure with sparsity.

NVIDIA’s January 2026 software update added hot-plug support for the ConnectX-7 adapter, which NVIDIA says saves up to 18 W while the network ports are unused. Tom’s Hardware, measuring at the USB-C input, saw idle power fall from about 37 W to between 22 and 25 W after that update.

Two nodes

Two Sparks connect directly with a QSFP cable and share a model across 2 × 128 GB, split by tensor or pipeline parallelism over the link. NVIDIA’s hardware documentation (updated 21 October 2026) still gives models up to 405B parameters this way, while the product page, as of October 2026, gives up to 400B on two 128 GB Sparks and up to 700B on four. Splitting the model by layers does little for single-user speed: in llama.cpp tests across two units published by DevelopersIO, prompt processing gained up to about 25 per cent and generation speed hardly changed in most runs. Two units are linked mainly to fit a larger model; tensor parallelism can win back some speed per request, and batch serving is the exception.

Who it suits, and who it does not

DGX Spark suits CUDA development on the same NVIDIA software stack as data-centre systems, built for Arm, without monthly cloud costs, work on data that must stay on the premises, serving a team, and QLoRA fine-tuning up to 70B when long training runs are acceptable. It does not suit fast interactive chat on the largest model available, Windows-only toolchains (DGX OS is aarch64 Linux), gaming or classic rendering.

Where one Spark is too slow for the number of users, DGX Station vs DGX Spark compares the next steps, including a workstation with one to four RTX PRO 6000 cards.

What we supply

We supply the NVIDIA DGX Spark Founders Edition (128 GB, 4 TB) across the EU on one contract and invoice, with manufacturer warranty; quantities and lead time are confirmed in writing with the quote. With the model, the context length and the number of people who will use it at once, we can tell you whether Spark is fast enough or whether a GPU server built to order fits better. For two to four linked units, NVIDIA Sync’s Cluster Assistant sets up the network between them; larger clusters, up to eight Sparks by NVIDIA’s blog of 21 October 2026, are configured manually. NVIDIA AI Enterprise for DGX Spark is a separate licence that we can add to the same quote.

FAQ