DGX Spark, one RTX PRO 6000 or a server with H200 NVL: which to start a first AI project on

Four questions that decide the hardware

Most first AI projects arrive at the same shortlist: an NVIDIA DGX Spark on a desk, one RTX PRO 6000 Blackwell in a workstation or a server, or a server with one or two H200 NVL cards. Four questions about the project decide between them.

How big is the model, and at what precision? The parameter count alone does not give the memory requirement. The same 70 billion parameters occupy 141.1 GB at BF16, about 72.7 GB at FP8 and about 42.7 GB at NVFP4, so the choice of precision, often not yet made at this stage, changes the hardware by a factor of more than three.

How many people use it at once? One developer iterating on prompts is a batch of one, which is also how NVIDIA measured its October 2025 Spark inference figures. Twenty people using an assistant are twenty concurrent sequences, each holding its own key-value cache, and on a single card the cache, rather than the weights, decides how many of them fit.

Where will the hardware be installed? A DGX Spark is 150 mm square, 50.5 mm high and 1.2 kg, operates ideally at 5 to 30 °C and draws at most 233.2 W by NVIDIA’s declaration, from an ordinary socket. An H200 NVL is a passively cooled dual-slot card that needs a named fan kit, a named air duct, a specific riser and the right 16-pin cable part number for its slot, in a server on the vendor’s supported list. The first is a desk purchase; the second is a server project for the data-centre team.

What happens after the pilot? Each option grows differently, and two of the three have a limit built into the hardware, so the upgrade path belongs in the first decision.

The three options side by side

Everything below comes from NVIDIA’s product pages, datasheets, user guides and product briefs, plus the OEM guides where an OEM overrides NVIDIA. The bandwidth row matters most, because it sets the speed of token generation.

NVIDIA DGX Spark product page and user guide; RTX PRO 6000 datasheets and the Blackwell PRO architecture whitepaper; H200 NVL product brief PB-12128-001 and datasheet; Dell, HPE and Lenovo product guides. September 2026.

The RTX PRO 6000 bandwidth most often quoted, 1,792 GB/s, is the figure for the Workstation Edition and the Max-Q. The Server Edition, the only edition passively cooled for a chassis and the only one on NVIDIA’s vGPU list, runs at 1,597 GB/s, about 11 per cent lower. Sizing a rack deployment on 1,792 GB/s therefore overstates its token generation by roughly 12 per cent. Our edition comparison covers the rest.

The entitlement differs too. H200 NVL is sold with a five-year NVIDIA AI Enterprise subscription, licensed per GPU and activated against the GPU serial number, so record the serials before the cards are racked. DGX Spark carries no such entitlement: NVIDIA states it exists only if purchased, requested as an evaluation or granted by certificate, and the free path is NVIDIA’s 90-day trial.

Memory capacity, bandwidth and tokens per second

Weights are parameters multiplied by bytes per parameter: two at BF16 or FP16, one at FP8, and at NVFP4 half a byte plus one FP8 scale per 16 weights, with the embeddings and output layer usually kept in BF16 in both FP8 and NVFP4 checkpoints. NVIDIA’s NVFP4 checkpoint of Llama 3.3 70B is 42.7 GB, about 0.61 bytes per parameter. MXFP4 checkpoints carry the same kind of overhead, a shared scale per block of weights and some layers kept at higher precision. Whether a model loads at all follows from this arithmetic.

Weights only. BF16 from the parameter counts of 8.03B, 32.76B and 70.55B; FP8 from the published FP8 checkpoints (NVIDIA’s for the Llama models, Qwen’s for Qwen3-32B), which keep the embeddings and output layer in BF16; NVFP4 from NVIDIA’s checkpoints of Qwen3-32B and Llama 3.3 70B (the same architecture as 3.1 70B), and for the 8B by our arithmetic, with the linear layers at 0.5625 bytes per weight and the embeddings and output layer at 2 bytes. Cache, activations and framework overhead come on top; the VRAM guide has the formula.

A 70B model at BF16 needs 141.1 GB, which is 131.4 GiB. It loads on an H200 NVL, whose driver reports 140.4 GiB, with about 9 GiB to spare and no room for users, and it fits neither the 96 GB RTX PRO 6000 nor the Spark’s 128 GB. NVIDIA’s NIM documentation listed a 70B model on the RTX PRO 6000 Server Edition at FP8 from one card and at BF16 only on four or eight cards; the current NIM 2 support matrix (October 2026) lists its profiles without card counts per GPU.

The Spark’s 128 GB comes with two caveats for sizing. It is unified system memory shared with a 20-core Arm CPU, DGX OS and the desktop session, not a framebuffer: nvidia-smi prints “Memory-Usage: Not Supported” on this platform, and NVIDIA warns that cudaMemGetInfo under-reports what can be allocated. NVIDIA publishes no usable-capacity figure. Its single-Spark vLLM launch script, linked from the playbook on build.nvidia.com, sets 0.7 of memory; its vLLM playbook on GitHub sets up to 0.8 on a single Spark and 0.8 or 0.9 on two, and its playbooks give TensorRT-LLM’s KV cache 0.80 to 0.9 of the memory still free after the weights load. Applied to the whole pool, the 0.8 and 0.9 settings put a working set at roughly 102 to 115 GB, and 0.7 at about 90 GB. That is our arithmetic from NVIDIA’s playbook settings, not a specification.

The second caveat concerns NVIDIA’s headline figure. NVIDIA says one Spark with 128 GB can “test, validate, and inference with AI models up to 200 billion parameters”. That holds at 4-bit, with little margin. At 4.25 to 4.5 bits per weight with the block scales, 200 billion parameters take 106 to 112 GB, leaving 16 to 22 GB of the 128 GB for cache, activations and the operating system. At FP8 the same system reaches roughly 110 billion parameters and at BF16 roughly 55 billion. Reading “200B on a desk” as a BF16 statement mis-sizes the project by nearly four times.

Once a model fits, its speed depends on memory bandwidth. Token generation re-reads every weight from memory for every token it emits, so the per-user rate at low batch is bandwidth divided by bytes of weights, not FLOPS.

The ceilings are our arithmetic, bandwidth divided by the 70.6 GB of weights each token reads; they are upper bounds that real systems do not reach, not measurements. At NVFP4 each token reads about 40.6 GB, the 4-bit layers plus the output layer in BF16, and every ceiling rises by about three quarters.

Silent reading of non-fiction averages 238 words per minute, which at roughly 0.75 words per token is about 5.3 tokens per second. A dense 70B model at FP8 on a Spark, with a ceiling of 3.9 tokens per second, cannot reach reading speed even in theory. Measured rates are lower than the ceiling. NVIDIA’s Spark measurements put Llama 3.1 8B at NVFP4 at 38.65 tokens per second against a ceiling of about 55, and Qwen3 14B at 22.71 against about 30. By our arithmetic from the bytes each token reads, both run at roughly 70 to 75 per cent of the ceiling, which puts the dense 70B nearer three tokens per second than four. NVIDIA’s October 2025 Spark inference figures are all 4-bit at batch size one; its March 2026 test in FP8 on vLLM stops at four tasks at once.

FLOPS headlines mislead for the same reason, and both headline figures in this comparison are sparsity figures by the vendor’s own documentation. NVIDIA’s DGX Spark hardware guide gives “up to 1 PFLOP (petaFLOP) at FP4 precision with sparsity”, and we found no dense, FP8 or BF16 figure for GB10 from NVIDIA. The RTX PRO 6000 Server Edition page prints 4 PFLOPS FP4, 2 PFLOPS FP8 and 1 PFLOP FP16 with no basis stated. The architecture whitepaper gives the Workstation Edition as dense and sparse pairs, 2,015.2 against 4,030.4 TFLOPS at FP4, footnoted “Effective TOPS / TFLOPS using the Sparsity Feature”, and its sparse values match the published Server Edition figures, so the dense rates are half the headline. Every H200 NVL tensor figure from TF32 to INT8 also carries “With sparsity”: 3,341 TFLOPS at FP8 and 1,671 TFLOPS at BF16, and the page lists no FP4 rate. By NVIDIA’s own figures none of the three reaches 4 PFLOPS at FP8, even with sparsity, since the RTX PRO 6000’s 4 PFLOPS is its FP4 figure. The FP8 figure close to 4 PFLOPS on NVIDIA’s H200 page, 3,958 TFLOPS with sparsity, belongs to the H200 SXM module, not to the H200 NVL card. A server reaches 4 PFLOPS of FP8 on NVIDIA’s per-card figures with two RTX PRO 6000 Server Edition cards counted with sparsity, or with four counted dense, as in the four-card inference node on our AI servers page; our GPU TFLOPS comparison works through the 4 PFLOPS FP8 server. Standard LLM serving does not use 2:4 structured sparsity, and none of these numbers predicts tokens per second at batch one, which depends on bandwidth.

Concurrent users and the KV cache on one 96 GB card

Serving twenty employees takes a different machine from serving one developer, because of the key-value cache. Every active conversation holds keys and values for every token it has seen, sized as 2 × layers × key-value heads × head dimension × bytes per element × sequence length. Grouped-query attention is what makes single-card serving possible: Llama 3.1 70B has 80 layers, 64 query heads and 8 key-value heads, so it costs 320 KiB per token at FP16 instead of the 2,560 KiB it would cost without GQA.

The table applies this to one 96 GB card: 90 per cent of the 95.6 GiB the driver reports, which was vLLM’s default up to release 0.19 (0.92 since 0.20.0 of April 2026) and is the figure we plan with, less about 3 GiB for activations and CUDA graphs.

Maximum concurrent sequences each holding a full-length context, with an FP16 cache / an FP8 cache. The counts are a floor rather than a ceiling, since PagedAttention allocates blocks on demand, but they decide whether a request is admitted or queued. FP8 weights are the published FP8 checkpoints, which keep the embeddings and output layer in BF16.

In the 70B rows, FP8 weights leave 15.3 GiB of cache, six full 8K sessions or one at 32K; NVFP4 weights leave 43.2 GiB and seventeen sessions. The choice of quantisation adds more concurrency on one card than a second card running a second copy would. Context length reduces concurrency faster than model size does, so ask first how long the conversations are, and only then how many users there will be.

Tell us your model and how long the conversations will be, and we calculate this table for your case and the number of cards it implies.

There is a gap in the published data. NVIDIA’s absolute LLM figures for the card are maximum-throughput numbers for the Server Edition in its TensorRT-LLM performance overview, measured with its own engine; for example, Qwen3-30B-A3B in FP4 on one GPU reaches 9,938 output tokens per second per GPU with 1,000-token input and output. We found no per-user figure from NVIDIA. The MLCommons-verified results for the card in MLPerf® Inference v6.0 of April 2026 are throughputs of multi-GPU servers, not per-user rates. NVIDIA’s inference performance page lists this GPU with vision, image- and video-generation models only, and the other LLM claims NVIDIA publishes are relative, against an L40S baseline with no precision or batch size stated. For your model and context, the reliable figure comes from your own card, where vLLM prints GPU KV cache size and Maximum concurrency at startup.

For example, entry 6.0-0004: MLPerf® Inference v6.0 Datacenter, Closed division, Available category, Llama 2 70B (llama2-70b-99), Interactive scenario, eight RTX PRO 6000 Blackwell Server Edition cards, 6,238.12 tokens per second with FP4 weights, retrieved from mlcommons.org on 24 September 2026. Result verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The interactive scenario of MLPerf® Inference v6.0 for Llama 2 70B, for comparison, requires the first token within 450 ms and at most 40 ms per token after it, 25 tokens per second per user, roughly five times average silent reading speed. The concurrency article works it through.

If the users arrive as virtual machines rather than API clients, the edition question returns. vGPU runs on the RTX PRO 6000 Server Edition from vGPU 19.0; the Workstation and Max-Q editions are not on NVIDIA’s supported list at all. MIG splits any of the three into four instances of 24 GB, two of 48 GB or one of 96 GB, and for compute profiles the ceiling is 12 time-sliced virtual machines of 8 GB. Our MIG and vGPU guide has the tables.

Limits of each option, and how to plan around them

Each option has its typical failure, and each has a way around it.

DGX Spark is limited by memory bandwidth. It loads models that then run too slowly for interactive use: a dense 70B model in FP8 stays below 4 tokens per second. Its 273 GB/s is 6.6 times below an RTX PRO 6000 Workstation card and 17.6 times below an H200 NVL. NVIDIA positions it as a machine to prototype, fine-tune and validate on before migrating the work to a data centre. So buy it for the people who write the code, and decide the serving hardware once the prototype has settled the model, the precision and the context length. NVIDIA gives 5 to 30 °C as its ideal operating temperature, and GPUDirect RDMA is not supported because of the unified memory model. DGX Spark is strong at fine-tuning. NVIDIA documents a full fine-tune at 3B, LoRA at 8B and QLoRA at 70B on one DGX Spark, measured at 13,519.54, 6,969.59 and 759.79 peak tokens per second. The benchmark article has the table.

With the RTX PRO 6000, problems come from the wrong edition and from long contexts. A Workstation Edition ordered for a rack is a 600 W card with double flow-through cooling that no server will accept, and it has no vGPU; a Server Edition ordered for a workstation is a passive card with no fan. Its 96 GB is comfortable for a 70B model at NVFP4 and tight at FP8, and what fills it is the cache of long conversations rather than the parameter count. One card is also one failure domain, and neither MIG nor vGPU changes that, since both partition a single physical GPU. Choose the edition for where the card will be installed, quantise before buying a second card, and if the service matters, put the second card in a second host.

With the H200 NVL, the server chassis decides. OEM support is defined per configuration, not per server model, and each requirement can block a retrofit. Dell lists the card only on 17th-generation systems, two maximum in an R770 or R7725, and not in the 15G or 16G tables, so a server you already own may not be on the list. Lenovo supports it in the SR650a V4 and states it is “not supported on SR650 V4”, two machine types of what reads like one 2U family. Beyond the model number there are named parts: Dell’s HPR Platinum fans, its riser kits and a different power-cable part number for each slot; Lenovo’s 600 W fan module, its front double-width air duct, slots 21 and 23 only, and an ambient cap of 30 °C. The drive backplane alone can rule the GPU out, because it sets the air impedance in front of the card. The 16-pin auxiliary cable must also be strapped for the 451 to 600 W class; otherwise the card does not boot at all rather than running throttled, and the maintenance window is lost. Validate machine type, backplane, riser, fan kit, air duct and cable part number before ordering: we found no airflow figure from NVIDIA or the OEMs to check a chassis against, so the supported-configuration list is the thermal specification. The retrofit checklist is the long version.

If the server is already in your rack, write to us with its exact machine type and configuration, and we check its power and airflow before you order the H200 NVL rather than after.

What the upgrade path looks like from each

DGX Spark grows by adding units, the RTX PRO 6000 by adding cards over PCIe, and the H200 NVL by adding bridged pairs or quads.

NVIDIA’s clustering documentation supports up to three systems connected directly through cables and up to four through a switch, and NVIDIA has announced support for clusters of up to eight Sparks in manual configuration (NVIDIA blog, 21 October 2026). NVIDIA’s product page now gives up to 400 billion parameters across two units and 700 billion across four (its hardware guide still says 405 billion for two), and both hold only at 4-bit. At 4.5 bits per weight, 405 billion parameters are about 228 GB against the 256 GB of two units, and 700 billion about 394 GB against the 512 GB of four. NVIDIA’s validated-model table flags exactly one entry as multi-node, Qwen3-235B-A22B at NVFP4, which NVIDIA measured in October 2025 at 11.73 tokens per second across two Sparks. Extra units add memory capacity first, and speed per request at most in part. The clustering article covers the cabling.

The RTX PRO 6000 has no NVLink on any edition, workstation or server. Lenovo’s product guide states this directly, NVIDIA’s specification tables omit it, and the reference architecture describes GPU placement purely in PCIe terms. Against a bridged card the gap is roughly seven times, 128 GB/s against 900 GB/s. That sets the order of steps. First quantise, because FP8 or NVFP4 weights on one card beat BF16 split across two cards with no fast link. Then add independent replicas, one model instance per card behind a load balancer, which creates no inter-GPU traffic and doubles availability. Split a model only when it does not fit one card, or when long conversations need more cache than one copy per card leaves, and then prefer pipeline parallelism to tensor parallelism, which is vLLM’s guidance for nodes without NVLink.

The H200 NVL path is the one designed for growth, with a caveat about geometry. Each card carries exactly one bridge, the biggest change from H100 NVL and its three, so an existing bridge set does not carry over. The cards must be physically adjacent. NVIDIA’s product brief says bridged cards should be within the same CPU domain and lists bridging across CPUs as allowed but not recommended; in a server whose risers split GPUs two and two across sockets, a 4-way bridge would span both sockets, which NVIDIA does not recommend. Plan the slot map before the order. Our head-to-head comparison takes those two cards further. Our DGX Spark vs H200 NVL comparison covers the step from development to production.

If the choice is still unclear, this sequence is a reasonable default. Start on a Spark while the model, the precision and the context length are still changing. Move to one RTX PRO 6000 when a team uses the service daily and generation speed becomes the limit. When several teams share one service, a server with four RTX PRO 6000 Server Edition cards runs four independent copies: with a 70B model in NVFP4 that is 68 full 8K conversations with an FP16 cache, or 136 with an FP8 cache. Move to H200 NVL when latency, long context or a model that must span GPUs makes the bridge and the 4.8 TB/s the point of the purchase.

What we supply

We supply all three on one EU contract and invoice: DGX Spark, the RTX PRO 6000 Blackwell in all three editions with the workstation or server around it, and the H200 NVL with its NVLink bridges. The cards go into AI servers built to order or, after a compatibility check of platform, power and cooling, into a server you already own. Quantities and lead time are confirmed in writing with the quote.

FAQ