Quantise your own LLM to NVFP4 or FP8 for the NVIDIA RTX PRO 6000 Blackwell Server Edition

When you need to quantise a model yourself

You quantise a model yourself when no ready-made NVFP4 or FP8 checkpoint exists for it: after a fine-tune, for a niche or regional model, or when you want a different recipe from the published one. For FP8, llm-compressor's dynamic scheme needs no calibration data; for NVFP4, llm-compressor and NVIDIA's Model Optimizer calibrate on a small set of samples. The server needs CPU memory for the 16-bit model above all; GPU memory matters less.

The sample sets range from 20 in llm-compressor's example to 2,048 for one of NVIDIA's own checkpoints. Host memory comes first because llm-compressor's sequential pipeline keeps most weights there while it works through the model layer by layer.

NVIDIA publishes NVFP4 versions of many popular models, listed with sizes and card counts in our NVFP4 checkpoint list for the RTX PRO 6000. Use one of those when it exists. This guide is for the other cases, with the NVIDIA RTX PRO 6000 Blackwell Server Edition as the serving target. What the formats are and how many bits they cost is in our guide to FP8, NVFP4, MXFP4 and INT4; here we deal with producing the checkpoint. Versions and documentation were read on 24 October 2026.

FP8 or NVFP4: which format to produce

The choice follows from the GPU that will serve the model.

vLLM documentation "FP8 W8A8", page dated 24 September 2026 ("FP8 computation is supported on NVIDIA GPUs with compute capability >= 8.9"); llm-compressor documentation on compression schemes and its NVFP4 example (below SM100, "vLLM will not run activation" quantisation). Read 24 October 2026.

FP8 halves the memory of a BF16 model and runs natively on every card we supply for LLM serving. NVFP4 takes about 56 per cent of the memory of FP8 and computes natively only on Blackwell, which among our cards means the RTX PRO Blackwell family and DGX Spark. A model that will run on H200 NVL cards as well as on RTX PRO 6000 cards is easier to operate as one FP8 checkpoint. A model that has to fit fewer RTX PRO 6000 cards, or leave more memory for the KV cache, is the case for NVFP4.

The memory difference is large enough to change the server. Our users per RTX PRO 6000 guide shows it for Llama 3.3 70B: 12 full 8K sessions per card with FP8 weights and an FP8 cache, 34 with NVFP4 weights.

The two tools: llm-compressor and Model Optimizer

llm-compressor documentation (model loading, sequential onloading, NVFP4 example) and release page of 0.13.0 (dated 11 August, no year shown); Model Optimizer README (install with pip install -U nvidia-modelopt[all]) and its hf_ptq example README on GitHub. Read 24 October 2026.

llm-compressor. The NVFP4 example in its documentation quantises all linear layers except the output head with one recipe line: a QuantizationModifier with targets="Linear", scheme="NVFP4" and ignore=["lm_head"]. It then runs oneshot with a calibration dataset and saves with save_compressed=True. NVFP4 stores weights with a local FP8 scale per block of 16 elements and a global scale per tensor; activations get a global scale from calibration and local scales computed at inference time. For FP8, vLLM's documentation recommends the FP8_DYNAMIC scheme on all linear layers with the output head excluded, and states that "we do not need any calibration data for this quantization flow". The 0.13.0 release added weight calibration in parallel across distributed workers and moved disk offloading into the model-loading context.

Model Optimizer. NVIDIA's library ships a post-training quantisation script, hf_ptq.py, called for example with --pyt_ckpt_path for the source model, --qformat fp8, --kv_cache_qformat fp8_cast and --export_path for the output. Its README lists nvfp4 among the formats and recommends nvfp4_mlp_only, nvfp4_experts_only or nvfp4_omlp_only "for higher NVFP4 accuracy". The KV cache format defaults to fp8_cast, with nvfp4 and none as alternatives. The recommended output is a unified Hugging Face checkpoint, which TensorRT-LLM loads directly and vLLM and SGLang load with quantization="modelopt". This is the path NVIDIA uses for its own NVFP4 checkpoints, whose cards name the Model Optimizer version, for example 0.48.0 for Qwen3.8-27B.

Which one to use depends mostly on the serving engine. A vLLM-only deployment can use either. For TensorRT-LLM, or to reproduce one of NVIDIA's recipes on a fine-tuned version of the same model, Model Optimizer is the direct route; its README marks NVFP4 support per model family, for example Llama 3.x and 4, Qwen 3, DeepSeek V3 to V3.2, Kimi K2, GPT-OSS and Nemotron 3, and marks Gemma 3 and Phi-4 as not supported.

Calibration data and sample counts

Calibration runs a small set of prompts through the model to measure the range of the activations, from which the scales are set. The published sample counts span two orders of magnitude:

llm-compressor NVFP4 example; Model Optimizer hf_ptq README (default calib_set); NVIDIA model cards. Read 24 October 2026.

The llm-compressor example gives the more important rule: match the calibration data to what the model will see in use, for a fine-tuned model for example samples from its training data. A legal assistant fine-tuned on German contracts should be calibrated on German contracts, not on an English chat dataset. Keep the calibration prompts at the length the service will run; a model that will answer long-context questions needs long calibration samples, as the Lightning recipe with its 32K-token samples shows.

If the model was fine-tuned with LoRA, merge the adapter first. PEFT's merge_and_unload() merges the adapter weights into the base model and returns a standalone model. PEFT's documentation also notes that on a quantised model "merging re-quantizes the base weight", so merge into the BF16 weights and quantise the merged model, not the other way round. Our guide to GPU memory for fine-tuning covers the training side.

We build servers with RTX PRO 6000 Server Edition cards that serve the quantised model and have the host memory to produce it; send us the base model, its size and the format you need, and our team replies with a configuration for both tasks.

What the server needs: host memory, GPU and disk

The sequential pipeline changes where memory is needed. llm-compressor's documentation describes it: "only a fraction of the model weights are moved to GPU memory for calibration" while "the rest of the weights remain offloaded to CPU or disk", and the calibration dataset is held on the CPU and moved to the GPU one batch at a time. By default the unit moved is one decoder layer; sequential_targets="Linear" moves single linear layers instead, which "will result in lower VRAM usage, but a longer runtime". For models too large for CPU memory, the model-loading guide recommends device_map="auto_offload", which tries the CPU first and falls back to disk, with offload_folder on "a file system with lots of storage"; it warns that "some operations may be slower due to disk offloading".

By our estimate, BF16 weights take two bytes per parameter. Set against the host memory of our reference configurations on the AI servers page, that gives:

our arithmetic, 2 bytes per parameter; Llama 3.3 70B at about 141 GB as in our VRAM guide; host memory of the entry and inference configurations on our AI servers page. An eight-card RTX PRO 6000 Server Edition server, which we quote with 1,536 GB, holds the 397B model in memory.

The operating system, the calibration data and the output checkpoint need room as well, so the fit is tighter than the column suggests for the 128B row on the entry server. On the GPU side, one decoder layer of a 70B dense model is about 1.8 GB in BF16 by our estimate (141 GB over 80 layers), far below the 96 GB of one RTX PRO 6000; the large MoE layers of the biggest models are where sequential_targets="Linear" helps. Model Optimizer has its own levers: --use_seq_device_map uses "up to 80% of each GPU's memory" across the GPUs, and --low_memory_mode compresses weights to low precision before calibration, supported for FP8 and NVFP4 with max calibration only.

Disk space is the third item. The BF16 source, any offload folder and the output must sit on fast local storage at the same time; for a 397B model that is well over a terabyte. Our guide to NVMe storage for an LLM server sizes the drives, and our guide to host memory for a GPU server explains why we quote twice the GPU memory as a starting point.

A procedure that holds up in production

1. Fix the source. Pin the exact BF16 checkpoint and, for a fine-tune, the merged model. Record the tool version, here llm-compressor 0.13.0 or the Model Optimizer version, in the output folder.
2. Choose the format by serving GPU. FP8 for a mixed fleet with H200 NVL or L40S cards; NVFP4 for RTX PRO 6000 Server Edition servers where memory per card is the limit.
3. Start with the standard recipe. Output head excluded, all other linear layers quantised; for mixture-of-experts models, try an experts-only recipe as NVIDIA does for several of its own checkpoints.
4. Calibrate on your own data. A few hundred samples from the deployment domain at realistic lengths.
5. Load it in the serving engine. In vLLM, read the start-up log for fallback warnings and the KV cache size, as our guide to vLLM KV cache size and maximum concurrency explains. On RTX PRO 6000 cards, a W4A16 or Marlin line means the native NVFP4 path was not taken.
6. Evaluate against BF16. vLLM's FP8 documentation evaluates with lm_eval, for example on 250 GSM8K samples, and notes that quantised models can be sensitive to the beginning-of-sequence token. Add your own task set, as in our guide to LLM evaluation before model upgrades.
7. Keep the BF16 or FP8 version. If the NVFP4 model fails a task, the fallback is ready without a second quantisation run.

Fine-tuned models need every step: the fine-tune changes the weight ranges, so a published NVFP4 recipe for the base model is only a starting point.

Which server to quantise on

Quantisation is a batch job and does not need its own server. The RTX PRO 6000 Server Edition server that will serve the model can produce it, as long as host memory and disk fit the source model. By the table above, the two-card entry configuration with 384 GB covers dense models up to about 128B in BF16, and the four-card inference node with 768 GB adds headroom. For the largest mixture-of-experts models, the choice is disk offload with a longer run, or a server with more memory, such as an eight-card RTX PRO 6000 Server Edition server with the 1,536 GB we quote for it. Our pages on the four-card RTX PRO 6000 Server Edition server and on CPU, RAM and storage for a four-GPU server give the component detail.

The quantised checkpoint is then served by the same engine as any published one. A fine-tuned 70B model in NVFP4, at about the size of NVIDIA's 42.7 GB Llama 3.3 70B checkpoint if the same recipe is used, runs on one card with room for 34 full 8K sessions with an FP8 cache by our users-per-card rule.

Our engineering partner Vixen.UNO deploys open models with vLLM on the servers we supply as part of our Private AI/ML service; describe the model and the format you need, and we check the tool and engine versions for it before you order.

FAQ