Apache Spark on GPU servers with the RAPIDS Accelerator: requirements, sizing and cards

What the plugin accelerates and when GPU nodes pay off

Apache Spark runs on GPUs through NVIDIA's cuDF plugin for Apache Spark, which NVIDIA's download page says "was formerly known as the RAPIDS Accelerator for Apache Spark". It executes Spark SQL and DataFrame operations on the GPU with the libcudf library, needs no code change and falls back to the CPU for any operation it does not support. GPU nodes pay off for long ETL and SQL jobs with joins, sorts and aggregations over large Parquet or ORC data; they do not help RDD code, short queries or jobs dominated by row-based user functions.

NVIDIA's FAQ puts numbers on that: "Generally, for complicated queries where all the processing can run on the GPU we see between 3x and 7x speedup." It lists aggregations over many distinct keys, joins and sorts, Parquet and ORC writes and CSV reads as operations that suit the GPU. On the other side, "the GPU typically takes about 2 to 3 seconds to initialize when an executor first starts", and "Other areas like GraphX or RDDs aren't accelerated", nor are APIs that take Row objects or Scala and Java functions.

The artifact names stay unchanged for now, so the jars are still called rapids-4-spark, and most teams still search for "Spark RAPIDS". This guide is for the data-engineering and platform team that runs Spark on its own servers and wants to know whether GPU nodes speed up its ETL, and what to buy. For single-node pandas and scikit-learn work on one card, our guide to a RAPIDS data science workstation is the better start.

Spark RAPIDS requirements as of October 2026

NVIDIA cuDF plugin for Apache Spark download page, release 26.08.1, read on 25 October 2026. Databricks 14.3 and 17.3 ML LTS and Dataproc 2.1 to 2.3 are also listed.

The driver line needs care on new cards. The R525 floor is the plugin's; the card sets its own. NVIDIA's product brief for the RTX PRO 6000 Blackwell Server Edition names R575 and CUDA 12.9 as minimums, so a Blackwell node runs a current production or long-term driver branch. The RTX PRO Blackwell Server Edition cards, the L40S and the H200 NVL fall under the listed architectures but are not among the tested models. NVIDIA's own user guide cites an AWS result on RTX PRO 4500 Blackwell GPUs: on eight-node clusters running 103 queries derived from TPC-DS at 3 TB, the GPU nodes finished in 4.7 minutes against 16.8 to 17.4 minutes for the CPU nodes. NVIDIA notes that these NDS results "do not comply with the TPC-DS Specification" and are not comparable to published TPC-DS results.

Start with the qualification tool

Before buying a GPU node, run NVIDIA's qualification tool on event logs from the existing CPU cluster. It parses the logs, predicts GPU duration and speedup per SQL query and per application with an XGBoost model called QualX, lists operators that would fall back to the CPU, and proposes Spark settings for the GPU. NVIDIA cautions that a high rank is no assurance that an application will gain the most.

The tool installs from PyPI as spark-rapids-user-tools and needs Python 3.10 to 3.12 and Java 8 or later. For an on-premise cluster the command is spark_rapids qualification --platform onprem with --eventlogs pointing at the log directory. A recommended cluster shape is produced only for cloud platforms, so for own servers the shape comes from the GPU count rules below. The qualification page also reports Iceberg write operators such as MergeRows and WriteDelta as unsupported on the GPU, which matters for lakehouse jobs.

The tool's estimate assumes a dedicated cluster. A useful first test is one GPU server running the top-ranked jobs next to the CPU cluster, with spark.rapids.sql.explain=NOT_ON_GPU set so that the logs show every operation that stayed on the CPU. We size GPU nodes from that report; send us its summary and your Spark version, and our team replies with a configuration.

One executor per GPU sets the cluster shape

The tuning guide states: "The cuDF for Apache Spark plugin only supports a one-to-one mapping between GPUs and executors." Neither several executors per GPU nor several GPUs per executor are supported, so a server with eight cards runs eight GPU executors, and a server with four runs four.

NVIDIA cuDF plugin for Apache Spark on-premise guide, tuning guide and FAQ, read on 25 October 2026. Full property names: spark.executor.resource.gpu.amount, spark.task.resource.gpu.amount, spark.rapids.sql.concurrentGpuTasks, spark.sql.files.maxPartitionBytes.

The FAQ adds that "the GPU typically benefits from having 2 tasks run in parallel on it at a time", and the tuning guide's example gives each executor 6 cores on a 24-core node with four GPUs. That matches the minimum of six physical cores per GPU in NVIDIA's guide for NVIDIA-Certified Systems, so an eight-card server needs at least 48 cores, and more where CPU-side stages remain. The Spark driver needs no GPU: "there is no benefit to having one available on the driver", according to the FAQ.

MIG is possible but not the plugin's preferred mode. The FAQ answers that it works when the cluster manager isolates each MIG instance per executor, that it "isn't recommended" in general, and that it "can be advantageous" on shared clusters. On YARN, MIG needs extra setup depending on the YARN version.

GPU memory, pinned memory and spill

GPU memory sets how much data each task holds and how many tasks run at once. The tuning guide recommends a columnar batch size of min(2GiB - 1 byte, (gpu_memory - 1 GiB) / gpu_concurrency / 4). By our arithmetic with the listed memory as the input, at two concurrent tasks every card in this article reaches the 2 GiB cap: a 24 GB L4 gives (24 − 1) ÷ 2 ÷ 4 = 2.9 GiB before the cap. At four concurrent tasks the L4 drops to 1.4 GiB and a 32 GB RTX PRO 4500 Server Edition to 1.9 GiB, while a 96 GB card stays at the cap. A larger card therefore mainly allows more concurrency without smaller batches, which matches the guide's note that larger GPUs handle more concurrency.

Host memory matters as much. The pinned pool, spark.rapids.memory.pinnedPool.size, is set per executor and should ideally hold the input partitions of the concurrent tasks; four executors at 4 GB each take 16 GB per node, and on YARN the executor memory overhead must cover it. The host spill store defaults to the pinned pool size plus 1 GB, and the guide advises a larger one when a job "spills frequently". Spill to a slow disk "can be especially expensive", which is an argument for NVMe drives as Spark local directories.

The plugin reserves 640 MiB of GPU memory for the system by default and builds an RMM memory pool; on a card shared with libraries that do not use RMM, pooling has to be turned off. For a node that also serves AI models, MIG or separate cards keep the two apart.

Shuffle and the network between nodes

The RAPIDS Shuffle Manager has two modes. MULTITHREADED, the default, is "similar to the built-in Spark shuffle" with more threads for compression and decompression, 20 by default, and stays compatible with Spark's external shuffle service. UCX mode, set with spark.rapids.shuffle.mode=UCX, keeps shuffle blocks in a spillable GPU, host and disk cache and moves them with Unified Communication X: GPU to GPU over cuda_ipc inside a node, and over InfiniBand or RoCE with GPUDirect RDMA between nodes.

UCX mode needs UCX 1.19.1 or later on hosts and in containers. It requires the external shuffle service to be disabled, and with it dynamic allocation, and NVIDIA's testing on Databricks ran with autoscaling off. The shuffle manager class must match the Spark version, for example com.nvidia.spark.rapids.spark353.RapidsShuffleManager for Spark 3.5.3.

NVIDIA gives no minimum link speed, only that the manager "relies on fast connections between executors". We size the network cards from the shuffle volume in your current Spark history: a cluster of two or three GPU nodes with heavy joins gets RDMA-capable NICs, a single node or a light shuffle does not. Our guide to GPU clusters without InfiniBand explains the RoCE option.

Which of our cards fit Spark

Each Spark executor uses one card, and only the UCX shuffle moves data between cards, so the choice is GPU count, memory per card and power, not the largest card.

NVIDIA product pages and datasheets as listed in our card guides, read in September and October 2026; "for Spark" is our assessment from the plugin's one-GPU-per-executor model.

The RTX PRO 4500 Server Edition is the card we suggest for a dedicated ETL node: the AWS result NVIDIA cites ran on RTX PRO 4500 Blackwell GPUs, the card is single-slot and passive, and eight of them give eight executors for about 2.2 kW at the wall by our estimate, as our eight-card RTX PRO 4500 Server Edition server works out. The L4 is the tested card and the lowest-power option. The RTX PRO 6000 Server Edition fits when the same node runs AI inference or training during the day and Spark at night. We supply all five cards in servers built to order; tell us your cluster manager and how many CPU nodes run your jobs today, and we propose the number of GPU nodes.

Scheduling GPUs on YARN and Kubernetes

On YARN, GPU scheduling needs YARN 2.10 or 3.1.1 or later, with 3.1.3 or later preferred for Docker support. NVIDIA recommends Docker with cgroup isolation; without isolation, each GPU is set to exclusive-process mode with nvidia-smi -c EXCLUSIVE_PROCESS. Executors find their GPU through Spark's getGpusResources.sh discovery script, installed on every node. In standalone mode, workers advertise their GPUs through SPARK_WORKER_OPTS, and the master needs no GPU.

On Kubernetes, executor pods request nvidia.com/gpu resources from the NVIDIA device plugin, which the GPU Operator installs along with the driver and toolkit, as our guide to Kubernetes GPU sharing describes. NVIDIA's FAQ suggests raising spark.kubernetes.executor.missingPodDetectDelta, for example to 120000 ms, if executor pods time out.

Spark GPU cluster sizing: configurations per node

These follow NVIDIA's configuration guide for NVIDIA-Certified Systems: at least six cores per GPU, system memory of at least twice the GPU memory and two sockets. Spark executors also need their own heap, so we add the executor memory your CPU jobs use today.

Host minimums are our arithmetic from NVIDIA's certified-systems rules; wall power is our estimate from our sibling guides (cards plus processors plus 15 per cent); a measurement on the delivered system confirms it.

The first three stay below the 2.94 kW we plan per 230 V 16 A feed. Eight RTX PRO 4500 Server Edition cards need a 3U to 5U chassis: Supermicro lists at most six in its 2U servers. We add NVMe drives for the Spark local directories and network ports sized from your shuffle volume. Our guide to CPU, RAM and storage for a 4-GPU server covers processor and memory channels.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with L4, RTX PRO 4500 Server Edition, L40S, RTX PRO 6000 Server Edition or H200 NVL cards, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, NVIDIA driver, CUDA and a container runtime on request. Quantities and lead time are confirmed in writing with the quote.

FAQ