AI surrogate models for CAE with NVIDIA PhysicsNeMo: GPUs, memory per sample and storage

What a surrogate model changes in a CAE workflow

An AI surrogate model is a neural network trained on a department's own simulation results that predicts the fields of a new design, such as surface pressure, wall shear stress or deformation, in seconds instead of a full solver run. NVIDIA PhysicsNeMo is an open-source framework for this work: it supplies the models, the data pipelines and the multi-GPU training for aerodynamics, crash and structural cases. Training needs a GPU server with large memory per card and fast storage, typically four or eight NVIDIA H200 NVL or NVIDIA RTX PRO 6000 Blackwell Server Edition cards; using the trained model needs one card.

The solver stays the reference. A surrogate ranks hundreds of variants, and the promising ones go to a full run. That shifts the hardware question: the solver runs need FP64 or single-precision throughput and memory per million cells, as our guide to GPUs for CFD simulation explains, while surrogate training needs memory per GPU for one training sample, GPU-to-GPU bandwidth and a dataset store measured in terabytes.

This guide covers the training side for the IT team that builds and runs it. Our guide to an AI server for automotive R&D shows where surrogates sit next to CFD, driving simulation and perception training in one department.

PhysicsNeMo in October 2026: release, install and requirements

PhysicsNeMo describes itself as "an open-source PyTorch framework for physics machine learning", and its models are ordinary torch.nn.Module objects. Release 2.1.0 ported the physics-informed examples to a physicsnemo.sym interface, replacing the separate physicsnemo-sym package.

NVIDIA PhysicsNeMo repository, releases page and changelog on GitHub, and the PhysicsNeMo system requirements on docs.nvidia.com, read on 25 October 2026. The releases page shows day and month; the year follows from the changelog, which dates release 2.1.0 to 26 May 2026.

The recommended hardware list names, among others, the B100 and RTX PRO 6000 Blackwell for Blackwell, and the H100 and H200 for Hopper, based on the PhysicsNeMo team's testing. NVIDIA advises its container to get "the benefits of all the performance improvements". The README's default install uses the cu13 extra; CUDA 13 needs driver 580.65.06 or later, as our RAPIDS workstation guide lists from NVIDIA's requirements. On RTX PRO Blackwell cards, a container or wheel built without sm_120 kernels fails with the error our guide to no kernel image on Blackwell explains.

Models and recipes: DoMINO aerodynamics, crash and structural (FEA) surrogates

The repository lists the surrogate architectures FNO, MeshGraphNet, Transolver, GeoTransolver, FLARE, DoMINO, FIGConvNet and the experimental GLOBE, among others. The CAE work is packaged as recipes.

External aerodynamics. The unified recipe trains GeoTransolver by default, a transformer for point-cloud regression with 12 layers, 256 hidden units and 8 heads, and also Transolver, GLOBE and FLARE; DoMINO is marked "DRAFT, doesn't run end-to-end" in the recipe gallery and has its own example. DrivAerML is the dataset the recipe supports today, with HiLift and SHIFT datasets listed as coming. The default precision is bfloat16, with float16 and float32 also supported.

Crash. The crash recipe trains on LS-DYNA d3plot results, which the separate PhysicsNeMo-Curator converts to VTP or Zarr. Its example configuration uses 121 training runs, 5 validation runs and 51 time steps, and compares one-shot, time-conditional and autoregressive schemes; the README calls the time-conditional scheme best for long horizons.

Structures. A deforming-plate example covers structural mechanics, and the gallery adds underfill dispensing and data-centre airflow cases for CFD.

The 121 runs of the crash example show the order of magnitude a department needs before training starts: hundreds of finished, consistent simulations of one product family. A team with that archive has the data; the server decides how fast it can iterate on the model.

Training data: storage for simulation results

Simulation output is large, and surrogate training reads it repeatedly. The DrivAerML dataset card on Hugging Face lists 500 variants of the DrivAer notchback, of which 484 runs are public, a total of 31.2 TB under CC BY-SA 4.0, computed with OpenFOAM v2212.

DrivAerML dataset card on Hugging Face, read on 25 October 2026. NVIDIA's DoMINO README gives "mesh sizes of 160M points per example".

By our arithmetic, 31.2 TB over 484 runs is about 64 GB per run. Curator preprocessing to Zarr "takes a couple of hours", according to the DoMINO README, and the processed copy sits beside the raw files. The DoMINO README also notes that by default the model "will read about 100x more data than necessary for the sampling size", which is why it shuffles volume data during preprocessing so that each read fetches random slices instead of the whole mesh.

For storage we plan three tiers: the raw archive on the department's file store, the processed training set on local NVMe in the training server, and checkpoints on both. A training set of a few hundred runs fits on several NVMe drives; the full raw DrivAerML set does not belong on the GPU server. Our guide to CPU, RAM and storage for a 4-GPU server gives the drive layout.

GPU memory per sample: the sampling resolution decides

A surrogate does not load a 160-million-point mesh into a card. Each training step samples a subset of cells or points, and the number sampled, together with the model, sets the memory per GPU. Batch size is 1 in the unified recipe: the README states that "batch_size > 1 is not supported by any model in the recipe today".

The recipe gives one published figure for GLOBE, its newest model "designed for large 3D use cases": "~143 GB/GPU at sampling_resolution=50000 in bf16; 200000 does not fit" 180 GB-class devices. The default sampling_resolution in train.yaml is 200,000, and the gallery runs other models at 50,000 to 200,000.

Our comparison of the recipe's ~143 GB per GPU with listed card memory from NVIDIA's product pages; GLOBE figure from the PhysicsNeMo unified external aerodynamics recipe README, read on 25 October 2026.

The model therefore sets the requirement as much as the card does. The recipe publishes no memory figure for GeoTransolver, the default, and the DoMINO README tells users who run out of memory to lower model.volume_points_sample and model.surface_points_sample. Before sizing a server, run the planned model on one card at two or three sampling resolutions and read peak memory with nvidia-smi; send us those figures, and we size the server from them. Where a sample does not fit one card, it can be split over several.

Several GPUs: DDP, FSDP2 and ShardTensor

PhysicsNeMo uses three ways to train on more than one card. With DistributedDataParallel (DDP), each GPU holds the whole model and processes its own sample; the recipe launches it with torchrun --nproc_per_node=N and averages losses with one fused all-reduce. FSDP2 shards parameters and optimiser state across GPUs, and release 2.2.0 added full save and load for FSDP2 checkpoints. ShardTensor, a "domain-parallel aware" tensor type, splits one high-resolution sample across GPUs and works alongside DDP or FSDP2.

The DoMINO README states the goal of sharding: "to enable higher resolution inputs and larger latent spaces without sacrificing substantial compute time". Its domain_size sets "the number of GPUs used for each batch", 1 being the standard regime, and requires shard_points or shard_grid when larger. Splitting one sample makes the GPUs exchange data within every step, so the link between them carries more traffic than in plain data parallelism. This is where the H200 NVL's NVLink bridge counts: 900 GB/s per GPU between bridged cards, against PCIe Gen5 for the RTX PRO 6000 Server Edition, which NVIDIA lists without NVLink.

For time to result, NVIDIA's DoMINO README reports that across several releases of performance work "overall training performance has decreased from about 5 days to just over 4 hours, with eight H100 GPUs". Memory pooling with RMM and reduced volume reads were among the changes. That figure is for one dataset and model on H100; it shows the effect of the data pipeline more than of any card.

H200 NVL or RTX PRO 6000 Server Edition for training

Both cards are on NVIDIA's PhysicsNeMo hardware list by family, and both run the bfloat16 default. They differ in memory per GPU, memory bandwidth and the link between cards.

NVIDIA product pages and product briefs for the H200 NVL and the RTX PRO 6000 Blackwell Server Edition, as cited in our CFD guide and four-card server guide; read on 24 October 2026.

The H200 NVL is the training card when a sample needs more than 96 GB, when ShardTensor splits samples across cards, or when the bandwidth-bound parts of a graph or transformer model dominate. Each H200 NVL also includes a five-year NVIDIA AI Enterprise subscription. The RTX PRO 6000 Server Edition fits when the model at its planned sampling resolution stays clearly below 96 GB per card and training runs as plain DDP; its 120 TFLOPS of FP32 and 96 GB also serve the GPU solvers in the same department.

We supply both cards; tell us your model, dataset size and training schedule, and our team replies with a configuration for either card.

Using the trained surrogate in the design loop

A trained surrogate answers one geometry at a time, without gradients or optimiser state, so it needs far less memory than training. DoMINO takes STL geometries as input, and its example includes an inference_on_stl.py script. NVIDIA publishes no inference timing for these models.

For designers, two layouts work. One is a shared inference service on the training server, or on one card of it, with MIG on the RTX PRO 6000 Server Edition dividing a card into up to four isolated 24 GB instances, as our MIG profiles guide shows. The other is a workstation per engineer with an RTX PRO 6000 Workstation Edition or Max-Q, which also runs CAD and pre-processing; our guide to GPU platforms for engineering firms compares the options.

Configurations for surrogate training

These start from the reference configurations on our AI servers page. Memory and cores follow NVIDIA's configuration guide for NVIDIA-Certified Systems: at least twice the GPU memory in system memory, at least six cores per GPU, two sockets.

Reference configurations of our AI servers page; wall power is our estimate (cards at 600 W plus two EPYC 9355 at 280 W or two EPYC 9555 at 360 W, plus 15 per cent for the rest), as in our sibling guides; a measurement on the delivered system confirms it.

A four-card server at full power is above the 2.94 kW we plan per 230 V 16 A feed and needs a 32 A or three-phase feed, as our GPU rack power and cooling guide works out. Training keeps the cards near full load for hours, so the inlet limits of the chosen chassis apply. Our H200 NVL NVLink bridge guide explains the two-way and four-way layouts.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with H200 NVL or RTX PRO 6000 Server Edition cards, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, NVIDIA driver, CUDA and a container runtime on request. Quantities and lead time are confirmed in writing with the quote. Setting up the training environment and serving the trained model belong to our Private AI/ML service.

FAQ