Power and cooling for a GPU rack: what to check before the servers arrive

Who checks power, cooling and floor loading before delivery

A GPU rack for inference needs 15 to 40 kW of power and cooling and a training rack 40 to 120 kW, against 8 to 15 kW for a classic enterprise rack. In Uptime Institute’s figures, room air cools up to about 20 to 25 kW per rack and containment with in-row cooling up to about 50 kW; above that, the rack needs a rear-door heat exchanger or liquid cooling. Before the servers arrive, three physical questions have to be answered: how much current the row can supply, where the hot air goes, and whether the floor can carry the weight.

None of these checks is difficult, but nobody owns them by default. The vendor sells the server, the integrator configures it, and the facilities team often hears about it only when the delivery arrives. This is the checklist we go through before we quote an AI server, and it is short enough to do yourself. Describe your rack position, and our team will help.

Rack density in kW: enterprise, inference and training racks

Racks for classic enterprise servers were designed for much lower densities than GPU servers need:

Densities per rack, our planning ranges; most first AI deployments fall into the middle band

The next table gives maximum power figures, per card or module for the GPUs and per system for the DGX rows; when you size a circuit, the system figure is the one that matters:

Four DGX H100 systems in one rack draw roughly 41 kW, the load of three to five legacy racks. A single 400 V three-phase 32 A feed delivers about 22 kW, and we plan on 80% of that as headroom. The US NEC makes 80% mandatory for continuous loads; IEC breakers are rated for continuous current at their nominal value, but the margin is still good practice. Planned at 80%, a feed carries about 17.7 kW, so 41 kW needs three such feeds before any A and B redundancy, and the building has to be able to supply them. Servers with PCIe cards sit far lower. By our estimate, the cards plus the processors plus 15 per cent for memory, drives, fans and conversion losses, a server with two RTX PRO 6000 Server Edition at 600 W draws about 1.8 kW, four cards about 3.4 kW (2.7 kW at a 450 W cap) and eight cards about 6.3 kW; Dell’s datasheet gives about 8 kW per eight-card node as an indicative figure, and the feed is planned on the maker’s figure. An existing air-cooled 10 kW rack therefore takes up to five two-card servers, two four-card servers (three at 450 W) or one eight-card server, as our guide to air-cooled AI servers in a 10 kW rack works out. A 230 V 16 A feed planned at 80 per cent carries 2.94 kW, enough for the two-card server; which feed the larger ones need is in our guide to GPU server power on 16 A, 32 A and three-phase.

Airflow per kW and floor tiles per rack

Every watt a server draws becomes heat that the air has to carry away. The usual planning figure is about 160 CFM per kW at an 11 °C rise across the server, and you can check it against what your raised floor delivers:

CFM figures assume a ~11 °C (20 °F) temperature rise; ~800 CFM per grate is a conservative planning figure at typical plenum pressures (ASHRAE cites up to ~1,900 CFM per tile on best-of-breed floors)

Two details matter as much as the totals. First, a raised floor delivers air per tile, and there is no room for six tiles in front of one rack. Second, without hot/cold aisle containment the hot exhaust flows back around the rack into the intake, so the servers take in warm air instead of cooled air. Above roughly 20 kW per rack, containment is required.

ASHRAE’s recommended inlet range for classes A1 to A4 is 18 to 27 °C, and 18 to 22 °C for its high-density class H1. If your measured inlet temperature is already at the top of the band with the current load, there is no headroom left for the new rack.

Where air stops and liquid starts

There is no hard cut-off, but there is a usual sequence of steps by rack density:

The rear-door heat exchanger is the step between air and direct-to-chip liquid cooling. It is a radiator that replaces the rear door of the rack: hot air passes through it and leaves the rack at close to room temperature, so the heat load does not reach the room. It needs a water loop to the rack, typically at 14 to 30 °C and above the dew point rather than chilled. That is a building project, but a smaller one than rebuilding the room, and the servers stay as they are.

Direct-to-chip liquid cooling is a different commitment. It changes the servers, the rack, the facility loop and the maintenance procedure, so it is worth it only when the rack density requires it.

Four checks before switch-on: circuit, PDU, floor loading, GPU cable

Each of these four can stop a delivered server from being switched on.

1. The circuit rating, in amps rather than kilowatts. Ask the facilities team for the rating of the circuit feeding the rack position and the current already drawn on it. A 32 A three-phase feed at 400 V is ~22 kW, and you plan at 80% of that as headroom. Two 10 kW systems on one such feed are already at the limit.

2. The socket type on the PDU. A C13 outlet carries 10 A, about 2.3 kW at 230 V. A GPU server with dual 2,700 W supplies needs C19 outlets; a PDU with only C13 outlets cannot feed it. A PDU with C19 outlets is a separate order, so check the outlet type before delivery.

3. Floor loading. A 64-GPU rack can approach one metric ton, and the worked example below puts four servers in one rack. Many raised floors are rated well below that per tile, and the route from the loading bay to the rack position has to carry it too. Check the point load, the weight on each foot or castor, as well as the total.

4. The GPU power cable supplied with the server. On cards with the 12V-2×6 connector (formerly 12VHPWR) the sense pins set the power limit, and some bundled server cables are configured for 450 W rather than 600 W. An RTX PRO 6000 Server Edition then runs and reports Max Power Limit 450 W in nvidia-smi without an error, but it works with a quarter less power budget than its 600 W rating; an H200 NVL, which supports only the 600 W level, does not boot at all. Some server makers set 450 W on purpose, for example to fit four cards in a 2U chassis, so check each cable against the server maker’s GPU configuration for that model and slot, and order a 600 W cable only where the maker supports 600 W there.

We check the rack, power and airflow before we quote an AI server. Write to us with how many servers you plan and which GPUs go into them, and we reply with a configuration and a quote within one business day.

Passive server cards need chassis airflow

Server-edition cards (the L40S, the RTX PRO 6000 Server Edition, the H100 and H200) have no fans of their own. They are designed to sit in front-to-back chassis airflow and they depend on it completely. A passive card in a workstation without directed airflow overheats even at idle, because no air moves through its heatsink; it needs a server chassis that pushes air through it.

Put a passive card only in a server chassis, with the chassis fan profile set for that card. A machine that goes under a desk needs an actively cooled card, the Workstation or Max-Q edition.

We supply the RTX PRO 6000 in all three editions, and the inference node on our AI servers page carries four Server Edition cards. Tell us the chassis and card count, and we reply within one business day with the edition that fits.

Worked example: four eight-GPU servers in one rack

A team plans four PCIe servers with eight 600 W cards each and wants them in one rack.

Eight 600 W cards and two 360 W processors such as the AMD EPYC 9555 come to 5,520 W per server; adding 15 per cent for memory, drives, fans and power-supply losses, we estimate about 6.3 kW at the wall, so four servers come to about 25 kW in one rack. Dell’s datasheet gives about 8 kW per eight-card node as an indicative figure, so plan the feeds on the maker’s figure for the exact configuration. That rules out the existing row, which is designed for 12 kW per rack. Airflow at 25 kW is about 4,000 CFM: five tiles, which will not fit in front of one rack, so containment is required. Two three-phase feeds are needed rather than one. Ask for the weight of each configured server and check it against the floor rating of the rack position; four DGX H100 systems, for comparison, weigh about 520 kg on their own, at 130 kg each.

The project is still feasible, but it needs a second three-phase feed and containment, and both are better planned before the order than found at switch-on. On these numbers, the alternative is one server in each of four racks of the existing row, about 6.3 kW each by our estimate and 8 kW by Dell’s figure, inside its 12 kW design, which needs no building work. For one 70B model and about thirty users, one server with one H200 NVL or two RTX PRO 6000 is enough, as our 70B sizing example works out.

What we supply

We build AI servers to order and supply the professional NVIDIA cards that go into them, including the RTX PRO 6000 Server Edition, with manufacturer warranty on one EU contract and invoice. Power supplies and cooling, air or liquid, are chosen by GPU count and site. If you want deployment engineering, our engineering partner Vixen.UNO designs, deploys and supports the system under the same contract.

FAQ