Cost & Energy
AI / GPU Power Reference
GPU TDP by model, the node- and facility-power build-up at typical server overhead and PUE, and the rack-density threshold where air cooling stops being enough. Use alongside the AI / GPU Infrastructure Power & Cost Calculator.
Last updated: July 2026
Terms
- GPUGraphics Processing Unit
- A chip originally built for rendering graphics that turned out to be extremely good at AI calculations too, because it can do huge numbers of simple sums at the same time. Modern AI training and inference runs almost entirely on GPUs rather than the regular processor (CPU).
- TDPThermal Design Power
- The maximum heat a chip is built to produce under sustained load, given in watts — used as a stand-in for its worst-case power draw. A GPU rated at 700 W TDP can pull close to that continuously under a full AI workload, unlike a regular processor that rarely sits at its peak.
- PUEPower Usage Effectiveness
- How much total electricity a facility uses for every unit that reaches the actual computers. 1.5 means 50% extra goes to cooling and power losses. Lower is better.
- NVLinkNVIDIA's high-speed GPU-to-GPU link
- A high-speed connection that lets multiple GPUs in the same server talk directly to each other and share data, much faster than going through the regular network. It's part of why a multi-GPU server draws meaningfully more power than simply adding up each GPU's own rating.
GPU Power by Model (8-GPU Node)
Typical rated TDP per GPU, and the resulting power for an 8-GPU node — the common building block of AI servers — at a 1.8× server overhead (CPUs, NVLink, RAM, PSU losses) and a 1.4 PUE.
| GPU model | TDP | Node power (8 GPU, ×1.8) | Facility power (×PUE 1.4) |
|---|---|---|---|
| NVIDIA A100 | 400 W | 5.8 kW | 8.1 kW |
| NVIDIA L40S | 350 W | 5.0 kW | 7.1 kW |
| NVIDIA H100 | 700 W | 10.1 kW | 14.1 kW |
| NVIDIA H200 | 700 W | 10.1 kW | 14.1 kW |
| NVIDIA B200 | 1000 W | 14.4 kW | 20.2 kW |
| NVIDIA B300 | 1400 W | 20.2 kW | 28.2 kW |
Read: one 8×H100 node draws ~10.1 kW of node power before facility overhead — not the 5.6 kW the raw GPU wattage alone suggests. Specs evolve; treat as planning-level, not a vendor datasheet.
Air vs Liquid Cooling — the ~30 kW/Rack Threshold
Rack density is compared against node power (before PUE) — this is the heat that has to be physically removed from the rack.
| Node power per rack | Verdict | What it means |
|---|---|---|
| Up to 20 kW | Air cooling OK | Standard hot/cold aisle handles it comfortably |
| 20 – 30 kW | Air cooling marginal | Needs good containment and high-capacity CRAC/CRAH units |
| Above 30 kW | Liquid cooling required | Air can no longer remove the heat — direct-to-chip or immersion needed |
Real dense AI racks (NVIDIA GB200 NVL72-class systems, ~72 GPUs) reach 120–140 kW per rack — well past the air-cooling ceiling, which is why liquid cooling is now standard for the densest AI deployments.
How Many GPUs Fit in a Rack Before Liquid Cooling Is Required?
Node power per GPU (TDP × 1.8 server overhead) divided into the ~30 kW/rack air-cooling ceiling — a rough cap before a rack needs liquid cooling, independent of how many physical servers that spreads across.
| GPU model | Node power per GPU | Approx. max GPUs/rack (air-cooled) |
|---|---|---|
| NVIDIA A100 | 0.72 kW | ~41 GPUs |
| NVIDIA L40S | 0.63 kW | ~47 GPUs |
| NVIDIA H100 / H200 | 1.26 kW | ~23 GPUs |
| NVIDIA B200 | 1.8 kW | ~16 GPUs |
| NVIDIA B300 | 2.52 kW | ~11 GPUs |
Read: a rack of H100s stays air-cooled up to roughly 23 GPUs (about three 8-GPU nodes); the newer, hungrier B300 crosses that line at around 11 GPUs — check your real count with the AI / GPU Power & Cost Calculator.
The Power Build-Up Formula
GPU power = TDP × GPU count × utilization
Node power = GPU power × server overhead (~1.8) — covers CPUs, NVLink, RAM and PSU losses
Facility power = node power × PUE — the number that actually shows up on the electricity bill
Annual cost = facility power × hours/year × electricity rate — AI inference commonly runs near-continuous (8,760 h), unlike typical office IT load.
Frequently Asked Questions
How much power does an H100 GPU actually use?
The H100 is rated at ~700 W TDP, but a full 8-GPU node draws closer to 10 kW once server overhead (CPUs, NVLink, RAM, PSU losses, ~1.8×) is included — and that's before facility/cooling overhead (PUE) is added on top.
Why do dense GPU racks need liquid cooling?
Air cooling tops out around 30 kW of heat removal per rack even with good containment. Dense AI racks with many high-TDP GPUs routinely exceed that — modern GB200 NVL72-class racks reach 120–140 kW — so direct liquid cooling becomes the only practical option.
Does higher utilization always mean higher cost?
Yes, roughly linearly — power scales with utilization in this model. The reason AI/inference deployments run near-continuously at high utilization (unlike bursty office workloads) is exactly why electricity becomes the dominant running cost.