What is inside an AI compute tray?

GPUs: performs parallel calculations

This NVIDIA teaching tray contains Blackwell Ultra GPU packages. Its compute dies and nearby memory work together on many calculations.

Several packages share a tray with host CPUs and supporting hardware. Cooling sits above them; electrical connections run through the boards below.

Grace and the GPUs communicate over coherent NVLink-C2C. The GPU can access CPU-attached memory, which has different bandwidth and latency from local HBM. Separate connections let GPUs communicate through the rack. These are different from the network adapter’s connections beyond the rack.

More GPUs need more data, electrical power, and heat removal. Useful performance depends on the supporting system as well as the chips.

How it is built. Tested packages are mounted on compute boards. The boards are installed in a tray and connected to cooling, power, and communication interfaces.

In the reference tray. NVIDIA’s GB300 reference tray contains two Grace CPUs and four Blackwell Ultra GPUs, the same counts used here.

CPUs: runs host software and coordinates work

The host CPU runs the operating system and general-purpose software. It helps organize work for the GPUs.

The CPU is near its own memory and the GPU connections. The smaller chips around each CPU represent host memory, which is separate from the HBM inside a GPU package.

CPU cores run instructions and access host memory. Coherent NVLink-C2C connects Grace to the GPUs, allowing GPU access to CPU-attached memory. That memory has different bandwidth and latency from local HBM.

A GPU-heavy workload still depends on host software and data preparation. A delay there can leave GPU resources waiting.

How it is built. A CPU and memory are assembled onto a board, with power regulation and interfaces. Cooling and system-level tests are added as the tray is assembled.

In the reference tray. A GB300 Grace Blackwell Superchip combines one Grace CPU with two Blackwell Ultra GPUs. A reference compute tray holds two of these combinations.

Network: exchanges data with other systems

A network adapter sends and receives data over the system network. It connects this tray’s work to other machines.

Ports sit at an accessible edge for cables. This model groups network functions into one adapter so their role is easy to see.

The adapter connects the computer to a network link. This is separate from the local CPU–GPU link and the rack’s NVLink connections. A network may use Ethernet or InfiniBand, depending on the system.

Fast processors can still wait for communication. Link bandwidth, delays, and how software exchanges data all matter.

How it is built. Controller chips, board connections, and external ports form the adapter. A system builder installs and tests it with the tray’s other hardware.

In the reference tray. The GB300 reference design uses four ConnectX-8 adapters for its compute network and two BlueField-3 DPUs for storage and management.

Power: brings electrical energy to the processors

Power delivery supplies electrical energy to the processors.

Power enters at the tray edge and is distributed across the board. Regulation near the chips helps supply the voltages they need.

In the DGX GB rack, power shelves feed a bus bar that distributes power to the trays. The model shows where that power enters the tray.

Power delivery and heat removal constrain the system together. A larger processor alone cannot remove either limit.

How it is built. Connectors, conductors, and regulation components are assembled with the board.

In the reference tray. NVIDIA documents rack power shelves and a shared bus bar for DGX GB systems.

Cooling: carries heat away from the chips

Cold plates collect heat from the processors. Liquid moving through the cooling system carries that heat away.

The plates sit above the hot chips. Learning view lifts them so the GPU packages and CPUs are visible underneath.

The DGX GB tray uses liquid cooling for CPUs and GPUs, while other components use air cooling.

Cooling needs pumps, heat exchange, and suitable infrastructure outside this tray.

How it is built. Cold plates are fitted over processors and connected to the coolant circuit. Flow and leak checks are part of system validation.

In the reference tray. DGX GB systems connect tray cold plates to rack manifolds. Fans cool other hardware.

How it developed

1. In 2024, NVIDIA shared portions of the GB200 NVL72 mechanical design with the Open Compute Project, including compute trays and cooling requirements.

2. Grace was designed for data-center workloads with tightly connected CPU, memory, and accelerator systems. NVIDIA’s architecture explanation describes its use of LPDDR5X memory.

3. Network adapters have evolved beyond basic connectivity to handle more communication work. NVIDIA’s ConnectX family includes hardware features for high-performance data movement.

4. The GB200 designs shared in 2024 treated rack power and cooling as part of the system design, alongside the compute hardware.

5. NVIDIA’s 2024 open hardware contribution included liquid-cooling and thermal requirements alongside the tray mechanical designs.

Key terms

GPU package

Compute chips, memory, and their package connections assembled together.

Local link

A connection between nearby processors within a system.

Host memory

Memory used by the CPU. It is distinct from the GPU’s local HBM.

LPDDR5X

A type of memory used in NVIDIA Grace systems.

NIC

Network interface card: hardware that connects a computer to a network.

Scale-out

Connecting multiple systems through a network so they can share work.

Bus bar

A substantial conductor used to distribute electrical power.

Regulation

Adjusting and maintaining the electrical supply needed by a component.

Cold plate

A surface that transfers heat from a component to coolant.

Manifold

A connection that distributes coolant to several branches.

More explainers

• What is high-bandwidth memory (HBM)?
• What is a GPU compute die?
• What is a silicon interposer?
• What does a GPU package substrate do?