HOW COMPUTE WORKSBy Michael Hsu

GPU package

What is high-bandwidth memory (HBM)?

A processor cannot work on data it has not received. High-bandwidth memory, or HBM, places stacked DRAM beside the compute dies, connected through a wide interface. It holds information such as model weights and intermediate results. The nearby position supports the many connections needed to move that information. Capacity determines how much fits; bandwidth determines how quickly it moves. If a workload is waiting on memory, adding more arithmetic units alone may not make it faster.

Explore it in the 3D GPU model

What it does

High-bandwidth memory (HBM) holds data the GPU needs while it works. Many connections let it move large amounts of data to the nearby processing chips.

Why it sits where it does

Memory sits beside the compute dies so they can share many short connections. Keeping them in the same package supports a wide path for data.

The package brings memory and compute into a shared physical assembly. This is different from placing ordinary memory modules elsewhere on a server board.

How it works

Several memory chips are stacked and connected vertically. Capacity is how much data they can hold. Bandwidth is how much data can move each second.

Limits and trade-offs

Stacking and advanced packaging add manufacturing and testing complexity. Many GPUs use GDDR memory instead.

In Blackwell Ultra and MI300X

NVIDIA Blackwell Ultra and AMD MI300X

HBM sits near compute to support a very wide interface: 1,024 data bits per HBM3/HBM3E stack, plus control and power connections. Carrying that many signals takes denser routing than an ordinary package substrate offers.

DRAM layers are thinned to fit the package height and thermal design. More layers complicate bonding, testing and heat removal.

NVIDIA Blackwell Ultra

288 GB HBM3E across eight stacks, each 12 DRAM layers high. Capacity measures stored bytes. Bandwidth measures bytes transferred each second. HBM4 doubles the data interface to 2,048 bits.

See it in the NVIDIA Blackwell Ultra model

AMD MI300X

192 GB HBM3 across eight stacks. Capacity measures stored bytes. Bandwidth measures bytes transferred each second. HBM4 doubles the data interface to 2,048 bits.

See it in the AMD MI300X model

How it is made

  1. Make the memory dies. DRAM circuits are fabricated on wafers. Through-silicon vias (TSVs) provide electrical paths through the silicon.
  2. Thin, connect, and stack. Wafers are thinned, connection bumps are formed, and dies are stacked. Precise alignment connects the layers.
  3. Protect and test the stack. Packaging protects the delicate assembly. A tested stack can then be integrated into a larger package.
  4. Bring memory beside compute. An interposer provides fine connections between chips; the package substrate connects the assembly to the board. They are distinct layers.

How it developed

  1. 2013

    Stacking becomes a memory architecture

    SK hynix dates its first HBM development to 2013. Vertical connections let multiple DRAM dies work together in a compact stack.

    SK hynix · Engineering HBM packaging ↗
  2. 2015

    A commercial graphics milestone

    AMD introduced HBM with the Radeon R9 Fury X. Memory moved into the GPU package instead of occupying a larger surrounding board footprint.

    AMD · Radeon R9 Fury X launch, 2015 ↗
  3. 2024

    Packaging enables another generation

    SK hynix reports beginning HBM3E production in March 2024. Its account emphasizes thin-die stacking and thermal and mechanical reliability.

    SK hynix · Engineering HBM packaging ↗
  4. 2026

    New generations enter evaluation

    SK hynix announced HBM4E samples in June. Customers test samples before volume production.

    SK hynix · HBM4E samples, June 2026 ↗

Where it is used

Key terms

DRAM
Memory that holds working data while powered and regularly refreshes its stored information.
TSV
Through-silicon via: an electrical connection that passes vertically through a silicon chip.
HBM
High-bandwidth memory: stacked memory designed to move large amounts of data.

Sources

The 3D models are illustrative teaching schematics, not manufacturer CAD. Counts and layouts are simplified where the sources say so.

More explainers