Fig. 7

CPU-GPU and GPU-GPU memory access models. In (a), red path is the optimal one, while pink path is inferior because CPU to GPU memory access across NUMA nodes is worse than local access. In (b), if GPUs support peer-to-peer (P2P) communication technology, green path indicates that two GPUs communicate by Nvlink, which is the fast path. Purple path indicates that two GPUs communicate by PCIe switch (PLX), which is the most efficient communication way without Nvlink. Yellow path means that two GPUs communicate by host (CPU). Red path is the worst one because P2P is not supported across NUMA nodes, and memory access between two GPUs in this way must be transferred by both CPUs