Project

General

Profile

Feature #1655

Feature #1497: Shared memory method to pass data between processes that share the same node

Enable use of pxshm/xpmem on mpi, ofi, and verbs builds

Added by Sam White 2 months ago. Updated 4 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Machine Layers
Target version:
Start date:
08/08/2017
Due date:
% Done:

0%


Description

pxshm is currently only supported on the GNI and NetLRTS layers. It should be trivial to allow its use on MPI and Verbs (and OFI when it gets merged), to say nothing of its performance.

History

#1 Updated by Sam White 2 months ago

  • Parent task set to #1497

#2 Updated by Sam White 2 months ago

  • Subject changed from Enable use of pxshm on mpi and verbs builds to Enable use of pxshm/xpmem on mpi and verbs builds

Also, './build charm++ gni-crayxe xpmem' fails to build because it tries to build pxshm and xpmem both. The issue is that we build with pxshm by default for gni-crayx* builds and don't disable that when explicitly building with xpmem. From what I've seen, xpmem offers performance nearly on par with user-space memcpy for Cray MPI, so that could potentially become the default on gni builds instead of pxshm if we implement it correctly. The key is to call xpmem_make() on the entire virtual address space during startup, avoiding the high cost of memory registration/deregistration during runtime.

#3 Updated by Sam White 4 days ago

  • Subject changed from Enable use of pxshm/xpmem on mpi and verbs builds to Enable use of pxshm/xpmem on mpi, ofi, and verbs builds

Also available in: Atom PDF