Project

General

Profile

Feature #1657

pxshm/xpmem support for nocopy sends across processes on the same host

Added by Sam White 2 months ago. Updated 3 days ago.

Status:
New
Priority:
Normal
Assignee:
Category:
Machine Layers
Target version:
Start date:
08/08/2017
Due date:
% Done:

0%

Tags:

Description

It should be straightforward to implement this at least for the transfer of the nocopy payload: the small metadata message can still go through the network, but the large message transfer of the nocopy array parameter should be done using pxshm or xpmem when on the same host.

History

#1 Updated by Sam White 3 days ago

  • Assignee set to Nitin Bhat

#2 Updated by Sam White 3 days ago

See the following paper for a description of how to use XPMEM efficiently. The key is that you can register the entire virtual address space with xpmem_make() during startup, then memory registration/deregistration/copy is cheap at runtime.

See section "XPMEM BTL – Vader" on pages 2-3 here: https://www.open-mpi.org/papers/cug-2012/cug_2012_open_mpi_for_cray_xe_xk.pdf
The implementation for this is in: openmpi-2.0.0/opal/mca/btl/vader/btl_vader_xpmem.{h,c}

We also have some existing code using xpmem in charm/src/arch/util/machine-xpmem.c but that code performs an extra copy into an intermediate xpmem buffer. Same for machine-pxshm.c.

Also available in: Atom PDF