Project

General

Profile

Feature #1657

CMA support for nocopy sends using the Entry Method API across processes on the same host

Added by Sam White almost 2 years ago. Updated 7 months ago.

Status:
Merged
Priority:
Normal
Assignee:
Category:
Machine Layers
Target version:
Start date:
08/08/2017
Due date:
% Done:

0%

Tags:

Description

It should be straightforward to implement this at least for the transfer of the nocopy payload: the small metadata message can still go through the network, but the large message transfer of the nocopy array parameter should be done using pxshm or xpmem when on the same host.


Related issues

Related to Charm++ - Feature #1655: Enable use of shm transport for regular messages in LRTS Merged 10/25/2017

History

#1 Updated by Sam White over 1 year ago

  • Assignee set to Nitin Bhat

#2 Updated by Sam White over 1 year ago

See the following paper for a description of how to use XPMEM efficiently. The key is that you can register the entire virtual address space with xpmem_make() during startup, then memory registration/deregistration/copy is cheap at runtime.

See section "XPMEM BTL – Vader" on pages 2-3 here: https://www.open-mpi.org/papers/cug-2012/cug_2012_open_mpi_for_cray_xe_xk.pdf
The implementation for this is in: openmpi-2.0.0/opal/mca/btl/vader/btl_vader_xpmem.{h,c}

We also have some existing code using xpmem in charm/src/arch/util/machine-xpmem.c but that code performs an extra copy into an intermediate xpmem buffer. Same for machine-pxshm.c.

#3 Updated by Nitin Bhat over 1 year ago

  • Subject changed from pxshm/xpmem support for nocopy sends across processes on the same host to shm support for nocopy sends across processes on the same host

Experimenting with different models has shown that CMA (Cross Memory Attach) is a good candidate for exploiting shm for within-host communication. CMA usage for nocopy sends through the direct API has been implemented here: https://charm.cs.illinois.edu/gerrit/#/c/3116/8/src/arch/util/machine-rdma.h

Was this request to add CMA support to the Zerocopy Entry Method API?

#4 Updated by Nitin Bhat over 1 year ago

  • Subject changed from shm support for nocopy sends across processes on the same host to shm support for nocopy sends using the Entry Method API across processes on the same host

#5 Updated by Nitin Bhat over 1 year ago

  • Related to Feature #1655: Enable use of shm transport for regular messages in LRTS added

#6 Updated by Sam White over 1 year ago

  • Subject changed from shm support for nocopy sends using the Entry Method API across processes on the same host to CMA support for nocopy sends using the Entry Method API across processes on the same host

#7 Updated by Nitin Bhat over 1 year ago

  • Target version changed from 6.9.0 to 7 (Next Generation Charm++)

Moving it out of 6.9 release as the Broadcast API has higher priority.

#8 Updated by Nitin Bhat 10 months ago

  • Target version changed from 7 (Next Generation Charm++) to 6.9.1
  • Status changed from New to Implemented

By implementing the Entry Method API internally using the Direct API, the Entry Method API internally gets CMA support.

The patch used for implementing this feature is the same patch that refactors the Entry Method API to use the Direct API: https://charm.cs.illinois.edu/gerrit/#/c/charm/+/4259/

#9 Updated by Eric Bohm 7 months ago

  • Target version changed from 6.9.1 to 6.10.0

#10 Updated by Nitin Bhat 7 months ago

  • Status changed from Implemented to Merged

Also available in: Atom PDF