Project

General

Profile

Feature #1497

Shared memory method to pass data between processes that share the same node

Added by Eric Bohm about 2 months ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
03/31/2017
Due date:
% Done:

0%


Description

PXSHM exists for this purpose on the net layers. However it is not generally used in SMP mode to exchange data when multiple comm threads share the same node.

Shared memory safety of our pxshm implementation and portability is a concern, so if the implementation choice were PXSHM, we would need to smoothly fail over to not using it on nodes which do not support it. Note, that is a runtime property as it can depend on kernel module load choices on compute nodes which can differ from the head node on which charm was compiled.

Some experimentation should be undertaken to determine where (if anywhere) this is provides any benefit.


Subtasks

Feature #1478: Investigate use of pxshm in CmiAllocNew

History

#1 Updated by Sam White about 2 months ago

The main things to investigate here appear to be xpmem, pxshm, knem and cma. OpenMPI's "vader" shared-memory BTL is able to use any of those implementations that are available (http://blogs.cisco.com/performance/the-vader-shared-memory-transport-in-open-mpi-now-featuring-3-flavors-of-zero-copy) and I've seen results from several papers (look at Nathan Hjelm's publications) that xpmem is the best performing of those. OpenMPI can get to <0.3us 1-byte message latency within a node using xpmem on Cray XE6 and XC40. I've also seen Intel MPI achieve ~0.5us shared-memory latency for 1-byte messages on KNLs and Haswells. From looking at Charm SMP pingpong we're usually 5-10x worse than those numbers.

Also available in: Atom PDF