Project

General

Profile

Feature #1721

Feature #1497: CMA support for passing data between processes on the same node

Feature #1655: Enable use of shm transport for regular messages in LRTS

pxshm in OFI

Added by Eric Bohm over 1 year ago. Updated over 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
10/25/2017
Due date:
% Done:

0%


Description

Check with OFI at intel regarding whether and how this should be done.

History

#1 Updated by Nitin Bhat over 1 year ago

Communication with Intel about SHM support in OFI :
Summary: Our OFI layer built on PSM2, which is mostly the default backend for the OFI layer, already uses SHM as PSM2 library internally supports SHM.

Hi Nitin,

Yes, OFI/PSM2 provider works over PSM2 library and PSM2 library does intra-node communication over shm.

Best regards,
Mikhail Shiryaev

From: Bhat, Nitin [mailto:]
Sent: 26 октября 2017 г. 19:28
To: Shiryaev, Mikhail <>; Burette, Yohann <>
Cc: White, Samuel T <>; Bohm, Eric J <>; Phil Miller <>; Kale, Laxmikant V <>
Subject: Re: pxshm support for Intra-host messages in OFI

Hi Mikhail and Yohann,

Thanks for that information.

Yes. We have a physical node API that’ll help us determine if the two PEs are on the same physical host.

About the point on PSM2 library, since it provides shm-transport internally, on machines where we use OFI with PSM2 backend, we are already taking advantage of that?

Yes. We mainly want to use the shared memory communication channel (PXSHM/XPMEM) for large messages across all applicable machine layers. May not help NAMD as the messages are capped to under a few KBs from what I know. But we do have other applications that use large messages like OpenAtom. For that reason, we’re planning to experiment with multiple shared memory models like CMA, XPMEM, KNEM and PXSHM and use them wherever applicable.

Thanks,
Nitin Bhat
Software Engineer,
Charmworks Inc.

From: "Shiryaev, Mikhail" <>
Date: Wednesday, October 25, 2017 at 4:20 PM
To: "Burette, Yohann" <>, "Bhat, Nitin" <>
Cc: "White, Samuel T" <>, "Bohm, Eric J" <>, Phil Miller <>, "Kale, Laxmikant V" <>
Subject: RE: pxshm support for Intra-host messages in OFI

Hi Nitin,

In addition to shm-provider: it is currently disabled in code, supports fi_msg API. There is pull request to add support for tagged API, and after that there are plans to add RMA API support. And as Yohann mentioned, the changes in Charm++/OFI code should be quite transparent. There should be some ability to detect if it is intra-node communication (as far as I understand currently it is impossible to detect that using CmiNode/Rank/PE/NodeSize …, correct me if I am wrong).

Also note that PSM2 library also has shm-transport for intra-node communication, so even if you work over PSM2-provider you actually do intra-node communication over shm but not over OPA loopback.

Regarding to other OFI runtimes: for example the current version of IMPI has own implementation of shm-transport (bypassing OFI transport for intra-node). But performance depends on application and shm:ofi and ofi:ofi configurations can give gain in different cases (cause adding of shm transport requires adding of extra logic and can lead to some limitations in performance optimization, for example handling of any-source messages).

Also as far as I understand using of PXSHM/XPMEM makes sense in case of large messages to avoid extra memcopies on critical path. How much it is important for NAMD (or other Charm-based applications) workloads?

Best regards,
Mikhail Shiryaev

From: Burette, Yohann
Sent: 25 октября 2017 г. 23:39
To: 'Bhat, Nitin' <>; Shiryaev, Mikhail <>
Cc: White, Samuel T <>; Bohm, Eric J <>; Phil Miller <>; Kale, Laxmikant V <>
Subject: RE: pxshm support for Intra-host messages in OFI

Hi Nitin,

There is some ongoing work being done on a new shared-memory provider in OFI:

https://github.com/ofiwg/libfabric/tree/master/prov/shm

The last pull request I see is adding tag messaging:

https://github.com/ofiwg/libfabric/pull/3457

Granted, I haven’t spent much time thinking all this through but, at first glance, I’d say that we would simply open a second endpoint for intra-node communications using that shm provider. Then, we would determine which endpoint to use when sending a message depending on its destination.

What do you think?

Yohann

From: Bhat, Nitin [mailto:]
Sent: Wednesday, October 25, 2017 12:38 PM
To: Shiryaev, Mikhail <>; Burette, Yohann <>
Cc: White, Samuel T <>; Bohm, Eric J <>; Phil Miller <>; Kale, Laxmikant V <>
Subject: pxshm support for Intra-host messages in OFI

Hi Mikhail and Yohann,

For intra-host (inter-process within the same physical node) communication in the OFI layer, we were planning to have a shared memory implementation like PXSHM or XPMEM. What would be the best way to go about this? Is there any specialized shared memory implementation or library (like Xpmem provided by Cray) that other OFI implementations use?

Thanks,
Nitin Bhat
Software Engineer,
Charmworks Inc.

#2 Updated by Sam White over 1 year ago

CMA will work on all Linux v3.2+ kernels

#3 Updated by Nitin Bhat over 1 year ago

  • Status changed from New to Rejected

Conversation with Intel revealed that psm2 fabric internally uses shm support. For that reason, Charm++'s OFI builds on psm2 should use shm transport. Additionally, since we're planning to have a generic LRTS based shm implementation using CMA ( https://charm.cs.illinois.edu/redmine/issues/1657), we could compare its performance with existing shm support in psm2. In any case, I'm rejecting this issue because there is no requirement of a network specific implementation.

Also available in: Atom PDF