Project

General

Profile

Feature #1184

SMP-safe CmiReference and CmiFree

Added by Sam White over 2 years ago. Updated 12 months ago.

Status:
Implemented
Priority:
Normal
Assignee:
Category:
SMP
Target version:
-
Start date:
08/24/2016
Due date:
% Done:

0%


Description

Two related issues that could be broken out into separate issues once this work is begun:

1. Currently CmiReference and CmiFree are not protected by a lock at all. Consequently CMK_BROADCAST_USE_CMIREFERENCE is set to 0 in conv-config.h for SMP mode builds, meaning that broadcasts have to be delivered to all PEs in SMP mode rather than just once to each logical Node. To fix this add locking to CmiReference/CmiFree.

2. We should avoid the overhead of locking if there are no actual shared references to an object, and we could set a threshold size for broadcast messages where if a message is large we will use CmiReference/Free within the logical Node and if it's smaller than the threshold we could potential just deliver it to every PE to avoid lock contention.


Related issues

Related to Charm++ - Bug #1633: NodeGroup Broadcasts creates many copies of the message for point to point sends In Progress 07/14/2017
Related to Charm++ - Feature #858: improve efficiency of exclusive entry methods In Progress 10/14/2015
Related to Charm++ - Feature #541: SMP mesage passing must enforce memory ordering consistency Merged 07/28/2014

History

#1 Updated by Eric Bohm over 2 years ago

  • Assignee set to Seonmyeong Bak

#2 Updated by Sam White over 2 years ago

  • Assignee changed from Seonmyeong Bak to Vipul Harsh

Reassigning to Vipul since this is mostly targeted at broadcasts.

See the note in conv-config.sh about CMK_BROADCAST_USE_CMIREFERENCE in SMP mode:

/**
    CmiReference broadcast/multicast optimization does not work for SMP
    due to race condition on memory reference counter, needs lock to protect
 */
#if CMK_SMP && CMK_BROADCAST_USE_CMIREFERENCE
#undef CMK_BROADCAST_USE_CMIREFERENCE
#define CMK_BROADCAST_USE_CMIREFERENCE                      0
#endif

#3 Updated by Sam White over 2 years ago

  • Tracker changed from Bug to Feature

This is more of a feature than a bug, meant to increase performance of large-message [nokeep] broadcasts.

Also, you may want to use these for thread-safe increment/decrement: https://charm.cs.illinois.edu/gerrit/#/c/1954/

#4 Updated by Sam White about 2 years ago

  • Category set to SMP

#5 Updated by Phil Miller almost 2 years ago

  • Target version changed from 6.8.0 to 6.9.0

#6 Updated by Eric Bohm almost 2 years ago

  • Assignee changed from Vipul Harsh to PPL

#7 Updated by Phil Miller almost 2 years ago

  • Related to Bug #1633: NodeGroup Broadcasts creates many copies of the message for point to point sends added

#8 Updated by Sam White over 1 year ago

  • Target version deleted (6.9.0)

#9 Updated by Phil Miller over 1 year ago

  • Related to Feature #858: improve efficiency of exclusive entry methods added

#10 Updated by Eric Bohm 12 months ago

  • Related to Feature #541: SMP mesage passing must enforce memory ordering consistency added

#11 Updated by Eric Bohm 12 months ago

Should be possible to fix this with a use of std::atomic that is similar to the approach used in the pcqueue.h fix. As the issue here is that in SMP we want the decrement on CmiReference to be safe and the old implementation does not enforce consistency.

#12 Updated by Eric Bohm 12 months ago

  • Assignee changed from PPL to Evan Ramos

#13 Updated by Evan Ramos 12 months ago

  • Status changed from New to Implemented

https://charm.cs.illinois.edu/gerrit/4108

Implementing this appears to have been trivial. It required renaming a large set of files to C++ though.

#14 Updated by Evan Ramos 12 months ago

Cross-referencing from Gerrit change:

Sam White wrote:

Turning CmiReference and CmiFree into atomics does solve this issue, but it will add overhead in the case of regular entry method invocations (not broadcasts). We'll want to measure that overhead. At the least I think we would want to only make them atomic in SMP mode.

Also available in: Atom PDF