SMP-safe CmiReference and CmiFree
Two related issues that could be broken out into separate issues once this work is begun:
1. Currently CmiReference and CmiFree are not protected by a lock at all. Consequently CMK_BROADCAST_USE_CMIREFERENCE is set to 0 in conv-config.h for SMP mode builds, meaning that broadcasts have to be delivered to all PEs in SMP mode rather than just once to each logical Node. To fix this add locking to CmiReference/CmiFree.
2. We should avoid the overhead of locking if there are no actual shared references to an object, and we could set a threshold size for broadcast messages where if a message is large we will use CmiReference/Free within the logical Node and if it's smaller than the threshold we could potential just deliver it to every PE to avoid lock contention.
#2 Updated by Sam White over 2 years ago
- Assignee changed from Seonmyeong Bak to Vipul Harsh
Reassigning to Vipul since this is mostly targeted at broadcasts.
See the note in conv-config.sh about CMK_BROADCAST_USE_CMIREFERENCE in SMP mode:
/** CmiReference broadcast/multicast optimization does not work for SMP due to race condition on memory reference counter, needs lock to protect */ #if CMK_SMP && CMK_BROADCAST_USE_CMIREFERENCE #undef CMK_BROADCAST_USE_CMIREFERENCE #define CMK_BROADCAST_USE_CMIREFERENCE 0 #endif
#14 Updated by Evan Ramos 12 months ago
Cross-referencing from Gerrit change:
Sam White wrote:
Turning CmiReference and CmiFree into atomics does solve this issue, but it will add overhead in the case of regular entry method invocations (not broadcasts). We'll want to measure that overhead. At the least I think we would want to only make them atomic in SMP mode.