Project

General

Profile

Feature #1352

CkArrayOptions callback for completion of chare array initialization

Added by Sam White over 2 years ago. Updated about 2 years ago.

Status:
Merged
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
01/08/2017
Due date:
% Done:

0%


Description

Add a callback to CkArrayOptions to be triggered when all of the initial objects are constructed in a chare array.
Right now we don't have a good way to do this short of QD, which is global.

AMPI's intercommunicator creation is an example where this is needed, and where QD is not possible (since intercommunicators are created dynamically and other messages may be in flight on other communicators during creation of an intercommunicator).


Related issues

Related to Charm++ - Bug #1323: megatest multisection test failures Merged 12/07/2016
Related to Charm++ - Bug #1507: ckio test failure on gni-crayxc Merged 04/16/2017

History

#1 Updated by Phil Miller over 2 years ago

  • Assignee set to Sam White

Could you get this implemented? Hand off to Ed, Karthik or such with your guidance if you think they can do it.

#2 Updated by Sam White over 2 years ago

One thing I am not sure of that you or someone might just know: Is there a single point of exit in the initialization sequence for all types of chare array creation from which to invoke the callback?

#3 Updated by Phil Miller over 2 years ago

I'm not sure of what you mean by 'all types of chare array creation' - I think we only need to be concerned with bulk construction, which is done when the code returns from the populateInitial call on each PE. The array manager can contribute to a reduction targeting the callback.

#4 Updated by Sam White over 2 years ago

AMPI's use case for this (intercomm creation) is one-at-a-time/element-by-element initial insertion. Perhaps we should refactor AMPI's array creation to use bulk insertion though...

One issue I ran into when initially trying to implement this was that the contribute() methods are not defined in CkArray.

#5 Updated by Phil Miller over 2 years ago

  • Priority changed from Normal to Urgent

#6 Updated by Sam White over 2 years ago

Reassign if urgent. Isomalloc hangs and other AMPI things are more important to me at the moment. This is only really seen during intercommunicator creation in AMPI, which is itself rare.

#7 Updated by Sam White over 2 years ago

  • Status changed from New to In Progress

Hacky implementation using pt2pt sends, rather than a reduction, here: https://charm.cs.illinois.edu/gerrit/#/c/2209/

I tried adding contribute() definitions/declarations to CkReductionMgr, which CkArray inherits from. But then when running with more chare array elements than PEs, we get a hang. The reduction mgr for some reason is expecting more contributions than it should...

#8 Updated by Sam White about 2 years ago

We decided that since getting a proper reduction done inside CkArray will be ugly and doing all-to-one pt2pt sends will blow up at large core counts, using pt2pt sends in a tree-like manner is acceptable (clean up can come later).

#9 Updated by Phil Miller about 2 years ago

  • Related to Bug #1507: ckio test failure on gni-crayxc added

#10 Updated by Phil Miller about 2 years ago

It looks like there's another use case for this outside AMPI - CkIO.

#11 Updated by Phil Miller about 2 years ago

  • Assignee changed from Sam White to Phil Miller

#12 Updated by Phil Miller about 2 years ago

  • Status changed from In Progress to Implemented

Hackishly re-using the reduction manager's spanning tree now.

#13 Updated by Sam White about 2 years ago

  • Status changed from Implemented to Merged

Also available in: Atom PDF