Isomalloc: refactor to associate migratable allocations with entities other than persistent threads
Isomalloc is currently very specialized to supporting the needs of AMPI/TCharm and BigSim emulation. These use cases have user-level threads that are created at startup, and persist for the duration of the run. However, there's nothing inherent in the migratable allocation functionality that limits it to this usage. In porting Chombo to run on Charm++, I've found that it would be convenient to have migratable allocations associated with an object instead. This could prove to be useful for porting in any kind of legacy code to Charm++, where recursively implementing PUP routines through the entire code base would be extraordinarily tedious and error-prone.
I intend to refactor Isomalloc so that it simply takes a 'context' argument that says which arena of co-migrating memory an allocation should come from. The TCharm/AMPI and BigSim cases would then associate a context with their threads, while the Chombo case would associate a context with each object.
#2 Updated by Phil Miller about 3 years ago
Took a lighter touch approach, of turning the existing
CmiIsomallocBlockList that was almost the necessary encapsulation of an allocation context into that context. Each block list now stores its associated memory pool, rather than using a
Ctv. Allocation is all fine, and external call sites now explicitly call for allocation from a particular block list.
I think the next step is to turn the active blocklist pointer from a
Cpv to a
Ctv so that it doesn't need to be explicitly en/disabled at thread context switches.
#11 Updated by Evan Ramos 5 months ago
Patch revival: https://charm.cs.illinois.edu/gerrit/4472
The remaining issue from the initial autobuild failures, which now shows up in Jenkins verification as well, is an abort from maxReduce in examples/bigsim/emulator.
The problem is the addition of a call to CmiIsomallocBlockListNew() in CthThreadBaseInit. Without the patch, CmiIsomallocBlockListNew is never called during maxReduce. With it, the debug prints in isomalloc say things like:
 tried to mmap 0x816dbf800000, but encountered error  tried to mmap 0x96ff5f400000, but encountered error Charm++> Isomalloc map failed to allocate 1048576 bytes at 0x816dbf800000, errno: 12. ------------- Processor 6 Exiting: Called CmiAbort ------------ Reason: Exiting Charm++> Isomalloc map failed to allocate 1048576 bytes at 0x96ff5f400000, errno: 12. ------------- Processor 7 Exiting: Called CmiAbort ------------ Reason: Exiting
errno 12 is ENOMEM.
#12 Updated by Evan Ramos 5 months ago
If I break on mempool_init in maxReduce, the crashing PE hits the breakpoint 16 times, and the other three PEs hit it exactly 169 times each by the time the crash occurs (measured with
info b in GDB). CthThreadBaseFree is never called.
Is the only solution here to make use lazy allocation on first use for isomallocBlockList, similar to what happens before the patch? Could another solution be to only allocate and use isomallocBlockList if the thread is migratable?
#13 Updated by Evan Ramos 5 months ago
- Status changed from In Progress to Implemented
- Target version deleted (
7 (Next Generation Charm++))
- Assignee changed from Eric Bohm to Evan Ramos
With my latest push implementing lazy allocation (which only takes place during the creation of a migratable stack, or setup of tlsglobals or swapglobals), all tests pass under netlrts-linux-x86_64-smp and netlrts-linux-smp.