20. Physical Node API

The following calls provide information about the division and mapping of physical hardware in Charm++. A processing element (PE) is a unit of mapping and scheduling, which takes the form of an OS thread in SMP mode and an OS process in non-SMP mode. A logical node (often shortened to “node”) refers to an OS process: a set of one or more PEs that share memory (i.e. an address space). A physical node refers to an individual hardware machine (or, more precisely, an operating system instance on which Charm++ processes execute, or, in networking terminology, a host).

Communication between PEs on the same logical node is faster than communication between different logical nodes because OS threads share the same address space and can directly interact through shared memory. Communication between PEs on the same physical node may also be faster than between different physical nodes depending on the availability of OS features such as POSIX shared memory and Cross Memory Attach, the abilities of the network interconnect in use, and the speed of network loopback.

PEs are ranked in the range 0 to CmiNumPes(). Likewise, logical nodes are ranked from 0 to CmiNumNodes(), and physical nodes are ranked from 0 to CmiNumPhysicalNodes().

Charm++ provides a set of functions for querying information about the mapping of PE's to physical nodes. Class cputopology.C, contains the following member functions:

int CmiPeOnSamePhysicalNode(int pe1, int pe2)
Returns 1 if PE's pe1 and pe2 are on the same physical node and 0 otherwise.

int CmiNumPhysicalNodes()
Returns the number of physical nodes that the program is running on.

int CmiNumPesOnPhysicalNode(int node)
Returns the number of PE's that reside within a physical node.

void CmiGetPesOnPhysicalNode(int node, int **pelist, int *num)
After execution pelist will point to a list of all PE's that reside within a physical node and num will point to the length of the list. One should be careful to not free or alter pelist since it points to system memory.

int CmiPhysicalRank(int pe)
Returns the rank of a PE among all PE's running on the same physical node.

int CmiPhysicalNodeID(int pe)
Returns the node ID of the physical node in which a PE resides.

int CmiGetFirstPeOnPhysicalNode(int node)
Returns the lowest numbered processor on a physical node.