2. Program Structure, Compilation and Utilities

A Charm++ program is essentially a C++ program where some components describe its parallel structure. Sequential code can be written using any programming technologies that cooperate with the C++ toolchain. This includes C and Fortran. Parallel entities in the user's code are written in C++ . These entities interact with the Charm++ framework via inherited classes and function calls.

2.1 Charm++ Interface (.ci) Files

All user program components that comprise its parallel interface (such as messages, chares, entry methods, etc.) are granted this elevated status by declaring them in separate charm++ interface description files. These files have a .ci suffix and adopt a C++-like declaration syntax with several additional keywords. In some declaration contexts, they may also contain some sequential C++ source code. Charm++ parses these interface descriptions and generates C++ code (base classes, utility classes, wrapper functions etc.) that facilitates the interaction of the user program's entities with the framework. A program may have several interface description files.

2.1.1 Syntax Highlighting of .ci Files Vim

To enable syntax highlighting of .ci files in Vim, do the following:

$ cp charm/contrib/ci.vim ~/.vim/syntax/.
$ vim ~/.vim/filetype.vim

And paste the following line in that file:

au! BufRead,BufNewFile *.ci set filetype=ci Sublime Text

Syntax highlighting in Sublime Text (version 3 or newer) can be enabled by installing the Charmci package through Package Control.

2.2 Modules

The top-level construct in a ci file is a named container for interface declarations called a module. Modules allow related declarations to be grouped together, and cause generated code for these declarations to be grouped into files named after the module. Modules cannot be nested, but each ci file can have several modules. Modules are specified using the keyword module. A module name must be a valid C++ identifier.

module myFirstModule {
    // Parallel interface declarations go here

2.3 Generated Files

Each module present in a ci file is parsed to generate two files. The basename of these files is the same as the name of the module and their suffixes are .decl.h and .def.h. For e.g., the module defined earlier will produce the files ``myFirstModule.decl.h'' and ``myFirstModule.def.h''. As the suffixes indicate, they contain the declarations and definitions respectively, of all the classes and functions that are generated based on the parallel interface description.

We recommend that the header file containing the declarations (decl.h) be included at the top of the files that contain the declarations or definitions of the user program entities mentioned in the corresponding module. The def.h is not actually a header file because it contains definitions for the generated entities. To avoid multiple definition errors, it should be compiled into just one object file. A convention we find useful is to place the def.h file at the bottom of the source file (.C, .cpp, .cc etc.) which includes the definitions of the corresponding user program entities.

It should be noted that the generated files have no dependence on the name of the ci file, but only on the names of the modules. This can make automated dependency-based build systems slightly more complicated. We adopt some conventions to ease this process. This is described in [*].

2.4 Module Dependencies

A module may depend on the parallel entities declared in another module. It can express this dependency using the extern keyword. externed modules do not have to be present in the same ci file.

module mySecondModule {

    // Entities in this module depend on those declared in another module
    extern module myFirstModule;

    // More parallel interface declarations

The extern keyword places an include statement for the decl.h file of the externed module in the generated code of the current module. Hence, decl.h files generated from externed modules are required during the compilation of the source code for the current module. This is usually required anyway because of the dependencies between user program entities across the two modules.

2.5 The Main Module and Reachable Modules

Charm++ software can contain several module definitions from several independently developed libraries / components. However, the user program must specify exactly one module as containing the starting point of the program's execution. This module is called the mainmodule. Every Charm++ program has to contain precisely one mainmodule.

All modules that are ``reachable'' from the mainmodule via a chain of externed module dependencies are included in a Charm++ program. More precisely, during program execution, the Charm++ runtime system will recognize only the user program entities that are declared in reachable modules. The decl.h and def.h files may be generated for other modules, but the runtime system is not aware of entities declared in such unreachable modules.

module A {

module B {
    extern module A;

module C {
    extern module A;

module D {
    extern module B;

module E {

mainmodule M {
    extern module C;
    extern module D;
    // Only modules A, B, C and D are reachable and known to the runtime system
    // Module E is unreachable via any chain of externed modules

2.6 Including other headers

There can be occasions where code generated from the module definitions requires other declarations / definitions in the user program's sequential code. Usually, this can be achieved by placing such user code before the point of inclusion of the decl.h file. However, this can become laborious if the decl.h file has to included in several places. Charm++ supports the keyword include in ci files to permit the inclusion of any header directly into the generated decl.h files.

module A {
    include "myUtilityClass.h"; //< Note the semicolon
    // Interface declarations that depend on myUtilityClass

module B {
    include "someUserTypedefs.h";
    // Interface declarations that require user typedefs

module C {
    extern module A;
    extern module B;
    // The user includes will be indirectly visible here too

2.7 The main() function

The Charm++ framework implements its own main function and retains control until the parallel execution environment is initialized and ready for executing user code. Hence, the user program must not define a main() function. Control enters the user code via the mainchare of the mainmodule. This will be discussed in further detail in [*].

Using the facilities described thus far, the parallel interface declarations for a Charm++ program can be spread across multiple ci files and multiple modules, permitting good control over the grouping and export of parallel API. This aids the encapsulation of parallel software.

2.8 Compiling Charm++ Programs

Charm++ provides a compiler-wrapper called charmc that handles all ci, C, C++ and Fortran source files that are part of a user program. Users can invoke charmc to parse their interface descriptions, compile source code and link objects into binaries. It also links against the appropriate set of charm framework objects and libraries while producing a binary. charmc and its functionality is described in B.

2.9 Utility Functions

The following calls provide basic rank information and utilities useful when running a Charm++ program.

void CkAssert(int expression)
Aborts the program if expression is 0.

void CkAbort(const char *message)
Causes the program to abort, printing the given error message. This function never returns.

void CkExit()
This call informs the Charm RTS that computation on all processors should terminate. This routine never returns, so any code after the call to CkExit() inside the function that calls it will not execute. Other processors will continue executing until they receive notification to stop, so it is a good idea to ensure through synchronization that all useful work has finished before calling CkExit().

double CkWallTimer()
Returns the elapsed time in seconds since the start of the program.

2.9.1 Information about Logical Machine Entities

As described in section 1.4, Charm++ recognizes two logical machine entities: ``node'' and PE (processing element). The following functions provide basic information about such logical machine that a Charm++ program runs on. PE and ``node'' are numbered starting from zero.

int CkNumPes()
returns the total number of PEs across all nodes.

int CkMyPe()
returns the index of the PE on which the call was made.

int CkNumNodes()
returns the total number of logical Charm++ nodes.

int CkMyNode()
returns the index of the ``node'' on which the call was made.

int CkMyRank()
returns the rank number of the PE on a ``node'' on which the call was made. PEs within a ``node'' are also ranked starting from zero.

int CkNodeFirst(int nd)
returns the index of the first PE on the logical node $nd$.

int CkNodeSize(int nd)
returns the number of PEs on the logical node $nd$ on which the call was made.

int CkNodeOf(int pe)
returns the ``node'' number that PE $pe$ belongs to.

int CkRankOf(int pe)
returns the rank of the given PE within its node.

2.9.2 Terminal I/O

Charm++ provides both C and C++ style methods of doing terminal I/O.

In place of C-style printf and scanf, Charm++ provides CkPrintf and CkScanf. These functions have interfaces that are identical to their C counterparts, but there are some differences in their behavior that should be mentioned.

Charm++ also supports all forms of printf, cout, etc. in addition to the special forms shown below. The special forms below are still useful, however, since they obey well-defined (but still lax) ordering requirements.

int CkPrintf(format [, arg]*)
This call is used for atomic terminal output. Its usage is similar to printf in C. However, CkPrintf has some special properties that make it more suited for parallel programming. CkPrintf routes all terminal output to a single end point which prints the output. This guarantees that the output for a single call to CkPrintf will be printed completely without being interleaved with other calls to CkPrintf. Note that CkPrintf is implemented using an asynchronous send, meaning that the call to CkPrintf returns immediately after the message has been sent, and most likely before the message has actually been received, processed, and displayed. As such, there is no guarantee of order in which the output for concurrent calls to CkPrintf is printed. Imposing such an order requires proper synchronization between the calls to CkPrintf in the parallel application.

void CkError(format [, arg]*))
Like CkPrintf, but used to print error messages on stderr.

int CkScanf(format [, arg]*)
This call is used for atomic terminal input. Its usage is similar to scanf in C. A call to CkScanf, unlike CkPrintf, blocks all execution on the processor it is called from, and returns only after all input has been retrieved.

For C++ style stream-based I/O, Charm++ offers ckout and ckerr in place of cout and cerr. The C++ streams and their Charm++ equivalents are related in the same manner as printf and scanf are to CkPrintf and CkScanf. The Charm++ streams are all used through the same interface as the C++ streams, and all behave in a slightly different way, just like C-style I/O.