28. Control Point Automatic Tuning

Charm++ currently includes an experimental automatic tuning framework that can dynamically adapt a program at runtime to improve its performance. The program provides a set of tunable knobs that are adjusted automatically by the tuning framework. The user program also provides information about the control points so that intelligent tuning choices can be made. This information will be used to steer the program instead of requiring the tuning framework to blindly search the possible program configurations. Warning: this is still an experimental feature not meant for production applications

28.1 Exposing Control Points in a Charm++ Program

The program should include a header file before any of its *.decl.h files:

    #include <controlPoints.h> 

The control point framework initializes itself, so no changes need to be made at startup in the program. The program will request the values for each control point on PE 0. Control point values are non-negative integers:

    my_var = controlPoint("any_name", 5, 10);
    my_var2 = controlPoint("another_name", 100,101);

To specify information about the effects of each control point, make calls such as these once on PE 0 before accessing any control point values:


For a complete list of these functions, see cp_effects.h in charm/include.

The program, of course, has to adapt its behavior to use these new control point values. There are two ways for a the control point values to change over time. The program can request that a new phase (with its own control point values) be used whenever it wants, or the control point framework can automatically advance to a new phase periodically. The structure of the program will be slightly different in these to cases. Sections 28.1.1 and 28.1.2 describe the additional changes to the program that should be made for each case.

28.1.1 Control Point Framework Advances Phases

The program provides a callback to the control point framework in a manner such as this:

    // Once early on in program, create a callback, and register it 
    CkCallback cb(CkIndex_Main::granularityChange(NULL),thisProxy); 
    registerCPChangeCallback(cb, true);

In the callback or after the callback has executed, the program should request the new control point values on PE 0, and adapt its behavior appropriately.

Alternatively, the program can specify that it wants to call gotoNextPhase(); itself when it is ready. Perhaps the program wishes to delay its adaptation for a while. To do this, it specifies false as the final parameter to registerCPChangeCallback as follows:

   registerCPChangeCallback(cb, false);

28.1.2 Program Advances Phases

     registerControlPointTiming(duration); // called after each program iteration on PE 0
     gotoNextPhase(); // called after some number of iterations on PE 0
    // Then request new control point values

28.2 Linking With The Control Point Framework

The control point tuning framework is now an integral part of the Charm++ runtime system. It does not need to be linked in to an application in any special way. It contains the framework code responsible for recording information about the running program as well as adjust the control point values. The trace module will enable measurements to be gathered including information about utilization, idle time, and memory usage.

28.3 Runtime Command Line Arguments

Various following command line arguments will affect the behavior of the program when running with the control point framework. As this is an experimental framework, these are subject to change.

The scheme used for tuning can be selected at runtime by the use of one of the following options:

     +CPSchemeRandom            Randomly Select Control Point Values
 +CPExhaustiveSearch            Exhaustive Search of Control Point Values
      +CPSimulAnneal            Simulated Annealing Search of Control Point Values
 +CPCriticalPathPrio            Use Critical Path to adapt Control Point Values
        +CPBestKnown            Use BestKnown Timing for Control Point Values
         +CPSteering            Use Steering to adjust Control Point Values
      +CPMemoryAware            Adjust control points to approach available memory

To intelligently tune or steer an application's performance, performance measurements ought to be used. Some of the schemes above require that memory footprint statistics and utilization statistics be gathered. All measurements are performed by a tracing module that has some overheads, so is not enabled by default. To use any type of measurement based steering scheme, it is necessary to add a runtime command line argument to the user program to enable the tracing module:


The following flags enable the gathering of the different types of available measurements.

        +CPGatherAll            Gather all types of measurements for each phase
+CPGatherMemoryUsage            Gather memory usage after each phase
+CPGatherUtilization            Gather utilization & Idle time after each phase

The control point framework will periodically adapt the control point values. The following command line flag determines the frequency at which the control point framework will attempt to adjust things.

     +CPSamplePeriod     number The time between Control Point Framework samples (in seconds)

The data from one run of the program can be saved and used in subsequent runs. The following command line arguments specify that a file named controlPointData.txt should be created or loaded. This file contains measurements for each phase as well as the control point values used in each phase.

         +CPSaveData            Save Control Point timings & configurations at completion
         +CPLoadData            Load Control Point timings & configurations at startup
     +CPDataFilename            Specify the data filename 

It might be useful for example, to run once with +CPSimulAnneal +CPSaveData to try to find a good configuration for the program, and then use +CPBestKnown +CPLoadData for all subsequent production runs.