6. Automatic Parameter Marshalling

Automatic Parameter Marshalling is a concise means of invoking functions on remote processors. The CPM module handles all the details of packing, transmitting, translating, and unpacking the arguments. It also takes care of converting function pointers into handler numbers. With all these details out of the way, it is possible to perform remote function invocation in a single line of code.

6.1 CPM Basics

The heart of the CPM module is the CPM scanner. The scanner reads a C source file. When it sees the keyword CpmInvokable in front of one of the user's function declarations, it generates a launcher for that particular function. The launcher is a function whose name is Cpm_ concatenated to the name of the user's function. The launcher accepts the same arguments as the user's function, plus a destination argument. Calling the launcher transmits a message to another processor determined by the destination argument. When the message arrives and is handled, the user's function is called. For example, if the CPM scanner sees the following function declaration

    CpmInvokable myfunc(int x, int y) { ... }

The scanner will generate a launcher named Cpm_myfunc. The launcher has this prototype:

    void Cpm_myfunc(CpmDestination destination, int x, int y);

If one were to call Cpm_myfunc as follows:

    Cpm_myfunc(CpmSend(3), 8, 9);

a message would be sent to processor 3 ordering it to call myfunc(8,9). Notice that the destination argument isn't just an integer processor number. The possible destinations for a message are described later. When the CPM scanner is applied to a C source file with a particular name, it generates a certain amount of parameter packing and unpacking code, and this code is placed in an include file named similarly to the original C file: the .c is replaced with .cpm.h. The include file must be included in the original .c file, after the declarations of the types which are being packed and unpacked, but before all uses of the CPM invocation mechanisms. Note that the .cpm.h include file is not for prototyping. It contains the C code for the packing and unpacking mechanisms. Therefore, it should only be included in the one source file from which it was generated. If the user wishes to prototype his code, he must do so normally, by writing a header file of his own. Each .cpm.h file contains a function CpmInitializeThisModule, which initializes the code in that .cpm.h file. The function is declared static, so it is possible to have one in each .cpm.h file without conflicts. It is the responsibility of the CPM user to call each of these CpmInitializeThisModule functions before using any of the CPM mechanisms.

We demonstrate the use of the CPM mechanisms using the following short program myprog.c:

 1:    #include "myprog.cpm.h"
 3:    CpmInvokable print_integer(int n)
 4:    {
 5:      CmiPrintf("%d\n", n);
 6:    }
 8:    user_main(int argc, char **argv)
 9:    {

10:      int i;

11:      CpmModuleInit();

12:      CpmInitializeThisModule();

13:      if (CmiMyPe()==0)

14:        for (i=1; i<CmiNumPes(); i++)

15:          Cpm_print_integer(CpmSend(i), rand());

16:    }


18:    main(int argc, char **argv)

19:    {

20:      ConverseInit(argc, argv, user_main, 0, 0);

21:    }

Lines 3-6 of this program contain a simple C function that prints an integer. The function is marked with the word CpmInvokable. When the CPM scanner sees this word, it adds the function Cpm_print_integer to the file myprog.cpm.h. The program includes myprog.cpm.h on line 1, and initializes the code in there on line 12. Each call to Cpm_print_integer on line 15 builds a message that invokes print_integer. The destination-argument CpmSend(i) causes the message to be sent to the i'th processor.

The effect of this program is that the first processor orders each of the other processors to print a random number. Note that the example is somewhat minimalist since it doesn't contain any code for terminating itself. Also note that it would have been more efficient to use an explicit broadcast. Broadcasts are described later.

All launchers accept a CpmDestination as their first argument. A CpmDestination is actually a pointer to a small C structure containing routing and handling information. The CPM module has many built-in functions that return CpmDestinations. Therefore, any of these can be used as the first argument to a launcher:

CpmSend(pe) - the message is transmitted to processor pe with maximum priority.

CpmEnqueue(pe, queueing, priobits, prioptr) - The message is transmitted to processor pe, where it is enqueued with the specified queueing strategy and priority. The queueing, priobits, and prioptr arguments are the same as for CqsEnqueueGeneral.

CpmEnqueueFIFO(pe) - the message is transmitted to processor pe and enqueued with the middle priority (zero), and FIFO relative to messages with the same priority.

CpmEnqueueLIFO(pe) - the message is transmitted to processor pe and enqueued with the middle priority (zero), and LIFO relative to messages with the same priority.

CpmEnqueueIFIFO(pe, prio) - the message is transmitted to processor pe and enqueued with the specified integer-priority prio, and FIFO relative to messages with the same priority.

CpmEnqueueILIFO(pe, prio) - the message is transmitted to processor pe and enqueued with the specified integer-priority prio, and LIFO relative to messages with the same priority.

CpmEnqueueBFIFO(pe, priobits, prioptr) - the message is transmitted to processor pe and enqueued with the specified bitvector-priority, and FIFO relative to messages with the same priority.

CpmEnqueueBLIFO(pe, priobits, prioptr) - the message is transmitted to processor pe and enqueued with the specified bitvector-priority, and LIFO relative to messages with the same priority.

CpmMakeThread(pe) - The message is transmitted to processor pe where a CthThread is created, and the thread invokes the specified function.

All the functions shown above accept processor numbers as arguments. Instead of supplying a processor number, one can also supply the special symbols CPM_ALL or CPM_OTHERS, causing a broadcast. For example,

Cpm_print_integer(CpmMakeThread(CPM_ALL), 5);

would broadcast a message to all the processors causing each processor to create a thread, which would in turn invoke print_integer with the argument 5.

6.2 CPM Packing and Unpacking

Functions preceded by the word CpmInvokable must have simple argument lists. In particular, the argument list of a CpmInvokable function can only contain cpm-single-arguments and cpm-array-arguments, as defined by this grammar:

    cpm-single-argument :== typeword varname
    cpm-array-argument  :== typeword '*' varname

When CPM sees the cpm-array-argument notation, CPM interprets it as being a pointer to an array. In this case, CPM attempts to pack an entire array into the message, whereas it only attempts to pack a single element in the case of the cpm-single-argument notation.

Each cpm-array-argument must be preceded by a cpm-single-argument of type CpmDim. CpmDim is simply an alias for int, but when CPM sees an argument declared CpmDim, it knows that the next argument will be a cpm-array-argument, and it interprets the CpmDim argument to be the size of the array. Given a pointer to the array, its size, and its element-type, CPM handles the packing of array values as automatically as it handles single values.

A second program, example2.c, uses array arguments:

 1:    #include "example2.cpm.h"
 3:    CpmInvokable print_program_arguments(CpmDim argc, CpmStr *argv)
 4:    {
 5:      int i;
 6:      CmiPrintf("The program's arguments are: ");
 7:      for (i=0; i<argc; i++) CmiPrintf("%s ", argv[i]);
 8:      CmiPrintf("\n");
 9:    }


11:    user_main(int argc, char **argv)

12:    {

13:      CpmModuleInit();

14:      CpmInitializeThisModule();

15:      if (CmiMyPe()==0)

16:        Cpm_print_program_arguments(CpmSend(1), argc, argv);

17:    }


19:    main(int argc, char **argv)

20:    {

21:      ConverseInit(argc, argv, user_main, 0, 0);

22:    }

The word CpmStr is a CPM built-in type, it represents a null-terminated string:

 typedef char *CpmStr;

Therefore, the function print_program_arguments takes exactly the same arguments as user_main. In this example, the main program running on processor 0 transmits the arguments to processor 1, which prints them out.

Thus far, we have only shown functions whose prototypes contain builtin CPM types. CPM has built-in knowledge of the following types: char, short, int, long, float, double, CpmDim, and CpmStr (pointer to a null-terminated string). However, you may also transmit user-defined types in a CPM message.

For each (non-builtin) type the user wishes to pack, the user must supply some pack and unpack routines. The subroutines needed depend upon whether the type is a pointer or a simple type. Simple types are defined to be those that contain no pointers at all. Note that some types are neither pointers, nor simple types. CPM cannot currently handle such types.

CPM knows which type is which only through the following declarations:


The user must supply such declarations for each type that must be sent via CPM.

When packing a value v which is a simple type, CPM uses the following strategy. The generated code first converts v to network interchange format by calling CpmPack_typename(&v), which must perform the conversion in-place. It then copies v byte-for-byte into the message and sends it. When the data arrives, it is extracted from the message and converted back using CpmUnpack_typename(&v), again in-place. The user must supply the pack and unpack routines.

When packing a value v which is a pointer, the generated code determines how much space is needed in the message buffer by calling CpmPtrSize_typename(v). It then transfers the data pointed to by v into the message using CpmPtrPack_typename(p, v) , where p is a pointer to the allocated space in the message buffer. When the message arrives, the generated code extracts the packed data from the message by calling CpmPtrUnpack_typename(p). The unpack function must return a pointer to the unpacked data, which is allowed to still contain pointers to the message buffer (or simply be a pointer to the message buffer). When the invocation is done, the function CpmPtrFree_typename(v) is called to free any memory allocated by the unpack routine. The user must supply the size, pack, unpack, and free routines.

The following program fragment shows the declaration of two user-defined types:

 2:    typedef struct { double x,y; } coordinate;    
 3:    CpmDeclareSimple(coordinate);
 5:    void CpmPack_coordinate(coordinate *p)
 6:    {
 7:      CpmPack_double(&(p->x));
 8:      CpmPack_double(&(p->y));
 9:    }


11:    void CpmPack_coordinate(coordinate *p)

12:    {

13:      CpmUnpack_double(&(p->x));

14:      CpmUnpack_double(&(p->y));

15:    }


17:    typedef int *intptr;

18:    CpmDeclarePointer(intptr);


20:    #define CpmPtrSize_intptr(p) sizeof(int)


22:    void CpmPtrPack_intptr(void *p, intptr v)

23:    {

24:      *(int *)p = *v;

25:      CpmPack_int((int *)p);

26:    }


28:    intptr CpmPtrUnpack_intptr(void *p)

29:    {

30:      CpmUnpack_int((int *)p);

31:      return (int *)p;

32:    }


34:    #define CpmPtrFree_intptr(p) (0)


36:    #include "example3.cpm.h"

37:    ...

The first type declared in this file is the coordinate. Line 2 contains the C type declaration, and line 3 notifies CPM that it is a simple type, containing no pointers. Lines 5-9 declare the pack function, which receives a pointer to a coordinate, and must pack it in place. It makes use of the pack-function for doubles, which also packs in place. The unpack function is similar.

The second type declared in this file is the intptr, which we intend to mean a pointer to a single integer. On line 18 we notify CPM that the type is a pointer, and that it should therefore use CpmPtrSize_intptr, CpmPtrPack_intptr, CpmPtrUnpack_intptr, and CpmPtrFree_intptr. Line 20 shows the size function, a constant: we always need just enough space to store one integer. The pack function copies the int into the message buffer, and packs it in place. The unpack function unpacks it in place, and returns an intptr, which points right to the unpacked integer which is still in the message buffer. Since the int is still in the message buffer, and not in dynamically allocated memory, the free function on line 34 doesn't have to do anything.

Note that the inclusion of the .cpm.h file comes after these type and pack declarations: the .cpm.h file will reference these functions and macros, therefore, they must already be defined.

6.3 Inventing New Types of CpmDestinations

It is possible for the user to create new types of CpmDestinations, and to write functions that return these new destinations. In order to do this, one must have a mental model of the steps performed when a Cpm message is sent. This knowledge is only necessary to those wishing to invent new kinds of destinations. Others can skip this section.

The basic steps taken when sending a CPM message are:

1. The destination-structure is created. The first argument to the launcher is a CpmDestination. Therefore, before the launcher is invoked, one typically calls a function (like CpmSend) to build the destination-structure.

2. The launcher allocates a message-buffer. The buffer contains space to hold a function-pointer and the function's arguments. It also contains space for an ``envelope'', the size of which is determined by a field in the destination-structure.

3. The launcher stores the function-arguments in the message buffer. In doing so, the launcher converts the arguments to a contiguous sequence of bytes.

4. The launcher sets the message's handler. For every launcher, there is a matching function called an invoker. The launcher's job is to put the argument data in the message and send the message. The invoker's job is to extract the argument data from the message and call the user's function. The launcher uses CmiSetHandler to tell Converse to handle the message by calling the appropriate invoker.

5. The message is sent, received, and handled. The destination-structure contains a pointer to a send-function. The send-function is responsible for choosing the message's destination and making sure that it gets there and gets handled. The send-function has complete freedom to implement this in any manner it wishes. Eventually, though, the message should arrive at a destination and its handler should be called.

6. The user's function is invoked. The invoker extracts the function arguments from the message buffer and calls the user's function.

The send-function varies because messages take different routes to get to their final destinations. Compare, for example, CpmSend to CpmEnqueueFIFO. When CpmSend is used, the message goes straight to the target processor and gets handled. When CpmEnqueueFIFO is used, the message goes to the target processor, goes into the queue, comes out of the queue, and then gets handled. The send-function must implement not only the transmission of the message, but also the possible ``detouring'' of the message through queues or into threads.

We now show an example CPM command, and describe the steps that are taken when the command is executed. The command we will consider is this one:

Cpm_print_integer(CpmEnqueueFIFO(3), 12);

Which sends a message to processor 3, ordering it to call print_integer(12).

The first step is taken by CpmEnqueueFIFO, which builds the CpmDestination. The following is the code for CpmEnqueueFIFO:

typedef struct CpmDestinationSend_s
  void *(*sendfn)();
  int envsize;
  int pe;

CpmDestination CpmEnqueueFIFO(int pe)
  static struct CpmDestinationSend_s ctrl;
  ctrl.envsize = sizeof(int);
  ctrl.sendfn  = CpmEnqueueFIFO1; = pe;
  return (CpmDestination)&ctrl;

Notice that the CpmDestination structure varies, depending upon which kind of destination is being used. In this case, the destination structure contains a pointer to the send-function CpmEnqueueFIFO1, a field that controls the size of the envelope, and the destination-processor. In a CpmDestination, the sendfn and envsize fields are required, additional fields are optional.

After CpmEnqueueFIFO builds the destination-structure, the launcher Cpm_print_integer is invoked. Cpm_print_integer performs all the steps normally taken by a launcher:

1. It allocates the message buffer. In this case, it sets aside just enough room for one int as an envelope, as dictated by the destination-structure's envsize field.

2. It stores the function-arguments in the message-buffer. In this case, the function-arguments are just the integer 12.

3. It sets the message's handler. In this case, the message's handler is set to a function that will extract the arguments and call print_integer.

4. It calls the send-function to send the message.

The code for the send-function is here:

void *CpmEnqueueFIFO1(CpmDestinationSend dest, int len, void *msg)
  int *env = (int *)CpmEnv(msg);
  env[0] = CmiGetHandler(msg);
  CmiSetHandler(msg, CpvAccess(CpmEnqueueFIFO2_Index));

The send-function CpmEnqueueFIFO1 starts by switching the handler. The original handler is removed using CmiGetHandler. It is set aside in the message buffer in the ``envelope'' space described earlier -- notice the use of CpmEnv to obtain the envelope. This is the purpose of the envelope in the message -- it is a place where the send-function can store information. The destination-function must anticipate how much space the send-function will need, and it must specify that amount of space in the destination-structure field envsize. In this case, the envelope is used to store the original handler, and the message's handler is set to an internal function called CpmEnqueueFIFO2.

After switching the handler, CpmEnqueueFIFO1 sends the message. Eventually, the message will be received by CsdScheduler, and its handler will be called. The result will be that CpmEnqueueFIFO2 will be called on the destination processor. Here is the code for CpmEnqueueFIFO2:

void CpmEnqueueFIFO2(void *msg)
  int *env;
  env = (int *)CpmEnv(msg);
  CmiSetHandler(msg, env[0]);

This function takes ownership of the message-buffer from Converse using CmiGrabBuffer. It extracts the original handler from the envelope (the handler that calls print_integer), and restores it using CmiSetHandler. Having done so, it enqueues the message with the FIFO queueing policy. Eventually, the scheduler picks the message from the queue, and print_integer is invoked.

In summary, the procedure for implementing new kinds of destinations is to write one send-function, one function returning a CpmDestination (which contains a reference to the send-function), and one or more Converse handlers to manipulate the message.

The destination-function must return a pointer to a ``destination-structure'', which can in fact be any structure matching the following specifications:

This pointer must be coerced to type CpmDestination.

The send-function must have the following prototype:

    void sendfunction(CpmDestination dest, int msglen, void *msgptr)

It can access the envelope of the message using CpmEnv:

    int *CpmEnv(void *msg);

It can also access the data stored in the destination-structure by the destination-function.