MultiRPC

Design Issues

The MultiRPC facility ¹ is an extension to RPC2 that provides a parallel RPC capability for sending a single request to multiple servers and awaiting their individual responses. Although the actual transmission is done sequentially, the resultant concurrent processing by the servers results in a significant increase in time and efficiency over a sequence of standard RPC calls. The RPC2 runtime overhead is also reduced as the number of servers increases. For the purposes of this discussion, the base RPC2 facility will be referred to simply as RPC2.

A noteworthy feature of the MultiRPC design is the fact that the entire implementation is contained on the client side of the RPC2 code. The packet which is finally transmitted to the server is identical to a packet generated by an RPC2 call, and the MultiRPC protocol requires only a normal response from a server.

A major design goal was the desire to automatically provide MultiRPC capability for any subsystem without requiring any additional support from the subsytem designer or implementor. This has been achieved through modifications to RP2Gen, the RPC2 stub generation package (see RP2Gen). RP2Gen generates an array of argument descriptor structures for each server operation in the specification file, and these arrays are inserted in the beginning of the client side stub file. These structures are made available to the client through definitions in the associated .h file, and allow the use of MultiRPC with any routine in any subsystem with RP2Gen generated interfaces.

The orthogonality of the MultiRPC modifications also extends to the side effect mechanism. Side effects for MultiRPC work exactly as in the RPC2 case except that the client must supply a separate SE_Descriptor for each connection. Side effects can be omitted on a subset of the connections by specifying a the Tag field value of OMITSE for those SE_Descriptor's. This used, for example, in the Coda file system where a MultiRPC call is made to obtain file version information from a set of servers, but data from only one of those servers.

Parameter packing and unpacking for MultiRPC is provided in the RPC2 runtime library by a pair of routines. These library routines provide the functionality of the client side interface generated by RP2Gen as well as some additional modifications to support MultiRPC. It was decided to perform the packing and unpacking in RPC2 library routines rather than in individual client side stub routines as in the RPC2 case; this requires some extra processing time, but saves a significant amount of space in the client executable file. This approach has the added advantage of modularity; execution of RPC2 calls will not be affected at all, and even for MultiRPC calls the additional processing time is negligable in comparison to the message transmission overheads imposed by the UNIX kernel.

Another feature of MultiRPC is the client supplied handler routine. Through the handler routine the client is allowed to process each server response as it arrives rather than waiting for the entire MultiRPC call to complete. After processing each response, the client can decide whether to continue accepting server responses or whether to abort the remainder of the call. This facility can be useful if only a subset of responses are required, or if one failed message renders the entire call useless to the client. This capability is discussed further in handler.

MultiRPC also provides the same correctness guarantees as RPC2 except in the case where the client exercises his right to terminate the call. RPC2 guarantees that a request (or response) will be processed exactly once in the absence of network and machine crashes; otherwise, it guarantees that it will be processed at most once. If the call completes normally, a return code of RPC2_Success guarantees that all messages have been received by the appropriate servers.

C Interface Specification

The following table shows the C type interface between the client routine and MRPC_MakeMulti for all the possible combinations of legal parameter declarations and types. In all cases it is the clients responsibility to allocate storage for all parameters, just as in the RPC2 case. For all types except RPC2_EncryptionKey, IN parameters are handled the same as in the single MakeRPC case; RPC2_EncryptionKeys must be passed as pointers in the MultiRPC case. For OUT and INOUT parameters, arrays of pointers to parameters must be supplied in order to hold the multiple server responses. The array for each parameter must contain the same number of items as the number of servers contacted, and they must be filled sequentially starting from element zero. For all INOUT parameters except for SE_Descriptors, only the first element of the array need be filled in. For SE_Descriptors, all elements must be filled in.

The following table should be consulted for specific formats.

RPC2 Type	in	out	in out
RPC2_Integer	long	long *[]	long *[]
RPC2_Unsigned	unsigned long	unsigned long *[]	unsigned long *[]
RPC2_Byte	unsigned char	unsigned char *[]	unsigned char *[]
RPC2_String	unsigned char *	unsigned char **[]	unsigned char **[]
RPC2_CountedBS	RPC2_CountedBS *	RPC2_CountedBS *[]	RPC2_CountedBS *[]
RPC2_BoundedBS	RPC2_BoundedBS *	RPC2_BoundedBS *[]	RPC2_BoundedBS *[]
RPC2_EncryptionKey	RPC2_EncryptionKey *	RPC2_EncryptionKey *[]	RPC2_EncryptionKey *[]
SE_Descriptor	illegal	illegal	SE_Descriptor *[]
RPC2_Enum name	name	name *[]	name *[]
RPC2_Struct name	name *	name *[]	name *[]
RPC2_Byte name[...]	name	name *[]	name *[]

Table 1. RP2Gen representation of MultiRPC parameters

The client is only responsible for understanding the parameter type interface to the MakeMulti and HandleResult routines, and for allocating all necessary storage. MRPC_MakeMulti and MRPC_UnpackMulti are included in the RPC2 libraries.

Runtime Calls

MRPC_MakeMulti

int MRPC_MakeMulti(
    in long ServerOpcode,
    in ARG ArgTypes[],
    in long HowMany,
    in RPC2_Handle CIDList[],
    out RPC2_Integer RCList[],
    in out RPC2_Multicast *MCast,
    in long (*HandleResult)(),
    in struct timeval *Timeout,
    ...
);

Pack arguments and initialize state for RPC2_MultiRPC. Individual server responses are processed by calling HandleResult. For all in or out parameters an array of HowMany of the appropriate type should be allocated and supplied by the client. For example, if one argument is an out integer, an array of HowMany integers (i.e. int foo[HowMany];) should be used. For structures an array of structures and NOT an array of pointers to structures should be used. in arguments are treated as in the RPC2_MakeRPC case.

Parameters

ServerOpcode: For server routine foo, foo_OP. RP2Gen generated opcode, defined in include file. Note that subsystems with overlapping routine names may cause problems in a RPC2_MakeMulti call.

ArgTypes: For server routine foo, foo_PTR. RP2Gen generated array of argument type specifiers. A pointer to this array is located in the generated include file foo.h.

HowMany: How many servers are being called.

CIDList: Array of HowMany connection handles, one for each of the servers.

RCList: Array of length HowMany, into which RPC2 will place return codes for each of the connections specified in ConnHandleList. May be specified as NULL if return codes will not be examined.

MCast: Pointer to multicast structure. Set to NULL for now.

HandleResult: User procedure to be called after each server response. Responses are processed as they come in. Client can indicate when he has received sufficient responses (see RPC2_UnpackMulti). MRPC_MakeMulti will return when all responses have been processed.

Timeout: Timeout for the entire MultiRPC call. If NULL, use default timeout.

...: Variable length argument list. This is the list of the server arguments as they are declared in the .rpc2 file. The types of these arguments are determined by the RP2Gen generated include file.

Completion Codes

RPC2_SUCCESS: Success.

RPC2_TIMEOUT: The user specified timeout expired before all server responses were received.

RPC2_FAIL: Something other than RPC2_SUCCESS or RPC2_TIMEOUT occured.

RPC2_MultiRPC

int RPC2_MultiRPC(
    in long HowMany,
    in RPC2_Handle ConnHandleList[],
    in RPC2_PacketBuffer *Request,
    in SE_Descriptor SDescList[],
    in long (*UnpackMulti) (),
    in out ARG(INFO) *ArgInfo,
    in struct timeval *Patience
);

Make a collection of remote procedure calls. Logically identical to iterating through ConnHandleList and making RPC2_MakeRPC calls to each specified connection using Request as the request block, but this call will be considerably faster than explicit iteration. The calling lightweight process blocks until either the client requests that the call abort or one of the following is true about each of the connections specified in ConnHandleList: a reply has been received, a hard error has been detected for that connection, or the specified timeout has elapsed.

The ArgInfo structure exists to supply argument packing and unpacking information in the case where RP2Gen is used. Since its value is not examined by RPC2, it can contain any pointer that a non-RP2Gen generated client wishes to supply.

Similarly, UnpackMulti will point to a specific unpacking routine in the RP2Gen case. If the RP2Gen interface is not used, you should assume that the return codes of the supplied routine must conform to the MRPC_UnpackMulti specifications.

Side effects are supported as in the RPC2 case except that the client must supply a separate SE_Descriptor for each connection. The format for the SE_Descriptor argument is described in Adding New Kinds of Side Effects. It will often be useful to supply connection specific information such as unique file names in the SE_Descriptor.

Parameters

HowMany: How many servers are being called.

ConnHandleList: Array of HowMany connection handles, one for each of the servers.

Request: A properly formatted request buffer.

SDescList: List of HowMany side effect descriptors.

UnpackMulti: Pointer to unpacking routine called by RPC2 when each server response as received. If RP2Gen is used, this will be supplied by MRPC_MakeMulti. Otherwise, it must be supplied by the client.

ArgInfo: A pointer to a structure containing argument information. This structure is not examined by RPC2; it is passed untouched to UnpackMulti. If RP2Gen is used, this structure will be supplied by MRPC_MakeMulti. Otherwise, it can be used to pass any structure desired by the client or supplied as NULL.

Patience: Maximum time to wait for remote sites to respond. A NULL pointer indicates infinite patience as long as RPC2 is running.

Completion Codes

RPC2_SUCCESS: Success.

RPC2_TIMEOUT: The user specified timeout expired before all server responses were received.

RPC2_FAIL: Something other than RPC2_SUCCESS or RPC2_TIMEOUT occured. More detailed information is supplied via UnpackMulti to the user handler routine supplied in the ArgInfo structure.

MRPC_UnpackMulti

int MRPC_UnpackMulti(
    in long HowMany,
    in RPC2_Handle ConnHandleList[],
    in out ARG(INFO) *ArgInfo,
    in RPC2_PacketBuffer *Response,
    in long rpcval,
    in long thishost,
);

Unpack server arguments and call client handler routine. This routine is fixed in the RP2Gen case, and can be ignored by the client. For the non-RP2Gen case, a pointer to a routine with the argument structure described must be supplied as an argument to RPC2_MultiRPC. The functionality of such a client-supplied routine is unconstrained, but note that the return codes have an important effect on the process of the MultiRPC call.

Parameters

HowMany: How many servers were included in the MultiRPC call.

ConnHandleList: Array of HowMany connection ids.

ArgInfo: Pointer to argument information structure. This pointer is the same one passed in to RPC2_MultiRPC, so for the non-RP2Gen case its type is determined by the client.

Response: RPC2 response buffer.

rpcval: Individual connection error code or server response code.

thishost: Index into ConnHandleList to identify the returning connection.

Completion Codes

0: Continue accepting and processing server responses.

-1: Abort MultiRPC call and return.

HandleResult

int HandleResult(
    in long HowMany,
    in RPC2_Handle ConnArray[],
    in long WhichHost,
    in long rpcval,
    ...
);

Process incoming server replies as they arrive.

This routine must return either 0 or -1. A return value of zero indicates that the client wants to continue receiving server responses as they come in (normal case). A return value of -1 indicates that the client has received enough responses and wants to terminate the MakeMulti call (in which the client is still blocked). This allows the client to call a large number or servers and terminate after the first n responses are received.

Note that the name of this routine is arbitrary and may be determined by the client. RPC2_MultiRPC sees it only as a pointer supplied as an argument to MRPC_MakeMulti. The parameter list is predefined, however, and the client must follow the structure specified here in writing the routine.

Parameters

HowMany: The number of servers contacted in the MultiRPC call.

ConnArray: Array of HowMany connection ids.

WhichHost: This is an offset into ConnArray and into any OUT or IN OUT parameters. Using this to index the arrays will yield the responding server and its corresponding argument values.

rpcval: This is the RPC2 return code from the specified server.

...: Variable length argument list. This is the list of the server arguments as they are declared in the .rpc2 file. The types of these arguments are determined by the RP2Gen generated include file.

Completion Codes

0: Continue accepting and processing server responses.

-1: Abort MultiRPC call and return.

M. Satyanarayanan and E. H. Siegel, "Parallel communication in a large distributed environment," IEEE Transactions on Computers, vol. 39, no. 3, pp. 328--348, 1990, doi: 10.1109/12.48864 ↩