test

ONC+ Developer's Guide

Chapter 7 Multithreaded RPC Programming

This manual does not cover basic topics and code examples for the Solaris implementation of multithreaded programming. Instead, refer to the Multithreaded Programming Guide for background on the following topics.

TI-RPC supports multithreaded RPC servers. The difference between a multithreaded server and a single-threaded server is that a multithreaded server uses threading technology to process incoming client requests concurrently. Multithreaded servers can have higher performance and availability compared with single-threaded servers.

MT Client Overview

In a multithread client program, a thread can be created to issue each RPC request. When multiple threads share the same client handle, only one thread at a time is able to make an RPC request. All other threads wait until the outstanding request is complete. On the other hand, when multiple threads make RPC requests using different client handles, the requests are carried out concurrently. Figure 4–1 illustrates a possible timing of a multithreaded client implementation consisting of two client threads using different client handles.

The following figure shows the client side implementation of a multithreaded rstat program. The client program creates a thread for each host. Each thread creates its own client handle and makes various RPC calls to the given host. Because the client threads are using different handles to make the RPC calls, they can carry out the RPC calls concurrently.

Figure 7–1 Two Client Threads Using Different Client Handles (Real Time)

Text describes graphic.

Note –

You must link in the thread library when writing any RPC multithreaded-safe application. The thread library must be the last named library on the link line. To link this properly, specify the -lthread option in the compile command.

Compile the program in the code example by typing cc rstat.c -lnsl -lthread.

MT Server Overview

RPC servers made available prior to the Solaris 2.4 release are single threaded. That is, they process client requests sequentially, as the requests come in. For example, say two requests come in, and the first takes 30 seconds to process, and the second takes only 1 second to process. The client that made the second request still has to wait for the first request to complete before it receives a response. This result is not desirable, especially in a multiprocessor server environment, where each CPU could be processing a different request simultaneously.Also, while one request is waiting for I/O to complete, sometimes other requests could be processed by the server.

Facilities in the RPC library for service developers can create multithreaded servers that deliver better performance to end users. Two modes of server multithreading are supported in TI-RPC: the Auto MT mode and the User MT mode.

In the Auto mode, the server automatically creates a new thread for every incoming client request. This thread processes the request, sends a response, and exits. In the User mode, the service developer decides how to create and manage threads for concurrently processing the incoming client requests. The Auto mode is much easier to use than the User mode, but the User mode offers more flexibility for service developers with special requirements.

Note –

You must link in the thread library when writing RPC multithreaded-safe applications. The thread library must be the last named library on the link line. To link this properly, specify the -lthread option in the compile command.

The two calls that support server multithreading are rpc_control() and svc_done(). The rpc_control() call is used to set the MT mode, either Auto or User mode. If the server uses Auto mode, it does not need to invoke svc_done() at all. In User mode, svc_done() must be invoked after each client request is processed so that the server can reclaim the resources from processing the request. In addition, multithreaded RPC servers must call on svc_run(). Note that svc_getreqpoll() and svc_getreqset() are unsafe in MT applications.

If the server program does not invoke any of the MT interface calls, it remains in single-threaded mode, which is the default mode.

You are required to make RPC server procedures multithreaded safe regardless of which mode the server is using. Usually, this means that all static and global variables need to be protected with mutex locks. Mutual exclusion and other synchronization APIs are defined in synch.h. See the condition(3THR) , rwlock(3THR), and mutex(3THR) man pages for a list of the various synchronization interfaces.

The following figure illustrates a possible timing of a server implemented in one of the MT modes of operation.

Figure 7–2 MT RPC Server Timing Diagram

Hosts A and B execute services on Host C via RPC.

Sharing the Service Transport Handle

The service transport handle, SVCXPRT, contains a single data area for decoding arguments and encoding results. Therefore, in the default, single-threaded mode, this structure cannot be freely shared between threads that call functions that perform these operations. However, when a server is operating in the MT Auto or User modes, a copy of this structure is passed to the service dispatch procedure in order to enable concurrent request processing. Under these circumstances, some routines that would otherwise be unsafe become safe. Unless otherwise noted, the server interfaces are generally MT safe. See the rpc_svc_calls(3NSL) man page for more details on safety for server-side interfaces.

MT Auto Mode

In the Auto mode, the RPC library creates and manages threads. The service developer invokes a new interface call, rpc_control(), to put the server into MT Auto mode before invoking the svc_run() call. In this mode, the programmer needs only to ensure that service procedures are MT safe.

rpc_control() enables applications to set and modify global RPC attributes. At present, this function supports only server-side operations. The following table shows the rpc_control() operations defined for Auto mode. See also the rpc_control(3NSL) man page for additional information.

Table 7–1 rpc_control() Library Routines

Routine 

Description 

RPC_SVC_MTMODE_SET()

Set multithread mode 

RPC_SVC_MTMODE_GET()

Get multithread mode 

RPC_SVC_THRMAX_SET()

Set maximum number of threads 

RPC_SVC_THRMAX_GET()

Get maximum number of threads 

RPC_SVC_THRTOTAL_GET()

Total number of threads currently active 

RPC_SVC_THRCREATES_GET()

Cumulative total number of threads created by the RPC library 

RPC_SVC_THRERRORS_GET()

Number of thr_create() errors within RPC library

Note –

All of the get operations in Table 7–1, except RPC_SVC_MTMODE_GET(), apply only to the Auto MT mode. If used in MT User mode or the single-threaded default mode, the results of the operations might be undefined.

By default, the maximum number of threads that the RPC server library creates at any time is 16. If a server needs to process more than 16 client requests concurrently, the maximum number of threads must be set to the desired number. This parameter can be set at any time by the server. It enables the service developer to put an upper bound on the thread resources consumed by the server. Example 7–1 is an example RPC program written in MT Auto mode. In this case, the maximum number of threads is set at 20.

MT performance is enhanced if the function svc_getargs() is called by every procedure other than NULLPROCS, even if there are no arguments (you can use xdr_void() in this case). This is true for both the MT Auto and MT User modes. For more information on this call, see the rpc_svc_calls(3NSL) man page.

Note –

You must link in the thread library when writing RPC multithreaded-safe applications. The thread library must be the last named library on the link line. Specify the -lthread option in the compile command.

The following example illustrates the server in MT Auto mode. To compile this program, type cc time_svc.c -lnsl -lthread.

Example 7–1 Server for MT Auto Mode

#include <stdio.h>

		#include <rpc/rpc.h>
		#include <synch.h>

		#include <thread.h>
		#include "time_prot.h"
 
		void time_prog();
 
		main(argc, argv)
		int argc;
		char *argv[];
		{
		int transpnum;
		char *nettype;
		int mode = RPC_SVC_MT_AUTO;
		int max = 20;      /* Set maximum number of threads to 20 */
 
		if (argc > 2) {
			fprintf(stderr, "usage: %s [nettype]\n", argv[0]);
			exit(1);
		}
 
		if (argc == 2)
			nettype = argv[1];
		else
			nettype = "netpath";
 
		if (!rpc_control(RPC_SVC_MTMODE_SET, &mode)) {
			printf("RPC_SVC_MTMODE_SET: failed\n");
			exit(1);
		}
		if (!rpc_control(RPC_SVC_THRMAX_SET, &max)) {
			printf("RPC_SVC_THRMAX_SET: failed\n");
			exit(1);
		}
		transpnum = svc_create( time_prog, TIME_PROG, TIME_VERS,
 			nettype);
 
		if (transpnum == 0) {
			fprintf(stderr, "%s: cannot create %s service.\n",
			argv[0], nettype);	
			exit(1);
		}
		svc_run();
	}
 
	/*
	 * The server dispatch function.
	 * The RPC server library creates a thread which executes
 * the server dispatcher routine time_prog().  After which
 * the RPC library destroys the thread.
 */
 
	static void
	time_prog(rqstp, transp)
		struct svc_req *rqstp;
		SVCXPRT *transp;
	{
 
		switch (rqstp->rq_proc) {
			case NULLPROC:
				svc_sendreply(transp, xdr_void, NULL);
				return;
			case TIME_GET:
				dotime(transp);
				break;
			default:
				svcerr_noproc(transp);
				return;
		}
	}
	dotime(transp)
	SVCXPRT *transp;
	{
	
		struct timev rslt;
		time_t thetime;
	
		thetime = time((time_t *)0);
		rslt.second = thetime % 60;
		thetime /= 60;
		rslt.minute = thetime % 60;
		thetime /= 60;
		rslt.hour = thetime % 24;
	if (!svc_sendreply(transp, xdr_timev,(caddr_t) &rslt)) {
			svcerr_systemerr(transp);
		}
	}

The following code example shows the time_prot.h header file for the server.

Example 7–2 MT Auto Mode: time_prot.h 

#include <rpc/types.h>

 
		struct timev {
			int second;

			int minute;
			int hour;
		};

 
		typedef struct timev timev;

		bool_t xdr_timev();
 
		#define TIME_PROG 0x40000001

		#define TIME_VERS 1
		#define TIME_GET 1

MT User Mode

In MT User mode, the RPC library does not create any threads. This mode works, in principle, like the single-threaded, or default mode. The only difference is that it passes copies of data structures, such as the transport service handle to the service-dispatch routine to be MT safe.

The RPC server developer takes the responsibility for creating and managing threads through the thread library. In the dispatch routine, the service developer can assign the task of procedure execution to newly created or existing threads. The thr_create() API is used to create threads having various attributes. All thread library interfaces are defined in thread.h. See the pthread_create(3THR) man page for more details.

This mode provides flexibility to the service developer. Threads can now have different stack sizes based on service requirements. Threads can be bound. Different procedures can be executed by threads with different characteristics. The service developer might choose to run some services single threaded. The service developer might choose to do special thread-specific signal processing.

As in the Auto mode, you use the rpc_control() library call to turn on User mode. Note that the rpc_control() operations shown in Table 7–1, except for RPC_SVC_MTMODE_GET(), apply only to MT Auto mode. If used in MT User mode or the single-threaded default mode, the results of the operations can be undefined.

Freeing Library Resources in User Mode

In the MT User mode, service procedures must invoke svc_done() before returning. svc_done() frees resources allocated to service a client request directed to the specified service transport handle. This function is invoked after a client request has been serviced, or after an error or abnormal condition that prevents a reply from being sent. After svc_done() is invoked, the service procedure should not reference the service transport handle. The following example shows a server in MT User mode.

Note –

svc_done() must only be called within MT User mode. For more information on this call, see the rpc_svc_calls(3NSL) man page.

Example 7–3 MT User Mode: rpc_test.h 

#define	SVC2_PROG 0x30000002
	#define	SVC2_VERS 1
	#define SVC2_PROC_ADD 1)
	#define SVC2_PROC_MULT 2

	struct intpair {
		u_short	a;
		u_short	b;
	};

	typedef struct intpair intpair;

	struct svc2_add_args {
		int argument;
		SVCXPRT *transp;
	};

	struct svc2_mult_args {
		intpair mult_argument;
		SVCXPRT *transp;
	};

	extern bool_t xdr_intpair();

	#define NTHREADS_CONST 500

The following code example is the client for MT User mode.

Example 7–4 Client for MT User Mode

#define	 _REENTRANT
#include <stdio.h>

#include <rpc/rpc.h>
#include <sys/uio.h>

#include <netconfig.h>
#include <netdb.h>

#include <rpc/nettype.h>
#include <thread.h>
#include "rpc_test.h"
void *doclient();
int NTHREADS;
struct thread_info {
	thread_t client_id;
	int client_status;
};
struct thread_info save_thread[NTHREADS_CONST];
main(argc, argv)
	int argc;
	char *argv[];
{
	int index, ret;
	int thread_status;
	thread_t departedid, client_id;
	char *hosts;
	if (argc < 3) {
		printf("Usage: do_operation [n] host\n");
		printf("\twhere n is the number of threads\n");
		exit(1);
	} else
		if (argc == 3) {
			NTHREADS = NTHREADS_CONST;
			hosts = argv[1];  /* live_host */
		} else {
			NTHREADS = atoi(argv[1]);
			hosts = argv[2];
		}
	for (index = 0; index < NTHREADS; index++){
		if (ret = thr_create(NULL, NULL, doclient,
		(void *)  hosts, THR_BOUND, &client_id)){
			printf("thr_create failed: return value %d", ret);
			printf(" for %dth thread\n", index);
			exit(1);
		}
		save_thread[index].client_id = client_id;
	}
	for (index = 0; index < NTHREADS; index++){

		if (thr_join(save_thread[index].client_id, &departedid,	
		(void *)
		&thread_status)){
			printf("thr_join failed for thread %d\n",
			save_thread[index].client_id);
			exit(1);
		}
		save_thread[index].client_status = thread_status;
	}
}
	void *doclient(host)
	char *host;
{
	struct timeval tout;
	enum clnt_stat test;
	int result = 0;
	u_short mult_result = 0;
	int add_arg;
	int EXP_RSLT;
	intpair pair;
	CLIENT *clnt;
	if ((clnt = clnt_create(host, SVC2_PROG, SVC2_VERS, "udp"
==NULL) {
		clnt_pcreateerror("clnt_create error: ");
		thr_exit((void *) -1);
	}
	tout.tv_sec = 25;
	tout.tv_usec = 0;
	memset((char *) &result, 0, sizeof (result));

	memset((char *) &mult_result, 0, sizeof (mult_result));
	if (thr_self() % 2){
		EXP_RSLT = thr_self() + 1;
		add_arg = thr_self();
		test = clnt_call(clnt, SVC2_PROC_ADD, (xdrproc_t) xdr_int,

		(caddr_t) &add_arg, (xdrproc_t) xdr_int, (caddr_t) &result,
		tout);
	} else {
		pair.a = (u_short) thr_self();
		pair.b = (u_short) 1;
		EXP_RSLT = pair.a * pair.b;
		test = clnt_call(clnt, SVC2_PROC_MULT, (xdrproc_t)
		xdr_intpair,
		(caddr_t) &pair, (xdrproc_t) xdr_u_short,

		(caddr_t) &mult_result, tout);
		result = mult_result;
	}
	if (test != RPC_SUCCESS) {
		printf("THREAD: %d clnt_call hav
		thr_exit((void *) -1);
	};
	thr_exit((void *) 0);
}

MT performance is enhanced if the function svc_getargs() is called by every procedure other than NULLPROC, even if there are no arguments. xdr_void may be used in this case. This result is true for both the MT Auto and MT User modes. For more information on this call, see the rpc_svc_calls(3NSL) man page.

Note –

You must link in the thread library when writing RPC multithreaded-safe applications. The thread library must be the last named library on the link line. Specify the -lthread option in the compile command.