This chapter deals with the aspects of messaging and interprocess communications that are native to ChorusOS systems. It also demonstrates the use of the application programming interfaces for handling message queues and IPC.
After implementing the examples provided in this chapter, you should understand how these APIs can be used in an application.
The MIPC
API is an optional feature that facilitates
message queues in ChorusOS systems. This feature enables an application,
composed of one or more actors, to create a shared communication environment --
often referred to as a message space. Within this message space, actors can
exchange messages efficiently. Supervisor and user actors of the same application
can use MIPC
to exchange messages. Furthermore, messages
may be initially allocated and sent by interrupt handlers to be processed
later by threads.
The MIPC
API includes the following system
calls:
Allocate a message
Free a message
Get a message
Get a message from any declared queues
Retrieve statistics about a message pool
Post a message
Retrieve statistics about a message queue
Remove a message from a queue
Create a message space
Open a message space.
Message queues are designed around the concept of a message space which encapsulates within a single entity:
A set of message pools shared by all actors of the application.
A set of message queues through which these actors exchange messages allocated from the shared message pools.
A message space is a temporary resource that must be created explicitly by one actor within the application. Once created, a message space can be opened by other actors within the application. Actors that have opened the same message space are said to share the message space. A message space is deleted automatically when the actor is deleted.
A message pool is defined by two parameters (message size and number of messages) provided by the application when it creates the message space. The configuration of the set of message pools defined within a message space depends on the application requirements.
A message is an array of bytes which can be structured and used at application level through any appropriate convention. Messages are presented to actors as pointers within their address space.
Messages are posted to message queues belonging to the same message space. All actors sharing a message space can allocate messages from the message pools. In the same way, all actors sharing a message space have send and receive rights on each queue of the message space.
Even though most applications need to create only a single message space, the message queue feature is designed to enable an application to create or open multiple message spaces. However, messages allocated from one message space cannot be sent to a queue of a different message space. The following example illustrates the typical use of message spaces:
The first actor, aware of the overall requirements of the application, creates the message space.
Other actors of the application open the shared message space.
An actor allocates a message from a message pool, and fills it with the data to be sent.
The actor which allocated the message then posts it to the appropriate queue and assigns a priority to the message.
The destination actor obtains the message from the queue. At this point, the message is removed from the queue.
After the destination actor has processed the message, the message can be freed. It is then available to be reallocated by the application. Alternatively, the destination actor may modify the message and post it to another queue.
To make the service as efficient as possible, physical memory is allocated for all messages and data structures of the message space at message space creation. At message space open time, this memory is transparently mapped by the system into the actor address space. Further operations such as posting and receiving a message are performed without any copy.
You can create a message space as follows:
#include <mipc/chMipc.h> int msgSpaceCreate (KnMsgSpaceId spaceGid, unsigned int msgQueueNb, unsigned int msgPoolNb, const KnMsgPool* msgPools);
The spaceGid parameter is an unique global identifier assigned by the application to the message space being created. This identifier is also used by other actors of the application to open the message space. Therefore, the identifier serves to bind actors participating in the application to the same message space. The K_PRIVATEID predefined global identifier indicates that the message space created will be private to the invoking actor. This means that its queues and message pools will only be accessible to threads executing within this actor. No other actor will be able to open the message space. The message space is described by the following three parameters:
msqgQueueNb indicates how many queues must be created within the message space. Each queue in the message space is then designated by its index within the set of queues. This may vary from 0 to msgQueueNb - 1.
msgPoolNb is the number of message pools to be created in the message space.
msgPools is a pointer to an array of msgPoolNb pool descriptions. Each pool is described by a KnMsgPool data structure, which includes the following information:
msgSize, which defines the size of each message belonging to the pool
msgNumber, which defines how many messages of msgSize bytes must be created in this pool
Figure 11-1 shows an example of a message space recently created by an actor.
The created message space is assigned a local identifier which is returned to the invoking actor as the return value of the msgSpaceCreate(2K). The scope of this local identifier is the invoking actor.
A message space may be opened by other actors through the following call:
#include <mipc/chMipc.h> int msgSpaceOpen(KnMsgSpaceId spaceGid);
The message space assigned with the spaceGid unique global identifier must have been created previously by a call to msgSpaceCreate(2K). A local identifier is returned to the invoking actor. This message space local identifier can then be used to manipulate messages and queues within the message space. Figure 11-2 shows an example of a message space recently opened by a second actor.
A message may be allocated by the following call:
#include <mipc/chMipc.h> int msgAllocate(int spaceLid, unsigned int poolIndex, unsigned int msgSize, KnTimeVal* waitLimit, char** msgAddr);
msgAllocate(2K) attempts to allocate a message from the appropriate pool of the message space identified by the spaceLid return value of a previous call to msgSpaceOpen(2K), or to msgSpaceCreate(2K). If poolIndex is not set to K_ANY_MSGPOOL, the allocated message is the first free (unallocated) message of the pool defined by poolIndex, regardless of the value specified by the msgSize parameter. Otherwise, if poolIndex is set to K_ANY_MSGPOOL, the message is allocated from the first pool for which the message size fits the requested msgSize. In this context, first pool means the pool with the lowest index in the set of pools defined at message space creation time. If the pool is empty, no attempt is made to allocate a message from another pool.
If the message pool is empty (all messages have been allocated and none are free), msgAllocate(2K) will block, waiting for a message in the pool to be freed. The invoking thread is blocked until the wait condition defined by waitLimit expires.
If successful, the address of the allocated message is stored at the location defined by msgAddr. The returned address is the address of the message within the address space of the actor. Remember that a message space is mapped to the address space of the actors sharing it. However, the message spaces (and therefore the messages themselves), may be mapped to different addresses in different actors. This is especially true for message spaces shared between supervisor and user actors.
Figure 11-3 illustrates two actors allocating two messages from two different pools of the same message space.
After a message has been allocated and initialized by the application, it may be posted to a message queue with:
#include <mipc/chMipc.h> int msgPut(int spaceLid, unsigned int queueIndex, char* msg, unsigned int prio);
msgPut(2K) posts the message (the address of which is msg) to the message queue queueIndex within the message space (the local identifier of which is spaceLid). The message has already been allocated previously by a call to msgAllocate(2K). The message is inserted into the queue according to its priority, prio. Messages with a high priority are taken from the queue first.
A message is posted to a queue without any message copy. Additionally, the message may be posted within an interrupt handler, or with preemption disabled.
Figure 11-4 illustrates how the actors from the previous figure post their messages to different queues.
To obtain a message from a queue (if any) use the following call:
#include <mipc/chMipc.h> int msgGet(int spaceLid, unsigned int queueIndex, KnTimeVal* waitLimit, char** msgAddr, KnUniqueId* srcActor);
msgGet(2K) enables the invoking thread to get the first message with the highest priority pending in the message queue queueIndex, within the message space whose local identifier is spaceLid. Messages with equal priority are posted and delivered in a first-in first-out order.
The address of the message delivered to the invoking thread is returned at the location defined by the msgAddr parameter. If no message is pending, the invoking thread is blocked until a message is sent to the message queue, or until the time-out (as defined by the expiration of the waitLimit parameter).
If the srcActor parameter is non-NULL, it points to a location where the unique identifier of the actor is to be stored. The actor that posted the message is referred to as the source actor).
No data copy is performed to deliver the message to the invoking thread. Multiple threads can be blocked, waiting in the same message queue. With the msgGetAny() call it is possible for one thread to wait for message arrival on all message queues, and to obtain the first posted message from any queue. It is not possible to define a subset of queues and to wait for a message to arrive on this subset. Consequently, msgGet() allows you to wait for a message on a queue described by an index held in its second argument, and msgGetAny() allows you to wait for a message from all queues.
The msgGetAny() call is defined as follows:
int msgGetAny(int spaceLid, unsigned int* msgQueueId, KnTimeVal* waitLimit, char** msgAddr, KnUniqueId* srcActor);
Figure 11-5 illustrates previously created actors receiving messages from queues.
A message that is of no further use to the application can be returned to its pool of messages and will be available for reallocation with the following call:
#include <mipc/chMipc.h> int msgFree(int spaceLid, char* msg);
To retrieve information about a specific message queue or message pool, use the following call:
#include <mipc/chMipc.h> int msgPoolStat (int spaceLid, unsigned int msgPoolId, KnMsgPoolStat* stat); int msgQueueStat(int spaceLid, unsigned int msgQueueId, KnMsgQueueStat* stat);
Information about a message pool is stored in the KnMsgPoolStat structure which contains:
the sizes of the messages in the pool.
the number of messages in the pool.
the number of free messages in the pool (messages that can be allocated using msgAllocate).
Information about a message queue is stored in the KnMsgQueueStat structure which contains:
the number of messages currently waiting in the queue.
msgFree and msgNumber may be invalid after the next time slice or thread schedule because other threads or applications have access to the message space.
Example 11-1 illustrates the basic use of the message queue feature. The example uses the posix_spawn() system call and demonstrates how MIPC can be used with a POSIX process.
For more information, refer to the msgSpaceCreate(2K), msgSpaceOpen(2K), msgAllocate(2K), msgPut(2K), msgGet(2K) and msgFree(2K) man pages.
(file: opt/examples/progov/msgSpace.c) #include <stdio.h> #include <stdlib.h> #include <string.h> #include <errno.h> #include <chorus.h> #include <mipc/chMipc.h> #include <spawn.h> char* spawn_args[3]; #define NB_MSG_POOLS 2 #define NB_MSG_QUEUES 3 #define SMALL_MSG_SZ 32 #define LARGE_MSG_SZ 256 #define NB_SMALL_MSG 13 #define NB_LARGE_MSG 4 #define SAMPLE_SPACE 1111 #define LARGE_POOL 0 #define SMALL_POOL 1 #define Q1 0 #define Q2 1 #define Q3 2 KnMsgPool samplePools[NB_MSG_POOLS]; char* tagPtr = "Spawned"; int main(int argc, char** argv, char**envp) { int res; int msgSpaceLi; char* smallMsg; char* smallReply; char* largeMsg; KnActorPrivilege actorP; res = actorPrivilege(K_MYACTOR, &actorP, NULL); if (res != K_OK) { printf("Cannot get actor privilege, error %d\n", res); exit(1); } if (argc == 1) { /* * This is the first actor (or spawning actor): * Create a message space, * Spawn another actor, * Allocate, modify and post a small message on Q2 * Get a large Message from Q3, print its contents, free it * Get reply of small message on Q1, print its contents, free it. */ samplePools[LARGE_POOL].msgSize = LARGE_MSG_SZ; samplePools[LARGE_POOL].msgNumber = NB_LARGE_MSG; samplePools[SMALL_POOL].msgSize = SMALL_MSG_SZ; samplePools[SMALL_POOL].msgNumber = NB_SMALL_MSG; msgSpaceLi = msgSpaceCreate(SAMPLE_SPACE, NB_MSG_QUEUES, NB_MSG_POOLS, samplePools); if (msgSpaceLi < 0) { printf("Cannot create the message space error %d\n", msgSpaceLi); exit(1); } /* * Message Space has been created, spawn the other actor, * argv[1] set to "Spawned" to differentiate the 2 actors. */ spawn_args[0] = argv[0]; spawn_args[1] = tagPtr; spawn_args[2] = NULL; res = posix_spawnp(NULL, spawn_args[0], NULL, NULL, spawn_args, NULL); if (res < 0) { printf("Cannot spawn second actor, error %d\n", res); exit(1); } /* * Allocate a small message */ res = msgAllocate(msgSpaceLi, SMALL_POOL, SMALL_MSG_SZ, K_NOTIMEOUT, &smallMsg); if (res != K_OK) { printf("Cannot allocate a small message, error %d\n", res); exit(1); } /* * Initialize the allocated message */ strncpy(smallMsg, "Sending a small message\n", SMALL_MSG_SZ); /* * Post the allocated small message to Q2 with priority 2 */ res = msgPut(msgSpaceLi, Q2, smallMsg, 2); if (res != K_OK) { printf("Cannot post the small message to Q2, error %d\n", res); exit(1); } /* * Get a large message from Q3 and print its contents */ res = msgGet(msgSpaceLi, Q3, K_NOTIMEOUT, &largeMsg, NULL); if (res != K_OK) { printf("Cannot get the large message from Q3, error %d\n", res); exit(1); } printf("Received large message contains:\n%s\n", largeMsg); /* * Free the received large message */ res = msgFree(msgSpaceLi, largeMsg); if (res != K_OK) { printf("Cannot free the large message, error %d\n", res); exit(1); } /* * Get the reply to small message from Q1 and print its contents */ res = msgGet(msgSpaceLi, Q1, K_NOTIMEOUT, &smallReply, NULL); if (res != K_OK) { printf("Cannot get the small message reply from Q1, " "error %d\n", res); exit(1); } printf("Received small reply contains:\n%s\n", smallReply); /* * Free the received small reply */ res = msgFree(msgSpaceLi, smallReply); if (res != K_OK) { printf("Cannot free the small reply message, error %d\n", res); exit(1); } } else { /* * This is the spawned actor: * Check we have effectively been spawned * Open the message space * Allocate, initialize and post a large message to Q3 * Get a small message from Q2, print its contents * Modify it and repost it to Q1 */ int l; if ((argc != 2) || (strcmp(argv[1], tagPtr) != 0)) { printf("%s does not take any argument!\n", argv[0]); exit(1); } /* * Open the message space, using the same global identifier */ msgSpaceLi = msgSpaceOpen(SAMPLE_SPACE); if (msgSpaceLi < 0) { printf("Cannot open the message space error %d\n", msgSpaceLi); exit(1); } /* * Allocate the large message */ res = msgAllocate(msgSpaceLi, K_ANY_MSGPOOL, LARGE_MSG_SZ, K_NOTIMEOUT, &largeMsg); if (res != K_OK) { printf("Cannot allocate a large message, error %d\n", res); exit(1); } strcpy(largeMsg, "Sending a very large large large message\n"); /* * Post the large message to Q3 with priority 0 */ res = msgPut(msgSpaceLi, Q3, largeMsg, 0); if (res != K_OK) { printf("Cannot post the large message to Q3, error %d\n", res); exit(1); } /* * Get the small message from Q2 */ res = msgGet(msgSpaceLi, Q2, K_NOTIMEOUT, &smallMsg, NULL); if (res != K_OK) { printf("Cannot get the small message from Q2, error %d\n", res); exit(1); } printf("Spawned actor received small message containing:\n%s\n", smallMsg); for (l = 0; l < strlen(smallMsg); l++) { if ((smallMsg[l]>= 'a') && (smallMsg[l] <= 'z')) { smallMsg[l] = smallMsg[l] - 'a' + 'A'; } } /* * Post the small message back to Q1, with priority 4 */ res = msgPut(msgSpaceLi, Q1, smallMsg, 4); if (res != K_OK) { printf("Cannot post the small message reply to Q1, error %d\n", res); exit(1); } } return 0; }
Two actors are used, one spawned by the other. The first actor:
Creates a message space with two pools of messages and three message queues (as shown in Figure 11-5).
Allocates a small message, initializes it and posts it to a queue.
Waits for a large message on a second queue, prints its contents and deallocates it.
Waits for the small message to come back on a third queue, prints its contents, deallocates it, then terminates.
During this time, the second actor:
Opens the message space, allocates a large message to be initialized and sends it to the first actor.
Receives the small message, converts all lower case characters to upper case, and posts it back to the third queue before terminating.
The IPC
feature enables threads to communicate
and synchronize when they do not share memory, for example, when they do not
run on the same node. Communications rely on the exchange of messages through
ports.
The IPC
feature includes a number of APIs. These are listed in the following table.
API |
Purpose |
---|---|
actorPi() |
Modify the PI of an actor |
portCreate() |
Create a port |
portDeclare() |
Declare a port |
portDelete() |
Destroy a port |
portDisable() |
Disable a port |
portEnable() |
Enable a port |
portGetSeqNum() |
Get a port sequence number |
portLi() |
Acquire the LI of a port |
portMigrate() |
Migrate a port |
portPi() |
Modify the PI of a port |
portUi() |
Acquire the UI of a port |
grpAllocate() |
Allocate a group name |
grpPortInsert() |
Insert a port into a group |
grpPortRemove() |
Remove a port from a group |
ipcCall() |
Send synchronously |
ipcGetData() |
Get the current message body |
ipcReceive() |
Receive a message |
ipcReply() |
Reply to the current message |
ipcRestore() |
Restore a message as the current message |
ipcSave() |
Save the current message |
ipcSend() |
Send asynchronously |
ipcSysInfo() |
Get information about the current message |
ipcTarget() |
Construct an address |
svMsgHandler() |
Connect a message handler |
svMsgHdlReply() |
Prepare a reply to a handled message |
The IPC feature enables threads to exchange messages in either asynchronous or demand/response mode. The demand/response mode is also known as a Remote Procedure Call (RPC).
In asynchronous mode, the sender of an asynchronous message is only blocked during the time of local processing of the message. The system does not guarantee that the message has been deposited at the destination location.
In RPC mode, the RPC protocol enables the construction of client-server applications using a demand/response protocol for the management of transactions. The client is blocked until a response is returned from the server, or a user-defined (optional) timeout occurs. RPC ensures at-most-once semantics for the delivery of the request. It also ensures that the response received by a client originates from the server. The response received by the client must also correspond to the request (and not to a former request to which the response might have been lost).
RPC also enables a client to be unblocked (which will generate an error) if the server is unreachable or has crashed before emitting a response.
Finally, the RPC protocol supports abort propagation. When a thread that is waiting for an RPC reply is aborted, the event will be propagated to the thread which is currently servicing the client request.
A thread that attempts to receive a message on a port is blocked until the new message is received or a until user-defined (optional) time-out occurs.
A thread can attempt to receive a message on several ports at a time. Among the set of ports attached to an actor, a subset of enabled ports is defined. A thread can also attempt to receive a message sent to any of the enabled ports in its actors.
Ports attached to an actor can be dynamically enabled or disabled. When a port is enabled, it receives a priority value. If several of the enabled ports hold a message when a thread attempts to receive the message on the enabled set of ports, the port with the highest priority will be selected.
An actor's default port is not automatically enabled. If a port has not been enabled, it is automatically disabled. This does not mean that the port cannot be used to send or to receive messages. It means that the port cannot be used in multiple port receive requests because the default value is disabled.
The conventional way for an actor to receive messages delivered to its ports is to have threads explicitly express receive requests. An alternative to this method is the use of message handlers. Instead of explicitly creating threads, an actor can attach a handler (a routine in its address space) to the port. When a message is delivered to the port, the handler is executed within the context of a thread provided by the microkernel.
Message handlers and explicit receive requests are exclusive. When a message handler has been attached to a port, any attempt by a thread to receive a message on that port will return an error.
The use of message handlers is restricted to supervisor actors. Message handlers enable significant optimization of the RPC protocol when both client and server reside on the same site, thereby avoiding thread context switches and memory copies. From the point of view of the microkernel, the client thread is used to run the handler and copying of the message into microkernel buffers is avoided.
The method in which messages are consumed (threads or handlers) is completely transparent to the client (the sender of the message). The strategy is selected on the server only.
The IPC_REMOTE feature enables support for communication among multiple sites in a network, using a location-transparent communication feature. Without this feature, IPC services may be used only within a single site.
The following example illustrates the use of the IPC mechanisms provided in the ChorusOS operating system.
#include <stdio.h> #include <stdlib.h> #include <string.h> #include <errno.h> #include <chorus.h> #include <spawn.h> #include "ipc/chIpc.h" #define ABORT_DELAY 1000 /* Delay for ipcReceive */ static KnUniqueId thePortUi; /* Our port unique identifier */ static int thePortLi; /* ....... local identifier */ static KnCap groupCap; /* Our group capability */ /* The outgoing annex and body */ static char sndAnnex[] = "Hello world from Chorus ...\n"; static char sndBody[] = "The sea is calm, the tide is full ...\n"; /* The received annex and body */ static char rcvAnnex[K_CMSGANNEXSIZE]; static char rcvBody[1000]; /* Argumentf for the spawnd process */ static char* spawn_args[3]; /* Port group stamp */ char* stamp; #define MYSTAMP 100 #define STAMPSIZE 10 int main(int argc, char** argv, char** envp) { int rslt; /* Work */ KnMsgDesc smsg; /* Descriptor for message being sent */ KnMsgDesc rmsg; /* Descriptor for message being received */ KnIpcDest ipcDest; /* IPC address */ if (argc == 1) { /* * Server actor * Create the destination port */ thePortLi = portCreate(K_MYACTOR, &thePortUi); if (thePortLi < 0) { printf("portCreate failed, returns %d\n", thePortLi); exit(1); } /* * Allocate a port group and insert the port into it */ rslt = grpAllocate(K_STATUSER, &groupCap, MYSTAMP); if (rslt < 0) { printf("grpAllocate failed, returns %d\n", rslt); exit(1); } rslt = grpPortInsert(&groupCap, &thePortUi); if (rslt < 0) { printf("grpPortInsert failed, returns %d\n", rslt); exit(1); } /* * Spawn the client actor * The group stamp is given in argument * Exit in case of malloc failure */ stamp = malloc(STAMPSIZE); if (stamp == NULL) { printf("Can't allocate %d bytes\n", STAMPSIZE); exit(1); } sprintf(stamp, "%d", MYSTAMP); spawn_args[0] = argv[0]; spawn_args[1] = stamp; spawn_args[2] = NULL; rslt = posix_spawnp(NULL, spawn_args[0], NULL, NULL, spawn_args, NULL); if (rslt < 0) { printf("Cannot spawn client actor, error %d\n", rslt); exit(1); } /* * Receive the message */ rmsg.flags = 0; rmsg.bodySize = sizeof(rcvBody); rmsg.bodyAddr = (VmAddr)rcvBody; rmsg.annexAddr = (VmAddr)rcvAnnex; rslt = ipcReceive(&rmsg, &thePortLi, ABORT_DELAY); if (rslt < 0) { printf("ipcReceive failed, returns %d\n", rslt); exit(1); } printf ("%s\n%s\n", rcvAnnex, rcvBody); rslt = portDelete(K_MYACTOR, thePortLi); if (rslt < 0) { printf("portDelete failed, returns %d\n", rslt); exit(1); } } else { /* * Get the port group capability giving the stamp. * Stamp has been received in argv[1] */ rslt = grpAllocate(K_STATUSER, &groupCap, (int) atoi(argv[1])); if (rslt < 0) { printf("grpAllocate failed, returns %d\n", rslt); exit(1); } /* * Prepare the message descriptor for the message to send */ smsg.flags = 0; smsg.bodySize = sizeof(sndBody); smsg.bodyAddr = (VmAddr)sndBody; smsg.annexAddr = (VmAddr)sndAnnex; /* * Prepare the IPC address for the message destination. * Send the message in broadcast mode. */ ipcDest.target = groupCap.ui; rslt = ipcTarget(&ipcDest.target, K_BROADMODE); if (rslt < 0) { printf("ipcTarget failed, returns %d\n", rslt); exit(1); } /* Send from our DEFAULT port */ rslt = ipcSend(&smsg, K_DEFAULTPORT, &ipcDest); if (rslt < 0) { printf("ipcSend failed, returns %d\n", rslt); exit(1); } } return 0; }
In this example, the main thread (which is implicitly created):
Creates a port
Creates a port group and inserts the port into it
Spawns another copy of itself, using posix_spawn(), and passes the port group stamp as an argument
Waits for a message on the created port
Prints the contents of the body and the annex
Frees all used resources and terminates
The spawned actor:
Retrieves the port group capability, passing the stamp received as an argument
Prepares and sends a message to this group in broadcast mode (the annex and body of the message are initialized with strings)
Terminates
The following comparison will help you decide which APIs to use for your specific application.
The MIPC service provides local communication only. MIPC has the following major benefits:
MIPC system calls are real-time compliant.
Messages are exchanged through a zero-copy interface.
Messages can be allocated and posted from interrupt handlers.
The MIPC service is therefore highly suited to real-time applications.
The IPC subsystem provides location-transparent, message-based communication services between applications. Two communication models are provided:
The RPC model (see the ipcCall(2K) ipcReceive(2K) and ipcReply(2K) man pages).
Asynchronous communication (see the ipcSend(2K) and ipcReceive(2K) man pages).
Although the IPC service implements fine-grained locking policies (which makes its services highly preemptable) it cannot enforce real-time constraints. IPC services are therefore intended specifically for distributed applications.