| Netra Data Plane Software Suite 1.1 User's Guide |    
|
| Netra Data Plane Software Suite 1.1 User's Guide |
|
| C H A P T E R 6 |
|
Interprocess Communication Software |
This chapter describes the interprocess communication (IPC) software. Topics include:
The interprocess communication (IPC) mechanism provides a means to communicate between processes that run in a domain under the NDPS Lightweight Runtime Environment (LWRTE) and processes in a domain with a control plane operating system. This chapter describes the generic APIs that are available to use the IPC mechanism, as well as the specific APIs and configuration interfaces used to operate in an LDoms environment with LWRTE and Solaris software.
The API described in this section is available on all operating environments that support IPC communications with LWRTE domain. The tnipc.h header located in the src/common/include directory of the SUNWndps package defines the interface and must be included in source files using the API. The header file defines a number of IPC protocol types. User-defined protocols must not be in conflict with these predefined types.
This function registers a consumer with an IPC channel. The opaque handle that is returned by a successful call to this function must be passed to access the channel using any of the other interface functions.
ipc_handle_t
ipc_connect(uint16_t channel, uint16_t ipc_proto)
channel: ID of channel
ipc_proto: Protocol type of IPC messages that are expected
NULL in case of failure
IPC handle otherwise. This handle needs to be passed to the tx/rx/free functions.
This function registers callback functions for the consumer of an IPC channel. When a message is received by the IPC framework, it strips the IPC header from the message and calls the rx_hdlr function with the content of the message. In future releases, the evt handler may be used to convey link events to the consumer.
event_handler_ft evt_hdler,
rx_handler_ft rx_hdler,
caddr_t arg)
ipc_hdl: Handle for IPC channel, obtained from ipc_register_callbacks().
evt_hdlr: Function to handle link events.
rx_hdlr: Function to handle received messages.
arg: Opaque argument that the framework will pass back to the handler functions.
IPC_SUCCESS
EFAULT invalid handle
This function transmits messages over IPC. The message is described by the mblk passed to the function. To make the function as efficient as possible, the function makes some nonstandard assumptions about the messages:
As the memory containing the message is not freed inside the function, the caller must deal with memory management accordingly.
int
ipc_tx(mblk_t *mp, ipc_handle_t ipc_hdl)
mp: Pointer to message block describing the messages.
ipc_hdl: Handle for IPC channel, obtained from ipc_register_callbacks().
EIO The write to the underlying media failed.
At this time, the only way to receive messages is through the callback function. In LWRTE, the callback function is called when the polling context finds a message on the channel. In Solaris user space, the callback is hidden in the framework, it makes the message available to be read by the read() system call.
The IPC framework allocates memory for messages that are received using its available memory pools. The consumer of an IPC message must call this function to return the memory to that pool.
void
ipc_free(mblk_t *mp, ipc_handle_t ipc_hdl)
mp Pointer to message block describing message to be freed.
ipc_hdl Handle for IPC channel, obtained from ipc_register_callbacks().
In the LWRTE domain, the interfaces described in the Programming Interfaces are used to communicate with other domains using IPC. Before this infrastructure can be utilized, you must initialize it. Once it is initialized, because there are no interrupts, you must ensure that every channel is polled periodically. This section describes the API for these tasks.
To use this function, the lwrte_ipc_if.h header file, which is located in the lib/ipc/include directory of the SUNWndps package, must be included where needed.
This function must be called in the initialization routine. This function must be called after the LDoms framework has been initialized, that is, mach_descrip_init(), lwrte_cnex_init(), and lwrte_init_ldc() must be called first.
0 Success
EFAULT Too many channels in machine description
ENOENT Global configuration channel not defined
To receive messages or event notifications for any IPC channel, this function must be called periodically. For example, it may be called as part of the main loop in the statistics thread. When a message is received, this ensures that the callback function registered for the channel and IPC type is called.
This function always returns 0.
Polling through the tnipc_poll() API is adequate for most IPC channels carrying low bandwidth control traffic. For higher throughput channels, the polling can be moved to a separate strand, using the following API functions:
This function removes the channel identified by the handle passed to the function from the pool of channels polled by the tnipc_poll() function. This function returns an opaque handle that must be passed to the tnipc_local_poll() function.
ipc_poll_handle_t
tnipc_register_local_poll(ipc_handle_t ipc_hdl)
ipc_hdl The channel handle obtained from the ipc_connect() API call.
NULL invalid input
Opaque handle to be passed to the tnipc_local_poll() call.
This function works the same way as tnipc_poll(), except that only the channel identified by the handle is polled. If there is data on the channel, the rx callback will be called.
int
tnipc_local_poll(ipc_poll_handle_t poll_hdl)
poll_hdl The handle obtained from the tnipc_register_local_poll() API call
This function always returns 0.
This function reverses the effect of the tnipc_register_local_poll() call and places the channel identified by the handle back into the common pool polled by tnipc_poll().
int
tnipc_unregister_local_poll(ipc_poll_handle_t poll_hdl)
poll_hdl The handle obtained from the tnipc_register_local_poll() API call
This function always returns 0.
In Solaris software, there are two different ways to use the IPC API, from user space and from kernel space.
To use an IPC channel from the Solaris user space, the character-driver interfaces are used. A program opens the tnsm device, issues an ioctl() call to connect the device to a particular channel, and then uses read() and write() calls to send and receive messages.
Before any of the interfaces can be used, the tnsm driver must be installed and loaded. This is done using the pkgadd system administration command to install the SUNWndpsd package on the Solaris domains that use IPC for communication.
The open(), close(), read(), and write() interfaces are described in their respective man pages.
The open() call on the tnsm driver will create a new instance for the specific client program. Before you can use the read() and write() calls, you must call TNIPC_IOC_CH_CONNECT ioctl. This ioctl takes the channel ID and IPC type to be used for messages by this instance.
In the kernel, the interfaces described in Programming Interfaces are used.
This section describes the configuration of the environment needed to use the IPC framework. This section covers setup of memory pools for the LWRTE application, the LDoms environment, and the IPC channels.
The IPC framework shares its memory pools with the basic LDoms framework. These pools are accessed through malloc() and free() functions that are implemented in the application. The ipfwd_ldom reference application contains an example implementation.
The file ldc_malloc_config.h contains definitions of the memory pools and their sizes. ldc_malloc.c contains the implementation of the malloc() and free() routines. These functions have the expected signatures:
In addition to these implementation files, the memory pools must be declared to the NDPS runtime. This is done in the software architecture definition in ipfwd_swarch.c.
In the LDoms environment, the IPC channels use Logical Domain Channels (LDCs) as their transport media. These channels are set up as Virtual Data Plane Channels using the ldm command (see the LDoms documentation). These channels are set up between a server and a client, and some basic configuration channels must be defined adhering to the naming convention described in LDoms Channel Setup. Each channel has a server defined in the LWRTE domain and a client defined in the link partner domain.
There must be a domain that has the right to set up IPC channels in the LWRTE domain. This domain can be the primary domain or a guest domain with the client for the configuration service. The administrator must only setup this channel, when the service (LWRTE) and the client domain are up (and the tnsm driver attached at the client), the special IPC channel with ID 0 is established automatically between the devices. The tnsmctl utility can then be used in the configuring domain to set up additional IPC channels (provided that the required virtual data plane channels have been configured.)
To enable IPC communications between the LWRTE domain and additional domains, a special configuration channel must be set up between these domains. Again, the channel names must adhere to a naming convention. In the LWRTE domain, the service name must begin with the prefix config-tnsm, whereas the client name in the other domain must be named config-tnsm0. For example, such a channel could be established using the ldm commands.
Additional channels can be added for data traffic between these domains, there are no naming conventions to follow for these channels, they are configured using the ldm commands.
Once the data plane channels are set up by the administrator in the primary domain, the tnsmctl utility is used to set up IPC channels from the IPC control domain. tnsmctl uses the following syntax:
The parameters to tnsmctl are described in TABLE 6-1. All of these parameters need to be present to set up an IPC channel.
The following is a sample environment, complete with all commands needed to set it up in a Sun Fire
T2000 server.
TABLE 6-1 describes the four environment domains:
The primary as well as the guest domains ldg2 and ldg3 run the Solaris 10
Update 3 Operating System (or higher) with the patch level required for LDoms operation. The SUNWldm package is installed in the primary domain, the SUNWndpsd package is installed in both ldg2 and ldg3.
Assuming 4 GB of memory for each of the domains, and starting with the factory default configuration, the environment can be set up using the following domain commands:
ldm remove-mau 8 primary
ldm remove-vcpu 28 primary
ldm remove-mem 28G primary (This assumes 32GB of total memory, adjust accordingly.)
ldm remove-io bus_b primary
ldm add-vsw mac-addr=you-mac-address net-dev=e1000g0 primary-vsw0
primary
ldm add-vds primary-vds0 primary
ldm add-vcc port-range=5000-5100 primary-vcc0 primary
ldm add-spconfig 4G4Csplit
ldm add-domain ldg1
ldm add-vcpu 16 ldg1
ldm add-mem 4G ldg1
ldm add-vnet mac-addr=your-mac-address-2 vnet0 primary-vsw0 ldg1
ldm add-var auto-boot\?=false ldg1
ldm add-io bus_b ldg1
ldm add-domain ldg2
ldm add-vcpu 4 ldg2
ldm add-mem 4G ldg2
ldm add-vnet mac-addr=your-mac-address-3 vnet0 primary-vsw0 ldg2
ldm add-vdsdev your-disk-file vol2@primary-vds0
ldm add-vdisk vdisk1 vol2@primary-vds0 ldg2
ldm add-var auto-boot\?=false ldg2
ldm add-var boot-device=/virtual-devices@100/channel-devices@200/disk@0 ldg2
ldm add-domain ldg3
ldm add-vcpu 4 ldg3
ldm add-mem 4G ldg3
ldm add-vnet mac-addr=your-mac-address-4 vnet0 primary-vsw0 ldg3
ldm add-vdsdev your-disk-file-2 vol3@primary-vds0
ldm add-vdisk vdisk1 vol3@primary-vds0 ldg3
ldm add-var auto-boot\?=false ldg3
ldm add-var boot-device=/virtual-devices@100/channel-devices@200/disk@0 ldg3
The disk files are created using the mkfile command. Solaris is installed once the domains are bound and started in a manner described in the LDoms Administrator's Guide.
While the domains are unbound, the virtual data plane channels are configured in the primary domain as follows:
ldm add-vdpcs primary-gc ldg1
ldm add-vdpcc tnsm-gc0 primary-gc ldg3
ldm add-vdpcs config-tnsm-ldg2 ldg1
ldm add-vdpcc config-tnsm0 config-tnsm-ldg2 ldg2
ldm add-vdpcs ldg2-vdpcs0 ldg1
ldm add-vdpcc vdpcc0 ldg2-vdpcs0 ldg2
Additional data channels can be added with names picked by you. Once all channels are configured, the domains can be bound and started.
The IPC channels are configured using the /opt/SUNWndpsd/bin/tnsmctl utility in ldg3.
Before you can use the utility, the SUNWndpsd package must be installed in both ldg3 and ldg2, using the pkgadd system administration command. After installing the package, you must add the tnsm driver by using the add_drv system administration command.
To be able to configure these channels, the output of ldm ls-bindings in the primary domain is needed to determine the LDC IDs. As an example, the relevant parts of the output for the configuration channel between ldg1 and ldg2 might appear as follows:
For ldg1:
Vdpcs: config-tnsm-ldg2
[LDom ldg2, name: config-tnsm0]
[LDC: 0x6]
For ldg2:
Vdpcc: config-tnsm0 service:config-tnsm-ldg2 @ ldg1
[LDC: 0x5]
The channel uses the local LDC ID 6 in the LWRTE domain (ldg1) and remote LDC ID 5 in the Solaris domain. Given this information, and choosing channel ID 3 for the control channel, this channel is set up using the following command-line:
After the control channel is set up, you can then set up the data channel between ldg1 and ldg2. Assuming local LDC ID 7, remote LDC ID 6, and IPC channel ID 4 (again, the LDC IDs must be determined using ldm ls-bindings), the following command-line sets up the channel:
Note that the -C 4 parameter is the ID for the new channel, -F 3 has the channel ID of the control channel set up previously. After the completion of this command, the IPC channel is ready to be used by an application connecting to channel 4 on both sides. An example application using this channel is contained in the SUNWndps package, and described in the following section.
The NDPS package contains an IP forwarding reference application that uses the IPC mechanism. The NDPS package contains an IP forwarding application in LWRTE and a Solaris utility that uses an IPC channel to upload the forwarding tables to the LWRTE domain (see Forwarding Application). NDPS chooses which table to use and where to gather some simple statistics, and displays them in the Solaris domain. The application is designed to operate in the example setup shown in IPC Channels.
The common header file fibtable.h, located in the src/common/include subdirectory, contains the data structures shared between the Solaris and the LWRTE domains. In particular, it contains the message formats for communication protocol used between the domains, and the IPC protocol number (201) that it uses. This file also contains the format of the forwarding table entries.
The code for the Solaris utility is in the src/solaris subdirectory and is composed of the single file fibctl.c. This file implements a simple CLI to control the forwarding application running in the LWRTE domain. The application is built using gmake in the directory and deployed into a domain that has an IPC channel to the LWRTE domain established. The program opens the tnsm driver and offers the following commands:
Connects to the channel with ID Channel_ID. The forwarding application is hard coded to use channel ID 4. The IPC type is hard coded on both sides. This command must be issued before any of the other commands.
Instructs the forwarding application to use the specified table. In the current code, the table ID must be 0 or 1.
Transmits the table with the indicated ID to the forwarding application. There are two predefined tables in the application.
Requests statistics from the forwarding application and displays them.
Reads an IPC message that has been received from the forwarding application. Currently not used.
Issues the TNIPC_IOC_CH_STATUS ioctl.
Contains program help information.
There are two components to the code implementing the forwarding application:
The hardware architecture is identical to the default architecture in all other reference applications.
The software application differs from other applications in that it contains code for the specific number of strands that the target LDoms will have. Also, the memory pools used in the malloc() and free() implementation for the LDoms and IPC frameworks are declared here.
The mapping file contains a mapping for each strand of the target LDom.
The rx.c and tx.c files contain simple functions that use the Ethernet driver to receive and transmit a packet, respectively.
ldc_malloc.c contains the implementation of the memory allocation algorithm. The corresponding header file, ldc_malloc_config.h, contains some configuration for the memory pools used.
user_common.c contains the memory allocation provided for the Ethernet driver, as well as the definition for the queues used to communicate between the strands. The corresponding header file, user_common.h contains function prototypes for the routines used in the application, as well as declarations for the common data structures.
ipfwd.c contains the definition of the functions that are run on the different strands. In this version of the application, all strands start the _main() function. Based on the thread IDs, the _main() function calls the respective functions for rx, tx, forwarding, a thread for IPC, the cli, and statistics gathering.
The main functionality is provided by the following processes:
The IP forwarding algorithm called by the forwarding thread is implemented in ipfwd_lib.c. The lookup algorithm used is a simple linear search through the forwarding table. The destination MAC address is set according to the forwarding entry found, and the TTL is decremented.
ipfwd_config.h contains configuration for the forwarding application, such as the number of strands and memory sizes used.
init.c contains the initialization code for the application. First, the queues are initialized. Initialization of the Ethernet interfaces is left to the rx strands, but the tx strands must wait until that initialization is done before they can proceed. The initialization of the LDoms framework is accomplished using calls to the functions mach_descrip_init(), lwrte_cnex_init(), and lwrte_init_ldc(). After this, the IPC framework is initialized by a call of tnipc_init(). The previous four functions must be called in this specific order. Finally, the data structures for the forwarding (that is, the tables) are initialized.
The forwarding application can be built using the build script located in the main application directory. For this application in an LDoms environment:
To deploy the application, the image must be copied to a tftp server. The image can then be booted using a network boot from either one of the Ethernet ports, or from a virtual network interface. See the README file for details. After booting the application, the IPC channels are initialized as described in Example Environment. Once the IPC channels are up, you can use the fibctl utility to manipulate the forwarding tables and gather statistics.
| Netra Data Plane Software Suite 1.1 User's Guide | 820-2374-10 |
|
Copyright © 2007, Sun Microsystems, Inc. All Rights Reserved.