Transparent Interprocess Communication

C H A P T E R 10

This chapter describes using the Transparent Interprocess Communication (TIPC) protocol. Topics include:

Transparent Interprocess Communication Introduction

TIPC Components

Installing TIPC

Programming Interfaces Overview

Configuring Environment for TIPC

Transparent Interprocess Communication Introduction

The Transparent Interprocess Communication (TIPC) protocol is designed for use in clustered computer environments. TIPC allows designers to create applications that can communicate quickly and reliably with other applications regardless of their location within the cluster.

You first must read the TIPC documentation, which is available at:
http://tipc.sourceforge.net/

TIPC is available for Oracle Solaris and can be downloaded at:
http://opensolaris.org/os/project/tipc/

TIPC Components

The TIPC implementation for Sun Netra DPS contains the components listed below. These components are required to run a TIPC application on Sun Netra DPS:

Sun Netra DPS runtime components - These components provide all APIs and the libraries for writing the TIPC application for Sun Netra DPS:

/opt/SUNWndps/lib/tipc/include - Contains the TIPC header files.

/opt/SUNWndps/lib/tipc/lwrtetipc.o - Provides the binaries for TIPC. The application writer needs to link the application with this binary.

Control plane (Oracle Solaris OS) components - To configure TIPC, the following is required:

/opt/SUNWndpsd/bin/tn-tipc-config - The utility tool for configuring TIPC.

/opt/SUNWndpsd/lib/libtipccfgsocket.so.1 - Contains Socket API implementation over IPC. This library is required for running the tipc-config utility tool.

Oracle Solaris TIPC packages - Oracle Solaris TIPC is modified to support IPC bearer media:

/opt/SUNWndps-tipc - Contains TIPC with IPC media support.

/opt/SUNWndps-tipc-headers - TIPC header files to develop TIPC applications in the Solaris OS.

/opt/SUNWndps-tipc-examples - Contains TIPC example applications.

Control plane (Linux) components - To configure Sun Netra DPS TIPC, the following is required:

/opt/SUNWndpsd/linux/src/lwrte-tipc-cfg-lnx - Contains the sources to build the utility tool called tn-tipc-config for Linux OS.

Note - The following packages should not be installed with the above mentioned packages: SUNWtipc, SUNWtipc-examples and SUNWtipc-headers.

TIPC source files

/opt/SUNWndps/src/libs/tipc - Contains the TIPC source files for Sun Netra DPS.

Installing TIPC

This section describes how to install TIPC.

To Install TIPC

1. Go to the /opt/SUNWndps/src/libs/tipc directory. Type:

# cd /opt/SUNWndps/src/libs/tipc

Or, copy /opt/SUNWndps/src/libs/tipc to a preferred location and change the directory to the copied location.

2. Compile the Oracle Solaris TIPC application.

Note - Compiling of Oracle Solaris TIPC packages is not necessary unless you want to change the Oracle Solaris source.

Run the following command to build the Oracle Solaris TIPC packages:

The above command builds the Sun Netra DPS binaries and Oracle Solaris packages under the ./bins directory.

Three packages are created for Oracle Solaris TIPC, as shown below:

SUNWndps-tipc - Contains the TIPC kernel module and associated library that includes support for IPC bearer.

SUNWndps-tipc-headers - Contains the TIPC public header files useful for development.

SUNWndps-tipc-examples - Contains ready-to-run examples.

The TIPC binaries for Sun Netra DPS are created as shown below:

./bins/include - This directory contains the TIPC header files required for writing Sun Netra DPS TIPC application

lwrtetipc.o - This is the TIPC library for the Sun Netra DPS application.

3. Compile the Sun Netra DPS TIPC application.

Note - Compiling of Sun Netra DPS TIPC is required if you want to change the IPC channel number used for the Sun Netra DPS TIPC configuration.

Compiling of Sun Netra DPS TIPC alone can be done by the following command.

# gmake TARGET_DIR=./bins

To build the Sun Netra DPS TIPC application with the TIPC library, the application should be compiled with the following options:

/opt/SUNWndps/lib/tipc/include - This directory path should be added to include the path to pick the header files.

/opt/SUNWndps/lib/tipc/lwrtetipc.o - This is added to USR_LIBS for linking to your application with the TIPC library.

4. Install TIPC.

Do the following to install Oracle Solaris TIPC.

a. Log on to your target machine as root.

b. Install the SUNWndps-tipc package using the pkgadd command as shown in the example below:

# pkgadd -d path-to-packages SUNWndps-tipc

c. If necessary, install the SUNWndps-tipc-examples package using the pkgadd command as shown below:

# pkgadd -d path-to-packages SUNWndps-tipc-examples

d. Reboot if necessary.

In certain conditions, the tipc module will not load automatically. In this case, reboot the system or load the tipc module manually by using the following command:

# add_drv tipc

e. You must set up the environment to preload the TIPC socket library as shown below before running the Oracle Solaris TIPC applications:

# LD_PRELOAD_32=/opt/SUNWndps-tipc/lib/libtipcsocket.so
# LD_PRELOAD_64=/opt/SUNWndps-tipc/lib/sparcv9/libtipcsocket.so
# export LD_PRELOAD_32 LD_PRELOAD_64

See SUNWndpsd and SUNWndps-tipc Binaries for a description the of tipc-config in the SUNWndpsd and SUNWndps-tipc binaries.

Programming Interfaces Overview

You can use the Transparent Interprocess Communication (TIPC) Socket APIs to write TIPC applications. The Sun Netra Data Plane Software Suite 2.1 Update 1 Reference Manual contains a detailed description of the TIPC APIs.

In a Sun Netra DPS application, you must call the framework initialization routines for the IPC, LDC, and TIPC frameworks, and must ensure that tipc_process() is called periodically.

The application developer must implement tipc_pbuf_alloc() and tipc_pbuf_free() to allocate and free message block buffers used by the TIPC stack. The buffers must be 8-byte aligned. The application developer must also implement tipc_eth_pbuf_alloc() and tipc_eth_pbuf_free() to allocate and free message block buffers to manage outgoing and incoming Ethernet or vnet packets.

To support Ethernet or vnet bearer in a Sun Netra DPS application, the programmer must implement the tipc_eth_get_mac() function to provide the MAC address of the Ethernet or vnet port to the TIPC stack. And also, the application must open the corresponding Ethernet or vnet port using eth_open(). To get the TIPC messages to be transmitted over the Ethernet or vnet media, the application needs to poll on the fastq returned by tipc_eth_get_fastq(dev_type, port, chan, TIPC_FASTQ_TX). Then, on receiving a TIPC message over the Ethernet or vnet, the application needs to enqueue it into the fastq, returned by tipc_eth_get_fastq(dev_type, port, chan, TIPC_FASTQ_RX) for the corresponding Ethernet or vnet port.

Note - poll() is not supported in Sun Netra DPS.

Configuring Environment for TIPC

This section describes the configuration of the environment needed to use the TIPC framework. This section also covers setup of the Ethernet bearer for the NDPS application

`SUNWndpsd` and `SUNWndps-tipc` Binaries

The tn-tipc-config-bin tool in the SUNWndpsd package configures Sun Netra DPS TIPC node from Oracle Solaris guest logical domain. This is the same binary as tipc-config that is present in the SUNWndps-tipc package. Only the libraries that are pre-loaded are different for Oracle Solaris TIPC and Sun Netra DPS TIPC. The tipc-config binary is placed in two packages because there are cases where both packages will not be installed on the same system. For example, if only the SUNWndpsd package is installed and in the control domain, the tn-tipc-config-bin and the corresponding tn-tipc-config script are required to configure the Sun Netra DPS TIPC. Thus, both scripts are in the SUNndpsd package.

Also, the packaging of SUNWndps-tipc is the same as the Oracle Solaris TIPC that can be found at http://tipc.sourceforge.net/. Therefore, SUNWndps-tipc also has the tipc-config binary.

The /opt/SUNWndpsd/bin/tn-tipc-config script sets the environment to pre-load /opt/SUNWndpsd/lib/libtipccfgsocket.so.1. The script then loads the /opt/SUNWndpsd/bin/tn-tipc-config-bin binary (this binary is the same as /opt/SUNWndps-tipc/sbin/tipc-config).

When a Linux guest logical domain is used, the tn-tipc-config-bin and the script file tn-tipc-config that are present in SUNWndpsd cannot be used. The tool tn-tipc-config-bin uses IPC which is not supported for the Linux environment. For the Linux environment, the source files for building the same tool is provided in /opt/SUNWndpsd/linux/src/lwrte-tipc-cfg-lnx. To build this tool, please do the following:

# cd /opt/SUNWndpsd/linux/src/lwrte-tipc-cfg-lnx
# tar -cvf lwrte-tipc-cfg-lnx.tar lwrite-tipc-cfg-lnx/

In a system that has a UltraSPARC T2 cross-compiler installed:

# tar -xvf lwrte-tipc-cfg-lnx.tar
# cd lwrte-tipc-cfg-lnx
# make

This creates a binary called tn-tipc-config. Deploy this binary in the Linux guest logical domain.

Configuring Sun Netra DPS TIPC Stack from an Oracle Solaris Guest Logical Domain

You can configure the TIPC stack using the tn-tipc-config tool located at /opt/SUNWndpsd/bin in the Solaris Control-Plane. This tool is the same as tipc-config which is distributed with the Oracle Solaris TIPC package. (Refer to the TIPC documentation available at http://tipc.sourceforge.net/ for the options supported by the tipc-config tool). The tn-tipc-config tool uses the IPC channel to configure the TIPC stack of Sun Netra DPS. tn-tipc-config uses TIPC_IPC_CHANNEL_ID channel, which is set to 10 by default in tipc_ipc_cfgsrv.h. This value can be modified by compiling the TIPC stack with a different TIPC_IPC_CHANNEL_ID and replacing /opt/SUNWndpsd/lib/libtipccfgsocket.so.1.

Configuring Sun Netra DPS TIPC Stack from a Linux Guest Logical Domain

The Sun Netra DPS TIPC stack can be configured from a Linux logical domain using the tn-tipc-config tool built from the sources in /opt/SUNWndpsd/linux/src/lwrte-tipc-cfg-lnx. This tool supports the same options as the tipc-config tool that is distributed with the Linux TIPC package (refer to the TIPC documentation available at http://tipc.sourceforge.net/ for information on the supported options). The tn-tipc-config tool uses a proprietary protocol to communicate with the Sun Netra DPS TIPC stack over a vnet interface. This protocol uses Ethernet encapsulated frames with an Ethernet Type of 0x3c21 to communicate between the stack and the tool running in Linux over a vnet interface. The vnet interface name that is to be used for this communication must be assigned to an environment variable called TN_TIPC_CFGDEV.

The prerequisites for using this tool are:

1. One vnet interface must be configured for the Oracle VM Server for SPARC software and the Linux Guest logical domain. These vnet interfaces must be reachable through a vswitch.

2. Ethernet multicast frame processing must be enabled on these vnet interfaces.

3. The Sun Netra DPS application must have vnet driver packet processing.

4. The TN_TIPC_CFGDEV environment variable to the vnet interface name that is used for tn-tipc-config exchanges.

Refer to the Sun Netra Data Plane Software 2.1 Update 1 Reference Manual for instructions on how to handle the tn-tipc-config frames.

The following commands set the TIPC address for Sun Netra DPS TIPC node from a Linux control plane logical domain:

# export TN_TIPC_CFGDEV=eth1
# ./tn-tipc-config -addr=10.3.7

To Set the TIPC Address

The TIPC addressing mechanism can be found at the TIPC homepage (http://tipc.sourceforge.net/).

Set the address using the -addr option of the tn-tipc-config tool.

The following example shows setting the TIPC address to 10.3.4:

# /opt/SUNWndpsd/bin/tn-tipc-config -addr=10.3.4

Enabling TIPC Ethernet Bearer

The TIPC Ethernet bearers in the Sun Netra Data Plane is named as a port number, for example, port0, port1, and so on.

For example, to enable eth:port0, the corresponding Ethernet port0 must have been opened by the application.

The following example enables the Ethernet port0 bearer for the Sun Netra DPS application with netmask of 10.3.0:

# /opt/SUNWndpsd/bin/tn-tipc-config -be=eth:port0/10.3.0

For more details on the -be option in tipc-config, refer to the TIPC documentation available at http://tipc.sourceforge.net.

Enabling the TIPC IPC Bearer

The TIPC IPC bearers must be named as channel.type, for example, 5.200, 6.100, and so on. Before enabling the bearer, ensure that the corresponding IPC channel is created. See Interprocess Communication Software to create an IPC channel. For example, to enable IPC bearer with IPC Channel 5 and type 200, the corresponding IPC channel 5 must be created using the tnsmctl tool.

Note the following requirements:

The IPC channel number must not be the same as TIPC_IPC_CHANNEL_ID which is used by TIPC configuration service.

The channel and type number should be the same in both Oracle Solaris and Sun Netra DPS for communication through the IPC bearer.

The following is an example of enabling IPC bearer for Sun Netra DPS using
tn-tipc-config:

# /opt/SUNWndpsd/bin/tn-tipc-config -be=ipc:5.200/10.3.0

The following is an example of enabling IPC bearer for Oracle Solaris using tipc-config:

# /opt/SUNWndps-tipc/sbin/tipc-config -be=ipc:5.200/10.3.0

Enabling TIPC vnet Bearer for a NDPS TIPC Node

The -be option of the tn-tipc-config command enables the vnet bearers for a Sun Netra DPS TIPC node. The media type in the bearer name must be specified as eth. The interface name in the bearer name must be vnet followed by the instance number, for example, vnet{instance number}. The instance number can be obtained by executing the following command from the primary Logical Domain

ldm list-bindings -e control-ldom

For example, the following command enables a vnet bearer using a vnet device with instance number 5:

# tn-tipc-config -be=eth:vnet5/10.3.0