Reference Applications

C H A P T E R 9

This chapter describes reference applications including IP Packet Forwarding and radio link protocol (RLP) applications. Topics include:

IP Packet Forwarding Application

Radio Link Protocol Application

IPSec Gateway Reference Application

IP Packet Forwarding Application

The IP packet forwarding application (ipfwd) simulates an IPv4 (Internet Protocol Version 4) forwarding operation. When each packet is received, the program performs a forward table search, rewrites the MAC source and destination address, rewrites the TTL, and recalculates the IP header checksum. The packet is then sent out to a specified port. The application can either simulate an IP packet forwarding lookup table or actually perform an IP packet forwarding lookup through the use of the lookup table. The goal of this application is to provide a framework for users to develop their own application based on their own lookup algorithm.

Source Files

All ipfwd source files are located in the following package directory: /opt/SUNWndps/src/apps/ipfwd. The contents include:

Makefiles for two different ethernet hardware devices include:

./Makefile.ipge

./Makefile.nxge

Information files for ipfwd and LDoms configuration include:

./README

./README.config

Build scripts for one step build include:

./build_10g

./build_10g_niu

./build_1g

./build_1g_1060

./build_4g

Application files and application configuration files include:

./src

./src/app

./src/app/common.h

./src/app/init.c

./src/app/ipfwd.c

./src/app/ipfwd_config.c

./src/app/ipfwd_config.h

./src/app/ipfwd_lib.c

./src/app/lb_objects.h

./src/app/ldc_malloc.c

./src/app/ldc_malloc_config.h

./src/app/radix

./src/app/radix/ipfwd_radix.h

./src/app/radix/radix.c

./src/app/radix/radix.h. /src/app/rx.c

./src/app/tx.c

./src/app/user_common.c

./src/app/user_common.h

./src/common

./src/common/include

./src/common/include/fibtable.h

System configuration for the application include:

./src/config

./src/config/ipfwd_hwarch.c

./src/config/ipfwd_map.c

./src/config/ipfwd_swarch.c

Solaris FIB Control applications for ipfwd LDoms include:

./src/solaris

./src/solaris/Makefile

./src/solaris/fibctl.c

Compiling the `ipfwd` Application

On a system that has SUNWndps installed, go to the src/app/ipfwd directory and use the build scripts shown in TABLE 9-1 to build the application.

Note - Copy the application files from the /opt/SUNWndps/src/apps/ipfwd directory of the package into your workspace before compiling the application.

Build Scripts

TABLE 9-1 shows the ipfwd application build scripts.

TABLE 9-1 `ipfwd` Application Build Scripts
Build Script	Usage
`build_1g <cmt> [ldoms]`	Build `ipfwd` application to run on `e1000g` Ethernet interface.
`build_1g_1060 [ldoms]`	Build `ipfwd` application to run on `e1000g` Ethernet interface on the Netra ATCA CP3060 System.
`build_4g <cmt> [ldoms]`	Build `ipfwd` application to run on Sun multithreaded 10GbE with NIU QGC (quad 1Gbps `nxge` Ethernet interface)
`build_10g <cmt> [ldoms]`	Build `ipfwd` application to run on Sun multithreaded 10G (dual 10Gbps `nxge` Ethernet interface).
`build_10g_niu [ldoms]`	Build `ipfwd` application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet Interface) on a CMT2 based system.

Note - This script does not have the <cmt> option because the Netra ATCA CP3060 system is UltraSPARC T1 based only.

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Argument Descriptions

<cmt>

This argument specifies whether to build the ipfwd application to run on the UltraSPARC T1 platform or UltraSPARC T2 platform.

cmt1 - Build for CMT1 (UltraSPARC T1) architecture

cmt2 - Build for CMT2 (UltraSPARC T2) architecture

This argument is required for scripts that expect <cmt>.

[ldoms]

This optional argument specifies whether to build the ipfwd application to run on the LDoms environment. When this flag is specified, the IPFWD LDoms reference application will be compiled. If this argument is not specified, then the non-LDoms application is compiled.

Build Example

In /opt/SUNWndps/src/apps/rlp, pick the correct build script and run it. For example, to build for Sun multithreaded ethernet adapter on Netra or Sun Fire T2000 systems, enter the following at your shell window.

% build_10g cmt1

In this example, build_10g is used to build IP forwarding application to run on the Sun multithreaded 10Gb ethernet. The <cmt> argument is specified as cmt1 to build the application to run on UltraSPARC T1-based Netra or Sun Fire T2000 systems.

To Run the ipfwd Application

1. Copy the binary into the /tftpboot directory of the tftpboot server.

2. On the tftpboot server, type:

% cp your_workspace/ipfwd/code/ipfwd/ipfwd /tftpboot/ipfwd_xxx

3. On the target machine, type the following at the ok prompt:

ok boot net:,ipfwd_xxx

Note - net is an OpenBoot PROM alias corresponding to the physical path of the network.

Default Configurations

This section shows the default configurations:

Default System Configuration

                    NDPS domain     Statistics      Other domains 
                    (Strand IDs)    (Strand ID)     (Strand IDs) 
                     ----------     -----------     -----------
CMT1 NON-LDOM:       0 - 31          31              N/A 
CMT1 LDOM:           0 - 19          19              20 - 31 
CMT2 NON-LDOM:       0 - 63          63              N/A 
CMT2 LDOM:           0 - 39          39              40 - 63

Main files that control the system configurations:

ipfwd/src/app/config/ipfwd_swarch.c

ipfwd/src/app/config/ipfwd_map.c

Default `ipfwd` Application Configuration

Application            # of    # of            Total #         Total # 
runs                   Ports   Channels        of Q            of strands 
on                     used    per Port        instances       used 
----------             -----   ---------       ---------       ------- 
1G (ipge):             2       1 (see note)       2               6 
1G (CP 1060):          2       1 (see note)       2               6 
4G-PCIE (nxge QGC):    4       1               4               12 
10G-PCIE (nxge 10G):   1       4               4               12 
10G-NIU (niu 10G):     1       8               8               24

Note - e1000g (ipge) only has one DMA channel per port.

Main files that control the application configurations:

ipfwd/src/app/ipfwd_config.c

ipfwd/src/app/ipfwd_config.h

Other Options

Profiling

To enable profiling, follow the instructions given in each build script that states:

# Replace the above gmake command with the following to# generate code with profiling enabled.

Radix Forwarding Algorithm

To enable the Radix Forwarding Algorithm for IP forwarding, uncomment the following line from Makfile.ipge or Makefile.nxge for e1000g 1Gb PCIe ethernet adapter or Sun multithreaded 10Gb and 1Gb PCIe Ethernet adapter, respectively:

- DIPFWD_RADIX

Bypassing the `ipfwd` Operation

To bypass the ipfwd operation (that is, receive --> transmit without forwarding operation), uncomment the following line from Makfile.ipge or Makefile.nxge for 1Gb PCIe ethernet adapter or Sun multithreaded 10 Gb and 1Gb PCIe Ethernet adapter, respectively:

-DIPFWD_RAW

Multiple Forward Port Destinations

When the Multiple Forward Destinations option is enabled, the output destination port is determined by the output of the Forwarding Table Lookup. Otherwise, the output destination port is the same as the input port. To enable this option, uncomment the following line from Makefile.ipge or Makefile.nxge for Ophir or Sun multithreaded 10GbE Ethernet, respectively:

-DIPFWD_MULTI_QS

Hash Policy for Spreading Traffic to Multiple DMA Channels

You can specify a policy for spreading traffic into multiple DMA flows by hardware hashing. TABLE 9-2 describes each policy:

TABLE 9-2 Hash Policy Descriptions
Name	Definition
`HASH_IP_ADDR`	Hash on IP destination and source addresses.
`HASH_IP_DA`	Hash on IP destination address.
`HASH_IP_SA`	Hash on IP source address.
`HASH_VLAN_ID`	Hash on VLAN ID.

To enable one of the above policies, add the following into the gmake line:

HASH_POLICY=policy

where policy is one of those specified in TABLE 9-2. For example, to enable hash on an IP destination and source address, add HASH_POLICY=HASH_IP_ADDR to the gmake line as shown below:

gmake -f Makefile.nxge NETHW_TYPE=2 CMT=N2 HASH_POLICY=HASH_IP_ADDR

If none of the policies listed in TABLE 9-2 are specified, a default policy is given. When you use the default policy, all L2/L3/L4 header fields are used for spreading traffic.

`ipfwd` Flow Configurations

The ipfwd_config.c file assists you in mapping application tasks to CPU core and hardware strands. Normally, mapping is set in the ipfwd_map.c file in the config directory. This configuration file is a productivity tool. This file provides a way to facilitate mapping in a quick manner without any modification to the ipfwd_map.c file.

This configuration file is not a replacement of ipfwd_hwarch.c, ipfwd_swarch.c and ipfwd_map.c. This framework is to conduct performance analysis and measurement with different system configurations. The default (*_def) configurations specified assumes a minimum of 16 threads of the system allocated for Netra DPS in ipfwd_map.c and all memory pool resources required are declared in ipfwd_swarch.c. You still need to specify system resources declarations and mapping in ipfwd_hwarch.c, ipfwd_swarch.c, and ipfwd_map.c. The configuration is assigned to a pointer named ipfwd_thread_config.

Note - You have the option to bypass this file entirely and perform all the mapping in ipfwd_map.c. In this case, you would also need to modify ipfwd.c so that it does not interpret the contents of this file.

Format

Each application configuration is represented in an array of a six-element entry. Each entry (each row) represents a software task and its corresponding resources:

Thread ID - Strand number of the hardware strand (0 ~ 31 on an UltraSPARC T1 system, 0 ~ 63 on an UltraSPARC T2 system) on which this software task is to be run.

HW init - If zero, it indicates no Ethernet port needs to be opened when this task is activated. If non-zero, it indicates ethernet port (port number specified by port#) needs to be opened. The contents of OPEN_OP consists of vendor and device id as (NXGE_VID << 16) | NXGE_DID.

Port# - This is the port number of the ethernet port to be opened. Port# should start from zero. Port# denotes the first ethernet port probed on the system for this given OPEN_OP.

Chan# - If this is a multichannel device (such as Sun multithreaded 10GbE with NIU), this entry indicates the channel number within each port. Sun multithreaded 10GbE device has 24 transmit channels (0~23) and 16 receive channels (0~16) in each port. Sun multithreaded 10GbE with NIU has 16 channels (both tx and rx) in each port. e1000g (ipge) Ethernet device has one channel in each port (it is a non-channelized device).

Role - This is the role of the software task.

TROLE_ETH_NETIF_RX -- This task performs a receive function. TROLE_ETH_NETIF_TX -- This task performs a transmit function. TROLE_APP_IPFWD -- This task performs IP forwarding function.

See common.h for all definitions. If you do not want to run any software task on this hardware strand, the role field should be set to -1. By default, during initialization of the ipfwd application, the hardware strand that encounters a -1 software role is parked.

Note - A parked strand is a strand that does not consume any pipeline cycles (an inactive strand).

MemPool# - This is the identity of the memory pool. Note that in this reference application, each Ethernet port has its own memory pool. Each channel within each port has its own memory pool. Memory pools are declared in ipfwd_swarch.c.

Note - The application can be configured such that a single memory pool is dedicated to a particular DMA channel or all DMA channels sharing a global memory pool. The default configuration is one memory pool per DMA channel.

Radio Link Protocol Application

The Radio Link Protocol (RLP) application (rlp) simulates Radio Link Protocol operation, which is one of the protocols in the CDMA-2000 High Rate Packet Data Interfaces (HRPD-A). This application implements the forwarding direction fully, with packets flowing from PDSN --> AN --> AT (that is, Packet Data Serving Node to Access Network to Access Terminal). Reverse direction support is also implemented, but requires an AT side application which can generate NAKs (Negative Acknowledges). The application must be modified to process reverse traffic.

Compiling the RLP Application

On a system that has SUNWndps installed, go to the src/app/rlp directory and use the build scripts shown in TABLE 9-1 to build the application.

Note - First copy the application files from the /opt/SUNWndps/src/apps/rlp directory of the package into your workspace before compiling the application.

Build Scripts

TABLE 9-1 shows the RLP (rlp) application build scripts.

TABLE 9-3 `rlp` Application Build Scripts
Build Script	Usage
`build_1g <cmt> [ldoms]`	Build `rlp` application to run on the `e1000g` Ethernet interface.
`build_1g_1060 [ldoms]`	Build `rlp` application to run on the e1000g Ethernet interface on the Netra ATCA CP3060 system.
`build_4g <cmt> [ldoms]`	Build `rlp` application to run on Sun multithreaded 10GbE with NIU QGC (quad 1Gbps `nxge` ethernet interface).
`build_10g <cmt> [ldoms]`	Build `rlp` application to run on Sun multithreaded 10G (dual 10Gbps `nxge` Ethernet Interface).
`build_10g_niu [ldoms]`	Build `rlp` application to run on NIU (dual 10Gbps UltraSPARC T2 ethernet interface) on a CMT2 (UltraSPARC T2) based system.

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Arguments Descriptions

<cmt>

This argument specifies whether to build the rlp application to run on the UltraSPARC T1 platform or UltraSPARC T2 platform.

cmt1 - Build for CMT1 (UltraSPARC T1) architecture

cmt2 - Build for CMT2 (UltraSPARC T2) architecture

This argument is required for scripts that expect <cmt>.

[ldoms]

This i optional argument specifies whether to build the rlp application to run on the LDoms environment. When this flag is specified, the RLP LDoms reference application will be compiled. If this argument is not specified, then the non-LDoms application is compiled.

Build Example

In /opt/SUNWndps/src/apps/ipfwd, pick the correct build script and run it. For example, to build for Sun multithreaded 10Gb PCIe ethernet adapter on Netra or Sun Fire T2000 systems, enter the following at your shell window:

% build_10g cmt1

In this example, build_10g is used to build RLP application to run on the Sun multithreaded 10Gb Ethernet. The <cmt> argument is specified as cmt1 to build the application to run on UltraSPARC T1-based Netra or Sun Fire T2000 systems.

To Run the Application

1. Copy the binary into the /tftpboot directory of the tftpboot server, and perform.

2. On the tftpboot server, type:

% cp your_workspace/rlp/code/rlp/rlp /tftpboot/rlp_xxx

3. On the target machine, type the following at the ok prompt:

ok boot net:,rlp_xxx

Note - net is an OpenBoot PROM alias corresponding to the physical path of the network.

Default Configurations

This section shows the default configurations.

Default System Configuration

                                     IPC polling,
                     NDPS domain     Statistics      Other domains 
                     (Strand IDs)    (Strand IDs)    (Strand IDs) 
                     ----------      -----------     -----------
CMT1 NON-LDOM:       0 - 31          31              N/A 
CMT1 LDOM:           0 - 19          18, 19          20 - 31 
CMT2 NON-LDOM:       0 - 63          63              N/A 
CMT2 LDOM:           0 - 39          38, 39          40 - 63

Main files that control the system configurations are:

ipfwd/src/app/config/rlp_swarch.c

ipfwd/src/app/config/rlp_map.c

Default `rlp` Application Configuration

Application            # of    # of            Total #         Total # 
runs                   Ports   Channels        of Q            of strands 
on                     used    per Port        instances       used 
----------             -----   ---------       ---------       ------- 
1G (ipge):             2       1 (see note)    2               6 
1G (CP 1060):          2       1 (see note)    2               6 
4G-PCIE (nxge QGC):    4       1               4               12 
10G-PCIE (nxge 10G):   1       4               4               12 
10G-NIU (niu 10G):     1       8               8               24

Note - e1000g (ipge) only has one DMA channel per port.

Main files that control the application configurations are:

ipfwd/src/app/rlp_config.c

ipfwd/src/app/rlp_config.h

Other RLP Options

To Enable Profiling

Follow the instructions given in each build script that states:

# Replace the above gmake command with the following to# generate code with profiling enabled.

To Bypass the rlp Operation

To bypass the rlp operation (that is, receive --> transmit without rlp_process operation), uncomment the following line from Makfile.ipge or Makefile.nxge for 1Gb PCIe ethernet adapter or Sun multithreaded 10 Gb and 1Gb PCIe Ethernet adapter, respectively:

-DIPFWD_RAW

Note - This action disables the RLP processing operation only, the queues are still used. This is not the default option.

To Use One Global Memory Pool

By default, the RLP application uses a single global memory pool for all the DMA channels.

1. Enable the single memory pool by using the following flag:.

-DFORCEONEMPOOL

2. Update the rlp_swarch.c file to use individual memory pools.

To Run RLP on Four Ports for ipge

By default, the RLP application is configured for two port-pairs for the ipge driver.

To run the application on eight ipge interfaces (four port-pairs), enable the following flag:

-DRLP_RUN_ALL_THDS

RLP Policy for Spreading Traffic to Multiple DMA Channels

You can specify a policy for spreading traffic into multiple DMA flows by hardware hashing or the TCAM lookup table. TABLE 9-4 describes each policy:

TABLE 9-4 RLP Policy Descriptions
Name	Description
`hash_ip_addr`	Hash on IP destination and source addresses.
`hash_ip_da`	Hash on IP destination address.
`hash_ip_sa`	Hash on IP source address.
`hash_vlan_id`	Hash on VLAN ID.
`hash_portnum`	Hash on physical MAC Port number
`hash_l2da`	Hash on L2 destination address.
`hash_proto`	Hash on protocol number.
`hash_src_port`	Hash on L4 source port number.
`hash_dst_port`	Hash on L4 destination port number.
`hash_all`	Hash on all of the above fields.
`tcam_classify`	Perform TCAM lookup.

To Enable an RLP Policy

Add the following into the gmake line:

FLOW_POLICY=policy

where policy is one of the above specified policies.

For example, to enable flow on an IP destination and source address, add FLOW_POLICY=hash_ip_addr to the gmake line as shown below:

gmake -f Makefile.nxge NETHW_TYPE=2 CMT=N2 FLOW_POLICY=HASH_IP_ADDR

If none of the policies listed in TABLE 9-4 are specified, a default policy is given. When you use the default policy, all L2/L3/L4 header fields are used for spreading traffic.

IPSec Gateway Reference Application

The IPSec Gateway Reference Application implements the IP Encapsulating Security Payload (ESP) protocol using Tunnel Mode. This application allows two Gateways (or a host and a gateway) to securely send packets over an unsecure network with the original IP packet tunneled and encrypted (privacy service). This application also implements the optional integrity service allowing the ESP header and tunneled IP packet to be hashed on transmit and verified on receipt.

IPSec Gateway Reference Application Architecture

The design calls for six Netra DPS threads in a classic architecture where four threads are dedicated to packet reception and transmission (two receivers, two senders). In this architecture, a thread takes plain text packets and encapsulates and encrypts them, as well as a thread that de-encapsulates and decrypts. The architecture is shown in FIGURE 9-1.

FIGURE 9-1 IPSec Gateway Reference Application Architecture

For testing purposes, this architecture uses four interfaces (NICS) as shown in TABLE 9-5.

TABLE 9-5 IPSec Gateway Reference Application Architecture
Interface	Description
NIC 0	Plaintext packets coming from the traffic generator.
NIC 1	IPSec-encapsulated ciphertext packets going to the peer Gateway (in this case it is a loopback cable going to NIC 2).
NIC 2	For testing purposes, this takes the packets sent over NIC 1 which are looped-back through a loopback cable. These packets are ciphertext. Normally, the packets come in on the NIC 1 interface from the peer gateway. Therefore, there is code that “reverses” source/destination addresses and port numbers to simulate an actual response packet.
NIC 3	IPSec de-encapsulated packets being “returned” to the host. These are plaintext packets that would normally be sent back over NIC 0.

Two extra packet receive threads are also used to listen on NIC 1 and NIC 3 where the test scenario above receives no packets. The main purpose of these threads are to assist with freeing TX buffers (see Netra Data Plane Software Suite 2.0 Reference Manual regarding the Netra DPS ethernet API).

Refer to Request for Comments (RFC) documents for a description of IPSec and the ESP protocol:

4301 - Security Architecture for the Internet Protocol

4303 - IP Encapsulating Security Payload (ESP)

The IPSec RFC refers to outbound and inbound packets. These design notes refer to these terms.

Outbound packets are those coming into the IPSec Gateway as plaintext (from the unprotected hosts) and being sent to the peer gateway as ciphertext packets (encrypted).

Inbound packets are the opposite, that is, IPSec-encapsulated (ciphertext) packets coming in from the peer gateway and being decrypted and sent to the unprotected hosts.

IPSec Gateway Reference Application Capabilities

IPSec is a complex protocol. This application handles the following most common processing:

Static Security Association Database (SADB)

Contains the type of service to provide (privacy, integrity), crypto and hashing types and keys to be used for a session, among other housekeeping items. An item in the SADB is called a Security Association (SA). An SA can be unique to one connection, or shared among many.

Static Security Policy database (SPD)

A partial implementation that is used to contain “selectors” that designate what action should be taken on a packet based on the source and destination IP addresses, protocol, and port numbers.

SPD cache

A critical cache used to quickly look up the SA to use for packets coming from the plaintext side. The packet source and destination addresses and ports are hashed to find the action to take on the packet (discard, pass-through, or IPSec protect) and the SA.

Security Parameter Index (SPI) hash

A critical cache used to quickly find an SA for ESP packets entering the system from the ciphertext side. The Security Parameter Index is in the ESP header

ESP Protocol, Tunnel Mode

This IPSec implementation uses the ESP protocol (it does not currently handle AH, though ESP provides most of the AH functionality). Tunnel Mode is used to encapsulate (tunnel) IP packets between hosts and interface to a peer gateway machine.

Privacy Service

Encrypt/Decrypt traffic

Tested Algorithms: AES EBC, CBC 128 and 256 bit

Integrity Service

Authenticate via optional hashing

Tested Algorithms: HMAC-SHA256

High-Level Packet Processing

The following describes functions of outbound and inbound packet processing:

Outbound Packets

Receive packet (from NIC 0)

Hash the source/destination IP and ports

Look up in (Security Policy Database caches (SPD-cache) to determine action to take and a pointer to the Security Association (SA)

If action is IPSec-protect:

Build (prepend) outer IP header, ESP header

Encrypt payload (original IP packet) using security parameters in SA

Optionally calculate and add a hash value

Xmit ciphertext packet over NIC 1

Inbound Packets

Receive packet (from NIC 2)

If action is an ESP packet:

hash Security Parameter Index (SPI) from ESP header to obtain SA

Optionally hash and verify hash value (integrity service)

Decrypt payload

Remove outer IP header, ESP header and trailer

Xmit plaintext packet over NIC 3

Security Association (SA) Database and Security Policy Database

The packet encapsulation and encryption code is straight-forward once you have a pointer to the SA. The SA contains the following information:

Crypto algorithm to use (AES, 3DES, and so on)

Key length

Type of service to apply (privacy-only or privacy + integrity)

Hash algorithm (SHA, etc)

Hash length

Hash key

Sequence number

Packet encapsulation and de-encapsulation is just a matter of determining where the new IP header goes or where the original IP header is, building the new IP header, and invoking the crypto APIs on the correct packet location and length. For the IPSec implementation, you need to find the SA to use when a packet is received (either outbound on inbound). You must use software hashing and hash table lookups for every packet. Note that when this is ported to Sun Multithreaded 10G Ethernet on PCIe, the packet classification features speed-up this hashing.

The following sections describe how the SA is obtained for each packet.

Outbound Packets and Inbound Packets

The following sections describe how the SA is obtained for each packet.

Outbound Packets

You must look at the packet selectors to determine what action to take (either DISCARD, PASS-THROUGH as is, or PROTECT). The selectors are the source and destination IP addresses, the source and destination ports, and the protocol (TCP, UDP, and so on).

The action to take is stored in the Security Policy Database (SPD). For this application, the complete SPD is not implemented. A static SPD exists that consists of rules that must be searched in order using the packet selectors.

For each selector (source IP, destination IP, source port, destination port, protocol), the rule states one of the following:

Single value (for example, matches on source addr of 129.1.2.3)

List of values (for example, matches either 129.1.2.3, 129.1.2.5, 129.1.2.10)

Range of values (for example, 129.1.1.3 - 129.1.1.10)

Match-all (for example, any source port)

Mask (for example, matches any source addr after applying mask 0x3F)

If all selectors match the rules, use the SP entry to determine what action to take. If it is PROTECTED (IPSec), the inbound and outbound Security Parameter Index (SPI) knows which SA to use. This implies the following:

An SA can be exclusive to a given connection

An SA can be shared among many connections (for example, a single SA can be used to protect all traffic between to hosts)

Each “connection” or flow of traffic has two SAs: one for outbound traffic and one for inbound traffic. Since a loopback cable is used to receive the ciphertext packets for the Xmit, use an SPI that is the SPI of the outbound packet plus 1.

The last rule in the SPD should be a catch-all that says DISCARD the packet.

The SPD structures and definitions can be found in spd.h

The source code for the SPD can be found in spd.c

The function used to lookup a rule is SPD_Search(), which is passed the selector values from the packet.

The above lookup is complex for every packet. Because of this, a cache called the SPD-Cache is maintained. The first time you lookup a particular connection, create a SPDC structure, hash the selectors, and place this SPDC in a hash table.

When a packet using the exact combination of selectors comes in, you first hash it (using SPDC_HASH()) and then look it up in the SPDC hash table. If found, immediate access to the SA is made.

The definitions for this SPDC can be found in sadb.h. The functions can be found in sadb.c.

This application does not hash on the protocol and assumes that it is either UDP or TCP because there are source and destination ports.

The SPDC hash table is defined as:

spdc_entry_t *spdc_hash_table[SPDC_HASH_TABLE_SIZE];

The primary function used to lookup an SPDC entry is:

spdc_e *spdc_hash_lookup_from_iphdr(iphdr)

For this hash table, take the hash value, mask off the hash table size - 1, then index into this table to get an entry. YOu then have to compare the entry for a match, and if it is not a match, walk the chain until you find one.

Inbound Packets

Inbound IPSec packets contain an ESP header with an SPI. Take the SPI, hash it using SPI_HASH_FROM_SPI(), look it up in the SPI hash table, and access the SA pointer from there. You cannot use the same hashing as done for outbound packets since the selectors (source/destination IP address and ports) have been encapsulated and encrypted. Decrypt cannot be done until the SA is looked up.

The SPI has table is defined as:

spi_entry_t *spi_hash_table[SPI_HASH_TABLE_SIZE];

Static Security Policy Database (SPD) and Security Association Database (SAD)

For the purposes of the application, statically define the test SPD and SAD in compile-time initialized C-code in the following C file: sa_init_static_data.c

SPD

Two SPD rules are defined.

The first rule appears as shown below:

sp_t sp_rule1 = {
    1,                      /* rule # */
    SA_OUTB1,               /* outb_spi */
    SA_INB1,                /* inb_spi */
    IPSEC_PROTECT,          /* action */
    SPD_PROTOCOL_MATCH_ALL, /* match on all protocols */
    { SPD_MATCH_ALL },      /* match all connections for now */
    { SPD_MATCH_ALL },
    { SPD_SINGLETON, 0, {6666} },   /* Only match UDP ports 6666, 7777 */
    { SPD_SINGLETON, 0, {7777} },   /* Only match UDP ports 6666, 7777 */
};

This rule matches any source or destination IP address and protocol (TCP or UDP), and a source port of 6666 and a destination port of 7777. The load generator is set to send UDP packets with those ports. This needs to be changed if other ports are used.

The second rule matches everything else and the action is set to IPSEC_DISCARD, which means drop the packet.

These rules are added to the SPD at init-time (init_ipsec() calls sa_init_static_data()) through the following call: SPD_Add()

Two other functions are defined but not currently used: SPD_Delete() and SPD_Flush().

SAD

The SAD is also statically defined in sa_init_static_data.c. There are currently two SA entries: one for the outbound SA and one for the inbound SA. Only the outbound SA needs to be defined as the inbound is just a copy except for the SPI.

To test out various encryption and hashing scenarios, this SA entry is where you need to make changes, as shown below:

sa_t sa_outb1 = {               /* First outbound SA */
        (void *)NULL,           /* auth ndps cctx */
        (void *)NULL,           /* encr ndps cctx */
        SA_OUTB1,               /* SPI */
        1,                      /* SPD rule # */
        0,                      /* seq # */
        0x0d010102,             /* local_gw_ip */
        0x0d010103,             /* remote_gw_ip */
        {{0x0,0x14,0x4f,0x3c,0x3b,0x18}},       /* remote_gw_mac */
        PORT_CIPHERTEXT_TX,     /* local_gw_nic */
//#define INTEGRITY
#ifdef INTEGRITY
        IPSEC_SVC_ESP_PLUS_INT, /* service type */
#else
        IPSEC_SVC_ESP,          /* service type */
#endif
        IPSEC_TUNNEL_MODE,      /* IPSec mode */
        0,                      /* dont use ESN */
 
        (int)NDP_CIPHER_AES128, /* encr alg */
        (int)NDP_AES128_ECB,    /* encr mode */
        /*(int)NDP_AES128_CBC,  /* encr mode */
        128/8,                  /* encr key len */
        0/8,                    /* encr IV len */
        16,                     /* encr block len */
 
        (int)NDP_HASH_SHA256,   /* auth alg */
        0,                      /* auth mode */
        256/8,                  /* auth key len */
        256/8,                  /* auth hash len - will get a default */
 
        {{TEST_ENCR_KEY_128}},  /* encr key */
        {{TEST_AUTH_KEY_256}},  /* auth key */
        //{{TEST_ENCR_IV_128}}, /* encr IV */
        {{’\000’}},             /* auth IV  - will get a default*/
        /* everything else is dynamic and does not need initing here */

The first element to note is the service type. If you want to test privacy (encryption), leave INTEGRITY commented out. No hashing will be done. If you want hashing, comment in the #define for INTEGRITY.

The next fields you might change are the encryption parameters: encr alg, encr mode, encr key len, encr IV len, encr block len, and the encr key. The IV is only used for certain modes, such as CBC for AES.

Be sure to put the proper key lengths and IV lengths in the table.

You might then need to modify the hashing algorithms in a similar manner assuming you chose INTEGRITY.

Eventually, the SPD and SAD need to be integrated with a Control Plane (CP) such that the CP determines the static databases. There are two scenarios on how this takes place: download the tables and shared memory.

Download the Tables

The CP uses the LDoms IPC mechanism to interface with Netra DPS to download (add) or modify the SPD and SA. Some functionality already exists to build these databases once the protocol is defined:

SPD_Add()

SPD_Delete()

SPD_Flush()

SADB_ADD()

Shared Memory

The CP sets up the tables in memory that is accessible from both the CP and Netra DPS and informs the Netra DPS application of updates through the LDoms IPC mechanism.

Packet Encapsulation and De-encapsulation

The main packet processing functions are called from the two processing threads which reside in ipsecgw.c.

The main plaintext packet processing thread is called PlaintextRcvProcessLoop() and it pulls a newly received packet off of a Netra DPS fast queue and calls:

IPSEC_Process_Plaintext_Pkt(mblk)

The main ciphertext packet processing thread is called CiphertextRcvProcessLoop() and it takes a packet off a fast queue and calls:

IPSEC_Process_Ciphertext_Pkt(mblk)

You can find the IPSEC_Process_Plaintext_Pkt() and IPSEC_Process_Ciphertext_Pkt() functions in:

ipsec_processing.c

The following two functions perform the hashing and invoke the actual processing code:

IPSEC_ESP_Encapsulate()

IPSEC_ESP_Deencapsulate()

The message block (mblk) contains pointers to the start and end of the incoming packets (b_rptr and b_wptr). Because plaintext packets must be prepended with a new outer IP header and ESP header, do not shift the incoming packet data down which is a copy. Therefore, when the ethernet driver asks for a new receive buffer through teja_dma_alloc(), a buffer is grabbed from the receive buffer Netra DPS memory pool. The memory pool is 2KB and returns an offset into that buffer which tells the driver where to place the packet data. This offset is set to 256 (MAX_IPSEC_HEADER), which is enough space to prepend the IPSec header information.

Packet Encapsulation

This section contains notes on how to calculate the location of the various parts of the ESP packet (outbound and inbound).

Outbound

Orig:
    OrigIPStart
    OrigIPLen (from original IP header, includes IP hdr + tcp/udp hdr + payload)
New:
    ETH_HDR_SIZE:       14
    IP_HDR_SIZE:        20
    ESP_HDR_FIXED:       8 (SPI + Seq#)
    EncIVLen:           variable - from SA or cryp_ctx
    EncBlkSize:         variable - from static structs
    AuthICVLen:         variable - from SA or cryp_ctx
 
    ESPHdrLen   = ESP_HDR_FIXED + EncIVLen
    ESPHdrStart = OrigIPStart - ESPHdrLen
    NewIPStart = OrigIPStart - (ETH_HDR_SIZE + IP_HDR_SIZE + ESP_HDR_FIXED +
                                EncIVLen)
    CryptoPadding = OrigIPLen % EncBlkSize
    ESPTrailerPadLen = 4

    HashStart = ESPHdrStart
    HashLen = ESPHdrLen + OrigIPLen + CryptoPadding + ESPTrailerPadLen
 
    CryptoStart = OrigIPStart
    CryptoLen = OrigLen + CryptoPadding + ESPTrailerPadLen
 
    NewIPLen = IP_HDR_SIZE + HashLen + AuthICVLen
 
NewPktStart---->0               1
                +---------------+
                |EtherHDR       |
                +---------------+
NewIPStart----->14              15
                +---------------+
                |IP HDR         |
                +---------------+
ESPHdrStart---->32              33
HashStart       +---------------+<====== to be hashed from here
                |ESP HDR        |
                +---------------+
                40              41
OrigIPStart---->+---------------+<====== to be crypted from here
                | Orig IP HDR   |
                +---------------+
                .
                .
                .
CryptoLen       +---------------+=== OrigIPLen + CryptoPadLen +
                                                        ESP_TRAILER_FIXED
 
 
ICVLoc--------->+---------------+=== HashStart + HashedBytesLen
HashedBytesLen                   === ESPHdrLen + OrigIPLen + CryptoPadLen +
                                                        ESP_TRAILER_FIXED;
 
        NDPSCrypt(OrigIPStart, CryptoLen)
        NDPSHashDirect(ICVLoc, HashStart, HashedBytesLen)

Inbound

OrigIPStart
OrigIPLen (from original IP header, includes IP hdr + tcp/udp hdr + payload)
HashStart = OrigIPStart + IP_HDR_SIZE
HashLen = OrigIPLen - (IP_HDR_SIZE + AuthICVLen)
 
CryptoStart = HashStart + ESP_HDR_FIXED + EncIVLen
CryptoLen = HashLen - (ESP_HDR_FIXED + EncIVLen)
 
PadOffset = HashStart + HashLen - 2
PadLen = *PadOffset
 
NewIPStart = CryptoStart
NewIPLen = same as tunneled IPLen - get from IP header

Memory Pools

The IPSec Gateway uses the Netra DPS memory pools shown in table. The names and sizes are defined in ipsecgw_config.h:

TABLE 9-6 Netra DPS Memory Pools
Memory Pool	Description
`SPDC_ENTRY_POOL`	Pool for SPDC entries stored in the SPDC hash table.
`SPI_ENTRY_POOL`	Pool for SPI entries stored in the SPI hash table. These hash tables are actually arrays indexed by the hash value (masked with the hash table size).
`SP_POOL`	Pool of SP entries.
`SA_POOL`	Pool of SA entries.
`CRYP_CTX_POOL`	Crypto context structures (maintained by the Crypto API library).

Pipelining

The two main processing threads (PlaintextRcvProcessLoop and CiphertextRcvProcessLoop) are pipelined into another thread each: one to do most of the packet encapsulation and de-encapsulation, and one to do the encryption and decryption and optional hashing.

An extra fast queue was inserted in each path, therefore, the pipeline for the eight threads are as shown below:

PlaintextRcvPacket -> 
        PlaintextRcvProcessLoop -> 
                EncryptAndHash -> 
                        CiphertextXmitPacket ->   NIC 1  ----> 
                                                                 LOOPBACK
                        <- CiphertextRcvPacket <- NIC 2  <----
                <- CiphertextRcvProcessLoop
        <- HashAndDecrypt
PlaintextXmitPacket

The two new threads (EncryptAndHash and HashAndDecrypt) reside in ipsec_processing.c rather than ipsecgw.c where the other threads reside.

The packet processing portion of this pipeline must pass the packet to the crypto part of the pipeline. Packets are normally passed on fast queues through the mblk pointer. Other vital information also needs to be passed, such as the SA pointer. Rather than allocation of a new structure to pass the data and the mblk, this data is piggy-back at the beginning of the receive buffer, which is not used. Refer to the cinfo structure defined in ipsec_processing.c.

Source Code File Description

The IPSec package comes with the following directories:

</opt/SUNWndpsc>/src/apps/ipsec-gw-ipge

This directory consists of IPSec code that supports the ipge (Ophia 1G Ethernet)

</opt/SUNWndpsc>/src/apps/ipsec-gw-nxge

This directory consists of IPSec code that supports the Sun multithreaded 10G Ethernet on PCI-E with NIU or on-chip (NIU)).

</opt/SUNWndpsc>/src/libs/ndps_crypto_api

This directory consists of Crypto API that interface to the Crypto hardware.

</opt/SUNWndpsc>/src/libs/ipsec

This directory consists of IPSec library functions.

The file descriptions in the following tables are based on the files in the
ipsec-gw-nxge directory.

TABLE 9-7 lists the build scripts.

TABLE 9-7 Build Scripts
Build Script	Description
`build_qgc`	Build for Sun Multi-thread Quad Gigabit Ethernet.
`build_10g_niu`	Build for Sun Multi-thread 10G Ethernet.
`build_niu_multi`	Build for Sun Multi-thread 10G Ethernet with multi instances configuration.
`build_ipcrypto`	Build for Crypto configuration only.

TABLE 9-8 lists the source files.

TABLE 9-8 Source Files
Source File	Description
`common.h`	Header file consists of common information.
`config.h`	Consists of receive buffer configuration information.
`debug.c`	Used when compiling in `DEBUG` mode (see `IPSEC_DEBUG` in the Makefile to turn on IPSec debugs). This file contains the debug thread that calls `teja_debugger_check_ctrl_c`().
`init.c`	Main initialization code called by Netra DPS runtime for setting up fast queues and initializing the Crypto library and the IPSec code.
`init_multi.c`	Main initialization code called by Netra DPS runtime for setting up fast queues used by the IPSec multiple instances code.
`ip_crypto.c`	Location of the main application threads for the IPSec crypto (Crypto only, no IPSec overhead).
`ipsec_niu_config.c`	Assists user to map application tasks to CPU core and hardware strands of the UltraSPARC T2 chip specific to the NIU (Network Interface Unit of the UltraSPARC T2 chip) configuration.
`ipsecgw.c`	Contains the main application threads.
`ipsecgw_config.c`	Assists user to map application tasks to CPU core and hardware strands.
ipsecgw_flow.c	Contains the classification flow entry.
ipsecgw_flow.h	Contains the definitions of the classification flow.
`ipsecgw_impl_config.h`	Contains the information related to `mblk`, receive buffer sizes, number of channels, SA, SPDC.
`ipsecgw_niu.c`	Main application thread for the NIU configuration.
`ipsecgw_niu_multi.c`	Main application thread for the NIU multi-instances configuration.
`lb_objects.h`	Contains memory pool definitions.
`mymalloc.c`	Used by the low-level crypto-code.
`mymalloc.h`	Memory pool definitions used by Crypto library.
`n2profile.c`	Contains the profiling code.
`n2profile.h`	Contains the profiling definitions.
`perf_tools.c`	Used for profiling (not available on UltraSPARC T2).
`perf_tools.h`	Used for profiling (not available on UltraSPARC T2).
`rx.c`	Packet receive code which uses ethernet API.
`skeleton.c`	Code needed to force Netra DPS runtime to include the `malloc` and `free`.
`stats.c`	Functions to collect statistics.
`tx.c`	Packet xmit code which uses ethernet API encryption and hashing algorithms.
`user_common.c`	Contains the callback functions used by the Netra DPS ethernet APIs.
`user_common.h`	Contains fast queue definitions and function prototypes.
`util.c`	Contains IPSec utility functions.

TABLE 9-9 lists the IPSec library files.

TABLE 9-9 IPSec Library Files
IPSec Library File	Description
`init_ipsec.c`	Code that is called at startup to initialize IPSec structures.
`ipsec_common.h`	Function prototypes, some common macros, other definitions.
`ipsec_defs.h`	IPSec protocol definitions and macros.
`ipsec_proc.c`	This is the main IPSec processing code. This is where all the encapsulation-encryption, de-encapsulation-decryption and hashing functions reside.
`ipsec_processing.c`	The `ipsec_proc.c` corresponding file for supporting `ipge` (Ophia 1G Ethernet).
`ipsecgw.c`	Major IPSec top level functions.
`netdefs.h`	Constant and macro definitions of common ethernet and IP protocols.
`sa_init_static_data.c`	Contains the statically-defined SAD and SPD. This is the file to modify for testing various SA configurations.
`sadb.c`	SADB functions.
`sadb.h`	SADB definitions.
`spd.c`	SPD functions.
`spd.h`	SPD definitions.

TABLE 9-10 lists the Crypto library files.

TABLE 9-10 Crypto Library Files
Crypto Library File	Description
`crypt_consts.h`	Contains various crypto constants.
`ndpscript.c`	Contains Crypto API implementations.
`ndpscrypt.h`	Contains data structures and function prototypes.
`ndpscrypt_impl.h`	Contains Crypto Context structure.

Each UltraSPARC T2 processor CPU core has a crypto unit. This unit leverages the Modular Arithmetic Unit and the Hash and Cipher Unit inside the SPU to accelerate crypto operations. There are a total of eight crypto units in an eight-core system.

Reference Applications Configurations

IPSec and Crypto has four reference application configurations:

IP with Encrypto and Decrypto

IPSec Gateway on Quad GE

IPSec Gateway on NIU 10G Interface (One Instance)

IPSec Gateway on NIU 10G Interface (Multiple Instances)

IP with Encrypto and Decrypto

This application tests how fast the pure accessing Crypto Engine can perform.

Build script: build_ipcrypto

IPSec Gateway on Quad GE

This application is the IPSec Gateway prototyped using the Sun multithreaded Quad Giga-bit Ethernet card.

Build script: build_qgc

IPSec Gateway on NIU 10G Interface (One Instance)

This application is the IPSec Gateway prototyped using the Sun multithreaded UltraSPARC T2 on-chip NIU.

Build script: build_10g_niu

The following is an example of one instance of the IPSec Gateway Application on the Sun multithreaded 10GbE with NIU interface.

Notes

If your connection is like that above, then you need to configure the traffic generator:

FrameBuilder: select IPv4 + UDP

DA MAC: 0x0,0x14,0x4f,0x9d,0x16,0xd0

IPv4: SA=69.235.4.0 DA=69.235.4.1

UDP: SP=6666 DP=7777; this has to be consistent with sp_rule1 in src/libs/ipsec/sa_init_static_data.c

Payload: Fill Pattern = 55

IPSec Gateway on NIU 10G Interface (Multiple Instances)

This application simulates multiple flow of IPSec, encrypt, and decrypt instances. It is designed to measure IPSec and Crypto overhead.

Note - This setting of the traffic generator applies to the Sun SPARC Enterprise T5120/T5220 systems. For Netra ATCA CP3260 systems, see Flow Policy for Spreading Traffic to Multiple DMA Channels).

The following is an example of multiple instances of the IPSec gateway application on the Sun multithreaded 10GbE with NIU interface.

Notes

For Sun SPARC Enterprise T5220 system (UltraSPARC T2 server) configurations:

If your connection is like that above, then you need to configure the traffic generator:

FrameBuilder: select IPv4 + UDP

DA MAC: 0x0,0x14,0x4f,0x9d,0x17,0x6

IPv4: SA=69.235.4.0 DA=69.235.4.1

UDP: SP=6666 DP=7777; this has to be consistent with sp_rule1 in src/libs/ipsec/sa_init_static_data.c

Payload: Fill Pattern = 55

In FrameBuilder, insert VLANtag L2 checked

Enable VLAN1-VLAN ID: Header select VLAN1

Field select VLAN ID, step num = 1

Start 25 End 28

In src/libs/ipsec/sa_init_static_data.c:

sa_outb1 remote_gw_mac must be {{0x0,0x14,0x4f,0x9d,0x17,0x6}}

In src/apps/ipsec-gw-nxge/src/app/ipsec_niu_config.c:

If it is port 0, do not do anything; if it is port 1, then add the following:

..., OPEN_OPEN, NXGE_10G_START_PORT+1, ...

Notes

For Netra ATCA CP3260 system (UltraSPARC T2 ATCA Blade) configurations:

If your connection is like that above, then you need to configure the traffic generator:

FrameBuilder: select IPv4 + UDP

DA MAC: 0x0,0x14,0x4f,0x9d,0x17,0x6

IPv4: SA=69.235.4.0 DA=69.235.4.1

Note - You need to select the SA IP to have COUNT 255 and step = 0.0.0.1 so that it will increment the SA IP from 0.0.0.1 to 0.0.0.255 with the step of 0.0.0.1

UDP: SP=6666 DP=7777; this has to be consistent with sp_rule1 in src/libs/ipsec/sa_init_static_data.c

Payload: Fill Pattern = 55

In src/libs/ipsec/ipsec_common.h:

If it is port 0, do not do anything; if it is port 1, then set the following:
#define PORT_PLAINTEXT_RX 1

In src/libs/ipsec/sa_init_static_data.c:

sa_outb1 remote_gw_mac must be {{0x0,0x14,0x4f,0x9d,0x17,0x6}}

In src/apps/ipsec-gw-nxge/src/app/ipsec_niu_config.c

If it is port 0, do not do anything; if it is port 1, then add the following:

..., OPEN_OPEN, NXGE_10G_START_PORT+1, ...

Flow Policy for Spreading Traffic to Multiple DMA Channels

You can specify a policy for spreading traffic into multiple DMA flows by hardware hashing or TCAM lookup (classification). TABLE 9-11 describes flow policies:

TABLE 9-11 Flow Policies
Flow Policy	Description
`HASH_IP_ADDR`	Hash on IP destination and source addresses.
`HASH_IP_DA`	Hash on IP destination address.
`HASH_IP_SA`	Hash on IP source address.
`HASH_VLAN_ID`	Hash on VLAN ID.
`HASH_PORTNUM`	Hash on physical MAC port number.
`HASH_L2DA`	Hash on L2 destination address.
`HASH_PROTO`	Hash on protocol number.
`HASH_SRC_PORT`	Hash on L4 source port number.
`HASH_DST_PORT`	Hash on L4 destination port number.
`HASH_ALL`	Hash on all of the above fields.
`TCAM_CLASSIFY`	Perform TCAM lookup.

To Enable a Flow Policy

Add the following into the gmake line:

HASH_POLICY=policy

whereas policy is one of the above specified policies

For example, to enable hash on an IP destination and source address, add FLOW_POLICY=HASH_IP_ADDR to the gmake line in the build script file build_niu_multi Remove # in front of FLOW_POLICY=HASH_IP_ADDR:

gmake -f Makefile.niu_multi NEPTUNE_CARD=2 FLOW_POLICY=HASH_IP_ADDR

Note - If you specify FLOW_POLICY=HASH_ALL which is backward compatible with Sun SPARC Enterprise T5120/T5220 systems, all fields are used.

If none of the policies in TABLE 9-11 are specified (do not specify the HASH_POLICY in the above gmake line, for example, #HASH_POLICY=HASH_IP_ADDR, a default policy will be given. When the default policy is used, all level (L2/L3/L4) header fields are used for spreading traffic.

IP Packet Forwarding Application

Source Files

Compiling the ipfwd Application

Build Scripts

Build Script Arguments

Argument Descriptions

Build Example

Default Configurations

Default System Configuration

Default ipfwd Application Configuration

Other Options

Profiling

Radix Forwarding Algorithm

Bypassing the ipfwd Operation

Multiple Forward Port Destinations

Hash Policy for Spreading Traffic to Multiple DMA Channels

ipfwd Flow Configurations

Format

Radio Link Protocol Application

Compiling the RLP Application

Build Scripts

Build Script Arguments

Arguments Descriptions

Build Example

To Run the Application

Default Configurations

Default System Configuration

Default rlp Application Configuration

Other RLP Options

RLP Policy for Spreading Traffic to Multiple DMA Channels

IPSec Gateway Reference Application

IPSec Gateway Reference Application Architecture

IPSec Gateway Reference Application Capabilities

High-Level Packet Processing

Outbound Packets

Inbound Packets

Security Association (SA) Database and Security Policy Database

Outbound Packets and Inbound Packets

Outbound Packets

Inbound Packets

Static Security Policy Database (SPD) and Security Association Database (SAD)

SPD

SAD

Download the Tables

Shared Memory

Packet Encapsulation and De-encapsulation

Packet Encapsulation

Outbound

Inbound

Memory Pools

Pipelining

Source Code File Description

Reference Applications Configurations

IP with Encrypto and Decrypto

IPSec Gateway on Quad GE

IPSec Gateway on NIU 10G Interface (One Instance)

Notes

IPSec Gateway on NIU 10G Interface (Multiple Instances)

Notes

Notes

Flow Policy for Spreading Traffic to Multiple DMA Channels

Compiling the `ipfwd` Application

Default `ipfwd` Application Configuration

Bypassing the `ipfwd` Operation

`ipfwd` Flow Configurations

Default `rlp` Application Configuration