C H A P T E R  10

Reference Applications

This chapter describes Netra DPS reference applications. Topics include:

Each UltraSPARC T2 processor CPU core has a crypto unit. This unit leverages the Modular Arithmetic Unit and the Hash and Cipher Unit inside the SPU to accelerate crypto operations. There are a total of eight crypto units in an eight-core system.

If you are using NIU ports under LDoms environment, refer to How Do I Calculate the Base PA Address for NIU/LDoms to Use with the tnsmctl Command?.


IP Packet Forwarding Application

The IP packet forwarding application (ipfwd) simulates an IPv4 (Internet Protocol Version 4) forwarding operation. When each packet is received, the program performs a forward table search (three memory searches), rewrites the MAC source and destination address, rewrites the TTL, and recalculates the IP header checksum. The packet is then sent out to a specified port. The application can either simulate the lookup or actually do a lookup.

Source Files

All ipfwd source files are located in the following package directory: /opt/SUNWndps/src/apps/ipfwd. The contents include:

Compiling the ipfwd Application

Copy the ipfwd reference application from the /opt/SUNWndps/src/apps/ipfwd directory to a desired directory location and do the build script in that location.

Build Script

TABLE 10-1 shows the IP packet forwarding (ipfwd) application build script.


TABLE 10-1 ipfwd Application Build Script

Build Script

Usage

./build

(See Build Script Arguments.)

Build ipfwd application to run on e1000g Ethernet interface on the Netra ATCA CP3060 System.

 

Build ipfwd application to run on Sun QGC (quad 1Gbps nxge Ethernet interface).

 

Build ipfwd application to run on Sun multithreaded 10G (dual 10Gbps nxge Ethernet interface).

 

Build ipfwd application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet interface) on a CMT2-based system.


Usage

./build <cmt> <type> [ldoms [diffserv] [acl] [gdb] [arp] [no_freeq]] [profiler] [2port] -hash <POLICY_NAME>

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Argument Descriptions

<cmt>

This argument specifies whether to build the ipfwd application to run on the CMT1 (UltraSPARC T1) platform or CMT2 (UltraSPARC T2) platform.

cmt1 - Build for CMT1 (UltraSPARC T1)
cmt2 - Build for CMT2 (UltraSPARC T2)

[type]

1g_1060:

Build the ipfwd application to run on e1000g Ethernet interface on a Netra ATCA CP3060 System.

4g:

Build ipfwd application to run on 10Gb Ethernet QGC (quad 1Gbps nxge Ethernet interface).

10g:

Build ipfwd application to run on 10Gb Ethernet (dual 10Gbps nxge Ethernet interface).

10g_niu:

Build ipfwd application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet interface) on a CMT2-based system.

[ldoms]:

This is an optional argument specifying whether to build the ipfwd application to run on the LDoms environment. When this flag is specified, the IP forwarding LDoms reference application will be compiled. If this argument is not specified, then the non-LDoms (standalone) application will be compiled. Refer to How Do I Calculate the Base PA Address for NIU/LDoms to Use with the tnsmctl Command?.

[acl]:

This is an optional argument to enable the Access Control List (ACL) reference application.

[gdb]:

This is an optional argument to enable gdb support in the LDoms environment.

[no_freeq]:

This option disables the use of free queues. May be used with the diffserv option in an LDoms environment.

[profiler]:

This is an optional argument that generate code with profiling enabled.

[2port]:

This is an optional argument to compile dual ports on the 10Gb Ethernet or the UltraSPARC T2 NIU.

[-hash POLICY_NAME]:

This is an optional argument used to enable flow policies. For more information, see Other IPFWD Options.

Build Example

In ...src/sys/lwrte/apps/ipfwd, pick the correct build script and run it. For example, to build for 10G Ethernet on a Netra or Sun Fire T2000 system, enter the following at your shell window.


% ./build cmt1 10g

In this example, the build script with the 10g option is used to build the IP forwarding application to run on the 10G Ethernet. The <cmt> argument is specified as cmt1 to build the application to run on UltraSPARC T1-based Netra or Sun Fire T2000 systems.


procedure icon  To Run the ipfwd Application

1. Copy the binary into the /tftpboot directory of the tftpboot server.

2. On the tftpboot server, type:


% cp your_workspace/ipfwd/code/ipfwd/ipfwd /tftpboot/ipfwd

3. On the target machine, type the following at the ok prompt:


ok boot net:,ipfwd



Note - net is an OpenBoot PROM alias corresponding to the physical path of the network.


Default Configurations

This section shows the default configurations:

Default System Configuration

 


NDPS domain
(Strand IDs)

Statistics
(Strand ID)

Other domains
(Strand IDs)

CMT1 NON-LDOM:

0 - 31

31

NA

CMT1 LDOM:

0 - 19

19

20 - 31

CMT2 NON-LDOM:

0 - 63

63

NA

CMT2 LDOM:

0 - 39

39

40 - 63


Main files that control the system configurations:

Default ipfwd Application Configuration

 


Applications
runs on

Number of Ports Used

Number of Channels per Port

Total Number of Q Instances

Total Number of Strands Used

1G (CP 1060):

2

1 (fixed)

2

6

4G-PCIE (nxge QGC):

4

1

4

12

10G-PCIE (nxge 10G):

1

4

4

12

10G-NIU (niu 10G):

1

8

8

24


Main files that control the application configurations:

Other IPFWD Options

Radix Forwarding Algorithm

To enable the Radix Forwarding Algorithm for IP forwarding, uncomment the following line from Makefile.nxge for e1000g 1Gb PCIe ethernet adapter or Sun multithreaded 10Gb and 1Gb PCIe Ethernet adapter, respectively:

- DIPFWD_RADIX

Bypassing the ipfwd Operation

To bypass the ipfwd operation (that is, receive --> transmit without forwarding operation), uncomment the following line from Makefile.nxge e1000g 1Gb PCIe ethernet adapter or Sun multithreaded 10 Gb and 1Gb PCIe Ethernet adapter, respectively:

-DIPFWD_RAW

Multiple Forward Port Destinations

When the Multiple Forward Destinations option is enabled, the output destination port is determined by the output of the Forwarding Table Lookup. Otherwise, the output destination port is the same as the input port. To enable this option, uncomment the following line from Makefile.nxge for e1000g 1Gb PCIe ethernet adapter or Sun multithreaded 10GbE Ethernet, respectively:

-DIPFWD_MULTI_QS

UltraSPARC T2 Version 2.1 Mode

The N2_1_MODE flag is enabled by default. This flag needs to be disabled when running Netra DPS on UltraSPARC T2 Version 2.2 and above.

Statistics Flags

IP Forward Static Cross Configuration

When IP Forwarding is configured as “Cross Configuration”, the IPFWD_STATIC_CROSS_CONFIG flag needs to be enabled. The following is one example of cross configuration:

Port0 ---> Port1
Port1 ---> Port0

Flow Policy for Spreading Traffic to Multiple DMA Channels

Specify a policy for spreading traffic into multiple DMA flows by hardware hashing. TABLE 10-2 describes each policy:


TABLE 10-2 Flow Policy Descriptions *

Name

Definition

IP_ADDR

Hash on IP destination and source addresses.

IP_DA

Hash on IP destination address.

IP_SA

Hash on IP source address.

VLAN_ID

Hash on VLAN ID.

PORTNUM

Hash on port number.

L2DA

Hash on L2 destination address.

PROTO

Hash on Protocol number.

SRC_PORT

Hash on source port number.

DST_PORT

Hash on destination port number.

ALL

Hash on all of the above fields.

TCAM_CLASSIFY

Performs TCAM lookup.


To enable one of the above policies, use the -hash option.

If none of the policies listed in TABLE 10-2 are specified, a default policy is given. The default policy is set to HASH_ALL. When you use the default policy, all L2/L3/L4 header fields are used for spreading traffic.

ipfwd Flow Configurations

The ipfwd_config.c file assists you in mapping application tasks to CPU core and hardware strands. Normally, mapping is set in the ipfwd_map.c file in the config directory. This configuration file is a productivity tool. This file provides a way to facilitate mapping in a quick manner without any modification to the ipfwd_map.c file.

This configuration file is not a replacement of ipfwd_hwarch.c, ipfwd_swarch.c and ipfwd_map.c. This framework is to conduct performance analysis and measurement with different system configurations. The default (*_def) configurations specified assumes a minimum of 16 threads of the system allocated for Netra DPS in ipfwd_map.c and all memory pool resources required are declared in ipfwd_swarch.c. The user still needs to specify system resources declarations and mapping in ipfwd_hwarch.c, ipfwd_swarch.c, and ipfwd_map.c. The configuration is assigned to a pointer named ipfwd_thread_config.



Note - The user has the option to bypass this file entirely and perform all the mapping in ipfwd_map.c. In this case, you would also need to modify ipfwd.c so that it does not interpret the contents of this file.


Format

Each application configuration is represented in an array of a six-element entry. Each entry (each row) represents a software task and its corresponding resources:

TROLE_ETH_NETIF_RX -- This task performs a receive function. TROLE_ETH_NETIF_TX -- This task performs a transmit function. TROLE_APP_IPFWD -- This task performs IP forwarding function.

See common.h for all definitions. If you do not want to run any software task on this hardware strand, the role field should be set to -1. By default, during initialization of the ipfwd application, the hardware strand that encounters a -1 software role is parked.



Note - A parked strand is a strand that does not consume any pipeline cycles (an inactive strand).




Note - The application can be configured such that a single memory pool is dedicated to a particular DMA channel or all DMA channels sharing a global memory pool. The default configuration is one memory pool per DMA channel.



Differentiated Services Application

The Differentiated Services (DiffServ) reference application is integrated with the IP forwarding application. The DiffServ data path consists of classifier, meter, marker, and policing components. These components provide Quality of Services (QoS) features for traffic entering the node and avoids congestion in the network. These components can be arranged in the pipeline such that each component performs specific task and propagates the result (traffic class and policing information) to the next component. The following are major features of DiffServ:

FIGURE 10-1 shows the arrangement of the components in the data path. The scheduler and queue manager are executed in a separate thread, whereas the other components are located in the forwarding thread. The following sections describe the functions of the different parts.

FIGURE 10-1 IPv4 DiffServ Internal Data Path

Classifier

This section describes the Diffserv classifiers.

Differentiated Services Code Point Classifier

The differentiated services code point (DSCP) classifier fast path component sets QoS variables (flow and color) based on the DSCP value extracted from the IPv4 packet header and directs packets to the proper next component (meter, marker, IPv4) for further processing. The DSCP classifier always remain enabled.

6-Tuple Classifier

The 6-tuple classifier fast path component performs an exact-match lookup on the IPv4 header. The classifier maintains a hash table with exact-match rules. Thus, a table lookup can fail only if there is no static rule defined. An empty rule corresponds to best-effort traffic. As a result, on a lookup failure a packet is assigned to the best-effort service (default rule) and passed on for further processing. The classifier slow path component configures the hash table used by the classifier fast path component. 6-tuple classifier can be enabled or disabled at run time.

Policing (Meter)

The three color (TC) meter implements two metering algorithms: single rate three color meter (SRTCM) and two rate three color meter (TRTCM).

Single Rate Three Color Marker (SRTCM)

The Single-Rate Three-Color Marker (SRTCM) meters an IP packet stream and marks its packets green, yellow, or red. Marking is based on a Committed Information Rate (CIR) and two-associated burst sizes, a Committed Burst Size (CBS) and an Excess Burst Size (EBS). A packet is marked green if it does not exceed the CIR; yellow if it does exceed the CBS, but not the EBS; and red, otherwise.

Two Rate Three Color Marker (TRTCM)

The Two-Rate Three-Color Marker (TRTCM) meters an IP packet stream and marks its packets green, yellow, or red. A packet is marked red if it exceeds the Peak Information Rate (PIR). Otherwise, it is marked either yellow or green depending on whether it exceeds or does not exceed the Committed Information Rate (CIR).

DSCP Marker

The DSPC marker updates the type of service (TOS) field in the IPv4 header and recomputes the IPv4 header checksum

Shaping

Deficit Round Robin Scheduler

The deficit round robin (DRR) scheduler schedules packets in a flexible queuing policy with priority concept. With this scheduler, the parameters are the number of sequential service slots that each queue can get during its service turn. The number of services for each queue depends on the value of its parameter called deficit factor. The deficit of queue is reduced as the scheduler schedules packets from that queue. The maximum deficit of each queue can be configured and is called weight of that queue. The DRR scheduler will schedule the packets by considering the packet size of the packet at the top of the queue. Queues are still served in round robin fashion (cyclically) in a preassigned order.

Queue Manager

The Queue Manager performs enqueue and dequeue operations on the queues. The queue manager manages an array of queues, with each queue corresponding to a particular per hop behavior (PHB), for queuing packets per port. The Queue Manager receives enqueue requests from the IPv4/DiffServ pipeline. Upon receiving the enqueue request, the queue manager places the packet into the queue corresponding to the PHB indicated by the DSCP value in the packet.The Queue Manager maintains the state for each queue and uses the tail drop mechanism in case of congestion.

The Queue Manager receives the dequeue requests from the scheduler. The dequeue request consists of the PHB and the output port. Packets from the queue corresponding to this PHB and output port is dequeued and the dequeued packet is placed on the transmit queue for the output port.

Building the DiffServ Application

To build the DiffServ application, specify the diffserv keyword on the build script command line. All files of the DiffServ data path implementation are located in the diffserv subdirectory of stc/app in the IP forwarding application. The DiffServ application requires an LDoms environment, as all configuration is through an application running on a Solaris control domain that communicates with the data plane application through IPC.

For example, to build the DiffServ application to make use of both NIU ports on an UltraSPARC T2-based system, use the following command:


./build cmt2 10g_niu ldoms diffserv no_freeq 2port

DiffServ Command-Line Interface Implementation

The IPv4 Forwarding Information Base (FIB) table configuration (fibctl) command-line interface (CLI) has been extended to support configuration of DiffServ tables. This support behavior is the same as the FIB table configuration protocol over IPC between the control plane and data plane LDoms. Support is provided for configuring (choosing) the following DiffServ tables:

To build the extended control utility, issue the following command in the src/solaris subdirectory of the IP forwarding reference application.


gmake DIFFSERV=on

Command-Line Interface for the IPv4-DiffServ Application

DSCP Classifier

Add DSCP Entry

Description

This command is used to add the DSCP classifier entry in DSCP table.

Command syntax

diffserv dscp add DSCP_value port_number flow_id color_id class_id next_block

Parameters

DSCP_value - DSCP value should be greater than 0 and less than 64.

port_number - Port number should be less than NUM_PORTS.

flow_id - ID used to identify the traffic flow to which the packet belongs.

color_id - ID should be green, yellow, or red.

class_id - ID used to identify the queue number within an output port.

next_block - Next block should be meter, marker, or fwder.

Example

diffserv dscp add 1 0 1 green 1 meter

Delete DSCP Entry

Description

This command is used to delete DSCP classifier entry from DSCP table.

Command syntax

diffserv dscp delete DSCP_value port_number

Parameters

DSCP_value - DSCP value should be greater than 0 and less than 64.

port_number - Port number should be less than NUM_PORTS.

Example

diffserv dscp delete 1 0

Update DSCP Entry

Description

This command is used to update the existing DSCP classifier entry in DSCP table.

Command syntax

diffserv dscp update DSCP_value port_number flow_id color_id class_id next_block

Parameters

DSCP_value - DSCP value should be greater than 0 and less than 64.

port_number - Port number should be less than NUM_PORTS.

flow_id - ID used to identify the traffic flow to which the packet belongs.

color_id - ID should be green, yellow, or red.

class_id - ID used to identify the queue number within an output port.

next_block - Next block should be meter, marker, or fwder.

Example

diffserv dscp update 1 0 1 yellow 1 fwder

Purge DSCP Table

Description

This command is used to purge the DSCP table.

Command syntax

diffserv dscp purge

Display DSCP Table Entries

Description

This command is used to display the DSCP table.

Command syntax

diffserv dscp display

6-Tuple Classifier

Add 6-Tuple Entry

Description

This command is used to add classifier 6-tuple entry in 6-tuple table.

Command syntax

diffserv class6tuple add SrcIp DestIp Proto Tos SrcPrt DestPrt IfNum flow_id color_id next_block class_id

Parameters

SrcIp - Source IP address (for example, 192.168.1.5) in the IP header of packet.

DestIp - Destination IP address (for example, 192.168.1.5) in the IP header of packet.

Proto - IP protocol field in the IP header of packet.

Tos - Differentiated services code point (6 bits of TOS field).

SrcPrt - Source port number in the TCP/UDP header packet.

DestPrt - Destination port number in the TCP/UDP header packet.

IfNum - Input port starting form port 0, on which the packet is received.

flow_id - ID used to identify the traffic flow to which the packet belongs.

color_id - ID used to identify the packet drop precedence level (green, yellow, or red).

next_block - Used to identify the next packet processing block meter, marker, and fwder.

class_id - ID used to identify the queue number within an output port
(for example: ef, af0, af1, af2, af3, be).

Example

diffserv class6tuple add 211.2.9.195 192.168.115.76 17 16 61897 2354 0 50 green meter 44

Delete 6-Tuple Entry

Description

This command is used to delete 6-tuple classifier entry from 6-tuple table.

Command syntax

diffserv class6tuple delete SrcIp DestIp Proto Tos SrcPrt DestPrt IfNum

Parameters

SrcIp - Source IP address (for example, 192.168.1.5) in the IP header of packet.

DestIp - Destination IP address (for example, 192.168.1.5) in the IP header of packet.

Proto - IP protocol field in the IP header of packet.

Tos - Differentiated services code point (6 bits of TOS field).

SrcPrt - Source port number in the TCP/UDP header packet.

DestPrt - Destination port number in the TCP/UDP header packet.

IfNum - Input port starting form port 0, on which the packet is received.

Example

diffserv class6tuple delete 211.2.9.195 192.168.115.76 17 16 61897 2354 0

Update 6-Tuple Entry

Description

This command is used to update the existing 6-tuple classifier entry in 6-tuple table.

Command syntax

diffserv class6tuple update SrcIp DestIp Proto Tos SrcPrt DestPrt IfNum flow_id color_id next_block class_id

Parameters

SrcIp - Source IP address (for example, 192.168.1.5) in the IP header of packet.

DestIp - Destination IP address (for example, 192.168.1.5) in the IP header of packet.

Proto - IP protocol field in the IP header of packet.

Tos - Differentiated services code point (6 bits of TOS field).

SrcPrt - Source port number in the TCP/UDP header packet.

DestPrt - Destination port number in the TCP/UDP header packet.

IfNum - Input port starting form port 0, on which the packet is received.

flow_id - ID used to identify the traffic flow to which the packet belongs.

color_id - ID used to identify the packet drop precedence level (green, yellow, or red).

next_block - Used to identify the next packet processing block meter, marker, and fwder.

class_id - ID used to identify the queue number within an output port
(for example: ef, af0, af1, af2, af3, be).

Example

diffserv class6tuple update 211.2.9.195 192.168.115.76 17 16 61897 2354 0 50 red marker 44

Purge 6-Tuple Table

Description

This command is used to purge the 6-tuple table.

Command syntax

diffserv class6tuple purge

Display 6-Tuple Table Entries

Description

This command is used to display the 6-tuple table.

Command syntax

diffserv class6tuple display

6-Tuple Enable/Disable

Description

This command is used to enable/disable the 6-tuple table.

Command syntax

diffserv class6tuple enable/disable

Example

diffserv class6tuple enable

diffserv class6tuple disable

TC Meter

Add TC Meter Entry

Description

This command is used to add a meter instance in TC meter table.

Command syntax

diffserv meter add flow_id CBS EBS CIR EIR green_dscp green_action yellow_dscp yellow_action red_dscp red_action meter_type stat_flag

Parameters

flow_id - ID used to identify the traffic flow to which the packet belongs.

CBS - The value of the Committed Burst Size (CBS) is larger than 0, it is larger than or equal to the size of the largest possible IP packet in the stream. cbs is measured in bytes.

EBS - The value of the Excess Burst Size (EBS) is larger than 0, it is larger than or equal to the size of the largest possible IP packet in the stream. EBS is measured in bytes.

CIR - Committed Information Rate (CIR) at which a traffic source is signed up to send packets to the meter instance. It is measured in bytes/sec. The cir should be in M-bytes per seconds.

EIR - Excess Information Rate (EIR) at which a traffic source is signed up to send packets to the meter instance. It is measured in bytes/sec. This is used only when TRTCM is enabled. The eir should be in M-bytes per seconds.

green_dscp - DSCP packet mark value for green packets.

green_action - Select the next packet processing block for green packets (drop, fwder, and marker).

yellow_dscp - DSCP packet mark value for yellow packets.

yellow_action - Select the next packet processing block for yellow packets (drop, fwder, and marker).

red_dscp - DSCP packet mark value for red packets.

red_action - Select the next packet processing block for red packets (drop, fwder, and marker).

meter_type

0 - TRTCM color aware

1 - TRTCM color blind

2 - SRTCM color aware

3 - SRTCM color blind

stat_flag

0 - Statistics Disable or 1 - Statistics Enable

Example

diffserv meter add 1 1500 1500 1 1 12 marker 13 drop 14 drop 1 1

Delete TC-Meter Entry

Description

This command is used to delete a meter instance in TC meter table.

Command syntax

diffserv meter delete flow_id

Parameters

flow_id - ID used to identify the traffic flow to which the packet belongs.

Example

diffserv meter delete 1

Update TC-Meter Entry

Description

This command is used to update a meter instance in TC meter table.

Command syntax

diffserv meter update flow_id CBS EBS CIR EIR green_dscp green_action yellow_dscp yellow_action red_dscp red_action meter_type stat_flag

Parameters

flow_id - ID used to identify the traffic flow to which the packet belongs.

CBS - The value of the Committed Burst Size (CBS) is larger than 0, it is larger than or equal to the size of the largest possible IP packet in the stream. cbs is measured in bytes.

EBS - The value of the Excess Burst Size (EBS) is larger than 0, it is larger than or equal to the size of the largest possible IP packet in the stream. EBS is measured in bytes.

CIR - Committed Information Rate (CIR) at which a traffic source is signed up to send packets to the meter instance. It is measured in bytes/sec. The cir should be in M-bytes per seconds.

EIR - Excess Information Rate (EIR) at which a traffic source is signed up to send packets to the meter instance. It is measured in bytes/sec. This is used only when TRTCM is enabled. The eir should be in M-bytes per seconds.

green_dscp - DSCP packet mark value for green packets.

green_action - Select the next packet processing block for green packets (drop, fwder, and marker).

yellow_dscp - DSCP packet mark value for yellow packets.

yellow_action - Select the next packet processing block for yellow packets (drop, fwder, and marker).

red_dscp - DSCP packet mark value for red packets.

red_action - Select the next packet processing block for red packets (drop, fwder, and marker).

meter_type

0 - TRTCM color aware

1 - TRTCM color blind

2 - SRTCM color aware

3 - SRTCM color blind

stat_flag

0 - Statistics Disable or 1 - Statistics Enable

Example

diffserv meter update 1 1500 1500 1 1 12 marker 13 drop 14 drop 0 0

Purge TC-Meter Table

Description

This command is used to purge meter table.

Command syntax

diffserv meter purge

Display TC-Meter Table Entries

Description

This command used to display the TC meter table.

Command syntax

diffserv meter display

Display TC-Meter statistics

Description

This command used to display the TC meter statistics.

Command syntax

diffserv meter stats flow_id

Parameters

flow_id - ID used to identify the traffic flow to which the packet belongs.

Example

diffserv meter stats 1

Scheduler

Add scheduler Entry

Description

This command is used to configure weight for all AF classes and maximum rate limit for EF class.

Command syntax

diffserv scheduler add output_port class_id weight

Parameters

output_port - Port number should be less than NUM_PORTS.

class_id - ID used to identify the queue number within an output port
(for example: ef, af0, af1, af2, af3, be).

weight - Maximum number of bytes to be scheduled. If class is ef the weight will be bytes/seconds, otherwise, number of bytes.

Example

diffserv scheduler add 1 af1 128

Update Scheduler Entry

Description

This command is used to update weight for all AF classes and maximum rate limit for EF class.

Command syntax

diffserv scheduler update output_port class_id weight

Parameters

output_port - Port number should be less than NUM_PORTS.

class_id - ID used to identify the queue number within an output port
(for example: ef, af0, af1, af2, af3, be).

weight - Maximum number of bytes to be scheduled. If class is ef the weight will be bytes/seconds, otherwise, number of bytes.

Example

diffserv scheduler update 1 af1 256

Display scheduler Entry

Description

This command is used to display scheduler table entries.

Command syntax

diffserv scheduler display output_port

Parameters

output_port - Port number should be less than NUM_PORTS.

Example

diffserv scheduler display 1

DiffServ References

TABLE 10-3 lists DiffServ references.


TABLE 10-3 DiffServ References

Reference

Document Descriptions

RFC 2474

Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers

RFC 2475

An Architecture for Differentiated Services

RFC 2597

Assured Forwarding PHB Group

RFC 2697

A Single Rate Three Color Marker

RFC 3246

An Expedited Forwarding PHB (Per-Hop Behavior)

RFC 3260

New Terminology and Clarifications for DiffServ

RFC 4115

A Differentiated Service Two-Rate, Three-Color Marker with Efficient Handling of in-Profile Traffic



Access Control List Application

The Access Control List (ACL) reference application is integrated with the IP forwarding application. The ACL component classifies IPv4 packets using a set of rules. The classification can be done using the source and destination addresses and ports, as well as the protocol and the priority of the packet.

The algorithms (trie, bspl, and hicut) are used in the ACL library trade memory for speed. The rules are preprocessed to achieve a high lookup rate while using a lot of memory.

Building the ACL Application

To build the IP packet forwarding (ipfwd) application with ACL, specify the acl keyword on the build script command line. For example:


% ./build cmt2 10g_niu ldoms acl

Running the ACL Application

The ipfwd application with ACL requires an LDoms environment, since all configurations are done through an application running on a Solaris control domain. The Netra DPS domain needs to be configured with at least 8 Gbytes of memory, which is a requirement for the ACL application.


procedure icon  To Configure the Environment for ACL

1. Enable shared memory by adding the following line to the /etc/system file:


set ldc:ldc_shmem_enabled = 1

2. Enable the ACL communication channel between the Netra DPS domain and the Solaris control domain.

To do this, a special configuration channel must be set up between these domains. The channel is established as follows:


% ldm add-vdpcs shmem-server Netra DPS domain name
% ldm add-vdpcc shmem-client shmem-server Solaris control domain name

3. Add /opt/SUNWndpsd/lib to LD_LIBRARY_PATH.

Command-Line Interface for the ACL Application

The acltool is a command-line tool that sends commands to the ACL engine running in the Netra DPS domain. The interface is similar to iptables(8). The major difference is that it does not take a chain as a parameter. The acltool can be found in the /opt/SUNWndpsd/bin directory.

Following is a description of the various acltool commands and options.


# acltool --help

Usage

acltool command [options]

Commands

Help command

-h --help Prints usage help.

Control commands

--init algorithm Initializes ACL engine using algorithm for packet lookup.

--start Starts the packet classification.

--stop Stops the packet classification.

--status Prints the status of the ACL engine.

-c --config-file filename Reads rule commands from the configuration file.

Rule commands

-A --append rule Appends a rule.

-D --delete rule Removes the matching rule.

-L --list Lists all rules.

-F --flush Flushes (removes) all rules.

Rule Specification Options

-p --protocol num Protocol (tcp, udp, icmp) or protocol number

-s --source ip[/mask] Source ip prefix

-d --destination ip[/mask] Destination ip prefix

-j --jump num Where to jump (action)

-g --goto num Same as --jump

--sport num[:num] Source protocol port

--source-port num[:num] Source protocol port

--dport num[:num] Destination protocol port

--destination-port num[:num] Destination protocol port

--tos num Type of service


Radio Link Protocol Application

The Radio Link Protocol (RLP) application (rlp) simulates Radio Link Protocol operation, which is one of the protocols in the CDMA-2000 High Rate Packet Data interfaces (HRPD-A). This application implements the forwarding direction fully, with packets flowing from PDSN --> AN --> AT (that is, Packet Data Serving Node to Access Network to Access Terminal). Reverse direction support is also implemented, but requires an AT side application which can generate NAKs (Negative Acknowledges). The application must be modified to process reverse traffic.

Compiling the RLP Application

Copy the rlp reference application from the /opt/SUNWndps/src/apps/rlp directory to a desired directory location and do the build script in that location.

Build Script

TABLE 10-4 shows the radio link protocol (rlp) application build script.


TABLE 10-4 rlp Application Build Script

Build Script

Usage

./build

(See Argument Descriptions.)

Build rlp application to run on e1000g Ethernet interface.

 

Build rlp application to run on e1000g Ethernet interface on the Netra ATCA CP3060 System.

 

Build rlp application to run on Sun QGC (quad 1Gbps nxge Ethernet interface).

 

Build rlp application to run on Sun multithreaded 10G (dual 10Gbps nxge Ethernet interface).

 

Build rlp application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet interface) on a CMT2-based system.


Usage

./build <cmt> <type> [ldoms] [arp] [profiler][-hash FLOW_POLICY]

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Argument Descriptions

<cmt>

This argument specifies whether to build the ipfwd application to run on the CMT1 (UltraSPARC T1) platform or CMT2 (UltraSPARC T2) platform.

cmt1 - Build for CMT1 (UltraSPARC T1)
cmt2 - Build for CMT2 (UltraSPARC T2)

[type]

1g_1060:

Build the rlp application to run on e1000g Ethernet interface on a CMT1-based system.

4g:

Build rlp application to run on QGC (quad 1Gbps nxge Ethernet interface).

10g:

Build rlp application to run on 10Gb Ethernet (dual 10Gbps nxge Ethernet interface).

10g_niu:

Build rlp application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet interface) on a CMT2-based system.

[ldoms]

This is an optional argument specifying whether to build the rlp application to run on the LDoms environment. When this flag is specified, the rlp LDoms reference application will be compiled. If this argument is not specified, then the non-LDoms (standalone) application will be compiled. Refer to How Do I Calculate the Base PA Address for NIU/LDoms to Use with the tnsmctl Command?.

[arp]

This is an optional argument to enable arp and can run only on the LDoms environment.

[profiler]

This is an optional argument that generate code with profiling enabled.

[-hash FLOW_POLICY]:

This is an optional argument used to enable flow policies. For more information, see Other RLP Options.

Build Example

In ...src/apps/rlp, pick the correct build script and run it. For example, For example, to build for 10G Ethernet on a Netra or Sun Fire T2000 system, enter the following at your shell window:


% ./build cmt1 10g

In this example, the 10g option is used to build the RLP application to run on the Sun multithreaded 10Gb Ethernet. The <cmt> argument is specified as cmt1 to build the application to run on UltraSPARC T1-based Netra or Sun Fire T2000 systems.


procedure icon  To Run the Application

1. Copy the binary into the /tftpboot directory of the tftpboot server, and perform.

2. On the tftpboot server, type:


% cp your_workspace/rlp/code/rlp/rlp /tftpboot/rlp

3. On the target machine, type the following at the ok prompt:


ok boot net:,rlp



Note - net is an OpenBoot PROM alias corresponding to the physical path of the network.


Default Configurations

This section shows the default configurations.

Default System Configuration

 


NDPS domain
(Strand IDs)

IPC Polling,

Statistics
(Strand ID)

Other domains
(Strand IDs)

CMT1 NON-LDOM:

0 - 31

31

NA

CMT1 LDOM:

0 - 19

18, 19

20 - 31

CMT2 NON-LDOM:

0 - 63

63

NA

CMT2 LDOM:

0 - 39

38, 39

40 - 63


Main files that control the system configurations are:

Default rlp Application Configuration

 


Applications
runs on

Number of Ports Used

Number of Channels per Port

Total Number of Q Instances

Total Number of Strands Used

1G (CP 1060):

2

1 (fixed)

2

6

4G-PCIE (nxge QGC):

4

1

4

12

10G-PCIE (nxge 10G):

1

4

4

12

10G-NIU (niu 10G):

1

8

8

24


Main files that control the application configurations are:

Other RLP Options


procedure icon  To Bypass the rlp Operation

single-step bullet  To bypass the rlp operation (that is, receive --> transmit without rlp_process operation), uncomment the following line from Makefile.nxge for 1Gb PCIe ethernet adapter or Sun multithreaded 10 Gb and 1Gb PCIe Ethernet adapter, respectively:

-DIPFWD_RAW



Note - This action disables the RLP processing operation only, the queues are still used. This is not the default option.



procedure icon  To Use One Global Memory Pool

By default, the RLP application uses a single global memory pool for all the DMA channels.

1. Enable the single memory pool by using the following flag:.

-DFORCEONEMPOOL

2. Update the rlp_swarch.c file to use individual memory pools.

Flow Policy for Spreading Traffic to Multiple DMA Channels

The user can specify a policy for spreading traffic into multiple DMA flows by hardware hashing or by hardware TCAM lookup (classification). Refer to TABLE 10-2 for flow policy options.


IPSec Gateway Application

The IPSec Gateway Reference Application implements the IP Encapsulating Security Payload (ESP) protocol using Tunnel Mode. This application allows two Gateways (or a host and a gateway) to securely send packets over an unsecure network with the original IP packet tunneled and encrypted (privacy service). This application also implements the optional integrity service allowing the ESP header and tunneled IP packet to be hashed on transmit and verified on receipt.

IPSec Gateway Application Architecture

The design calls for six Netra DPS threads in a classic architecture where four threads are dedicated to packet reception and transmission (two receivers, two senders). In this architecture, a thread takes plain text packets and encapsulates and encrypts them, as well as a thread that de-encapsulates and decrypts. The architecture is shown in FIGURE 10-2.

FIGURE 10-2 IPSec Gateway Application Architecture

Refer to Request for Comments (RFC) documents for a description of IPSec and the ESP protocol:

The IPSec RFC refers to outbound and inbound packets. These design notes refer to these terms.

IPSec Gateway Application Capabilities

IPSec is a complex protocol. This application handles the following most common processing:

Contains the type of service to provide (privacy, integrity), crypto and hashing types and keys to be used for a session, among other housekeeping items. An item in the SADB is called a Security Association (SA). An SA can be unique to one connection, or shared among many.

A partial implementation that is used to contain “selectors” that designate what action should be taken on a packet based on the source and destination IP addresses, protocol, and port numbers.

A critical cache used to quickly look up the SA to use for packets coming from the plaintext side. The packet source and destination addresses and ports are hashed to find the action to take on the packet (discard, pass-through, or IPSec protect) and the SA.

A cache is used to quickly look up an SA for ESP packets entering the system from the ciphertext side. The Security Parameter Index is in the ESP header.

This IPSec implementation uses the ESP protocol (it does not currently handle AH, though ESP provides most of the AH functionality). Tunnel Mode is used to encapsulate (tunnel) IP packets between hosts and interface to a peer gateway machine.

High-Level Packet Processing

The following describes functions of outbound and inbound packet processing:

Outbound Packets

Inbound Packets

Security Association (SA) Database and Security Policy Database

The packet encapsulation and encryption code is straight-forward once you have a pointer to the SA. The SA contains the following information:

Packet encapsulation and de-encapsulation is just a matter of determining where the new IP header goes or where the original IP header is, building the new IP header, and invoking the crypto APIs on the correct packet location and length. For the IPSec implementation, you need to find the SA to use when a packet is received (either outbound on inbound). The user must use software hashing and hash table lookups for every packet. Note that when this is ported to Sun multithreaded 10G Ethernet on PCIe, the packet classification features speed-up this hashing.

The following sections describe how the SA is obtained for each packet.

Outbound Packets and Inbound Packets

The following sections describe how the SA is obtained for each packet.

Outbound Packets

The user must look at the packet selectors to determine what action to take, either DISCARD, PASS-THROUGH (as is), or PROTECT. The selectors are the source and destination IP addresses, the source and destination ports, and the protocol (TCP, UDP, and others).

The action to take is stored in the Security Policy Database (SPD). For this application, the complete SPD is not implemented. A static SPD exists that consists of rules that must be searched in order using the packet selectors.

For each selector (source IP, destination IP, source port, destination port, protocol), the rule states one of the following:

If all selectors match the rules, use the SP entry to determine what action to take. If it is PROTECTED (IPSec), the inbound and outbound Security Parameter Index (SPI) knows which SA to use. This implies the following:

The last rule in the SPD should be a catch-all that says DISCARD the packet.

The SPD structures and definitions can be found in spd.h.

The source code for the SPD can be found in spd.c.

The function used to lookup a rule is SPD_Search(), which is passed the selector values from the packet.

The above lookup is complex for every packet. Because of this, a cache called the SPD-Cache is maintained. The first time you lookup a particular connection, create a SPDC structure, hash the selectors, and place this SPDC in a hash table.

When packet that uses the exact combination of selectors comes in, it needs to be looked up in the SPDC hash table using the SPDC_HASH() function. If found, immediate access to the SA is made.

The definitions of this SPDC and the function can be found in sadb.h and sadb.c, respectively.

This application does not hash on the protocol type because UDP or TCP protocols types are assumed due to the presence of the source and destination ports in the packets.

The SPDC hash table is defined as:


spdc_entry_t *spdc_hash_table[SPDC_HASH_TABLE_SIZE];

The primary function used to lookup an SPDC entry is:


spdc_e *spdc_hash_lookup_from_iphdr(iphdr)

For this hash table, take the hash value, mask off the hash table size - 1, then index into this table to get an entry. The application then compares the entry for a match, and if there is not a match, the function will walk the chain until one is found.

Inbound Packets

Inbound IPSec packets contain an ESP header with an SPI. The application parses the SPI, hashes it using SPI_HASH_FROM_SPI(), looks it up in the SPI hash table, and accesses the SA pointer from there. The application cannot use the same hashing as done for outbound packets since the selectors (source/destination IP address and ports) have been encapsulated and encrypted. Decryption cannot be done until the SA is looked up.

The SPI hash table is defined as:


spi_entry_t *spi_hash_table[SPI_HASH_TABLE_SIZE];

Static Security Policy Database (SPD) and Security Association Database (SAD)

For the purposes of the application, statically define the test SPD and SAD in compile-time initialized C-code in the following C file: sa_init_static_data.c

SPD

Two SPD rules are defined.

 


sp_t sp_rule1 = {
    1,                      /* rule # */
    SA_OUTB1,               /* outb_spi */
    SA_INB1,                /* inb_spi */
    IPSEC_PROTECT,          /* action */
    SPD_PROTOCOL_MATCH_ALL, /* match on all protocols */
    { SPD_MATCH_ALL },      /* match all connections for now */
    { SPD_MATCH_ALL },
    { SPD_SINGLETON, 0, {6666} },   /* Only match UDP ports 6666, 7777 */
    { SPD_SINGLETON, 0, {7777} },   /* Only match UDP ports 6666, 7777 */
};

This rule matches any source or destination IP address and protocol (TCP or UDP), and a source port of 6666 and a destination port of 7777. The load generator is set to send UDP packets with those ports. This needs to be changed if other ports are used.

These rules are added to the SPD at init-time (init_ipsec() calls sa_init_static_data()) through the following call: SPD_Add()

Two other functions are defined but not currently used: SPD_Delete() and SPD_Flush().

SAD

The SAD is also statically defined in sa_init_static_data.c. There are currently two SA entries: one for the outbound SA and one for the inbound SA. Only the outbound SA needs to be defined since the inbound SA is just a copy of the outbound SA, except for the SPI.

To perform various encryption and hashing scenarios, this SA entry is where the user needs to make changes, as shown below:


sa_t sa_outb1 = {               /* First outbound SA */
        (void *)NULL,           /* auth ndps cctx */
        (void *)NULL,           /* encr ndps cctx */
        SA_OUTB1,               /* SPI */
        1,                      /* SPD rule # */
        0,                      /* seq # */
        0x0d010102,             /* local_gw_ip */
        0x0d010103,             /* remote_gw_ip */
        {{0x0,0x14,0x4f,0x3c,0x3b,0x18}},       /* remote_gw_mac */
        PORT_CIPHERTEXT_TX,     /* local_gw_nic */
//#define INTEGRITY
#ifdef INTEGRITY
        IPSEC_SVC_ESP_PLUS_INT, /* service type */
#else
        IPSEC_SVC_ESP,          /* service type */
#endif
        IPSEC_TUNNEL_MODE,      /* IPSec mode */
        0,                      /* dont use ESN */
 
        (int)NDP_CIPHER_AES128, /* encr alg */
        (int)NDP_AES128_ECB,    /* encr mode */
        /*(int)NDP_AES128_CBC,  /* encr mode */
        128/8,                  /* encr key len */
        0/8,                    /* encr IV len */
        16,                     /* encr block len */
 
        (int)NDP_HASH_SHA256,   /* auth alg */
        0,                      /* auth mode */
        256/8,                  /* auth key len */
        256/8,                  /* auth hash len - will get a default */
 
        {{TEST_ENCR_KEY_128}},  /* encr key */
        {{TEST_AUTH_KEY_256}},  /* auth key */
        //{{TEST_ENCR_IV_128}}, /* encr IV */
        {{’\000’}},             /* auth IV  - will get a default*/
        /* everything else is dynamic and does not need initing here */

The first element to note is the service type. If the user wants to test privacy (encryption), leave INTEGRITY commented out. No hashing will be done. If the user wants hashing, comment in the #define for INTEGRITY.

The next fields you might change are the encryption parameters: encr alg, encr mode, encr key len, encr IV len, encr block len, and the encr key. The IV is only used for certain modes, such as CBC for AES.

It is important to ensure the proper key lengths and IV lengths in the table.

The user might then need to modify the hashing algorithms in a similar manner assuming you chose INTEGRITY.

Eventually, the SPD and SAD need to be integrated with a Control Plane (CP) such that the CP determines the static databases. There are two scenarios on how this takes place: download the tables and shared memory.

Download the Tables

The CP uses the LDoms IPC mechanism to interface with Netra DPS to download (add) or modify the SPD and SA. Some functionality already exists to build these databases once the protocol is defined:

Shared Memory

The CP sets up the tables in memory that is accessible from both the CP and Netra DPS and informs the Netra DPS application of updates through the LDoms IPC mechanism.

Packet Encapsulation and De-encapsulation

The main packet processing functions are called from the two processing threads which reside in ipsecgw.c.

The main plaintext packet processing thread is called PlaintextRcvProcessLoop() and it pulls a newly received packet out of a Netra DPS fast queue and calls:

IPSEC_Process_Plaintext_Pkt(mblk)

The main ciphertext packet processing thread is called CiphertextRcvProcessLoop() and it takes a packet off a fast queue and calls:

IPSEC_Process_Ciphertext_Pkt(mblk)

Find the IPSEC_Process_Plaintext_Pkt() and IPSEC_Process_Ciphertext_Pkt() functions in:
ipsec_proc.c

The following two functions perform the hashing and invoke the actual processing code:

The message block (mblk) contains pointers to the start and end of the incoming packets (b_rptr and b_wptr). Because plaintext packets must be prepended with a new outer IP header and ESP header, the user application should not shift the incoming packet data down which is a copy. Therefore, when the ethernet driver asks for a new receive buffer through teja_dma_alloc(), a buffer is grabbed from the receive buffer Netra DPS memory pool. The memory pool size is 2KB and the memory pool function returns an offset into that buffer which tells the driver where to place the packet data. This offset is set to 256 (MAX_IPSEC_HEADER), which is enough space to prepend the IPSec header information.

Packet Encapsulation

This section contains notes on how to calculate the location of the various parts of the ESP packet (outbound and inbound).

Outbound

 


Orig:
    OrigIPStart
    OrigIPLen (from original IP header, includes IP hdr + tcp/udp hdr + payload)
New:
    ETH_HDR_SIZE:       14
    IP_HDR_SIZE:        20
    ESP_HDR_FIXED:       8 (SPI + Seq#)
    EncIVLen:           variable - from SA or cryp_ctx
    EncBlkSize:         variable - from static structs
    AuthICVLen:         variable - from SA or cryp_ctx
 
    ESPHdrLen   = ESP_HDR_FIXED + EncIVLen
    ESPHdrStart = OrigIPStart - ESPHdrLen
    NewIPStart = OrigIPStart - (ETH_HDR_SIZE + IP_HDR_SIZE + ESP_HDR_FIXED +
                                EncIVLen)
    CryptoPadding = OrigIPLen % EncBlkSize
    ESPTrailerPadLen = 4

 


    HashStart = ESPHdrStart
    HashLen = ESPHdrLen + OrigIPLen + CryptoPadding + ESPTrailerPadLen
 
    CryptoStart = OrigIPStart
    CryptoLen = OrigLen + CryptoPadding + ESPTrailerPadLen
 
    NewIPLen = IP_HDR_SIZE + HashLen + AuthICVLen
 
NewPktStart---->0               1
                +---------------+
                |EtherHDR       |
                +---------------+
NewIPStart----->14              15
                +---------------+
                |IP HDR         |
                +---------------+
ESPHdrStart---->32              33
HashStart       +---------------+<====== to be hashed from here
                |ESP HDR        |
                +---------------+
                40              41
OrigIPStart---->+---------------+<====== to be crypted from here
                | Orig IP HDR   |
                +---------------+
                .
                .
                .
CryptoLen       +---------------+=== OrigIPLen + CryptoPadLen +
                                                        ESP_TRAILER_FIXED
 
 
ICVLoc--------->+---------------+=== HashStart + HashedBytesLen
HashedBytesLen                   === ESPHdrLen + OrigIPLen + CryptoPadLen +
                                                        ESP_TRAILER_FIXED;
 
        NDPSCrypt(OrigIPStart, CryptoLen)
        NDPSHashDirect(ICVLoc, HashStart, HashedBytesLen)

Inbound

 


OrigIPStart
OrigIPLen (from original IP header, includes IP hdr + tcp/udp hdr + payload)
HashStart = OrigIPStart + IP_HDR_SIZE
HashLen = OrigIPLen - (IP_HDR_SIZE + AuthICVLen)
 
CryptoStart = HashStart + ESP_HDR_FIXED + EncIVLen
CryptoLen = HashLen - (ESP_HDR_FIXED + EncIVLen)
 
PadOffset = HashStart + HashLen - 2
PadLen = *PadOffset
 
NewIPStart = CryptoStart
NewIPLen = same as tunneled IPLen - get from IP header

Memory Pools

The IPSec Gateway uses the Netra DPS memory pools shown in table. The names and sizes are defined in ipsecgw_config.h:


TABLE 10-5 Netra DPS Memory Pools

Memory Pool

Description

SPDC_ENTRY_POOL

Pool for SPDC entries stored in the SPDC hash table.

SPI_ENTRY_POOL

Pool for SPI entries stored in the SPI hash table. These hash tables are actually arrays indexed by the hash value (masked with the hash table size).

SP_POOL

Pool of SP entries.

SA_POOL

Pool of SA entries.

CRYP_CTX_POOL

Crypto context structures (maintained by the Crypto API library).


Pipelining

The two main processing threads (PlaintextRcvProcessLoop and CiphertextRcvProcessLoop) are pipelined into two threads: one to perform most of the packet encapsulation and de-encapsulation, and the other to perform the encryption and decryption and optional hashing.

An extra fast queue is inserted in each path. For example, the pipeline for the eight threads configurations is shown as follows:


PlaintextRcvPacket -> 
     PlaintextRcvProcessLoop -> 
           EncryptAndHash -> 
                  CiphertextXmitPacket -> Network port 1  ----> 
                                                                 LOOPBACK
                <- CiphertextRcvPacket <- Network port 2  <----
           <- CiphertextRcvProcessLoop
     <- HashAndDecrypt
PlaintextXmitPacket

The two new threads (EncryptAndHash and HashAndDecrypt) reside in ipsec_processing.c rather than ipsecgw.c where the other threads reside.

The packet processing portion of this pipeline must pass the packet to the crypto part of the pipeline. Packets are normally passed on fast queues through the mblk pointer. Other vital information also needs to be passed, such as the SA pointer. Rather than allocation of a new structure to pass the data and the mblk (message block), this data is piggy-backed at the beginning of the receive buffer, which is not used. Refer to the cinfo structure defined in ipsec_processing.c.

Source Code File Description

The IPSec package comes with the following directories:

This directory consists of IPSec code that supports the Sun multithreaded 10G Ethernet on PCI-E or on-chip NIU in UltraSPARC T2.

This directory consists of Crypto API that interface to the Crypto hardware.

This directory consists of IPSec library functions.

Build Script

Usage

./build <cmt> <type> [auth] [-hash FLOW_POLICY]

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Argument Descriptions

<cmt>

This is an argument specifying whether to build the IPSec Gateway application to run on the CMT1 platform or CMT2 platform.

cmt1 - Build for CMT1 (UltraSPARC T1)
cmt2 - Build for CMT2 (UltraSPARC T2)

<type>

This argument specifies the application type. Available application types are shown as follows.

ipcrypto

Build the ipsecgw application to run Crypto on IP packets only (no IPSec protocol). This application configuration can be used to measure raw Crypto overheads.

qgc

Build the ipsecgw application to run on the Sun Multithreaded Quad Gigabit Ethernet.

10g_niu

Build the ipsecgw application to run one application instance on the UltraSPARC T2 10G Ethernet (NIU).

niu_multi

Build the ipsecgw application to run up to four application instances on the UltraSPARC T2 10G Ethernet (NIU).

niu_tunnel_in

Build the ipsecgw application to run up to eight application instances on the UltraSPARC T2 10G Ethernet (NIU).

niu_tunnel_out

Build the ipsecgw application to run up to eight application instances on the UltraSPARC T2 10G Ethernet (NIU).

[auth]

This is an optional argument to apply Authentication (hashing protocol) to the packet stream along with Crypto. The hash algorithm is specified in the sa_init_static_data.c source file.

[-hash FLOW_POLICY]

This is an optional argument used to enable flow policies. Refer to TABLE 10-2 for the Flow Policies for all flow policy options.

The file descriptions in the following tables are based on the files in the
ipsec-gw-nxge directory.

TABLE 10-6 lists the source files.


TABLE 10-6 Source Files

Source File

Description

common.h

Header file consists of common information.

config.h

Consists of receive buffer configuration information.

debug.c

Used when compiling in DEBUG mode (see IPSEC_DEBUG in the Makefile to turn on IPSec debugs). This file contains the debug thread that calls teja_debugger_check_ctrl_c().

init.c

Main initialization code called by Netra DPS runtime for setting up fast queues and initializing the Crypto library and the IPSec code.

init_multi.c

Main initialization code called by Netra DPS runtime for setting up fast queues used by the IPSec multiple instances code.

ip_crypto.c

Location of the main application threads for the IPSec crypto (Crypto only, no IPSec overhead).

ipsec_niu_config.c

Assists user to map application tasks to CPU core and hardware strands of the UltraSPARC T2 chip specific to the NIU (Network Interface Unit of the UltraSPARC T2 chip) configuration.

ipsecgw.c

Contains the main application threads.

ipsecgw_config.c

Assists user to map application tasks to CPU core and hardware strands.

ipsecgw_flow.c

Contains the classification flow entries.

ipsecgw_flow.h

Contains the definitions of the classification flow.

ipsecgw_impl_config.h

Contains the information related to mblk, receive buffer sizes, number of channels, SA, SPDC.

ipsecgw_niu.c

Main application thread for the NIU configuration.

ipsecgw_niu_multi.c

Main application thread for the NIU multi-instances configuration.

lb_objects.h

Contains memory pool definitions.

mymalloc.c

Used by the low-level crypto-code.

mymalloc.h

Memory pool definitions used by Crypto library.

perf_tools.c

Used for profiling (not available on UltraSPARC T2).

perf_tools.h

Used for profiling (not available on UltraSPARC T2).

rx.c

Packet receive code which uses ethernet API.

tx.c

Packet xmit code which uses ethernet API encryption and hashing algorithms.

user_common.c

Contains the callback functions used by the Netra DPS ethernet APIs.

user_common.h

Contains fast queue definitions and function prototypes.

util.c

Contains IPSec utility functions.


TABLE 10-7 lists the IPSec library files.


TABLE 10-7 IPSec Library Files

IPSec Library File

Description

init_ipsec.c

Code that is called at startup to initialize IPSec structures.

ipsec_common.h

Function prototypes, some common macros, other definitions.

ipsec_defs.h

IPSec protocol definitions and macros.

ipsec_proc.c

This is the main IPSec processing code. This is where all the encapsulation-encryption, de-encapsulation-decryption and hashing functions reside.

netdefs.h

Constant and macro definitions of common ethernet and IP protocols.

sa_init_static_data.c

Contains the statically-defined SAD and SPD. This is the file to modify for testing various SA configurations.

sadb.c

SADB functions.

sadb.h

SADB definitions.

spd.c

SPD functions.

spd.h

SPD definitions.


TABLE 10-8 lists the Crypto library files.


TABLE 10-8 Crypto Library Files

Crypto Library File

Description

crypt_consts.h

Contains various crypto constants.

ndpscript.c

Contains Crypto API implementations.

ndpscrypt.h

Contains data structures and function prototypes.

ndpscrypt_impl.h

Contains Crypto Context structure.


Reference Applications Configurations

IPSec and Crypto has five reference application configurations:

IP with Encryption and Decryption

This configuration can be used to evaluate the raw performance of the Crypto Engine. Two UltraSPARC T2 crypto engines are used. One for encryption, one for decryption.

Default Configuration

IPSec Gateway on Quad GE

This configuration implements one traffic flow on the PCIE Quad Gigabit Ethernet card.

Default Configuration

IPSec Gateway on NIU 10G Interface (One Instance)

This configuration runs one instance of IPSec Gateway Application on the NIU 10G Ethernet interface. Two UltraSPARC T2 Crypto engines are used. One for encrypt/hash, the other for hash/decrypt. This configuration is not yet supported on the Netra CP3260 platform.

Default Configuration

IPSec Gateway on NIU 10G Interface (Up to Four Instances)

This configuration implements multiple instances of IPSEC Gateway Application on the NIU interface through internal loopback. Eight UltraSPARC T2 Crypto engines are used. Four to perform encrypt/hash, four to perform decrypt/hash.

Default Configuration



Note - To build for running on Netra ATCA CP3260 systems, HASH_POLICY options are limited to the following policies: IP_ADDR, IP_DA, and IP_SA.


- If <FLOW_POLICY> is TCAM_CLASSIFY
SA=69.235.4.0
DA=69.235.4.1 ~ 69.235.4.4 (increment by 1 and repeat every 4 counts)



Note - This setting of the traffic generator applies to the Sun SPARC Enterprise T5120 and T5220 systems. For Netra ATCA CP3260 systems, see Flow Policy for Spreading Traffic to Multiple DMA Channels.




Note -
To build for Netra CP3260, in src/libs/ipsec/sa_init_static_data.c, the sa_outb1 remote_gw_mac must be set to the port address of the outgoing Ethernet port.




Note - In the application configuration file (for example, ipsecgw_niu_config.c), if port 0 is used, no action is required; if port 1 is used, add: ..., OPEN_OPEN, NXGE_10G_START_PORT+1, ...


Multiple Instances (Up to Eight Instances) Back-to-Back Tunneling Configuration

This configuration implements multiple instances of the IPSec Gateway Application on the NIU interfaces through back-to-back between two systems.

Default Configuration

System1 (tunnel in):


System1 (tunnel out):


- System1:

For Crypto only:
./build cmt2 niu_tunnel_in -hash <FLOW_POLICY>

For Crypto and Authentication:
./build cmt2 niu_tunnel_in auth -hash <FLOW_POLICY>

- System2:

For Crypto only:
./build cmt2 niu_tunnel_out -hash TCAM_CLASSIFY

For Crypto and Authentication:
./build cmt2 niu_tunnel_out auth -hash TCAM_CLASSIFY



Note - Although other hash policies may still be used to generate binary for System2, traffic may not spread evenly on the System2 Rx input. TCAM_CLASSIFY policy will guarantee that traffic will spread evenly among the 8 DMA channels for this particular configuration.


-If <FLOW_POLICY> is TCAM_CLASSIFY
SA=69.235.4.0
DA=69.235.4.1 ~ 69.235.4.8 (increment by 1 and repeat every 8 counts)



Note - In the application configuration file (for example, ipsecgw_niu_config.c), if port 0 is used, no action is required; if port 1 is used, add: ..., OPEN_OPEN, NXGE_10G_START_PORT+1, ...


Flow Policy for Spreading Traffic to Multiple DMA Channels

The user can specify a policy for spreading traffic into multiple DMA flows by hardware hashing or by hardware TCAM lookup (classification). Refer to TABLE 10-2 for flow policy options.


procedure icon  To Enable a Flow Policy

single-step bullet  Add the following into the gmake line:

FLOW_POLICY=policy

whereas policy is one of the above specified policies.

For example, to enable hash on an IP destination and source address, run the build script with the following arguments:


% ./build cmt2 niu_multi -hash FLOW_POLICY=HASH_IP_ADDR



Note - If you specify FLOW_POLICY=HASH_ALL which is backward compatible with Sun SPARC Enterprise T5120/T5220 systems, all fields are used.


If none of the policies in TABLE 10-2 are specified (do not specify the FLOW_POLICY in the above gmake line, for example, #FLOW_POLICY=HASH_IP_ADDR, a default policy will be given. When the default policy is used, all level (L2/L3/L4) header fields are used for spreading traffic.


Traffic Generator Application

This section explains how to compile Netra DPS traffic generator tool (ntgen), how to use the tool, and the options provided by this tool.

The traffic generator (ntgen) tool allows the generation of IPv4 UDP packets with options to modify various header fields in headers of all layers.

The traffic generator operates only with LDoms enabled. The user interface application runs in the Solaris LDoms and the ntgen tool runs in the Netra DPS domain.

The user interface application provides a template packet to ntgen with user-provided options for modifications. The traffic generator creates new packets using the template packet and applies the modifications as per the user options and transmits the packets.

The template packets are read by the user interface application from a snoop capture file (see the templates/ directory in the ntgen application directory.). This chapter explains the various options provided by the ntgen tool, methods on using ntgen, and compiling with the ntgen build script.



Note - tnsmctl -P -v is required to start the traffic generator on systems that use NIU. The user interface application must be run as root in the Solaris LDoms. On Sun SPARC Enterprise T5120 and T5220 systems, 4Gb of memory is required.


User Interface

The ntgen control plane application is represented by the binary ntgen.

single-step bullet  To start ntgen, type:


% ./ntgen

Usage

./ntgen [options ...] filename

See TABLE 10-9 for the list of options.

Parameters

filename - Snoop file

See Parameters Description for further descriptions and examples.

TABLE 10-9 lists the options for the ntgen control plane application. See -I for further descriptions and examples.


TABLE 10-9 Traffic Generator Control Plane Application Options

Option

Description

-h

Prints this message.

-D

Sets destination MAC address.

-S

Sets source MAC address.

-A

Sets source and destination IPv4 addresses.

-P

Sets payload size.

-p

Sets UDP source and destination ports.

-V

Sets VLAN ID range.

-k

Sets GRE key range.

-iD

Destination MAC address increment mask.

-iS

Increment source IP address, destination IP address host or network.

-iA

Increment SIP,DIPs host or network.

-ip

Increment UDP source/destination port.

-iV

Increment or decrement VLAN ID.

-ik

Increment or decrement GRE key.

-dD

Destination MAC address decrement mask.

-dS

Source MAC address decrement mask.

-dA

Decrement source IP address, destination IP address host or network.

-dp

Decrement UDP source or destination port.

-c

Continuous generation.

-n

Generate number of packets specified.

-I

Ingress/receive only mode.


Option Descriptions

-h

Prints displayed message.

Example

ntgen -h

-D xx:xx:xx:xx:xx:xx
Change destination MAC address

Allows the user to change the destination MAC address of a packet. The destination MAC address is specified in the colon format.

Example

ntgen -D aa:bb:cc:dd:ee:00 filename

-S xx:xx:xx:xx:xx:xx
Change source MAC address

Allows the user to change the source MAC address of a packet. The source MAC address is specified in the colon format.

Example

ntgen -S 11:22:33:44:55:00 filename

-A xx.xx.xx.xx, yy.yy.yy.yy
Change IP addresses

Allows the user to change the source and destination IP addresses in the packet. The IP addresses are specified in the dotted decimal notation.

The first argument in the option is the source IP address. The second argument in the option is the destination IP address. An asterisk (*) can be specified for either the source IP address or the destination IP address to imply that no change needs to be done for that parameter.

Examples

In this example, the source IP address is changed to 192.168.1.1 and the destination IP address is changed to 192.168.2.1.

In this example, the source IP is changed to 192.168.1.10 and the destination IP is unchanged. The destination IP in this case is retained as it is in the template packet.

-p xx,yy
Change UDP port numbers

Allows the user to change the UDP source port and destination port numbers.

The first argument is the UDP source port number and the second argument is the UDP destination port number. An asterisk (*) can be specified for either the source port or the destination port to imply that no change needs to be done to that parameter. In that case, the value present in the template packet is retained.

Examples

In this example, the source port number is changed to 1111 and the destination port number is changed to 2222.

In this example, the source port number remains unchanged from its value in the template packet. The destination port number is changed to 2222 in the packets generated.

-P x
Change UDP payload size

Allows the user to increase the UDP payload size. The value specified must be between 1 and 65536. The value denotes the number of bytes that needs to be added to the payload.

Example

ntgen -P 1024 filename

In this example, the UDP packet payload size gets incremented by 1024 bytes, that is, original size plus 1024 bytes is the new payload size.



Note - Currently, the -P option is not supported for GRE encapsulated UDP datagrams.


-V VLAN ID Start Value, VLAN ID End Value
Change VLAN IDs

Allows the user to create Ethernet frames with 802.1Q VLAN tags in the traffic packets. The Ethernet header of each packet that is generated is appended with a VLAN tag. The VLAN Identifier (VLAN ID) in the VLAN tags of the outgoing frames vary between VLAN ID Start Value and VLAN ID End Value. Two methods of VLAN ID variation are provided through the -iV option. When the -iV option is used with an argument of 1, the VLAN IDs are incremented. When the -iV option is used with an argument of 0, the VLAN IDs are decremented. See the explanation of the
-iV option for further details and examples.

Example

In this example, Ethernet frames with VLAN tags are generated where the VLAN IDs in the VLAN tags of all frames are set to 100, that is, the VLAN ID start value. The VLAN IDs do not vary in this example since the -iV option is not used.

In this example, Ethernet frames with VLAN tags are generated where the VLAN IDs in the VLAN tags vary from 1 to 4094 in an incremental fashion.

In this example, Ethernet frames with VLAN tags are generated where the VLAN IDs in the VLAN tags vary from 1 to 4094 in a decremental fashion.

-k GRE Key Start Value, GRE Key End Value
Change GRE Keys

Allows the user to change the GRE key of GRE encapsulated packets in the range specified using this option. The GRE Key field in the generated packets will vary between the GRE Key Start value and the GRE Key End value. Two methods of the GRE Key variation are provided with the -ik option. When the -ik option is used with value 1, GRE Keys are incremented. When the -ik option is used with value 0, the GRE keys are decremented. See -ik 1/0 Increment or decrement GRE Key for further details.

Example

In this example, GRE keys in the generated traffic start from 1 and increase to 1000.

In this example, GRE keys in the generated traffic start from 1000 and decrease to 1.



Note - Only the file_gre_novlan template file can be used with this option.


-iD xx:xx:xx:xx:xx:xx
Increment destination MAC address

Allows the user to specify the bytes in the destination MAC address that is specified using the -D option, to be incremented. The option is followed by the byte mask. An ff implies increment the byte and 0 implies no increment for that byte.

Examples

In this example, only byte 0 is incremented.

In this example, all bytes are incremented.

-iS xx:xx:xx:xx:xx:xx
Increment source MAC address

Allows the user to specify the bytes in the source MAC address that is specified using the -S option, to be incremented. The option is followed by the byte mask. An ff implies increment the byte and a 0 implies no increment for that byte.

Examples

In this example, only byte 0 is incremented.

In this example, all bytes are incremented.

-iA host/net/[pfx]/*, host/net/[pfx]/*
Increment the host or the network portion of an IP address

Allows the user to increment the source IP address and destination IP address (that were specified using the -A option) based on the IP address class or based on a prefix. The first argument corresponds to the source IP address of a packet. The second argument corresponds to the destination IP Address of a packet.

A class-based increment can be done by using the host or net arguments with the
-iA option. ntgen determines the class of IP address (class A, class B, class C or class D) that is specified with the -A option. From the class, it determines the length of the host part and the network part of the IP address. Then, based on the parameters passed through the -iA option, either the host part or the network part of the IP address is incremented. If an asterisk (*) is passed, then the IP address is not incremented.

The string net denotes that the network portion of the corresponding IP address needs to be incremented. The string host denotes that the host part of the IP address needs to be incremented.

Prefix-based increment can be done by providing the prefix length argument with the -iA option. The user provides a prefix length for each IP address (source and destination) as an arguments to the -iA option. This is used to calculate the portion of the IP address that must be incremented. If an asterisk (*) is passed, then the corresponding IP address is not incremented.



Note - Currently, only 16 bits of an IP address can be incremented using either class-based or prefix-based methods.


Examples

In this example, the network portion of the source IP address and the host portion of the destination IP address are incremented

In this example, the host portion of both the source and destination IP addresses are incremented.

In this example, the host portion of the source IP address is incremented. The destination IP address is not incremented.

In this example, the source IP address is incremented with a prefix length of 10 and the destination IP address is incremented with a prefix length of 12.

In this example, the source IP address is incremented with a prefix length of 10. The destination IP address is not incremented.

-ip 0/1, 0/1
Increment the UDP source and destination port

Allows the user to increment the UDP source port and destination port numbers. The first argument corresponds to the UDP source port and the second argument to the UDP destination port. 0 implies no increment and 1 implies that increment is needed.

Examples

In this example, the source port is not incremented, but the destination port is incremented.

In this example, both the source and destination ports are incremented.

-iV 1/0
Increment or decrement VLAN ID

Allows the user to increment or decrement VLAN IDs in the VLAN tag s of the generated Ethernet frames. Argument 1 denotes increment operation and a 0 denotes decrement operations.

The VLAN IDs are provided by the user using the -V option. For increment operation, the first VLAN ID is the VLAN ID Start Value that is provided in the -V option. Thereon, the VLAN ID is incremented for each subsequent frame until VLAN ID End Value provided with the -V option is reached. Then the VLAN ID wraps around to the VLAN ID Start Value and the sequence is repeated.

For the decrement operation, the first VLAN ID is the VLAN ID End Value that is provided with the -V option. Thereon, the VLAN ID is decremented for each subsequent frame until VLAN ID Start Value provided with the -V option is reached. Then the VLAN ID wraps around to the VLAN ID End Value and the sequence is repeated.

Examples

In this example, Ethernet frames are appended with a VLAN tag that contain VLAN ID in the range 100 to 200. The VLAN IDs are incremented for each frame starting at 100 until 200.

In this example, Ethernet frames are appended with a VLAN tag that contain VLAN ID in the range 200 to 100. The VLAN IDs are decremented for each frame starting at 200 until 100.

-ik 1/0
Increment or decrement GRE Key

Allows the user to increment or decrement GRE Keys in the GRE header of the generated GRE packets. An argument of 1 denotes increment operation and a 0 denotes decrement operation. The GRE Keys are provided using the -k option.

For the increment operation, the first GRE key is the GRE Key Start Value provided with the -k option. Thereon, the GRE Key is incremented for each subsequent packet until the GRE Key End value provided with the -k option is reached. The GRE Key then wraps around to the GRE Key Start Value and the sequence is repeated.

For the decrement operation, the first GRE key is the GRE Key End Value provided with the -k option. Thereon, the GRE Key is decremented for each subsequent packet until the GRE Key Start Value provided with the -k option is reached. The GRE Key then wraps around to the GRE Key End Value and the sequence is repeated.

Examples

In this example, GRE packets with key values in the range 1 to 100 are generated. The key value is incremented for each packet starting at 1 until 100.

In this example, GRE packets with key values in the range 100 to 1 are generated. The key value is decremented for each packet starting at 100 until 1.

-dD xx:xx:xx:xx:xx:xx
Decrement destination MAC address

Allows the user to specify the bytes in the destination MAC address, that is specified using the -D option, to be decremented. The option is followed by a byte mask. An ff implies decrement the byte and a 00 implies no decrement for that byte.

Examples

In this example, only byte 0 of the MAC address is decremented.

In this example, all bytes of the MAC address are decremented.

-dS xx:xx:xx:xx:xx:xx
Decrement source MAC address

Allows the user to specify the bytes in the source MAC address, that is specified using the -S option, to be decremented. The option is followed by a byte mask. An ff implies decrement the byte and a 00 implies no decrement for that byte.

Examples

In this example, only byte 0 of the MAC address is decremented.

In this example, all bytes of the MAC address are decremented.

-dA host/net/[pfx]/*, host/net/[pfx]/*
Decrement the host or the network portion of an IP address

Allows the user to decrement the source IP address and destination IP address (that were specified using the -A option) based on the IP address class or based on a prefix. The first argument corresponds to the source IP address of a packet. The second argument denotes the destination IP address of a packet.

A class-based decrement can be done by using the host or net arguments with the -dA option. ntgen determines the class of the IP address (class A, class B, class C or class D) that is specified using the -A option. From the class, it determines the length of the host part and the network part of the IP address. Then, based on the parameters passed through the -iA option, either the host part or the network part of the IP address is decremented. If an asterisk (*) is passed, then the IP address is not decremented.

The string net denotes that the network portion of the corresponding IP address needs to be decremented. The string host denotes that the host part of the corresponding IP address needs to be decremented.

Prefix-based decrement can be done by providing the prefix length argument with the -dA option. The user provides a prefix length for each IP address (source and destination) as an argument to the -dA option. This is used to calculate the portion of the IP address that needs to be decremented. If an asterisk (*) is passed, then the corresponding IP address is not decremented.



Note - Currently, only 16 bits of an IP address can be decremented using either class-based or prefix-based methods.


Examples

In this example, the network portion of the source IP address and the host portion of the destination IP address are decremented.

In this example, the host portion of both the source and destination IP addresses are decremented.

In this example, the host portion of the source IP address is decremented. The destination IP address is not decremented.

In this example, the source IP address is decremented using a prefix length of 10 and the destination IP address is decremented using a prefix length of 12.

In this example, the source IP address is decremented using a prefix length of 10. The destination IP address is not decremented.

-dp 0/1,0/1
Decrement the UDP source and destination port

Allows the user to decrement the UDP source port and destination port numbers. The first argument corresponds to the UDP source port and the second argument to the UDP destination port. A 0 implies no decrement and a 1 implies that decrement is needed.

Examples

In this example, the UDP source port is not decremented, but the destination port is decremented.

In this example, both the source and destination ports are decremented.

-c
Generate packets continuously

Allows the user to generate packets continuously.

Example

In this example, the packets in the file are generated continuously without applying any modifications.

In this example, all the modifications pertaining to the options specified are applied and the packets are generated continuously.

-n <number of packets>
Generate a specified number of packets

Allows the user to specify the number of packets that need to be generated.

Examples

ntgen -n 1000000 filename

In this example, a million packets are generated.

-I
Ingress or Receive only mode

Allows the user to run the traffic generator in ingress mode. In this mode the traffic generator only receives packets, displays statistics about the ingress traffic and discards the received traffic. This option takes no arguments.

Parameters Description

filename - The Snoop input file

Allows the user to specify a snoop file that contains the template packet to be used for creating the traffic packets. One of the files in the templates/ directory in the ntgen application directory can be used. These files contain packets whose fields can be modified with the ntgen tool options. These snoop files can be analyzed by the user using the snoop program in Solaris. The protocol header fields can be modified using the options provided by ntgen. A detailed explanation of the template snoop files is provided in Template Files.



Note - Only one packet (the first) from a snoop is used by ntgen for generating traffic.


Notes

The increment options (-iD, -iS, -iA and -ip) and the decrement options (-dD,
-dS, -dA and -dp) have effect only when the values that need to be incremented/decremented are also being modified.

For example, the following commands have no effect:

This command has no effect and the destination MAC address will not be incremented.

This command has no effect. The source and destination IP addresses will not be incremented.

This command has no effect. The port numbers will not be incremented.

The following commands will have effect:

This command increments the destination MAC address after changing it to aa:bb:cc:dd:ee:00. Since -D option is being used, the -iD option takes effect.

This command increments the source and destination IP addresses. Since the -A option is being used, the -iA option takes effect.

This command increments the source and destination UDP ports. Since the -p option is being used, the -ip option takes effect.

Traffic Generator Output

TABLE 10-10 shows an example of the traffic generator output.


TABLE 10-10 Traffic Generator Output Example

Port,Chan

Tx Rate (pps)

Tx Rate (mbps)

Rx Rate (pps)

Rx Rate (mbps)

0, 0

947550.5506

485.1459

32224.4898

386.6939

1, 0

947550.5506

485.1459

32224.4898

386.6939

2, 0

947550.5506

485.1459

32224.4898

386.6939

3, 0

947550.5506

485.1459

32224.4898

386.6939


TABLE 10-11 describes the traffic generator output.


TABLE 10-11 Traffic Generator Output Description

Column

Description

Port,Chan

Port is the port number and Chan is the channel number for which the statistics are displayed.

In the example output shown in TABLE 10-10 for NxGE QGC, port varies from 0 to 3 and Chan is 0 for all ports.

Tx Rate (pps)

Transmission rate in packets per second.

Tx Rate (mbps)

Transmission rate in megabits per second.

Rx Rate (pps)

Receive rate in packets per second.

Rx Rate (mbps)

Receive rate in megabits per second.


Template Files

The following template files are provided with the application to be used with ntgen.

Snoop file that contains a single 64-byte Ethernet frame that has no VLAN tag. This file has a UDP/IPv4 payload.

Snoop file that contains a single 256 bytes Ethernet frame that has no VLAN tag. The file has a UDP/IPv4 payload.

Snoop file that contains a single 1514 bytes Ethernet frame that has no VLAN tag. This file has a UDP/IPv4 payload.

Snoop file that contains a GRE packet with an IPv4 as the delivery protocol and IPv4 as the payload protocol. The payload is a UDP datagram. The UDP datagram has a payload of 22 bytes. Both IPv4 headers have no IP options. GRE header consists of GRE Key and GRE checksum values.

Using the Traffic Generator

This section describes configuring, starting, and stopping the ntgen tool.

Configuring LDoms for the Traffic Generator

TABLE 10-12 shows the domain role in the configuration.


TABLE 10-12 LDoms Configuration

Domain

Operating System

Role

primary

Solaris

Owns one of the PCI buses, and uses the physical disks and networking interfaces to provide virtual I/O to the Solaris guest domains.

ldg1

LWRTE

Owns the other PCI bus (bus_b) with its two network interfaces and runs an LWRTE application.

ldg2

Solaris

Runs control plane application (ntgen), add_drv tnsm (SUNWndpsd package) and uses ntgen to control traffic generation.

ldg3

Solaris

Controls lwrte (global configuration channel), add_drv tnsm (SUNWndpsd package) and uses tnsmctl to setup configuration.


TABLE 10-13 shows the LDC channels configured.


TABLE 10-13 LDC Channels Configured

Server

Client

ldg1 primary-gc

ldg3 tnsm-gc0

ldg1 config-tnsm-ldg2

ldg2 config-tnsm0

ldg1 ldg2-vdpcs0

ldg2 vdpcc0

ldg1 ldg2-vdpcs1

ldg2 vdpcc1


These LDC channels can be added with the following LDoms manager commands:


ldm add-vdpcs primary-gc ldg1
ldm add-vdpcc tnsm-gc0 primary-gc ldg3
ldm add-vdpcs config-tnsm-ldg2 ldg1
ldm add-vdpcc config-tnsm0 config-tnsm-ldg2 ldg2
 
ldm add-vdpcs ldg2-vdpcs0 ldg1
ldm add-vdpcc vdpcc0 ldg2-vdpcs0 ldg2
etc.

In the Solaris domains, the tnsm driver needs to be added. To do this,

1. Install the SUNWndpsd package.

2. Type the following to install the driver:


add_drv tnsm

The primary-gc and tnsm-gc0 combination is the global configuration channel. LWRTE accepts configuration messages on this channel.

The config-tnsm-ldgx and config-tnsm0 combination is for setup messages between LWRTE and the control plane domain.

To find out what the LDC IDs are on both sides, use the following:

Example Output from LDoms 1.0

 


ldm list-bindings
In ldg1:
Vdpcs:  config-tnsm-ldg2
        [LDom  ldg2, name: config-tnsm0]
        [LDC: 0x6]
In ldg2:
Vdpcc:  config-tnsm0    service: config-tnsm-ldg2 @ ldg1
        [LDC: 0x5]

Example Output from LDoms 1.0.1

 


ldm list-bindings -e
In ldg1:
VDPCS
    NAME
    config-tnsm-ldg2
        CLIENT                    LDC
        config-tnsm0@ldg2         6
In ldg2:
VDPCC
    NAME               SERVICE                     LDC
    config-tnsm0       config-tnsm-ldg2@ldg1       5

Pick a channel number to be used for the control IPC channel that uses this LDC channel, for example, 3. Bring up the control channel with the following command:


tnsmctl -S -C 3 -L 6 -R 5 -F 3

Description of parameters:

In the above tnsmctl command example,

Control channel 3 can now be used to set up general purpose IPC channels between LWRTE and Solaris.

For example, set up channel ID 4 for use by the ntgen to ndpstgen communication:

Look up the LDC IDs on both ends.

Example Output from LDoms 1.0

 


ldg1:
Vdpcs:  ldg2-vdpcs0
        [LDom  ldg2, name: vdpcc0]
        [LDC: 0x7]
ldg2:
Vdpcc:  vdpcc0  service: ldg2-vdpcs0 @ ldg1
        [LDC: 0x6]

Example Output from LDoms 1.0.1

 


ldg1:
VDPCS
    NAME
    ldg2-vdpcs0
        CLIENT                    LDC
        vdpcc0@ldg2               7
ldg2:
VDPCC
    NAME              SERVICE                      LDC
    vdpcc0‘           ldg2-vdpcs0@ldg1             6

With this information, do the following in ldg3:


tnsmctl -S -C 4 -L 7 -R 6 -F 3

Note that the -C 4 parameter is the ID for the new channel, the -F 3 has the channel set up before.

Also, note that the global configuration channel between ldg3 and LWRTE comes up automatically as soon as the application is booted in LWRTE and the tnsm device driver is added in ldg3.

Build the ntgen utility in the Solaris subtree. Once the channel to be used is initialized using tnsmctl (must be channel ID 4 which is hard coded into the ndpstgen application), use ntgen to generate traffic (refer to the NTGEN User’s Manual).


procedure icon  To Prepare Building the ntgen Utility

1. Build the Netra DPS image.

2. Build the ntgen user interface application (in the src/solaris subdirectory)


procedure icon  To Setup and Use LDoms for the Traffic Generator

1. Configure the primary domain.

2. Save the configuration (ldm add-spconfig) and reboot.

3. Configure the Netra DPS domain (including the vdpcs services).

4. Configure the Solaris domains (including vdpcc clients).

5. Bind the Netra DPS domain (ldg1).

6. Bind the Solaris domains (ldg2 and ldg3).

7. Start and boot all domains (can be in any order).

8. Install the SUNWndpsd package in the Solaris domains.

9. Load the tnsm driver in the Solaris domains (add_drv tnsm).

10. In the global configuration Solaris domain (ldg3), use /opt/SUNWndpsd/bin/tnsmctl to set up the control channel between the Netra DPS domain (ldg1) and the control domain (ldg2).

11. In the global configuration Solaris domain (ldg3), use /opt/SUNWndpsd/bin/tnsmctl to set up the ntgen control channel (channel ID 4).

12. In the control domain (ldg2), use the ntgen utility to start traffic generation.

Starting the Traffic Generation

Start the traffic generator tool by using the ntgen application binary.

For example, type:


% ./ntgen -c file_64B_novlan

Stopping Traffic Generation

The traffic generation can be stopped at any point of time by pressing Ctrl-C from the keyboard.

Compiling the Traffic Generator

Copy the ntgen reference application from the /opt/SUNWndps/src/apps/ntgen directory to a desired directory location and do the build script in that location.

Build Script

TABLE 10-14 shows the traffic generator (ntgen) application build script.


TABLE 10-14 ntgen Application Build Script

Build Script

Usage

./build

(See Argument Descriptions.)

Build ntgen application to run on e1000g Ethernet interface.

 

Build ntgen application to run on e1000g Ethernet interface on the Netra ATCA CP3060 System.

 

Build ntgen application to run on Sun QGC (quad 1Gbps nxge Ethernet interface).

 

Build ntgen application to run on Sun multithreaded 10G (dual 10Gbps nxge Ethernet interface).

 

Build ntgen application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet interface) on a CMT2-based system.


Usage

./build <cmt> <app> [profiler] [2port]

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Argument Descriptions

<cmt>

This is an argument specifying whether to build the traffic generator application to run on the CMT1 platform or CMT2 platform.

cmt1 - Build for CMT1 (UltraSPARC T1)
cmt2 - Build for CMT2 (UltraSPARC T2)

<app>

4g - Builds the traffic generator application to run on QGC (quad 1Gbps nxge Ethernet interface).

10g - Builds the traffic generator application to run on 10Gb Ethernet (dual 10Gbps nxge Ethernet interface).

10g_niu - Builds the traffic generator application to run on NIU (dual 10Gbps UltraSPARC T2 Ethernet interface) on a CMT2 based system.

[profiler]:

This is an optional argument that generates code with profiling enabled.

[2port]:

This is an optional argument to compile dual ports on the 10Gb Ethernet card or the UltraSPARC T2 Network Interface Unit (NIU).

For example, to build for 10Gb Ethernet on the Netra T2000 system, type:


% ./build cmt1 10g

In this example, the build script is used to build the traffic generator application to run on the 10Gb Ethernet. <cmt> argument is specified as cmt1 to build the application to run on the Netra T2000 system which is an UltraSPARC T1-based system. The <app> argument is specified as 10g to run on 10Gb Ethernet.


procedure icon  To Run

On a tftpboot server, type:


% cp your_workspace/ntgen/code/ndpstgen/ndpstgen /tftpboot/ndpstgen

At the target machine OK prompt, type:


ok boot net:,ndpstgen

Default Configurations

The following is the default system configuration:

 


NDPS domain
(Strand IDs)

Statistics
(Strand ID)

Other domains
(Strand IDs)

CMT1 LDOM:

0 - 19

NA

20 - 31

CMT2 LDOM:

0 - 39

NA

40 - 63


The main files that control the system configuration are:

The following is the default ntgen application configuration:


Applications
runs on

Number of Ports Used

Number of Channels per Port

Total Number of Q Instances

Total Number of Strands Used

4G-PCE (nxge QGC):

4

1

4

12

10G-PCIE (nxge 10G):

1

4

4

12

10G-NIU (niu 10G):

1

8

8

40


The main files that control the application configurations are:


Interprocess Communication (IPC) Reference Application

The IPC reference application showcases the programming interfaces of the IPC framework (see Chapter 5, Interprocess Communication Software and the Netra Data Plane Software Suite 2.0 Update 2 Reference Manual).

The IPC reference application consists of the following three components:

The application is designed to run in an LDoms environment very similar to the one described in (see Example Environment for UltraSPARC T1 Based Servers and Example Environment for UltraSPARC T2 Based Servers).

IPC Reference Application Content

The complete source code for the IPC reference application can be found in the SUNWndps package in the /opt/SUNWndps/src/apps/ipc_test directory. The source code files include the following:

Building the IPC Reference Application

Usage

build <cmt> [single_thread] | solaris

Build Script Arguments

< > - Required arguments

[ ] - Optional arguments

Argument Descriptions

<cmt>

This argument specifies whether to build the ipc_test application to run on the CMT1 (UltraSPARC T1) platform or CMT2 (UltraSPARC T2) platform.

cmt1 - Build for CMT1 (UltraSPARC T1)
cmt2 - Build for CMT2 (UltraSPARC T2)

This argument is required to build the Netra DPS application.

[single_thread]

With this option, two data IPC channels are polled by the same thread. In the default case, three channels are polled, each one on its own thread. The interfaces and usage for the Solaris side remain unchanged.

solaris

Build the Solaris user space application and the STREAMS module in their respective source directories.

Build Example

The commands shown below build the Netra DPS application for single thread polling on a UltraSPARC T2 processor and the Solaris components, respectively.


# ./build cmt2 single_thread
# ./build solaris

Running the Application

In addition to the channels described in Example Environment for UltraSPARC T1 Based Servers, two IPC channels with IDs 5 and 6, respectively, need to be set up using the ldm and tnsmctl commands.

The Netra DPS application is booted from either a physical or a virtual network interface assigned to its domain. For example, if a tftp server has been set up in the subnet, and there is a vnet interface for the Netra DPS domain, the IPC test application can be booted with the following command at the OpenBoot Prom:


ok boot /virtual-devices@100/channel-devices@200/network@0:,ipc_test

Using the ipctest Utility

First, the ipc_test application must be booted in the Netra DPS domain, and the IPC channels to be used must be set up using the tnsmctl utility from the control domain. Copy the ipctest binary from the src/solaris/cmd directory to the Solaris domain (for example, ldg2 as shown in the Solaris user space application in src/solaris/cmd).

The ipctest utility drives a single IPC channel, which is selected by the connect command (see Commands). Multiple channels can be driven by separate instances of the utility, and the utility can be used at the same time as the STREAMS module (see Installing the lwmod STREAMS Module). In this case, however, the IPC channel with ID 5 is not available for this utility. For example, the utility can be used on channel 4 to read statistics of the traffic between the Netra DPS application and the Solaris module on channel 5.

Commands

The ipctest utility opens the tnsm driver and offers the following commands:

connect Channel_ID

Connects to the channel with ID Channel_ID. The forwarding application is hard coded to use channel ID 4. The IPC type is hard coded on both sides. This command must be issued before any of the other commands.

stats

Requests statistics from the ipc_test application and displays them.

perf-stats iterations

Requests statistics from the ipc_test application for iterations times and displays the time used.

perf-pkts-rx num_messages message_size

Send request to the Netra DPS to send num_messages messages with a data size of message_size and receive the messages.

perf-pkts-tx num_messages message_size

Send num_messages messages with a data size of message_size to the Netra DPS domain.

perf-pkts-rx-tx num_messages message_size

Sends request to the Netra DPS to send num_messages messages with a data size of message_size and receive the messages. Also, spawns a thread that sends as many messages of the same size to the Netra DPS domain.

exit
x
quit
q

Exits the program.

help

Contains program help information.

Installing the lwmod STREAMS Module

After copying the lwmod module from the rc/solaris/module/sparcv9 directory to the Solaris domain (for example, ldg2 as shown in the Solaris STREAMS module in src/solaris/module), the module needs to be loaded and inserted just above the driver for either a virtual or a physical networking device.

To use a physical device, modify the configuration such that the primary domain is connected through IPC channel 5, or, on an UltraSPARC T1-based system, assign the second PCI bus to ldg2.



Note - Before inserting the module, the ipc_test application must have been booted in the Netra DPS domain, and the IPC channels must have been set up.


Set up the module on a secondary vnet interface by entering the following sequence of commands:


# modload lwmod
# ifconfig vnet1 modinsert lwmod@2

After this sequence, the modules position can be displayed by entering the following command:


# ifconfig vnet1 modlist
0 arp
1 ip
2 lwmod
3 vnet

With the module installed, all packets sent to vnet1 will be diverted to the Netra DPS domain, where the application will reverse the MAC addresses and echo the packets back to the Solaris module. The module will transmit the packet on the same interface.



Note - No packet will be delivered to the stack above the module. If networking to the domain is needed, the module should not be inserted in the primary interface.


The module can be removed from the interface by entering the following command:


# ifconfig vnet1 modremove lwmod@2