test

NFS Server Performance and Tuning Guide for Sun Hardware

Appendix A Using NFS Performance-Monitoring and Benchmarking Tools

This appendix discusses tools that help you monitor NFS and network performance. These tools generate information that you can use to tune and improve performance. See Chapter 3, Analyzing NFS Performance," and Chapter 4, Configuring the Server and the Client to Maximize NFS Performance."

For more information about these tools, refer to their man pages (where applicable). For third-party tools, refer to the product documentation.

This chapter also describes SPEC SFS 2.0, an NFS file server benchmarking tool.

NFS Monitoring Tools

Table A-1 describes the tools that you can use to monitor NFS operations and performance.

Table A-1 NFS Operations and Performance-Monitoring Tools

Tool 

Function 

iostat

Reports I/O statistics, including disk I/O activity 

nfsstat

Reports NFS statistics: NFS and RPC (Remote Procedure Call) interfaces to the kernel; can also be used to initialize statistical information 

nfswatch

Shows NFS transactions classified by file system; nfswatch is a public domain tool with source code available on the URL: http://www.ers.ibm.com/~davy/software/nfswatch.html

 

sar

Reports system activity such as CPU utilization, buffer activity, and disk and tape device activity 

SharpShooter* 

Pinpoints bottlenecks, balances NFS load across clients and servers; shows effect of distributed applications and balances network traffic across servers; accounts for disk usage by user or group 

vmstat

Reports virtual memory statistics including disk activity 

* by Network General Corporation, formerly AIM Technology 

For additional networking and network monitoring utilities, see the URL: http://www.alw.nih.gov/Security/prog-network.html

Network Monitoring Tools

Use the tools described in Table A-2 to monitor network performance as it relates to NFS.

Table A-2 Network Monitoring Tools

Tool 

Function 

snoop

Displays information about specified packets on Ethernet 

netstat

Displays the contents of network-related data structures 

ping

Sends ICMP ECHO_REQUEST packets to network hosts

NetMetrix Load Monitor 

Handles network load monitoring and characterization of load in terms of time, source, destination, protocol, and packet size 

SunNet Manager 

Performs network device monitoring and troubleshooting 

LAN analyzers: Network General Sniffer, Novell/Excelan Lanalyzer 

Performs packet analysis 

snoop

The snoop command is part of the Solaris 2.x software environment. The snoop command must run by root (#) to capture packets in promiscuous mode. To capture packets in non-promiscuous mode, or to analyze packets from a captured file, you do not need to be superuser.

In promiscuous mode, the interface turns off its filter, which enables you to see all packets on the subnet, whether or not they are addressed to your system. You can observe other packets not destined for your system. Promiscuous mode is limited to root.

Using the snoop command turns a Sun system into a network sniffer, which can detect network problems. It also captures a certain number of network packets, enables you to trace the calls from each client to each server, and displays the contents of the packets. You can also save the contents of the packets to a file, which you can inspect later.

The snoop command does the following:

The snoop command can display packets in a single-line summary or in expanded form. In summary form, only the data pertaining to the highest level protocol is displayed. For example, an NFS packet will have only NFS information displayed. The underlying RPC, UDP (User Datagram Protocol), IP (Internet Protocol), and network frame information is suppressed, but can be displayed if you choose either of the verbose (-v or -V) options.

The snoop command uses both the packet filter and buffer modules of the Data Link Provider Interface (DLPI) so the packets can be captured efficiently and transmitted to or received from the network.

To view or capture all traffic between any two systems, run snoop on a third system.

The snoop command is a useful tool if you are considering subnetting, since it is a packet analysis tool. You can use the output of the snoop command to drive scripts that accumulate load statistics. The program is capable of breaking the packet header in order to debug it, and to investigate the source of incompatibility problems.

The following shows some examples of how to use snoop.

Looking at Selected Packets in a Capture File (pkts)

The statistics show which client is making a read request, and the left column shows the time in seconds, with a resolution of about 4 microseconds.

When a read or write request is made, be sure the server doesn't time-out. If it does, the client has to re-send again, and the client's IP code will break up the write block into smaller UDP blocks. The default write time is .07 seconds. The time-out factor is a tunable parameter in the mount command.

Example A-1 Output of the snoop -i pkts -p99, 108 Command

# snoop -i pkts -p99,108
99   0.0027   boutique -> sunroof     NFS C GETATTR FH=8E6C
100   0.0046   sunroof -> boutique     NFS R GETATTR OK
101   0.0080   boutique -> sunroof     NFS C RENAME FH=8E6C MTra00192
to .nfs08
102   0.0102   marmot -> viper          NFS C LOOKUP FH=561E screen.r.13.i386
103   0.0072   viper -> marmot          NFS R LOOKUP No such file or
directory
104   0.0085   bugbomb -> sunroof    RLOGIN C PORT=1023 h
105   0.0005   kandinsky -> sparky    RSTAT C Get Statistics
106   0.0004   beeblebrox -> sunroof  NFS C GETATTR FH=0307
107   0.0021   sparky -> kandinsky    RSTAT R
108   0.0073   office -> jeremiah        NFS C READ FH=2584 at 40960
for 8192

The following table describes the arguments to the snoop command.

Table A-3 Arguments to the snoop Command

-i pkts

Displays packets previously captured in the pkts file

-p99,108

Selects packets 99 through 108 to be displayed from a capture file; the first number 99, is the first packet to be captured; the last number, 108, is the last packet to be captured; the first packet in a capture file is packet 1 

  1. To get more information on a packet, type: 

    # snoop -i pkts -v 101

The command snoop -i pkts -v 101 obtains more detailed information on packet 101. Table A-4 describes the command arguments.

Table A-4 Description of Arguments to the snoop -i pkts -v 101 Command

-i pkts

Displays packets previously captured in the pkts file

-v

Verbose mode; prints packet headers in detail for packet 101; use this option only when you need information on selected packets 

To view NFS packets, type: 

# snoop -i pkts rpc nfs and sunroof and boutique
1   0.0000   boutique -> sunroof    NFS C GETATTR FH=8E6C
2   0.0046    sunroof -> boutique   NFS R GETATTR OK
3   0.0080   boutique -> sunroof    NFS C RENAME FH=8E6C MTra00192 to .nfs08

This example gives a view of the NFS packets between the systems sunroof and boutique. Table A-5describes the arguments to the previous snoop command.

Table A-5 Arguments to the snoop -i pkts rpc nfs and sunroof and boutique Command

-i pkts

Displays packets previously captured in the pkts file

rpc nfs

Displays packets for an RPC call or reply packet for the NFS protocol; the option following nfs is the name of an RPC protocol from /etc/rpc or a program number

and

Performs a logical and operation between two boolean values; for example, sunroof boutique is the same as sunroof and boutique

  1. To save packets to a new capture file, type: 

    # snoop -i pkts -o pkts.nfs rpc nfs sunroof boutique

Table A-6describes the arguments to the previous snoop command.

Table A-6 Description of Arguments to the snoop -i pkts -o.nfs rpc nfs sunroof boutique Command

-i pkts

Displays packets previously captured in the pkts file

-o pkts.nfs

Saves the displayed packets in the pkts.nfs output file

rpc nfs

Displays packets for an RPC call or reply packet for the NFS protocol; the option following nfs is the name of an RPC protocol from /etc/rpc or a program number

See the snoop man page for additional details on options used with the snoop command and additional information about using snoop.

SPEC System File Server 2.0

SPEC System File Server (SFS) 2.0 measures NFS file server throughput and response time. It is a one test benchmark suite consisting of 097.LADDIS. It contains an updated workload that was developed based on a survey of more than 1,000 NFS servers in different application environments. The workload is larger and the response-time threshold is lower than those used in SFS 1.1, due to advances in server technologies. Because of these and other changes, you cannot compare SPEC SFS 2.0 results to SFS 1.1 or SFS 1 results.

In addition to general code improvemenets, SPEC SFS 2.0 includes these enhancements

Two reference points are considered when reporting 097.LADDIS:

LADDIS is designed so that its workload can be incrementally increased until the target server performance falls below a certain level. That point is defined as an average response time exceeding 50ms. This restriction is applied when deriving the maximum throughput in NFS operations per second for which the response time does not exceed 50 ms.

If throughput continues to increase with the workload, the throughput figure at 50 ms is reported. In many cases, throughput will start to fall off at a response time below the 50 ms limit. In these cases, the tables in this chapter report the response time at the point of maximum throughput.

0.97 LADDIS Benchmark

The SPEC SFS 2.0 (0.97 LADDIS)benchmark is a synthetic NFS workload based on an application abstraction, an NFS operation mix, and an NFS operation request rate. The workload generated by the benchmark emulates an intensive software development environment at the NFS protocol level. The LADDIS benchmark makes direct RPC calls to the server, eliminating any variation in NFS client implementation. This makes it easier to control the operation mix and workload, especially for comparing results between vendors. However, this also hides the benefits of special client implementations, such as the cache file system client.

Table A-7 shows the NFS operations mix. These percentages indicate the relative number of calls made to each operation.

Table A-7 NFS Operations Mix by Call

NFS Operation 

Percent Mix 

Lookup

34 

Read

22 

Write

15 

GetAttr

13 

ReadLink

8 

ReadDir

3 

Create

2 

Remove

1 

Statfs

1 

SetAttr

1 

The LADDIS benchmark for NFS file systems uses an operation mix that is 15 percent write operations. If your NFS clients generate only one to two percent write operations, LADDIS underestimates your performance. The greater the similarity between the operation mixes, the more reliable the maximum throughput in NFS operations is as a reference.

Running the benchmark requires the server being benchmarked to have at least two NFS clients (the NFS load generators), and one or more isolated networks. The ability to support multiple networks is important because a single network may become saturated before the server maximum performance point is reached. One client is designated as the LADDIS Prime Load Generator. The Prime Load Generator controls the execution of the LADDIS load generating code on all load-generating clients. It typically controls the benchmark. In this capacity, it is responsible for collecting throughput and response time data at each of the workload points and for generating results.

To improve response time, configure your NFS server with the NVRAM-NVSIMM Prestoserve NFS accelerator. NVSIMMs provide storage directly in the high-speed memory subsystem. Using NVSIMMs results in considerably lower latency and reduces the number of disks required to attain a given level of performance.

Since there are extra data copies in and out of the NVSIMM, the ultimate peak throughput is reduced. Because NFS loads rarely sustain peak throughput, the better response time using the NVSIMMs is preferable. For information on the Prestoserve NFS accelerator, see "Prestoserve NFS Accelerator" in Chapter 4, Configuring the Server and the Client to Maximize NFS Performance.

SPEC SFS 2.0 Results

Sun Microsystems has run SPEC SFS 2.0 benchmarks on the Sun Enterprise 3000-6000 family of servers. The benchmarks were run with NFS Version 2 and NFS Version 3. Table A-8 shows the results with Version 2. Table A-9 shows the results with Version 3.

Table A-8 SPEC SFS 2.0 Results With Version 2

System 

Number of CPUs 

Result 

Overall Response Time 

Sun Enterprise 3000 

6 

11806 

5.1 

Sun Enterprise 4000 

12 

20406 

6.7 

Sun Enterprise 6000 

18 

25639 

9.9 

Table A-9 SPEC SFS 2.0 Results With Version 3

System 

Number of CPUs 

Result 

Overall Response Time 

Sun Enterprise 3000 

6 

5903 

5.4 

Sun Enterprise 4000 

12 

10592 

5.6