This appendix discusses tools that help you monitor NFS and network performance. These tools generate information that you can use to tune and improve performance. See Chapter 3, Analyzing NFS Performance," and Chapter 4, Configuring the Server and the Client to Maximize NFS Performance."
For more information about these tools, refer to their man pages (where applicable). For third-party tools, refer to the product documentation.
This chapter also describes SPEC SFS 2.0, an NFS file server benchmarking tool.
Table A-1 describes the tools that you can use to monitor NFS operations and performance.
Table A-1 NFS Operations and Performance-Monitoring Tools
Tool |
Function |
---|---|
iostat |
Reports I/O statistics, including disk I/O activity |
nfsstat |
Reports NFS statistics: NFS and RPC (Remote Procedure Call) interfaces to the kernel; can also be used to initialize statistical information |
nfswatch |
Shows NFS transactions classified by file system; nfswatch is a public domain tool with source code available on the URL: http://www.ers.ibm.com/~davy/software/nfswatch.html
|
sar |
Reports system activity such as CPU utilization, buffer activity, and disk and tape device activity |
SharpShooter* |
Pinpoints bottlenecks, balances NFS load across clients and servers; shows effect of distributed applications and balances network traffic across servers; accounts for disk usage by user or group |
vmstat |
Reports virtual memory statistics including disk activity |
* by Network General Corporation, formerly AIM Technology |
For additional networking and network monitoring utilities, see the URL: http://www.alw.nih.gov/Security/prog-network.html
Use the tools described in Table A-2 to monitor network performance as it relates to NFS.
Table A-2 Network Monitoring Tools
Tool |
Function |
---|---|
snoop |
Displays information about specified packets on Ethernet |
netstat |
Displays the contents of network-related data structures |
ping |
Sends ICMP ECHO_REQUEST packets to network hosts |
NetMetrix Load Monitor |
Handles network load monitoring and characterization of load in terms of time, source, destination, protocol, and packet size |
SunNet Manager |
Performs network device monitoring and troubleshooting |
LAN analyzers: Network General Sniffer, Novell/Excelan Lanalyzer |
Performs packet analysis |
The snoop command is part of the Solaris 2.x software environment. The snoop command must run by root (#) to capture packets in promiscuous mode. To capture packets in non-promiscuous mode, or to analyze packets from a captured file, you do not need to be superuser.
In promiscuous mode, the interface turns off its filter, which enables you to see all packets on the subnet, whether or not they are addressed to your system. You can observe other packets not destined for your system. Promiscuous mode is limited to root.
Using the snoop command turns a Sun system into a network sniffer, which can detect network problems. It also captures a certain number of network packets, enables you to trace the calls from each client to each server, and displays the contents of the packets. You can also save the contents of the packets to a file, which you can inspect later.
The snoop command does the following:
Logs or displays packets selectively
Provides accurate time stamps for checking network Remote Procedure Call (RPC) response time
Formats packets and protocol information in a user-friendly manner
The snoop command can display packets in a single-line summary or in expanded form. In summary form, only the data pertaining to the highest level protocol is displayed. For example, an NFS packet will have only NFS information displayed. The underlying RPC, UDP (User Datagram Protocol), IP (Internet Protocol), and network frame information is suppressed, but can be displayed if you choose either of the verbose (-v or -V) options.
The snoop command uses both the packet filter and buffer modules of the Data Link Provider Interface (DLPI) so the packets can be captured efficiently and transmitted to or received from the network.
To view or capture all traffic between any two systems, run snoop on a third system.
The snoop command is a useful tool if you are considering subnetting, since it is a packet analysis tool. You can use the output of the snoop command to drive scripts that accumulate load statistics. The program is capable of breaking the packet header in order to debug it, and to investigate the source of incompatibility problems.
The following shows some examples of how to use snoop.
The statistics show which client is making a read request, and the left column shows the time in seconds, with a resolution of about 4 microseconds.
When a read or write request is made, be sure the server doesn't time-out. If it does, the client has to re-send again, and the client's IP code will break up the write block into smaller UDP blocks. The default write time is .07 seconds. The time-out factor is a tunable parameter in the mount command.
# snoop -i pkts -p99,108 99 0.0027 boutique -> sunroof NFS C GETATTR FH=8E6C 100 0.0046 sunroof -> boutique NFS R GETATTR OK 101 0.0080 boutique -> sunroof NFS C RENAME FH=8E6C MTra00192 to .nfs08 102 0.0102 marmot -> viper NFS C LOOKUP FH=561E screen.r.13.i386 103 0.0072 viper -> marmot NFS R LOOKUP No such file or directory 104 0.0085 bugbomb -> sunroof RLOGIN C PORT=1023 h 105 0.0005 kandinsky -> sparky RSTAT C Get Statistics 106 0.0004 beeblebrox -> sunroof NFS C GETATTR FH=0307 107 0.0021 sparky -> kandinsky RSTAT R 108 0.0073 office -> jeremiah NFS C READ FH=2584 at 40960 for 8192
The following table describes the arguments to the snoop command.
Table A-3 Arguments to the snoop Command
-i pkts |
Displays packets previously captured in the pkts file |
-p99,108 |
Selects packets 99 through 108 to be displayed from a capture file; the first number 99, is the first packet to be captured; the last number, 108, is the last packet to be captured; the first packet in a capture file is packet 1 |
The command snoop -i pkts -v 101 obtains more detailed information on packet 101. Table A-4 describes the command arguments.
Table A-4 Description of Arguments to the snoop -i pkts -v 101 Command
-i pkts |
Displays packets previously captured in the pkts file |
-v |
Verbose mode; prints packet headers in detail for packet 101; use this option only when you need information on selected packets |
# snoop -i pkts rpc nfs and sunroof and boutique 1 0.0000 boutique -> sunroof NFS C GETATTR FH=8E6C 2 0.0046 sunroof -> boutique NFS R GETATTR OK 3 0.0080 boutique -> sunroof NFS C RENAME FH=8E6C MTra00192 to .nfs08
This example gives a view of the NFS packets between the systems sunroof and boutique. Table A-5describes the arguments to the previous snoop command.
Table A-5 Arguments to the snoop -i pkts rpc nfs and sunroof and boutique Command
-i pkts |
Displays packets previously captured in the pkts file |
rpc nfs |
Displays packets for an RPC call or reply packet for the NFS protocol; the option following nfs is the name of an RPC protocol from /etc/rpc or a program number |
and |
Performs a logical and operation between two boolean values; for example, sunroof boutique is the same as sunroof and boutique |
Table A-6describes the arguments to the previous snoop command.
Table A-6 Description of Arguments to the snoop -i pkts -o.nfs rpc nfs sunroof boutique Command
-i pkts |
Displays packets previously captured in the pkts file |
-o pkts.nfs |
Saves the displayed packets in the pkts.nfs output file |
rpc nfs |
Displays packets for an RPC call or reply packet for the NFS protocol; the option following nfs is the name of an RPC protocol from /etc/rpc or a program number |
See the snoop man page for additional details on options used with the snoop command and additional information about using snoop.
SPEC System File Server (SFS) 2.0 measures NFS file server throughput and response time. It is a one test benchmark suite consisting of 097.LADDIS. It contains an updated workload that was developed based on a survey of more than 1,000 NFS servers in different application environments. The workload is larger and the response-time threshold is lower than those used in SFS 1.1, due to advances in server technologies. Because of these and other changes, you cannot compare SPEC SFS 2.0 results to SFS 1.1 or SFS 1 results.
In addition to general code improvemenets, SPEC SFS 2.0 includes these enhancements
Measures results for both NFS Version 2 and 3
Adds support for TCP (either TCP or UDP can be used as the network transport)
Operation mix more closely matches real-world NFS workloads
Improved interface to accomodate both accomplished and novice users
Two reference points are considered when reporting 097.LADDIS:
NFS operation throughput--The peak number of NFS operations the target server can complete in a given number of milliseconds. The larger the number of operations an NFS server can support, the more users it can serve.
Response time--The average time needed for an NFS client to receive a reply from a target server in response to an NFS request. The response time of an NFS server is the client's perception of how fast the server is.
LADDIS is designed so that its workload can be incrementally increased until the target server performance falls below a certain level. That point is defined as an average response time exceeding 50ms. This restriction is applied when deriving the maximum throughput in NFS operations per second for which the response time does not exceed 50 ms.
If throughput continues to increase with the workload, the throughput figure at 50 ms is reported. In many cases, throughput will start to fall off at a response time below the 50 ms limit. In these cases, the tables in this chapter report the response time at the point of maximum throughput.
The SPEC SFS 2.0 (0.97 LADDIS)benchmark is a synthetic NFS workload based on an application abstraction, an NFS operation mix, and an NFS operation request rate. The workload generated by the benchmark emulates an intensive software development environment at the NFS protocol level. The LADDIS benchmark makes direct RPC calls to the server, eliminating any variation in NFS client implementation. This makes it easier to control the operation mix and workload, especially for comparing results between vendors. However, this also hides the benefits of special client implementations, such as the cache file system client.
Table A-7 shows the NFS operations mix. These percentages indicate the relative number of calls made to each operation.
Table A-7 NFS Operations Mix by Call
NFS Operation |
Percent Mix |
---|---|
Lookup |
34 |
Read |
22 |
Write |
15 |
GetAttr |
13 |
ReadLink |
8 |
ReadDir |
3 |
Create |
2 |
Remove |
1 |
Statfs |
1 |
SetAttr |
1 |
The LADDIS benchmark for NFS file systems uses an operation mix that is 15 percent write operations. If your NFS clients generate only one to two percent write operations, LADDIS underestimates your performance. The greater the similarity between the operation mixes, the more reliable the maximum throughput in NFS operations is as a reference.
Running the benchmark requires the server being benchmarked to have at least two NFS clients (the NFS load generators), and one or more isolated networks. The ability to support multiple networks is important because a single network may become saturated before the server maximum performance point is reached. One client is designated as the LADDIS Prime Load Generator. The Prime Load Generator controls the execution of the LADDIS load generating code on all load-generating clients. It typically controls the benchmark. In this capacity, it is responsible for collecting throughput and response time data at each of the workload points and for generating results.
To improve response time, configure your NFS server with the NVRAM-NVSIMM Prestoserve NFS accelerator. NVSIMMs provide storage directly in the high-speed memory subsystem. Using NVSIMMs results in considerably lower latency and reduces the number of disks required to attain a given level of performance.
Since there are extra data copies in and out of the NVSIMM, the ultimate peak throughput is reduced. Because NFS loads rarely sustain peak throughput, the better response time using the NVSIMMs is preferable. For information on the Prestoserve NFS accelerator, see "Prestoserve NFS Accelerator" in Chapter 4, Configuring the Server and the Client to Maximize NFS Performance.
Sun Microsystems has run SPEC SFS 2.0 benchmarks on the Sun Enterprise 3000-6000 family of servers. The benchmarks were run with NFS Version 2 and NFS Version 3. Table A-8 shows the results with Version 2. Table A-9 shows the results with Version 3.
Table A-8 SPEC SFS 2.0 Results With Version 2
System |
Number of CPUs |
Result |
Overall Response Time |
---|---|---|---|
Sun Enterprise 3000 |
6 |
11806 |
5.1 |
Sun Enterprise 4000 |
12 |
20406 |
6.7 |
Sun Enterprise 6000 |
18 |
25639 |
9.9 |
Table A-9 SPEC SFS 2.0 Results With Version 3
System |
Number of CPUs |
Result |
Overall Response Time |
---|---|---|---|
Sun Enterprise 3000 |
6 |
5903 |
5.4 |
Sun Enterprise 4000 |
12 |
10592 |
5.6 |