Networking Enhancement

This section describes a networking enhancement in this release.

SSH, SCP, and SFTP Speed Improvements

Secure Copy, scp(1) shows up to 8X improved performance in bulk data transfers over high-bandwidth, high-latency networks in Oracle testing.

This scenario typically occurs during long distant communications, for example data transfer between the US and Europe using a high bandwidth connection. scp uses ssh(1) to do the actual transfers. Prior to Oracle Solaris 10 1/13, you could not adjust the receive window buffer used internally by ssh to tune performance for a particular situation of high bandwidth and high latency.

In Oracle Solaris 10 1/13 (and Oracle Solaris 11.1), the size of the ssh receive buffer is linked to the system tcp receive buffer setting (tcp_recv_hiwat). With this relationship established, it is now possible to tune for this specific data transfer situation, i.e. a high bandwidth and high latency connection. For high latency communications (in other words high round trip time communications), the copy performance can be significantly improved by allowing more packets to be in transit. With this strategy, the round trip time to send a packet and receive an acknowledgement does not limit throughput because many packets are allowed to be in transit. The limit for the amount of data that could be in transit is determined by the tcp_recv_hiwat setting. However, this strategy only works if the bandwidth is large enough that data in the link does not impact the new data sent. If the bandwidth is not large enough, there is little advantage in increasing the TCP receive window size.

The following table shows the improvements measured using a high bandwidth connection with latencies between 50-200 milliseconds:

By default, tcp_recv_hiwat is set to 48 KB in Oracle Solaris 10. This value is optimized for 100 MB networks. For Oracle Solaris 11, the default value was increased to 128 KB to optimize for 1 GB LANs. Neither of those buffer size choices show any appreciable difference in performance between Oracle Solaris 10 1/13 and Oracle Solaris 10 8/11 for high-latency or high-bandwidth networks. You can see meaningful improvements in speed when the TCP receive window buffer is set to 256 KB.

Increasing the tcp_recv_hiwat:

• Increases the TCP receive buffer requirements for every receiver on the system.

• Impacts every TCP sender because they likely will need to buffer that much data due to the length of time to receive packet acknowledgements.

• Puts additional load on the route between senders and receiver to handle the increased traffic enabled by increasing the buffer window size.

For more benchmark details, see https://blogs.oracle.com/BestPerf/entry/20130208_solaris_scp.

Additional observations:

• sftp(1) demonstrated similar performance improvements as it also uses ssh.

• tcp_recv_hiwat is a system wide setting that typically only the administrator can influence.

• You can also increase the value of the -recvpipe option in the route(1M) utility as an alternative per-connection tuning facility. See http://docs.oracle.com/cd/E23823_01/html/817-0404/chapter4-64.html#scrolltoc.