Double network throughput by tuning network parameters on Solaris and Linux

The default network buffer parameter in Solaris is too conservative, Linux from kernel 2.6.x is ok. The tuning applies to network environment where high throughput is needed e.g. ISCSI/NFS/CIFS storage Server, it is not wise to raise network buffer on firewall, because the parameters are per connection based, It just waste large memory to handle small data flow.

ENV

Virtualbox 3.1+ Centos 5.3 VM + OpenSolaris-2009.06 VM + Intel Pro/1000 desktop NIC for each VM +1G RAM for each VM. I use Iperf to test throughput by transfer 900Mb data (memory to memory, no disk IO involved),

Firstly, raise MTU to 9000 on the Gigabit Ethernet interface (no point to change default network buffer for Fast Ethernet).

Before Tuning

Start iperf on Solaris as Server, the TCP window size detected is 48KB.

root@opensolaris:~# /usr/local/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.1.12 port 5001 connected with 172.16.1.11 port 39019
[ ID] Interval       Transfer     Bandwidth

Start iperf on Linux to transfer 900M data,the TCP window size detected is 27.5KB Please note the MSS is 8948 as expected. Bandwidth is 390 Mbits/sec

[root@centos1 ~]# iperf   -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size: 27.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.11 port 39019 connected with 172.16.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-19.3 sec    900 MBytes    390 Mbits/sec
[  3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

Tuning commands

Solaris tuning:

ndd -set /dev/tcp tcp_xmit_hiwat 983040
ndd -set /dev/tcp tcp_recv_hiwat 983040
ndd -set /dev/tcp tcp_max_buf  4194304

Save the command in startup script e.g /etc/rc3.d/S99local to survive reboot

Linux Tuning:

sysctl -w net.ipv4.tcp_rmem="40960       1048560 4194304"
sysctl -w net.ipv4.tcp_wmem="40960       196608  4194304"
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=4194304

All values are per connection based in bytes. net.core.rmem_max/net.core.wmem_max is for all protocols. No need to change net.core.[rw]mem_default because the 2nd value of net.ipv4.tcp_[rw]mem overrides it.

You don't need to change net.ipv4.tcp_mem, the default values are just fine, It is in pages(normally 4KB) for overall cap for all connections. There are other advanced parameters e.g net.ipv4.tcp_sack/net.ipv4.tcp_timestamps, I don’t change them ,because the effect seems unpredictable in complex network environment. Save the commands without leading "sysctl -w" in /etc/sysctl.conf to survive reboot.

run "man tcp" for more information.

After Tuning

After tuning the bandwidth is more than doubled: 955 Mbits/sec Please note the default Window Size was changed on both Solaris and Linux

root@opensolaris:~# /usr/local/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:   960 KByte (default)

[root@centos1 ~]# iperf   -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size:   192 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.11 port 36932 connected with 172.16.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 7.9 sec    900 MBytes    955 Mbits/sec
[  3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

Reference:

Solaris TCP Tunable Parameters

Change MTU for Solaris

Download iperf for Solaris 10