The default network buffer parameter in Solaris is too conservative, Linux from kernel 2.6.x is ok. The tuning applies to network environment where high throughput is needed e.g. ISCSI/NFS/CIFS storage Server, it is not wise to raise network buffer on firewall, because the parameters are per connection based, It just waste large memory to handle small data flow.
ENV
Virtualbox 3.1+ Centos 5.3 VM + OpenSolaris-2009.06 VM + Intel Pro/1000 desktop NIC for each VM +1G RAM for each VM. I use Iperf to test throughput by transfer 900Mb data (memory to memory, no disk IO involved),
Firstly, raise MTU to 9000 on the Gigabit Ethernet interface (no point to change default network buffer for Fast Ethernet).
Before Tuning
Start iperf on Solaris as Server, the TCP window size detected is 48KB.
root@opensolaris:~# /usr/local/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[ 4] local 172.16.1.12 port 5001 connected with 172.16.1.11 port 39019
[ ID] Interval Transfer Bandwidth
Start iperf on Linux to transfer 900M data,the TCP window size detected is 27.5KB Please note the MSS is 8948 as expected. Bandwidth is 390 Mbits/sec [root@centos1 ~]# iperf -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size: 27.5 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.1.11 port 39019 connected with 172.16.1.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-19.3 sec 900 MBytes 390 Mbits/sec
[ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
Tuning commands
Solaris tuning:
ndd -set /dev/tcp tcp_xmit_hiwat 983040
ndd -set /dev/tcp tcp_recv_hiwat 983040
ndd -set /dev/tcp tcp_max_buf 4194304
Save the command in startup script e.g /etc/rc3.d/S99local to survive reboot
Linux Tuning:
sysctl -w net.ipv4.tcp_rmem="40960 1048560 4194304"
sysctl -w net.ipv4.tcp_wmem="40960 196608 4194304"
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=4194304
All values are per connection based in bytes.
net.core.rmem_max/net.core.wmem_max is for all protocols. No need to change net.core.[rw]mem_default because the 2nd value of net.ipv4.tcp_[rw]mem overrides it. You don't need to change net.ipv4.tcp_mem, the default values are just fine, It is in pages(normally 4KB) for overall cap for all connections. There are other advanced parameters e.g net.ipv4.tcp_sack/net.ipv4.tcp_timestamps, I don’t change them ,because the effect seems unpredictable in complex network environment. Save the commands without leading "sysctl -w" in /etc/sysctl.conf to survive reboot.
run "man tcp" for more information.
After Tuning
After tuning the bandwidth is more than doubled: 955 Mbits/sec Please note the default Window Size was changed on both Solaris and Linux
root@opensolaris:~# /usr/local/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 960 KByte (default)
[root@centos1 ~]# iperf -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size: 192 KByte (default)
------------------------------------------------------------
[ 3] local 172.16.1.11 port 36932 connected with 172.16.1.12 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 7.9 sec 900 MBytes 955 Mbits/sec
[ 3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)
Reference:
Solaris TCP Tunable Parameters
Change MTU for Solaris
Download iperf for Solaris 10
Posts
