Configure multipath on Solaris 11 for IBM V7000 SAN storage

IBM V7000 is not listed from command “mpathadm show mpath-support libmpscsi_vhci.so”, but it is still supported, Solaris 11 mpxio supports any third party storage device that is T10/T11 standards-compliant.
Procedures to setup multipath

#Setup zoning in SAN switch
#Login to V7000 management UI to map Solaris host to the volume, select host type ‘TPGS’
#Rescan new SAN disks without rebooting
$cfgadm -o force_update -c configure cX (X is the port id as shown cfgadm -al)
#verify SAN disks are detected.
$echo|format
#create scsi_vhci.conf
#scsi_vhci.conf doesn’t need to customized, scsi-vhci-failover-override parameter is optional, IBM V7000 is detected as f_tpgs with the standard probe.
$cp /kernel/drv/scsi_vhci.conf /etc/driver/drv/scsi_vhci.conf

#Obtain the device path of the fc ports of a single HBA
$ls -l /dev/cfg
lrwxrwxrwx   1 root     root          60 Feb 28 10:20 c4 -> ../../devices/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0/fp@0,0:fc
lrwxrwxrwx   1 root     root          62 Feb 28 10:20 c5 -> ../../devices/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0:fc

the path needed is the string between ./devices/ and fp@, so the paths are  
/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0
/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1

#Create fp.conf
cp /kernel/drv/fp.conf /etc/driver/drv/fp.conf
#edit fp.conf to enable multipath for the two fc ports only
mpxio-disable="yes";
name="fp" parent="/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0" port=0 mpxio-disable="no";
name="fp" parent="/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1" port=0 mpxio-disable="no";
#run the command to enable multiple path on fc ports only, server will need to be rebooted.
$stmsboot –u –D fp

#verify 
$ echo| format will show only a single disk
$stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c4t50050768024046D8d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
/dev/rdsk/c4t50050768022046D9d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
/dev/rdsk/c5t50050768022046D8d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
/dev/rdsk/c5t50050768024046D9d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
$mpathadm list lu
        /dev/rdsk/c0t5000CCA0164251B8d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t5000CCA01642CDDCd0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t6005076802830163A000000000000005d0s2
                Total Path Count: 4
                Operational Path Count: 4
$mpathadm show lu /dev/rdsk/c0t6005076802830163A000000000000005d0s2

#the disk is deteced as f_tpgs as shown in messages log
$grep f_ /var/adm/messages
Mar 21 13:56:46 dnmsovm1 scsi: [ID 583861 kern.info] ssd4 at scsi_vhci0: unit-address g6005076802830163a000000000000005: f_tpgs

Setup Perl in Solaris 10 to talk to Microsoft SQL Server 2000.

I tested following solution in Solaris 10 to connect to Microsoft SQL Server 2000, But it should work with MS SQL 2005. (Maybe with MS SQL 2008 as well)
In Unix/Linux, there are two typical ways to have Perl to talk to Microsoft SQL Server.
1)  Perl DBI + Perl DBD::ODBC + ODBC Driver
2)  Perl DBI + Perl DBD::Sybase + Freetds driver
I choose Freetds solution  because Unix ODBC driver is not free (http://www.unixodbc.org/).

Install complier and build tools

Compiler choices for Solaris: 1) GCC For Sun Systems: Compatible with gcc and optimised for Sun hardware
free download from Oracle.
2) Solaris Studio: Solaris Native complier. free download from Oracle 
3) gcc GNU compiler
Free download from  http://www.sunfreeware.com/  or retrieve from Solaris 10 CD (SUNWgcc)
My Personal opinion: “GCC For Sun Systems” is the best choice for compatibility  and performance,GNU gcc is the last choice. Build tools choices for Solaris: 1) GNU build tools
NAME: SUNWgmake  SUNWbinutils SUNWgm4 
DOWNLOAD: Free download from http://www.sunfreeware.com/ or retrieve from Solaris 10 CD
PATH: /usr/sfw/bin/{gmake,gld,gnm,gar,gas,ggprof,gm4,granlib,gsize,gstrip}
2) SUN build tools
NAME:SUNWbtool
DOWNLOAD: Retrieve from Solaris 10 CD
PATH:/usr/ccs/bin/{make,ld,nm,ar,as,gprof,m4,ranlib,size,strip}
I recommend to start with SUN build tools, if it doesn’t work try GNU build tools.
If you choose GNU build tools, you may need to create links to remove prefix “g” and rename dir /usr/ccs/bin (in case SUN build tools are picked up first)
 for i in   {gmake,gld,gnm,gar,gas,ggprof,gm4,granlib,gsize,gstrip}; do echo ln -s $i ${i#g}; done

Setup Freetds, Perl DBI and Perl DBD::sybase

Install Freetds
##Download
  Download source code from: http://www.freetds.org/
##Install
./configure --with-tdsver=8.0; make ; make install
tdsver=8.0 is compatible with MS SQL 2000, but it has been reported that SQL 2005 works with it as well.
By default, freetds binaries are installed in /usr/local/bin and libraries are installed in /usr/local/lib. you can override this with --prefix.
##add /usr/local/lib to search path.
If your crle output doesn't include /usr/local/lib, you can append it current lib dirs e.g 
crle -u -l /lib:/usr/lib:/usr/local/lib
(the equivalent Linux command is ldconfig)

##Test Login
   /usr/local/bin/tsql -H 172.1.1.1 -p 1433 -U 'username' -P password -D DB-NAME
（tsql is just for testing connection, The interactive shell working with freetds is http://shellsql.sourceforge.net/, so you can run sql command in shell script with shellsql!）

make sure above command works before continue to next step

Install Perl DBI
download from http://search.cpan.org/
perl Makefile.PL; make ; make install
Install Perl DBD::Sybase
download from http://search.cpan.org/
export SYBASE=/usr/local;
perl Makefile.PL; make ; make install
Sample Perl Script

#!/usr/local/bin/perl -w
use DBI;
my $dbh = DBI->connect('DBI:Sybase:server=172.1.1.1;port=1433','username,'password') or die $DBI::errstr;
$dbh->do("use DB-NAME");
$SQL= "SELECT uid,name from employee;";
my $ids = $dbh->selectall_arrayref( "$SQL", { Slice => {} } );
foreach my $id ( @$ids ) {
print "$id->{uid} | $id->{name} \n";
}

Change Solaris user password with Expect script

Changing Linux user password  without user keyboard input  is very easy by  builtin command "chpasswd "or "passwd --stdin".
Solaris doesn't have such commands , the result can be achieved  by Expect script.

#Expect script
$cat chpwd.sh

#!/usr/local/bin/expect --
# Input: username password
set USER [lindex $argv 0]
set PASS [lindex $argv 1] 

if { $USER == "" || $PASS == "" }  {
   puts "Usage:  ./scriptname username password\n"
   exit 1
 }

spawn sudo passwd $USER 
expect "assword:"
send "$PASS\r"
expect "assword:"
send "$PASS\r"
expect eof

#Shell script: Generate a random password and call the Expect script to set it


$cat createpwd.sh

#!/bin/ksh
# Generate a random password  for a user and set the new password 
USER=$1
PASS=`tr -dc [:alnum:] </dev/urandom | fold -8 | head -1`
if [ -z "$USER" ]; then
echo "Usage $0 username"
exit
fi
./chpwd.sh $USER $PASS 
echo "username=$USER"
  
echo "password=$PASS"

Improve OpenSolaris COMSTAR iSCSI Server performance.

I setup OpenSolaris iSCSI and Linux Open iSCSI in Virtualbox to compare the performace. I can get 70~80% of local FS throughput on Linux iSCSI, but it is 10~20% on OpenSolaris iSCSI.
Some tuning can boost OpenSolaris iSCSI performance to be on par with Linux iSCSI

#Environment
Server: OpenSolaris 2009.06 VS Openfiler 2.3 on Virutalbox 3.1
Client: Centos 5.2
Regardless of the iSCSI Server type, the network has to be optimized by tuning network parameters
#OpenSolaris iSCSI tunning
1. Use the bock device(/dsk) as backstore, NOT character device (/rdsk), because the block device (/dev/dsk) has buffer.

$iscsitadm create target -b /dev/zvol/dsk/data/iscsi filera.dbvol

2. Turn on fast-write-ack

$iscsitadm modify admin --fast-write-ack enable

reactivate the zpool for existing target.

$svcadm disable iscsi/target iscsitgt
$zpool export poolname ;zpool import poolname
$svcadm enable iscsi/target iscsitgt

Use the option with care, from iscsitadm man pages:
Enables or disables fast-write acknowledgment. You should enable this option only if a system is connected to the power grid through a UPS.
Otherwise, data corruption could occur if power is lost and some writes that were acknowledged have not been completely flushed to the backing store.

Setup OpenSolaris ISCSI server for Solaris and Linux Client

Setup ISCSI server in Opensolaris 2009.06 and configure Opensolaris/Centos 5.2 client to connect to ISCSI server

Simple ISCSI setup without access-list

####ISCSI Server

##enable iscsi service
svcadm enable iscsi/target iscsitgt

## Create zpool on  whole disk ( XXp0 is whole disk convention for  i86 arch, SPARC arch use XXs2)
zpool create data c8t1d0p0
##create volume on the zpool
zfs create -V 1G data/iscsi

##Create ISCSI target filera.dbvol, the name is arbitrary, it doesn't need to be volume name

[Server]$iscsitadm create target -b /dev/zvol/dsk/data/iscsi filera.dbvol

[Server]$iscsitadm list target  -v
Target: filera.dbvol                                    
    iSCSI Name: iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol
    Connections: 0                                                                     
    ACL list:                                                                          
    TPGT list:                                                                         
    LUN information:                                                                   
        LUN: 0                                                                         
            GUID: 0                                                                    
            VID: SUN                                                                   
            PID: SOLARIS                                                               
            Type: disk                                                                 
            Size: 1.0G                                                                 
            Backing store: /dev/zvol/dsk/data/iscsi                                   
            Status: online        

Command  "zfs set shareiscsi=on data/iscsi " can also create target, but the target name wil be volume name

## if ISCSI server has multiple NICS, the target will be exposed on all NICS, following is to limit the target to the desired NIC
[Server]$iscsitadm create tpgt 1
[Server]$iscsitadm modify tpgt -i 172.16.1.12 1
[Server]$iscsitadm modify target -p 1 filera.dbvol

####Client: Solaris

Client-Solaris]$svcadm enable iscsi_initiator
[Client-Solaris]$iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:946808d8ea81.4b6a1021
##Change the long name to a short  friendly name
[Client-Solaris]$iscsiadm modify initiator-node  -N iqn.1986-03.com.sun:01:opensolaris01
#add the server address to discover
[Client-Solaris]$iscsiadm modify discovery -t  enable
[Client-Solaris]$iscsiadm add discovery-address 172.16.1.12
[Client-Solaris]$iscsiadm list target -v
#check new disk discovered
[Client-Solaris]$echo  format

####Client: Linux

##assign a friendy name to client
[Client-Linux]$ cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:centos01
[Client-Linux]$service iscsi start
[Client-Linux]$iscsiadm --mode discovery --type sendtargets --portal 172.16.1.12
##login to ISCSI discovered in previous step
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol " --portal "172.16.1.12:3260" --login
##Check the new discovered disk
[Client-Linux]$fidsk -l

ISCSI access list and CHAP authentication

Access list is a must for security reason, it is just like LUN mapping in SAN. If you need additional security, add chap-name and chap-secret, Access list alone works fine without chap initialized


##On Server, Set up CHAP information for Linux client

##Create mapping of initiator ISCSI name to an arbitrary name

[Server]$iscsitadm create initiator --iqn iqn.1994-05.com.redhat:centos01   centos01
[Server]$iscsitadm modify initiator --chap-name user_centos01 centos01
[Server]$iscsitadm modify initiator --chap-secret centos01
[Server]$iscsitadm modify  target --acl centos1 filera.dbvol

##On Server, Set up CHAP information for Solaris client

[Server]$iscsitadm create initiator --iqn iqn.1986-03.com.sun:01:opensolaris01   opensolaris01
[Server]$iscsitadm modify initiator --chap-name user_opensolaris01 opensolaris01
[Server]$iscsitadm modify initiator --chap-secret opensolaris01
[Server]$iscsitadm modify  target --acl opensolaris01 filera.dbvol

##On Linux Client, use the CHAP info to login

[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --op=update --name node.session.auth.authmethod --value=CHAP
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --op=update --name node.session.auth.username --value=user_centos01
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --op=update --name node.session.auth.password --value=Password0Password0
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --login

##On Solaris Client, use the CHAP info to login

[Client-Solaris]$ iscsiadm modify initiator-node --authentication CHAP
[Client-Solaris]$ iscsiadm modify initiator-node --CHAP-name user_opensolaris01
[Client-Solaris]$ iscsiadm modify initiator-node --CHAP-secret
Enter secret :
Re - enter secret :

Double network throughput by tuning network parameters on Solaris and Linux

The default network buffer parameter in Solaris is too conservative, Linux from kernel 2.6.x is ok. The tuning applies to network environment where high throughput is needed e.g. ISCSI/NFS/CIFS storage Server, it is not wise to raise network buffer on firewall, because the parameters are per connection based, It just waste large memory to handle small data flow.

ENV

Virtualbox 3.1+ Centos 5.3 VM + OpenSolaris-2009.06 VM + Intel Pro/1000 desktop NIC for each VM +1G RAM for each VM. I use Iperf to test throughput by transfer 900Mb data (memory to memory, no disk IO involved),

Firstly, raise MTU to 9000 on the Gigabit Ethernet interface (no point to change default network buffer for Fast Ethernet).

Before Tuning

Start iperf on Solaris as Server, the TCP window size detected is 48KB.

root@opensolaris:~# /usr/local/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[  4] local 172.16.1.12 port 5001 connected with 172.16.1.11 port 39019
[ ID] Interval       Transfer     Bandwidth

Start iperf on Linux to transfer 900M data,the TCP window size detected is 27.5KB Please note the MSS is 8948 as expected. Bandwidth is 390 Mbits/sec

[root@centos1 ~]# iperf   -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size: 27.5 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.11 port 39019 connected with 172.16.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-19.3 sec    900 MBytes    390 Mbits/sec
[  3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

Tuning commands

Solaris tuning:

ndd -set /dev/tcp tcp_xmit_hiwat 983040
ndd -set /dev/tcp tcp_recv_hiwat 983040
ndd -set /dev/tcp tcp_max_buf  4194304

Save the command in startup script e.g /etc/rc3.d/S99local to survive reboot

Linux Tuning:

sysctl -w net.ipv4.tcp_rmem="40960       1048560 4194304"
sysctl -w net.ipv4.tcp_wmem="40960       196608  4194304"
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=4194304

All values are per connection based in bytes. net.core.rmem_max/net.core.wmem_max is for all protocols. No need to change net.core.[rw]mem_default because the 2nd value of net.ipv4.tcp_[rw]mem overrides it.

You don't need to change net.ipv4.tcp_mem, the default values are just fine, It is in pages(normally 4KB) for overall cap for all connections. There are other advanced parameters e.g net.ipv4.tcp_sack/net.ipv4.tcp_timestamps, I don’t change them ,because the effect seems unpredictable in complex network environment. Save the commands without leading "sysctl -w" in /etc/sysctl.conf to survive reboot.

run "man tcp" for more information.

After Tuning

After tuning the bandwidth is more than doubled: 955 Mbits/sec Please note the default Window Size was changed on both Solaris and Linux

root@opensolaris:~# /usr/local/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:   960 KByte (default)

[root@centos1 ~]# iperf   -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size:   192 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.11 port 36932 connected with 172.16.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 7.9 sec    900 MBytes    955 Mbits/sec
[  3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

Reference:

Solaris TCP Tunable Parameters

Change MTU for Solaris

Download iperf for Solaris 10

Change MTU for Solaris on e1000g interface

I have Linux and OpenSolaris installed on Virtualbox with Intel Pro/1000 network interface, Changing MTU on Linux worked fine but it failed on Solaris

$ifconfig e1000g1 mtu 9000
ifconfig: setifmtu: SIOCSLIFMTU: e1000g1: Invalid argument

It turns out that Solaris's driver doesn't have jumbo frame enabled by default, you have to enable it manually. Following enables jumbo frame on e1000g1 only.

$/kernel/drv/e1000g.conf 
MaxFrameSize=0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
        # 0 is for normal ethernet frames.
        # 1 is for upto 4k size frames.
        # 2 is for upto 8k size frames.
        # 3 is for upto 16k size frames.
        # These are maximum frame limits, not the actual ethernet frame
        # size. Your actual ethernet frame size would be determined by
        # protocol stack configuration (please refer to ndd command man pages)
        # For Jumbo Frame Support (9k ethernet packet)
        # use 3 (upto 16k size frames)

Now set MTU to 9000 instead of 16K by editing file hostname.e1000g1. hostname.e1000g1 is interface configuration file, the first entry is your ip address or a name, the name must be resolvable in /etc/hosts.

$/etc/hostname.e1000g1
ip/name mtu 9000

#shutdown -i6 to reboot to take effect.

dladm is supposed to be the new method, But it didn't work.

#dladm show-linkprop -p mtu e1000g1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
e1000g1      mtu             rw   16298          1500           -- 
#dladm set-linkprop -p mtu=9000  e1000g1
dladm: warning: cannot set link property 'mtu' on 'e1000g1': try again later

Troubleshooting a high system CPU usage issue on Linux/Solaris

A Linux server has high %system CPU usage, following are steps to find the root cause of the issue and how to resolve it.

Vmstat show %system CPU usage is high.
# vmstat 2

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

1  0      0 191420   8688  35780    0     0     0     0 1006   31  1  4 96  0  0
1  0      0 124468   9208  98020    0    0 15626  2074 1195  188  0 76  0 24  0
0  1      0 110716   9316 110996    0    0  3268  4144 1366   84  0 94  0  7  0
0  3      0  97048   9416 122272    0    0  2818 11855 1314  109  1 80  0 20  0
0  4      0  80476   9544 137888    0    0  3908  2786 1272  172  0 54  0 46  0
2  1      0  72860   9612 145848    0    0  1930     0 1193  141  0 42  0 58  0
0  1      0  74300   9620 145860    0    0     0     6 1208   67  0 38  0 62  0
0  0      0  75680   9620 145860    0    0     0  6929 1364  101  0 70  6 24  0

Let’s run mpstat to show more detailed CPU usage,it showed CPU was busy with interruptions.

# mpstat 2
Linux 2.6.18-92.el5 (centos-ks)         01/14/2010

02:03:50 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
02:04:04 AM  all    1.33    0.00   41.78    0.00    0.44    3.56    0.00   52.89   1015.56
02:04:06 AM  all    0.00    0.00    8.04   38.69   29.65   23.62    0.00    0.00   1326.63
02:04:08 AM  all    0.00    0.00    8.70   30.43   27.54   28.50    0.00    4.83   1327.54
02:04:10 AM  all    0.00    0.00    5.47   46.77   27.36   20.40    0.00    0.00   1280.10
02:04:12 AM  all    0.50    0.00    6.47   63.18   19.40   10.45    0.00    0.00   1183.08
02:04:14 AM  all    1.01    0.00    6.53   62.31   21.11    9.05    0.00    0.00   1190.95
02:04:16 AM  all    0.00    0.00    8.04   26.63   43.72   21.61    0.00    0.00   1365.83
02:04:18 AM  all    0.00    0.00    1.50    0.00    0.00    0.50    0.00   98.00   1006.50

Use sar to find out which interrupt number was culprit. #9 was the highest excluding system interrupt #0.
# sar -I XALL 2 10

02:07:10 AM      INTR    intr/s
02:07:12 AM         0    992.57
02:07:12 AM         1      0.00
02:07:12 AM         2      0.00
02:07:12 AM         3      0.00
02:07:12 AM         4      0.00
02:07:12 AM         5      0.00
02:07:12 AM         6      0.00
02:07:12 AM         7      0.00
02:07:12 AM         8      0.00
02:07:12 AM         9    350.50

[ Solaris equivalent command]
Solaris# intrstat 2 

device |      cpu0 %tim      cpu1 %tim 
-------------+------------------------------ 
bge#0 |         0  0.0       128  0.6 
cpqary3#0 |         0  0.0        14  0.0

# cat /proc/interrupts

CPU0
0:     702980          XT-PIC  timer
1:        439          XT-PIC  i8042
2:          0          XT-PIC  cascade
6:          2          XT-PIC  floppy
8:          1          XT-PIC  rtc
  9:      14464          XT-PIC  acpi, eth2
11:         12          XT-PIC  eth0
12:        400          XT-PIC  i8042
14:       6091          XT-PIC  ide0
15:         22          XT-PIC  ide1
NMI:          0
LOC:     700623
ERR:          0
MIS:          0

[ OpenSolaris equivalent command ]

Solaris#echo ::interrupts | mdb –k
Native Solaris has to search the interrupt from output of prtconf -v

Solution:
When the card transmits or receives a frame, the system must be notified of the event. If the card interrupts the system for each transmitted and received frame, the result is a high degree of processor overhead. To prevent that, Gigabit Ethernet provides a feature called Interrupt Coalescence. Effective use of this feature can reduce system overhead and improve performance.

Interrupt Coalescence essentially means that the card interrupts the system after sending or receiving batch of frames. 

you can enable adaptive moderation ( Adaptive RX: off  TX: off) to let system choose value automatically or set individual values manually.

A interrupt is generated by the card to the host when either frame counter or timer counter is met. Values 0 means disabled.

RX for example:
Timer counter in microseconds: rx-usecs/rx-usecs-irq
Frames counter:rx-frames/rx-frames-irq

# A sample output with default values.
# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: off  TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 6
rx-usecs-irq: 18
rx-frames-irq: 6

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 80
tx-frames-irq: 20

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
<>

[ Solaris equivalent command]

Varying on driver, Find out the driver's capability. 
Solaris#ndd -get /dev/e1000g0 \? | egrep ‘interrupt |intr’
The Value shoud be set in driver conf file:
Solaris#/platform/`uname -m`/kernel/drv/*.conf

Alternative Workaround:

I couldn't config Interrupt Coalescence because virtual machine NIC didn't support it. but as workaround, Increasing mtu can also decrease interrupt, ifconfig eth2 mtu 9000 resolved the issue. It needs to set on both hosts peer, if they are not directly connected, make sure the switch supports jumbo frames. 
You don't need to care Interrupt Coalescence if your CPU resource is abundant, But for high load NFS/CIFS/ISCSI/ NAS servers, it is very useful.

Command to get system hardware serial number.

SMBIOS/DMI standard includes system manufacturer, model name, serial number, BIOS version, asset tag as well as a lot of other details of varying level of interest and reliability depending on the manufacturer. This will often include usage status for the CPU sockets, expansion slots (e.g. AGP, PCI, ISA) and memory module slots, and the list of I/O ports (e.g. serial, parallel, USB).
Solaris X86:

$smbios
  ..
  Manufacturer: HP
  Product: ProLiant DL360 G3
  Serial Number: B038XXXXX
  ..

Solaris Sparc:
smbios is not supported in SPARC yet, the traditional command prtdiag works for both X86 and SPARC ,but it reports less detailed hardware information.

Linux/BSD:

$dmidecode

Create restricted login account

Create a login in restricted shell and doesn’t allow user to change password.

rsh is a limiting version of the standard command inter-
preter sh, used to restrict logins to execution environments
whose capabilities are more controlled than those of sh (see
sh(1) for complete description and usage).

The actions of rsh are identical to those of sh, except that
the following are disallowed:

changing directory (see cd(1)),

setting the value of $PATH,

pecifying path or command names containing /,

redirecting output (> and >>).

Set restricted shell as login shell

Solaris:
usermod  -s /usr/lib/rsh userid
Linux:
usermod  -s /usr/bin/rbash userid

Set minimum number of days between password changes to large number, so user can’t change password until min days

Solaris:
passwd  -n 9999 –x 9998 userid
(Solaris needs to set both Min Max days and Min is greater than Max)
Linux:
passwd –n 9999 userid

Dtrace Basics

- D Program Structure
probe descriptions
/ predicate /
{
action statements
}
-- Probe Descriptions
provider : subsystem : function : probeName
syscall::*lwp*:entry, syscall::*sock*:entry #support wildcards, one or more fields of the probe description are omitted(means any value)
--- list available provider moduel function name
$dtrace -l more
ID PROVIDER MODULE FUNCTION NAME
1 dtrace BEGIN
2 dtrace END
3 dtrace ERROR
4 nfsmapid209 nfsmapid check_domain daemon-domain
5 nfsmapid209 nfsmapid resolv_query_thread thread-domain
6 syscall nosys entry
-- Predicates
Predicates are expressions enclosed in slashes / / that are evaluated at probe firing time to determine whether the associated actions should be executed.
D language doesn't has control-flow constructs such as if-statements and loops. it use Predicates
-- Actions
Probe actions are described by a list of statements separated by semicolons (;) and enclosed in braces { }.If no additional action need, an empty set of braces with no statements inside
-- example
$dtrace -n syscall::read:entry #

-n means to match probename from command line, -m=match module name
$dtrace -s counter.d #- s read input from script
$vi counter.d
dtrace:::BEGIN
{
i = 10;
}
profile:::tick-1sec
/i > 0/
{
trace(i--);
}
profile:::tick-1sec
/i == 0/
{
trace("blastoff!");
exit(0);
}

dtrace/profile are providers
tick-5sec #tick-xsec is the function name of provider profile, like sleep in shell
trace (100) #print out a value or string(needs enclosed by " ",trace ("hello")), like echo in shell
printf ("%s","hello") # print out in particular format
/* ... */ #comment lines
if no
dtrace:::END statement, you need to press ctrl+c to see to the result.

--Use built-in variables for predicates
syscall::read:entry,
syscall::write:entry
/pid == 12345/
{
}


execname: Name of the current process's executable file
pid:Process ID of the current process
tid: Thread ID of the current thread
...

- Aggregations
DTrace stores the results of aggregating functions in objects called aggregations. The aggregation results are indexed using a tuple of expressions similar to those used for associative arrays. In D, the syntax for an aggregation is
@name[ keys ] = aggfunc ( args );
Aggregations is used for result data,the entire data set need not be stored
Arrgegations are printed out by default no need to print statement
-- Example
# Syscall count by process,
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

-- DTrace Aggregating Functions
count: The number of times called.
sum: The total value of the specified expressions.
.....

- Structs
If you have programmed in the Java programming language, think of aDstruct as a class, but one with data members only and no methods.
struct callinfo {
uint64_t ts; /* timestamp of last syscall entry */
uint64_t elapsed; /* total elapsed time in nanoseconds */
uint64_t calls; /* number of calls made */
size_t maxbytes; /* maximum byte count argument */
};
You can use the operator -> to access struct members through a pointer. callinfo->ts
- Further reading..
Solaris Dynamic Tracing Guide - Official Dtrace guide
DTrace Tools - collection of useful scripts

Solaris/Linux: find port number for a program and vice-versa

#== Find port number for a program
- lsof tool(Platform independent)

$lsof -nc | sshd grep TCP

sshd 1962 root 3u IPv6 6137 TCP *:ssh (LISTEN)
sshd 2104 root 3u IPv6 7425 TCP 172.16.31.3:ssh->172.16.31.2:cs-services (ESTABLISHED
- Linux

$netstat -anp |grep sshd

tcp 0 0 :::22 :::* LISTEN 1962/sshd
- Solaris
$ pfiles 16976
...
sockname: AF_INET 172.18.126.148 port: 22
..

#==Find program name for port number
- lsof tool(Platform independent)
$lsof -i TCP:22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 1962 root 3u IPv6 6137 TCP *:ssh (LISTEN)
sshd 2104 root 3u IPv6 7425 TCP 172.16.31.3:ssh->172.16.31.2:cs-services (ESTABLISHED)
- Linux
$netstat -anp grep 22
tcp 0 0 :::22 :::* LISTEN 1962/sshd
- Solaris
list open files for all process,then search the file for "port: 22"

$ ps -e -o pid | xargs pfiles > /tmp/pfiles.log 

When some processes stop Solaris Zone from being shutdown

If Solaris Zone takes long time to shutdown, you may need to examine the process with '*' on the state status.

$ svcs -a grep sendmail
*online Mar_09 svc:/network/smtp:sendmail

#==find the process id of the offending process
$svcs -p sendmail
STATE STIME FMRI
*online Mar_09 svc:/network/smtp:sendmail
Mar_09 309 sendmail
Mar_09 310 sendmail

#==Then kill with kill cmd



#==If it happens quite offen, You may find the following script handy.

#!/bin/ksh
SVCNAMES=$@
CNT=0
DELAY=5
#
getpid () {
           
        SVC=$1
        /usr/bin/svcs $SVC >/dev/null
        if [ $? -ne 0 ];then
                PID=0       
                return 1
        fi
        PID=`/usr/bin/svcs -Hp $SVC|tail +2 | awk '{print $2}'| tail -1`
        if [ -z "$PID" ];then
                PID=1            
        fi                       
        return 0    
}

[ -z $SVCNAMES ] && echo "Usage $0 svcname1 [svcname2] .."

for SVCNAME in $SVCNAMES
do

getpid  $SVCNAME 

if [ $PID -lt 1 ];then
        echo "No pid found for $SVCNAME"
                exit 1
fi

while [ $PID -gt 1 ]
do
        echo "Delay for " $DELAY " secs"
        sleep $DELAY;
        getpid $SVCNAME
        CNT=`expr $CNT + 1 `    
        if [ $CNT -le 7 ] && [ $PID -gt 1 ];then
         echo "Service $SVCNAME is still running after " `expr $CNT \* $DELAY ` "secs, Gracefully  kill it: kill $PID"
           kill $PID
        elif [ $CNT -gt 7 ] && [ $PID -gt 1 ];then
         echo "Service $SVCNAME is still running after " `expr $CNT \* $DELAY ` "secs, Forcefully  kill it: kill -9 $PID"
        kill -9 $PID

        fi  
done
sleep 2;
/usr/bin/svcs $SVCNAME | grep disabled  

if [ $?  -eq 0   ]; then
        echo "Service is stopped"
else
        echo "Service is till running, please kill it mannually"
        exit 1
fi

done