Change MTU for Solaris on e1000g interface

I have Linux and OpenSolaris installed on Virtualbox with Intel Pro/1000 network interface, Changing MTU on Linux worked fine but it failed on Solaris

$ifconfig e1000g1 mtu 9000
ifconfig: setifmtu: SIOCSLIFMTU: e1000g1: Invalid argument

It turns out that Solaris's driver doesn't have jumbo frame enabled by default, you have to enable it manually. Following enables jumbo frame on e1000g1 only.

$/kernel/drv/e1000g.conf 
MaxFrameSize=0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0;
        # 0 is for normal ethernet frames.
        # 1 is for upto 4k size frames.
        # 2 is for upto 8k size frames.
        # 3 is for upto 16k size frames.
        # These are maximum frame limits, not the actual ethernet frame
        # size. Your actual ethernet frame size would be determined by
        # protocol stack configuration (please refer to ndd command man pages)
        # For Jumbo Frame Support (9k ethernet packet)
        # use 3 (upto 16k size frames)

Now set MTU to 9000 instead of 16K by editing file hostname.e1000g1. hostname.e1000g1 is interface configuration file, the first entry is your ip address or a name, the name must be resolvable in /etc/hosts.

$/etc/hostname.e1000g1
ip/name mtu 9000

#shutdown -i6 to reboot to take effect.

dladm is supposed to be the new method, But it didn't work.

#dladm show-linkprop -p mtu e1000g1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
e1000g1      mtu             rw   16298          1500           -- 
#dladm set-linkprop -p mtu=9000  e1000g1
dladm: warning: cannot set link property 'mtu' on 'e1000g1': try again later

Troubleshooting a high system CPU usage issue on Linux/Solaris

A Linux server has high %system CPU usage, following are steps to find the root cause of the issue and how to resolve it.

Vmstat show %system CPU usage is high.
# vmstat 2

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

1  0      0 191420   8688  35780    0     0     0     0 1006   31  1  4 96  0  0
1  0      0 124468   9208  98020    0    0 15626  2074 1195  188  0 76  0 24  0
0  1      0 110716   9316 110996    0    0  3268  4144 1366   84  0 94  0  7  0
0  3      0  97048   9416 122272    0    0  2818 11855 1314  109  1 80  0 20  0
0  4      0  80476   9544 137888    0    0  3908  2786 1272  172  0 54  0 46  0
2  1      0  72860   9612 145848    0    0  1930     0 1193  141  0 42  0 58  0
0  1      0  74300   9620 145860    0    0     0     6 1208   67  0 38  0 62  0
0  0      0  75680   9620 145860    0    0     0  6929 1364  101  0 70  6 24  0

Let’s run mpstat to show more detailed CPU usage,it showed CPU was busy with interruptions.

# mpstat 2
Linux 2.6.18-92.el5 (centos-ks)         01/14/2010

02:03:50 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
02:04:04 AM  all    1.33    0.00   41.78    0.00    0.44    3.56    0.00   52.89   1015.56
02:04:06 AM  all    0.00    0.00    8.04   38.69   29.65   23.62    0.00    0.00   1326.63
02:04:08 AM  all    0.00    0.00    8.70   30.43   27.54   28.50    0.00    4.83   1327.54
02:04:10 AM  all    0.00    0.00    5.47   46.77   27.36   20.40    0.00    0.00   1280.10
02:04:12 AM  all    0.50    0.00    6.47   63.18   19.40   10.45    0.00    0.00   1183.08
02:04:14 AM  all    1.01    0.00    6.53   62.31   21.11    9.05    0.00    0.00   1190.95
02:04:16 AM  all    0.00    0.00    8.04   26.63   43.72   21.61    0.00    0.00   1365.83
02:04:18 AM  all    0.00    0.00    1.50    0.00    0.00    0.50    0.00   98.00   1006.50

Use sar to find out which interrupt number was culprit. #9 was the highest excluding system interrupt #0.
# sar -I XALL 2 10

02:07:10 AM      INTR    intr/s
02:07:12 AM         0    992.57
02:07:12 AM         1      0.00
02:07:12 AM         2      0.00
02:07:12 AM         3      0.00
02:07:12 AM         4      0.00
02:07:12 AM         5      0.00
02:07:12 AM         6      0.00
02:07:12 AM         7      0.00
02:07:12 AM         8      0.00
02:07:12 AM         9    350.50

[ Solaris equivalent command]
Solaris# intrstat 2 

device |      cpu0 %tim      cpu1 %tim 
-------------+------------------------------ 
bge#0 |         0  0.0       128  0.6 
cpqary3#0 |         0  0.0        14  0.0

# cat /proc/interrupts

CPU0
0:     702980          XT-PIC  timer
1:        439          XT-PIC  i8042
2:          0          XT-PIC  cascade
6:          2          XT-PIC  floppy
8:          1          XT-PIC  rtc
  9:      14464          XT-PIC  acpi, eth2
11:         12          XT-PIC  eth0
12:        400          XT-PIC  i8042
14:       6091          XT-PIC  ide0
15:         22          XT-PIC  ide1
NMI:          0
LOC:     700623
ERR:          0
MIS:          0

[ OpenSolaris equivalent command ]

Solaris#echo ::interrupts | mdb –k
Native Solaris has to search the interrupt from output of prtconf -v

Solution:
When the card transmits or receives a frame, the system must be notified of the event. If the card interrupts the system for each transmitted and received frame, the result is a high degree of processor overhead. To prevent that, Gigabit Ethernet provides a feature called Interrupt Coalescence. Effective use of this feature can reduce system overhead and improve performance.

Interrupt Coalescence essentially means that the card interrupts the system after sending or receiving batch of frames. 

you can enable adaptive moderation ( Adaptive RX: off  TX: off) to let system choose value automatically or set individual values manually.

A interrupt is generated by the card to the host when either frame counter or timer counter is met. Values 0 means disabled.

RX for example:
Timer counter in microseconds: rx-usecs/rx-usecs-irq
Frames counter:rx-frames/rx-frames-irq

# A sample output with default values.
# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: off  TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 6
rx-usecs-irq: 18
rx-frames-irq: 6

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 80
tx-frames-irq: 20

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
<>

[ Solaris equivalent command]

Varying on driver, Find out the driver's capability. 
Solaris#ndd -get /dev/e1000g0 \? | egrep ‘interrupt |intr’
The Value shoud be set in driver conf file:
Solaris#/platform/`uname -m`/kernel/drv/*.conf

Alternative Workaround:

I couldn't config Interrupt Coalescence because virtual machine NIC didn't support it. but as workaround, Increasing mtu can also decrease interrupt, ifconfig eth2 mtu 9000 resolved the issue. It needs to set on both hosts peer, if they are not directly connected, make sure the switch supports jumbo frames. 
You don't need to care Interrupt Coalescence if your CPU resource is abundant, But for high load NFS/CIFS/ISCSI/ NAS servers, it is very useful.

vSphere PowerCLI Quick Start

vSphere Command-Line Interface vCLI or vSphere PowerCLI can be used to manage ESX/ESXi host.  vCLI is supported on  both Windows and Linux Client; PowerCLI is supported Windows Client only,  but it is more powerful than vCLI.

Windows PowerShell basics

 
###Windows Powershell supports wildcard
* ? []
###Window Powershell help
help
help get*
help get-vm
help get-vm -full
###list alias
get-alias
e.g
sort sort-object
ft format-table
fl format-list
###Variable, store result into variable
$var=get-process
#$var becomes array, $var[0] is first process, so it can be used in foreach loop for more complex operations
foreach ($proc in $var) {  $proc.ProcessName}
#The output  can be achieved by format-table
get-process | ft name
###sort, ascending is  default order
get-process| sort cpu -descending
###filter, find notepad process
get-process| where-object { $_.name -eq "notepad" }

vSphere PowerCLI basics

###Install in following order
- install Windows  PowerShell on Window XP/Windows 2003/Windows 2008.
- install vSphere PowerCLI
### First time use
#you will receive certificate warning,  type A to accept it for always run.
#after this you will receive the other  warning about signing,  type this command:
Set-ExecutionPolicy RemoteSigned.
#restart PowerCLI to check if warning disappears.
### Login and execute command
#login
Connect-VIServer -server ServerName
##Some useful commands
#list all vms and sort them by Memory in descending order
#The column real name is MemoryMB, but it is displayed as "Memory (MB)"
# So you need to use fl command to find out the realname ; get-vm vmname | fl
get-vm | sort MemoryMB -descending
#Restart all VMs which are currently Poweredon
#Don't use Restart-VM  because it  is like poweron and poweroff, not #graceful restart
get-vm | where {$_.powerstate -eq "poweredon"} | reset-vmguest
#Read hosts from file
# if you want to exclude some hosts from previous example, you certainly # can add more filter expression but i just want to show how to read file,
#save output to a file
get-vm | where {$_.powerstate -eq "poweredon"}  >d:\temp\host.list
#Edit the file and remove unwanted hosts
#Read the file and restart all hosts in the file, trimend is to remove trailing space
get-content d:\temp\host.list | foreach { $x=$_.trimend() ; reset-vmguest  $x }

Links:

PowerShell quick reference
VBScript-to-Windows PowerShell Conversion Guide
VMware: vSphere PowerCLI Blog

NFS/Samba alternative: Mount remote directory over SSH.

It is possible to mount a remote directory as a local file system through SSH by using SSHFS, which is based on FUSE: a library to create filesystems in userspace. There are many other files system based on FUSE .

Advantage:

Secure and easy to setup, only client side needs install SSHFS and FUSE, the server side just needs SSH server with sftp support.

Disadvantage:

Performance Penalty, because data transferred has to be encrypted and decrypted, which is time consuming and CPU intensive.

Install:

Search and install sshfs using package management tool of your Linux flavour.

Example:

- Mount any remote dir:

sshfs user@server:dir /mnt/sshfs

Only root and users in fuse group can use this command.

- Unmount sshfs

fuserumount -u /mnt/sshfs

or

umount /mnt/sshfs (root user only)

How to exclude directories with find

To exclude directories with find,grep is the obvious choice  but it is not efficient.Following shows three ways to exclude directories with find

#Sample directories and file, the aim is to exclude directory /tmp/test/test1

$find /tmp/test
/tmp/test
/tmp/test/test1
/tmp/test/test1/test1.file
/tmp/test/test2
/tmp/test/test2/test2.file
/tmp/test/test3

(1) Simple most efficient way, but I can only get it working in ksh

$find /tmp/test/!(test1) -type f
/tmp/test/test2/test2.file
/tmp/test/test3

(2)With not expression “! “

$find /tmp/test ! -path "/tmp/test/test1*" -type f
/tmp/test/test2/test2.file
/tmp/test/test3

(3)With prune, which means exclude preceding path

$find /tmp/test  -path  /tmp/test/test1 -prune  -o -type f -print
/tmp/test/test2/test2.file
/tmp/test/test3

Zenoss monitor Windows Server 2008 via WMI

Zenoss supports Windows SNMP, it can get partition and interfaces infomation, but it couldn't get CPU/MEMORY info. WMI script can get almost any info in Windows. Zenoss supports Windows WMI by zenpack
The agent account in remote Windows Server doens't need to be admin user as long as following previleges granted.

Enable DCOM
The easy way is to add the user to group "Distributed COM users"

Alternatively, grant specific rights to the user
Start DCOM GUI by DCOMCNFG command-> Component Services -> Computers->Right-click My Computer, and then select Properties->COM Security tab
Give access permission and launch and activation permission.

Enabling Account Privileges in WMI
Computer Management -> Services and Applications-> WMI Control->right click select Properties->Security
Select CIMV2 under root

Select security button add new user with
Enable Account
Remote Enable

Allowing WMI through the Windows Firewall
Allow pre-defined rule: Windows Management Instrumentation (WMI)

Deny ssh interactive login but allow sftp

SSH interactive login need tty to be allocated but sftp/scp doesn't need tty. So you can disable SSH interactive login by no-pty option in OPENSSH. But no-pty option is valid only in public key authentication, so you have to disable password for the user with “passwd –l username” command.

I have attempted to use pam_listfile.so tty option to achieve this, I found it is impossible because pam_tty name ssh will be allocated in either ssh login or sftp.

All you need to is to put no-pty parameter in ~/.ssh/authorized_keys, it must be in the same line with the public key, multiple options are separated by comma e.g

no-pty,no-X11-forwarding  ssh-dss AAAAB3Nz ... key-comment

Another useful feature of public key authentication is forced command, which means the command is invoked whenever the key is authenticated, it is great security feature for remote execution e.g backup job. you can also limit client source with "from= " option.

#Force to run command date only
$ cat /home/test/.ssh/authorized_keys
command="date" ssh-dss AAAAB3NzaC1kc3 ..

#date command was executed even given  command is ls
$ ssh   test@localhost ls
Wed Oct 14 10:29:39 EST 2009

#forced command can literally disable SSH interactive login.
$ ssh test@localhost
Wed Oct 14 10:29:44 EST 2009
Connection to localhost closed.