Fix out of order network interfaces in Linux

I removed two old NICs and assigned two news NICs for a Vmware VM (SLES 10) and I expect the interface name to be eth0, eth1, but they appear as eth2, eth3. 
dmesg output revealed that eth0 was renamed to eth2 and eth1 was renamed to eth3 at some stage, It turned out udev rules renamed it.

Why?

30-net_persistent_names.rules had four entries, the first 2 recorded MAC address of two old NICs ; the last 2 entries recorded MAC address of current two NICs. Upon matching current MAC address, the udev rule renamed the interfaces to eth2 and eth3.

$cat /etc/udev/rules.d/30-net_persistent_names.rules
# This rules are autogenerated from /lib/udev/rename_netiface.
# But you can modify them, but make sure that you don't use an interface name
# twice. Also add such interface name rules only in this rules file. Otherwise
# rename_netiface will create wrong rules for new interfaces.
# It is safe to delete a rule, as long as you did not disable automatic rule
# generation. Only if all interfaces get a rule the renaming will work
# flawlessly. See also /etc/udev/rules.d/31-net_create_names.rules.
# 
# Read /usr/share/doc/packages/sysconfig/README.Persistent_Interface_Names for
# further information.
#
# Use only a-z, A-Z and 0-9 for interface names!
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:6d:df", IMPORT="/lib/udev/rename_netiface %k eth0"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:0b:2c", IMPORT="/lib/udev/rename_netiface %k eth1"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:1a:26", IMPORT="/lib/udev/rename_netiface %k eth2"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:14:a6", IMPORT="/lib/udev/rename_netiface %k eth3"

How to fix it?

The fix is easy you can delete all four entries then reboot, the file will be populated with correct entries automatically. if you don't want to reboot, edit the file with correct entries then run “/lib/udev/rename_netiface oldname newname” manually.

Further discussion.

udev rule makes device naming very easy. you can ensure interfaces are named according to PCI order, for example, you want to name onboard NIC as eth0 and name PCI NIC as eth1. ( some times the order is reversed).
 Check if the NIC name follows PCI order.

$ ls -l /sys/class/net/eth*/device
lrwxrwxrwx 1 root root 0 2010-08-26 09:46 /sys/class/net/eth0/device -> ../../../devices/pci0000:00/0000:00:11.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 2010-08-26 09:46 /sys/class/net/eth1/device -> ../../../devices/pci0000:00/0000:00:11.0/0000:02:01.0

Resolving Perl CGI buffering issue

It is typical issue in Perl CGI that output doesn’t appear immediately until the whole script ends.  It is not desired for lengthy operations, because users want to see helpful message in advance rather than a blank page.
It is because that buffering of STDOUT is enabled ($|=0) by default, The following are two solutions to resolve the issue.

Requirements:

You need Apache2 Perl mod_perl2 and Perl module: CGI.pm.
Following is the environment where my scripts were tested ( However the versions  are NOT  minimum requirements)

Apache/2.2.11 (Ubuntu)
mod_perl/2.0.4
Perl/v5.10.0
libcgi-pm-perl 3.42-1 

Apache Server configuration

The handler has to be perl-script, handler cgi-script won’t work.

<Directory /var/www/scripts>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
</Directory> 

Solution #1

Enable buffering and use mod_perl API ($r is mod_perl request object) to print.

#!/usr/bin/perl -w
use CGI qw(:standard);
$|=1;
my $r = shift;
print header('text/html');
print start_html;
$r->print ("Processing ... <br>");
foreach $i (1..2) {
sleep 2;
$r->print ("Item .. $i <br>");
}
$r->print ("DONE <br>");
print end_html; 

Solution #2

Disable buffering and use mod_perl API rflush function to flush buffer as needed.
“print br” is CGI.pm function to print a new line.
“$r->rflush” is mod_perl function to flush any buffered data to the client

#!/usr/bin/perl -w
use CGI qw(:standard);
my $r = shift;
print header('text/html');
print start_html;
print br ("Processing ... ");
$r->rflush; 
foreach $i (1..2) {
sleep 2;
print br  ("Item .. $i ");
$r->rflush;
}
print br ("DONE");
print end_html;

Tune max open files parameter in Linux

A busy websever serving thousands of connections may encounter error like “java.net.SocketException: Too many open files”, That is because the default value open files per process in Linux is 1024 and each connection consume 1 file handle.

An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.) 
Linux has global setting and per process setting to control the max number of file descriptor
Global Setting:
 Max number of file handles for whole system. This value varies with memory size.
 

$sysctl fs.file-max
fs.file-max = 65535

Per process setting:
This value is per process (values in multiple child process don’t count towards parent process ).
The default value is 1024, which doesn’t seem to vary with memory size.

$ulimit -a | grep "open files"
open files                    (-n) 1024 

The value can be changed with “ulimit –n” command, but it is only effective on current shell session.  To impose limit on each new shell session, enable PAM module pam_limits.so.

There are many ways to start new shell session: login, sudo, and su.
each need to be enabled with pam_limits  by PAM config file  /etc/pam.d/login,  /etc/pam.d/sudo, or  /etc/pam.d/su

/etc/security/limits.conf is the configuration file for pam_limits.so to set values.
e.g Increase max number of open files  from 1024 to 4096 for Apache web server, which is started with user apache
apache       -       nofile       4096
pam_limits.so is session PAM, so change become effective for new session, no reboot required.
Count the number of open files for a process.
 ls -l  /proc/PID/fd | wc –l
or use lsof to count open files  excluding memory-mapped file (mem)
sudo lsof –n -p PID | awk '$4 != "mem" {print}' | wc –l
lsof is slow, but it can count all processes belong to a user “lsof –n –u username”
Count the number of open files for whole system.
 The first column of fs.file-nr output is current number of open files

$sysctl fs.file-nr
fs.file-nr = 1530       0       65535

Test ulimit.
You will be disappointed to test open files directly in shell by command tail –f  etc, because the limit is imposed on  process, but each tail –f   will start new process.

The following Perl script can open 10 files in a single process.

#!/usr/bin/perl -w
foreach $i (1..10) {
$FH="FH${i}";
open ($FH,'>',"/tmp/Test${i}.log") || die "$!";
print $FH "$i\n";
} 

nfile has been set to 8 with: ulimit –n 8

$ ulimit -a | grep files
open files                      (-n) 8

Too many open files error appeared while halfway creating files

$ ./testnfiles.pl
Too many open files at ./ testnfiles.pl line 4

Dissect Linux memory cache

Command “free” has buffers and cached columns, What is the difference? How to dig further to find the size of Dentry cache, inode cache and page cache in cached column.

Difference between Buffers and cached

$free -m
total       used       free     shared    buffers     cached
Mem:          3777       3746         31          0        160        954
-/+ buffers/cache:       2631       1145
Swap:          753          1        751

Buffer Pages
Whenever the kernel must individually address a block, it refers to the buffer page that holds the block buffer and checks the corresponding buffer head.
Here are two common cases in which the kernel creates buffer pages:
· When reading or writing pages of a file that are not stored in contiguous disk blocks. This happens either because the filesystem has allocated noncontiguous blocks to the file, or because the file contains "holes"
· When accessing a single disk block (for instance, when reading a superblock or an inode block).


Raw disk operation such dd use buffers.
Read 10M of raw disk block

$dd if=/dev/sda6 of=/dev/zero bs=1024k count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.209051 seconds, 50.2 MB/s

Buffers sized increased by 10MB( 170M-160M)

$free -m
total used free shared buffers cached
Mem: 3777 3754 23 0 170 952
-/+ buffers/cache: 2631 1145
Swap: 753 1 751

Dentry cache, inode cache and page cache in cached column.
Dentry cache: Directory Entry Cache, pathname (filename) lookup cache.
Inode cache: Cache for inode, not actual data block.
Page cache: Cache for actual data block
[FROM: http://www.mjmwired.net/kernel/Documentation/filesystems/vfs.txt ]
The combined value of dentry and inode cache is not bigger than whole slab size

$grep -i slab /proc/meminfo 
Slab: 183896 kB

Examine in detail by checking /proc/slabinfo.

$ awk '/dentry|inode/ { print $1,$2,$3,$4}' /proc/slabinfo 
nfs_inode_cache 122787 123312 984
rpc_inode_cache 24 25 768
ext3_inode_cache 9767 9770 776
mqueue_inode_cache 1 4 896
isofs_inode_cache 0 0 624
minix_inode_cache 0 0 640
hugetlbfs_inode_cache 1 7 576
ext2_inode_cache 0 0 728
shmem_inode_cache 441 455 776
sock_inode_cache 231 235 704
proc_inode_cache 670 756 608
inode_cache 2415 2415 576
dentry_cache 99060 110162 200
#sum up them (bytes)
$awk '/dentry|inode/ { x=x+$3*$4} END {print x }' /proc/slabinfo 
153348952

view live stats with slabtop by sorting by cache size

$slabtop -s c
Active / Total Objects (% used) : 332028 / 361849 (91.8%)
Active / Total Slabs (% used) : 45630 / 45631 (100.0%)
Active / Total Caches (% used) : 102 / 139 (73.4%)
Active / Total Size (% used) : 171443.16K / 175338.12K (97.8%)
Minimum / Average / Maximum Object : 0.02K / 0.48K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
125948 125864 99% 0.96K 31487 4 125948K nfs_inode_cache
113962 105262 92% 0.20K 5998 19 23992K dentry_cache
14609 14460 98% 0.52K 2087 7 8348K radix_tree_node
9770 9765 99% 0.76K 1954 5 7816K ext3_inode_cache
59048 44057 74% 0.09K 1342 44 5368K buffer_head
2415 2410 99% 0.56K 345 7 1380K inode_cache

I haven’t found to way to get page_cache size directly, It needs bit of calculation page_cache=~ cached – inode – dentry
alternatively observe the value change  by releasing pagecache

#write dirty pages to disk
sync
#To free pagecache:
echo 1 > /proc/sys/vm/drop_caches
#To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches
#To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation and dirty objects are not freeable, it is highly recommended to run command “sync” first.
#Suppress page cache usage.
sysctl -w vm.swappiness=0

vm.swappiness  # value range: 0 - 100, lower value tends to shrink page cache to get free memory, higher value  tends to use  swap to get free memory

Debugging issues with strace in Linux.

strace runs the specified command until it exits.  It intercepts and records the system calls. -T option shows the time spent in system calls. It is particularly useful to troubleshoot slow response issues, because you can pinpoint the step taking the longest time. (-r option  Print  a  relative timestamp upon entry to each system call, which is actually  the time spent in last system call, It is easier to read than –T output because it is displayed in first column)

#The following Telnet command took 20 secs to response, Was it issue with DNS or web server?  (It is an obvious DNS issue, but I just want to demonstrate how can strace pinpoint the issue.)

$time strace -f -F -i -r -t -T -v -o /tmp/trace.log telnet  www.google.com 80
telnet: could not resolve www.google.com/80: Name or service not known
real    0m20.080s
user    0m0.010s
sys     0m0.050s

#List line number and time spent by sorting time

$awk '{ print "LINE#"NR, $1}'  /tmp/trace.log | sort -nk2 | tail -5
LINE#55 0.010000
LINE#140 5.000075
LINE#154 5.000075
LINE#136 5.000076
LINE#150 5.000076

 #Print out the lines in question. it is clear that DNS timed out on waiting response from DNS server 100.0.0.23, it tried four times(the remaining 3 timeout were not included here) each time took 5 secs.

$awk '{  if ( NR > 125  &&  NR <= 136 ) {print "LINE#"NR, $0 } }' /tmp/trace.log
LINE#126      0.000000 [b7e601d1] stat64("/etc/resolv.conf", {st_dev=makedev(117, 0), st_ino=50235, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=8, st_size=83, st_atime=2010/07/16-09:47:25, st_mtime=2010/07/16-09:45:02, st_ctime=2010/07/16-09:45:02}) = 0 <0.000000>
LINE#127      0.000000 [b7e2a0f1] gettimeofday({1279237645, 625155}, NULL) = 0 <0.000000>
LINE#128      0.000000 [b7e72402] socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4 <0.000000>
LINE#129      0.000000 [b7e71f0c] connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("100.0.2.3")}, 28) = 0 <0.000000>
LINE#130      0.000000 [b7e61e88] fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR) <0.000000>
LINE#131      0.000000 [b7e61e88] fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0 <0.000000>
LINE#132      0.000000 [b7e2a0f1] gettimeofday({1279237645, 625155}, NULL) = 0 <0.000000>
LINE#133      0.000000 [b7e67296] poll([{fd=4, events=POLLOUT}], 1, 0) = 1 ([{fd=4, revents=POLLOUT}]) <0.000000>
LINE#134      0.000000 [b7e7220c] send(4, "B\262\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1"..., 32, MSG_NOSIGNAL) = 32 <0.000000>
LINE#135      0.000000 [b7e67296] poll([{fd=4, events=POLLIN}], 1, 5000) = 0 (Timeout) <5.000076>
LINE#136      5.000076 [b7e2a0f1] gettimeofday({1279237650, 625231}, NULL) = 0 <0.000000>

Master Bash command line

You may already know the bash shortcut keys: CTL+A/CTL+E/CTL+R ... , but do you know how the shortcut keys are defined and any other useful shortcut keys?
The functions of short keys come from readline library, bash seems to be the only shell supports readline.
In addition to the shell capability, its editing mode has to be set to emacs instead of vi

$set -o | egrep '^vi | ^emacs'
emacs           on
vi              off

To check current key bindings:

$bind -p | grep  \[CM]
…
"\M-.": yank-last-arg
"\M-_": yank-last-arg
"\M-\C-y": yank-nth-arg
"\M-y": yank-pop

The text C-k is read as `Control-y', Control is Ctrl key.
The text M-k is read as `Meta-k', Meta key is ESC/ALT key
The following is not a complete list of short keys,check readline readme for a complete list)

Example command:

$ echo one two three
one two three

TASK #1 paste the last argument: the word “three”
m-. insert last argument of the previous command( special variable $_ also refer to last argument, it works for ksh/bash)


TASK #2 paste the 1st argument or the command.
m-c-y : to paste 1st argument “one”

TASK #3 paste the command or the nth argument
m-2-m-c-y : to paste the 2nd argument “two”;
m-0-m-c-y : to paste the command “echo” itself.

TASK #4 delete word “three” to change it to “echo one two”
Step #1) Retrieve last command:
Use either of thee options: Up arrow key ; c-p key ; c-r key to search, keep pressing c-r to find the next match
Step #2) Delete the word.
c-w Delete from the cursor to the previous whitespace(you don’t need to type backspace five times to delete the word)
m-d is the opposite of c-w is , so to delete “echo”; the key sequence will be c-p c-a m-d
Less used text keys to cut text: c-k and c-u
C-k Kill the text from the current cursor position to the end of the line
C-u Kill backward from the cursor to the beginning of the current line.
To delete text “echo one”: c-p m-b m-b c-u (m-b is to back one word)

TASK #5 swap “three” “two” change it to “echo one three two”
c-p m-b c-w c-e c-y the key-point is to cut (c-w) word “two” and paste(c-y) it to end of line

TASK #6 undo the changes
m-r Undo all changes made to this line
c-_ Incremental undo, separately remembered for each line

###Other shell tips
CDPATH: directory search path Instead of typing full path, firstly adding the parent dir to CDPATH ENV variable(save it to .profile for permanent change), then cd dir-name will go to the dir

$ export CDPATH=/var/log/
$cd audit
/var/log/audit

OLDPWD: - is equivalent to $OLDPWD; When it is used as the operand, this shall be equivalent to the command:
cd "$OLDPWD" && pwd ; (reference man ksh or man bash)

$cd /tmp
$cd /var/tmp
$cd -
/tmp

DIRS
“-“ only remember last dir, for dirs >2 , you can use “pushd .” to remember any number of dirs;
the dir names can be displayed with “dirs” command or go to the last dir with “popd” command

How to generate core dump for an application in Linux

In order to troubleshoot application issue, a process content in memory can be dumped to a file and then the file can be analysed with debug tool, such as gdb in Linux. 
Another way of doing this is to send QUIT signal to the PID (kill –3  PID), but the thread dump will be directed to stdout, which can be viewed with “cat /proc/PID/fd/1 |tee /tmp/dump.log”, messages will be constantly directed to  /proc/PID/fd/1 until the process is stopped. So it is useful for real time debugging.
The following java application example use gcore command in gdb. (gcore or kill –3 will not stop the process )
Linux default core file size is 0, which means core dump is disabled, It needs to be changed to unlimited

#ulimit -a | grep core
core file size          (blocks, -c) 0
#ulimit –c unlimited
#ulimit -a | grep core
core file size          (blocks, -c) unlimited

Firstly, find the virtual memory size of the process, the PID is 10008 in following example.
# ps aux | egrep 'VSZ| 10008'

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       10008  0.4  8.3 2231312 660100 ?      Sl   Jun03  43:57 /opt/sunjdk6/bin/java

The process VSZ=2.2GB, it will be the size of the core file. Go to a dump dir which has more than 2.2GB free space
$cd /var/tmp

Attach to running process PID.
$gdb --pid=10008

At the gdb prompt, enter gcore command
gdb>gcore

wait for few minutes for core file to be generated. type in quit to exit gdb, answer yes to detach the process.
gdb>quit

core file is generated
$ ls -lh /var/tmp/core.10008

-rw-r--r-- 1 root root 2.2G Jun 10 11:59 /var/tmp/core.10008

file command reveals the source program name

$ file /var/tmp/core.10008
/var/tmp/core.10008: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from 'java'