Fix out of order network interfaces in Linux

I removed two old NICs and assigned two news NICs for a Vmware VM (SLES 10) and I expect the interface name to be eth0, eth1, but they appear as eth2, eth3. 
dmesg output revealed that eth0 was renamed to eth2 and eth1 was renamed to eth3 at some stage, It turned out udev rules renamed it.

Why?

30-net_persistent_names.rules had four entries, the first 2 recorded MAC address of two old NICs ; the last 2 entries recorded MAC address of current two NICs. Upon matching current MAC address, the udev rule renamed the interfaces to eth2 and eth3.

$cat /etc/udev/rules.d/30-net_persistent_names.rules
# This rules are autogenerated from /lib/udev/rename_netiface.
# But you can modify them, but make sure that you don't use an interface name
# twice. Also add such interface name rules only in this rules file. Otherwise
# rename_netiface will create wrong rules for new interfaces.
# It is safe to delete a rule, as long as you did not disable automatic rule
# generation. Only if all interfaces get a rule the renaming will work
# flawlessly. See also /etc/udev/rules.d/31-net_create_names.rules.
# 
# Read /usr/share/doc/packages/sysconfig/README.Persistent_Interface_Names for
# further information.
#
# Use only a-z, A-Z and 0-9 for interface names!
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:6d:df", IMPORT="/lib/udev/rename_netiface %k eth0"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:0b:2c", IMPORT="/lib/udev/rename_netiface %k eth1"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:1a:26", IMPORT="/lib/udev/rename_netiface %k eth2"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:50:56:b7:14:a6", IMPORT="/lib/udev/rename_netiface %k eth3"

How to fix it?

The fix is easy you can delete all four entries then reboot, the file will be populated with correct entries automatically. if you don't want to reboot, edit the file with correct entries then run “/lib/udev/rename_netiface oldname newname” manually.

Further discussion.

udev rule makes device naming very easy. you can ensure interfaces are named according to PCI order, for example, you want to name onboard NIC as eth0 and name PCI NIC as eth1. ( some times the order is reversed).
 Check if the NIC name follows PCI order.

$ ls -l /sys/class/net/eth*/device
lrwxrwxrwx 1 root root 0 2010-08-26 09:46 /sys/class/net/eth0/device -> ../../../devices/pci0000:00/0000:00:11.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 2010-08-26 09:46 /sys/class/net/eth1/device -> ../../../devices/pci0000:00/0000:00:11.0/0000:02:01.0

Resolving Perl CGI buffering issue

It is typical issue in Perl CGI that output doesn’t appear immediately until the whole script ends.  It is not desired for lengthy operations, because users want to see helpful message in advance rather than a blank page.
It is because that buffering of STDOUT is enabled ($|=0) by default, The following are two solutions to resolve the issue.

Requirements:

You need Apache2 Perl mod_perl2 and Perl module: CGI.pm.
Following is the environment where my scripts were tested ( However the versions  are NOT  minimum requirements)

Apache/2.2.11 (Ubuntu)
mod_perl/2.0.4
Perl/v5.10.0
libcgi-pm-perl 3.42-1 

Apache Server configuration

The handler has to be perl-script, handler cgi-script won’t work.

<Directory /var/www/scripts>
SetHandler perl-script
PerlResponseHandler ModPerl::Registry
PerlOptions +ParseHeaders
Options +ExecCGI
</Directory> 

Solution #1

Enable buffering and use mod_perl API ($r is mod_perl request object) to print.

#!/usr/bin/perl -w
use CGI qw(:standard);
$|=1;
my $r = shift;
print header('text/html');
print start_html;
$r->print ("Processing ... <br>");
foreach $i (1..2) {
sleep 2;
$r->print ("Item .. $i <br>");
}
$r->print ("DONE <br>");
print end_html; 

Solution #2

Disable buffering and use mod_perl API rflush function to flush buffer as needed.
“print br” is CGI.pm function to print a new line.
“$r->rflush” is mod_perl function to flush any buffered data to the client

#!/usr/bin/perl -w
use CGI qw(:standard);
my $r = shift;
print header('text/html');
print start_html;
print br ("Processing ... ");
$r->rflush; 
foreach $i (1..2) {
sleep 2;
print br  ("Item .. $i ");
$r->rflush;
}
print br ("DONE");
print end_html;

Tune max open files parameter in Linux

A busy websever serving thousands of connections may encounter error like “java.net.SocketException: Too many open files”, That is because the default value open files per process in Linux is 1024 and each connection consume 1 file handle.

An open file may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.) 
Linux has global setting and per process setting to control the max number of file descriptor
Global Setting:
 Max number of file handles for whole system. This value varies with memory size.
 

$sysctl fs.file-max
fs.file-max = 65535

Per process setting:
This value is per process (values in multiple child process don’t count towards parent process ).
The default value is 1024, which doesn’t seem to vary with memory size.

$ulimit -a | grep "open files"
open files                    (-n) 1024 

The value can be changed with “ulimit –n” command, but it is only effective on current shell session.  To impose limit on each new shell session, enable PAM module pam_limits.so.

There are many ways to start new shell session: login, sudo, and su.
each need to be enabled with pam_limits  by PAM config file  /etc/pam.d/login,  /etc/pam.d/sudo, or  /etc/pam.d/su

/etc/security/limits.conf is the configuration file for pam_limits.so to set values.
e.g Increase max number of open files  from 1024 to 4096 for Apache web server, which is started with user apache
apache       -       nofile       4096
pam_limits.so is session PAM, so change become effective for new session, no reboot required.
Count the number of open files for a process.
 ls -l  /proc/PID/fd | wc –l
or use lsof to count open files  excluding memory-mapped file (mem)
sudo lsof –n -p PID | awk '$4 != "mem" {print}' | wc –l
lsof is slow, but it can count all processes belong to a user “lsof –n –u username”
Count the number of open files for whole system.
 The first column of fs.file-nr output is current number of open files

$sysctl fs.file-nr
fs.file-nr = 1530       0       65535

Test ulimit.
You will be disappointed to test open files directly in shell by command tail –f  etc, because the limit is imposed on  process, but each tail –f   will start new process.

The following Perl script can open 10 files in a single process.

#!/usr/bin/perl -w
foreach $i (1..10) {
$FH="FH${i}";
open ($FH,'>',"/tmp/Test${i}.log") || die "$!";
print $FH "$i\n";
} 

nfile has been set to 8 with: ulimit –n 8

$ ulimit -a | grep files
open files                      (-n) 8

Too many open files error appeared while halfway creating files

$ ./testnfiles.pl
Too many open files at ./ testnfiles.pl line 4