Collectl, an all-in-one tool for collecting Linux statistical data

Collectl,collect for Linux, is a single tool which integrates functions of various tools:sar,iostat,mpstat,top,slaptop,netstat,nfstat,ps .. .
- Supported: Linux
- Requirement: Perl
Collectl features:
- run in command line or run as daemon
- Various output formats: raw,gunplot,gexprt(ganglia),sexpr,lexpr,csv(--sep ,)
- Send data to other programs (ganglia) remotely via socket instead of writing to a file
- IPMI monitoring for fans and temperature sensors
- Support module (Perl scripts)  for customized checks
- Monitor process’s disk read/write, find the top processes keeping disk busy
The last one is the most impressive feature, I haven’t found other Linux tools can do it. (DTtrace can in Solaris)
collectl  examples

#help, all options 
$collect –x
#-s?, what to monitor:c – cpu  d – disk “collectl   --showsubsys”
#-c 5 : collect 5 samples and exit
#-oT:  T - preface output with time only ; “collectl   --showoptions”
$collectl   -sc -c5 -i2 --verbose -oT
waiting for 2 second sample...
# CPU SUMMARY (INTR, CTXSW & PROC /sec)
#Time      User  Nice   Sys  Wait   IRQ  Soft Steal  Idle  CPUs  Intr  Ctxsw  Proc  RunQ   Run   Avg1  Avg5 Avg15
12:39:34      0     0     0     0     0     1     0    97     1  1082     23     0    76     1   0.42  0.42  0.44
12:39:36      0     0     0     0     0     1     0    97     1  1088     24     0    76     1   0.42  0.42  0.44

The following demonstrates how collectl identify the process reading/writing most data to disk
#Hammer disk by writing 50mb data with dd

$dd if=/dev/urandom of=test bs=1k count=50000
#collectl identifies the “dd” process
#in top mode, sort by  “iokb   total I/O KB” ; “collectl –showtopopts”
$collectl -i2  --top iokb
TOP PROCESSES sorted by iokb (counters are /sec) 12:50:31
# PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct  AccuTime  RKB  WKB MajF MinF Command
6861  root     18  6784    0 R    3M  572K  0  0.91  0.00  45   0:00.91    0 3680    0   97 dd
1  root     15     0    0 S    2M  632K  0  0.00  0.00   0   0:28.21    0    0    0    0 init
2  root     RT     1    0 S     0     0  0  0.00  0.00   0   0:00.00    0    0    0    0 migration/0

Understanding Red Hat Linux recovery runlevels

If Linux system can boot but hang during starting a service, booting to “recovery runlevels” can skip the service and gain shell to troubleshoot.
If Linux system can’t boot at all,  booting from rescue CD (first installation media) and type “linux rescue” to gain shell to troubleshoot
Red Hat Linux boot order

The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/inittab->
/etc/rc.d/rc.sysinit->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S

Recovery runlevels
- runlevel  1
Execute up to /etc/rc.d/rc.sysinit and /etc/rc.d/rc1.d/
Runlevel 1 is identical to singleuser mode. It is switched to singleuser mode in last step, just a number of trivial scripts executed before that.
 $ls  /etc/rc.d/rc1.d/S*
 /etc/rc.d/rc1.d/S02lvm2-monitor  /etc/rc.d/rc1.d/S13cpuspeed  /etc/rc.d/rc1.d/S99singlesingleuser

- single
Execute up to /etc/rc.d/rc.sysinit

- Emergency
Does not execute /etc/rc.d/rc.sysinit.
 Because rc.sysinit is not executed, file system is mounted in read-only mode. You need run “mount –o rw,remount /” to remount it in read-write mode.
emergency runlevel is Red Hat term, it is identical to  “init=/bin/sh” in any Linux distribution
How to go to a  runlevel
In the grub menu, type “a” to append one of following options to boot line.
1  
single  
emergency   
init=/bin/sh
When Centos hung on starting up boot services, how to get to shell without rescue CD
RHCE Notes - Troubleshooting booting issue

Advanced RPM topics

Query
“queryformat”  option can query every piece information of a rpm package, the  information tags (macros ) are returned  by “rpm –querytags” command

#list top 2 rpm packages sorted by installation time
$rpm -qa  | xargs -I{} rpm -q --queryformat "{}        %{installtime}\n" {} | sort -rn -k2 | head -2
collectl-3.5.1-1        1317864013
git-1.7.4.1-1.el5        1316484590
#unfortunately the time returned is unixtime.  You can convert it to human readable format by  “date –d @timestring” e.g 
$date -d @1317864013
Thu Oct  6 12:20:13 EST 2011
#but there is a shortcut  “--last”
$ rpm -qa --last  | head -2
collectl-3.5.1-1                              Thu 06 Oct 2011 12:20:13 PM EST
git-1.7.4.1-1.el5                             Tue 20 Sep 2011 12:09:50 PM EST

"rpm -qa" supports regular expression itself, rather than pipe to grep e.g “rpm -qa | grep perl”
“rpm –qa perl\*” also works. There is no improvement on speed but typing become lesser.
requires and provides
#You can check the package dependency before install the package

$ rpm -qp --requires git-1.7.6-1.el5.rf.i386.rpm
..
libssl.so.6 
…
#To meet the dependency, you want to check who provides libssl.so.6 
$yum whatprovides libssl.so.6
openssl-0.9.8e-20.el5.i686 : The OpenSSL toolkit
Repo        : base
Matched from:
Other       : libssl.so.6
#if openssl has been installed, “rpm -q –whatprovides” can also provide the answer
$rpm -q --whatprovides libssl.so.6
openssl-0.9.8e-12.el5_4.6

rpm scriptlets

#query all nopre|nopost|nopreun|nopostun  scripts
$rpm -q --scripts xinetd
postinstall scriptlet (using /bin/sh):
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
fi
preuninstall scriptlet (using /bin/sh):
if [ $1 = 0 ]; then
/sbin/service xinetd stop > /dev/null 2>&1
/sbin/chkconfig --del xinetd
fi
postuninstall scriptlet (using /bin/sh):
if [ $1 -ge 1 ]; then
/sbin/service xinetd condrestart >/dev/null 2>&1
Fi
#query postinstall script only
$ rpm -q --queryformat "%{POSTIN}"  xinetd
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
#Don’t run the scripts during install/remove
rpm –i –noscripts|nopre|nopost|nopreun|nopostun   pkgname
rpm –e –noscripts|nopre|nopost|nopreun|nopostun   pkgname

Extract rpm contents without install

#use rpm2cpio to extract everything
$mkdir /tmp/epel
$ cd /tmp/epel
$ rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd
./etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
..
#use rpm2cpio to extract particular file
$rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd  ./usr/share/doc/epel-release-5
#another way is to use rpm install with alternative root
$ rpm --root /tmp/epel/ -ivh  --nodeps /root/epel-release-5-4.noarch.rpm

Recover corrupted rpm database
Build RPM from source file

Get information for inode or block of a file and vice versa

Debugfs can retrieve a file’s inode or block information and vice versa, it is provided by e2fsprogs package, which should be installed by default.
Find a file’s information for inode or block

#find a file’s inode id by ls, the sample file’s inode number is 12
$ls -li /boot/message
12 -rw-r--r-- 1 root root 80032 Mar 13  2009 /boot/message
#find a file’s inode or block info by debugfs
$echo "stat message" | debugfs /dev/sda1
debugfs 1.39 (29-May-2006)
debugfs:  Inode: 12   Type: regular    Mode:  0644   Flags: 0x0   Generation: 3354433043
User:     0   Group:     0   Size: 80032
File ACL: 4640    Directory ACL: 0
Links: 1   Blockcount: 162
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4e2371f6 -- Mon Jul 18 09:36:22 2011
atime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
mtime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
BLOCKS:
(0-11):7681-7692, (IND):7693, (12-78):7694-7760
TOTAL: 80

Find a file based on inode or block info
#find a file based on inode number by find, but it may take long time to search

$find /boot/ -inum 12
/boot/message
#find a file based on inode number by debugfs
$echo -e "ncheck 12" | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message
##find a file based on block number by debugfs
#use icheck to find inode number based on block number first
$echo -e "icheck 7693 " | debugfs /dev/sda1
debugfs:  Block Inode number
7693  12
#then use ncheck find the file by inode number
$echo -e "ncheck 12 " | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message

Recover corrupted RPM database

RPM database consists of a number Berkeley DB   files in  /var/lib/rpm, the exception  is  __db.* files, which like cache files are updated for every rpm operation and they can be safely deleted.

#tested in Centos 5.5 
$ ls /var/lib/rpm
Basenames     __db.001  __db.003  Filemd5s  Installtid  Packages      Provideversion  Requireversion  Sigmd5
Conflictname  __db.002  Dirnames  Group     Name         Providename   Pubkeys         Requirename  Sha1header      Triggername
$ file /var/lib/rpm/Packages
/var/lib/rpm/Packages: Berkeley DB (Hash, version 8, native byte-order)

If one of the DB files is partially corrupted and it is still readable by /usr/lib/rpm/rpmdb_dump, you can reload the DB file and rebuild db.
$cd /var/lib/rpm

$rm -f __db*      
$mv Packages Packages.orig
$/usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages
$/usr/lib/rpm/rpmdb_verify Packages
#if you got this error: db_verify: PANIC: fatal region error detected; run recovery
#make sure /var/lib/rpm/__db.* are cleaned
#It is unlikely to rebuilddb if rpmdb_verify fails
$rpm -v –rebuilddb

If one of the DB files is completely corrupted and it is not readable by rpmdb_dump, you have to restore from backup,
$cd /var/lib/rpm

$cp Packages Packages.bak
#simulate a damaged RPM DB file
$ >Packages
$ cp Packages.bak  Packages
# Simply restoring  from backup file won’t work
#file verification is successful 
$ /usr/lib/rpm/rpmdb_verify Packages
#but any rpm operation fails
$rpm -qa 
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Even “rpm –rebuilddb” fails
$rm -f __db.*
$rpm –rebuilddb
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Notice the error about signature: BAD? The Pubkeys have to be cleaned as well.
$ mv Pubkeys Pubkeys.bak
#all good after removing Pubkeys file, a new Pubkeys is generated automatically on “rpm –rebuilddb”
$ rm -f __db.*
$ rpm –rebuilddb
$ rpm -qa | head -2
man-pages-2.39-15.el5_4
bash-3.2-24.el5

Authenticate Linux users by Windows AD: LDAP+Kerberos or Winbind method

Authenticating Linux users by Windows AD has become popular in many organizations for the convenience of centralized account management. It makes sense, if you need existing Windows account to login to Linux servers, but it doesn’t make sense to install a new Windows AD for the sole purpose of authenticating Linux users, just because Windows has a nice GUI. Redhat Directory Server/Centos Directory Server/389 Directory Server all  are capable of doing such task beautifully by ldap+StartTLS or ldap+Kerberos.
At least, there are 4 ways to authenticate Linux users by Windows AD:
1.    Linux Clients use Linux LDAP server synchronized with Windows AD.
     Replicate a subtree of Windows AD accounts to Linux Ldap Server (Redhat Directory Server/Centos Directory Server/389 Directory Server), Linux users authenticate against the   Linux LDAP Server either by Ldap+StartTLS or Ldap+Kerberos
A more complex approach, it is for those who don’t like a Windows AD sitting in datacenter.
2.    Linux Client use LDAP or LDAP+SSL(ldaps) or LDAP+StartTLS
User information store: LDAP
Authentication: LDAP or LDAP+SSL(ldaps) or LDAP+StartTLS
Limitation: Client can login but can’t change its password
3.    Linux Client use ldap+Kerberos
User information store: LDAP
Authentication: Kerberos
The most popular approach, work out of the box, no need to install additional software. User can change  its password.
A vbscript to add unix user to windows AD.
http://honglus.blogspot.com/2009/04/add-unix-user-to-windows-ad-by-vbscript.html
4.    Linux Client use samba Winbind
User information store: Winbind local file or winbind ldap
Authentication: Winbind
Cons to solution #3 ldap+Kerberos approach:
- /etc/init.d/windbind dameon need to be running
- The Linux client  has to be joined to the domain, it is showed in computers container.
- The UID/GID is derived from Windows SID(ID = RID - BASE_RID + LOW RANGE ID, "man 8 idmap_rid"), you don’t seem to have a choice to set to a specific number.
Pros to solution #3 ldap+Kerberos approach:
- No need to insall Service for Unix in Windows
This post demonstrate method #3 ldap+kerberos and #4  Winbind.
Linux OS: Centos 5.5
Windows OS: Window 2003
FQDN: server1.ad.example.com
Domainname/Realm: ad.example.com

Linux Clients use  ldap+Kerberos

Setup Windows Server
- Install  “Windows Services for UNIX Version 3.5” for Windows 2003
   Windows 2003 R2 or Windows 2008 has built-in component “Server for NIS Service”
- Create a generic user account for Linux nss-ldap client to run ldap query with the credentials.
     e.g username=Linux-bind-user  password=Redhat123
- Create a test group, go to “Unix Attribute tab” to set GID etc
- Create a test user, go to “Unix Attribute tab”, to set uid/homedir/shell/gid etc
- Modify default domain security group policy to allow users change password immediately, by default it is 1 day, change it to 0 day.
Set up Linux client

#make sure the following pkgs are installed 
krb5-workstation
openldap-clients
nss_ldap
pam_krb5
#Edit configuration files: /etc/nsswitch.conf; /etc/ldap.conf; /etc/krb5.conf; /etc/pam.d/system-auth
#The easiest way is to run authconfig-tui.
#Before run the command, empty config files for easy reading
$cat /dev/null >/etc/ldap.conf ; cat /dev/null > /etc/krb5.conf
$authconfig-tui       #select use ldap for “User information” and “use md5 passwords, use shadow passwords, use Kerberos,local authorization is sufficient” for authentication.
$cat /etc/ldap.conf
uri ldap://server1.ad.example.com/
base cn=Users,dc=ad,dc=example,dc=com
ssl no
binddn cn=linux-bind-user,cn=users,dc=ad,dc=example,dc=com
bindpw Redhat123
##Map Unix account attributes to Winnows SFU 3.5 attributes
##NOTE: Windows 2003 R2/Windows 2008 use different name
##In either scheme, the attribute can be retrieved by ldapsearch command
#$ldapsearch –x –W –b ”cn=Users,dc=ad,dc=example,dc=com” –D “cn=linux-bind-user,cn=users,dc=ad,dc=example,dc=com” 
#....
#msSFU30Name: jsmith
#msSFU30UidNumber: 10001
#...
nss_map_objectclass posixAccount User
nss_map_objectclass shadowAccount User
nss_map_objectclass posixGroup Group
nss_map_attribute uid sAMAccountName
nss_map_attribute uidNumber msSFU30uidNumber
nss_map_attribute gidNumber msSFU30gidNumber
nss_map_attribute gecos         name
nss_map_attribute homeDirectory msSFU30HomeDirectory
nss_map_attribute loginShell msSFU30loginShell
pam_login_attribute sAMAccountName
pam_filter objectclass=User
nss_base_password cn=Users,dc=ad,dc=example,dc=com
nss_base_shadow cn=Users,dc=ad,dc=example,dc=com
nss_base_group cn=Users,dc=ad,dc=example,dc=com
pam_password ad
#Optional, "referrals yes" try to connect both ldap://server1.ad.example.com and ldap://ad.example.com
referrals       no
$cat /etc/krb5.conf
##[logging] section is not needed, I think, it is  for KDC daemon running locally only
[libdefaults]
default_realm = AD.EXAMPLE.COM                    #realm name MUST be in capitals 
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
AD.EXAMPLE.COM = {
kdc = server1.ad.example.com
admin_server = server1.ad.example.com
}
##domain to realm mapping is also optional for user login, it is for the host added to KDC as principal to provide service
[domain_realm]
.example.com = AD.EXAMPLE.COM
example.com = AD.EXAMPLE.COM

#nsswitch.conf system-auth don’t need to be further customization after “authconfig-tui” is run, just post them for reference
$cat /etc/nsswitch.conf
passwd:     files ldap
shadow:     files ldap
group:      files ldap
$grep krb5  /etc/pam.d/system-auth
auth        sufficient    pam_krb5.so use_first_pass
account     [default=bad success=ok user_unknown=ignore] pam_krb5.so
password    sufficient    pam_krb5.so use_authtok
session     optional      pam_krb5.so

Test LDAP+Kerberos

#Query LDAP user information or run getent password|shadow|group to list all 
$id jsmith 
uid=10001(jsmith) gid=10000(unix-test-group) groups=10000(unix-test-group)

Troubleshooting
Error:
pam_krb5[5330]: password change failed for jsmith@AD.EXAMPLE.COM: Password change rejected, Password change rejected
Solution:
test changing password in Windows GUI, Make sure the password meet password complexity policy and minimum change day is 0

Error:
 Authentication failure (Cannot find KDC for requested realm)
$ kinit
kinit(v5): Cannot find KDC for requested realm while getting initial credentials
Solution:
check the relam spelling is correct and all in UPPER case, e.g AD.EXAMPLE.COM

ERROR:
 Authentication failure (Clock skew too great)
Solution:
synchronize datetime with AD and enable ntpd
$ntpdate -s server1.ad.example.com

Enable DEBUG:
$cat /etc/ldap.conf
 debug 7
Linux Clients use  samba Winbind

#Windows required software
Windows Services for UNIX is not needed
#Linux required software
samba-client
samba-common
krb5-workstation
#Clear /etc/ldap.conf and /etc/krb5.conf in last test
$cat /dev/null >/etc/ldap.conf; $cat /dev/null >/etc/krb5.conf; 
$authconfig-tui
#select use winbind for “User information” and “use md5 passwords, use shadow passwords, use winbind,local authorization is sufficient” for authentication.
#supply following information and  type in "domain admin"'s username password to join domain
$cat /etc/samba/smb.conf
[global]
#--authconfig--start-line--
workgroup = AD
password server = server1.ad.example.com
realm = AD.EXAMPLE.COM
security = ads
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
template shell = /bin/bash
winbind offline logon = false
##above lines are generated by authconfig-tui
##following 3 parameters need mannual customization
winbind use default domain = true
winbind enum users = yes
winbind enum groups = yes
#--authconfig--end-line--
##Other files don't need further customization, just paste here for reference
$cat /etc/krb5.conf
[libdefaults]
default_realm = AD.EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
AD.EXAMPLE.COM = {
kdc = server1.ad.example.com
}
$grep winbind /etc/pam.d/system-auth
auth        sufficient    pam_winbind.so use_first_pass
account     [default=bad success=ok user_unknown=ignore] pam_winbind.so
password    sufficient    pam_winbind.so use_authtok
$grep winbind /etc/nsswitch.conf
passwd:     files winbind
shadow:     files winbind
group:      files winbind

Test winbind

$wbinfo -u
jsmith
$gettent passwd
jsmith:*:16777216:16777216:John smith:/home/AD/jsmith:/bin/bash
##Group membership is added by opening the group, then click add member button in Windows GUI
$id jsmith
uid=16777216(jsmith) gid=16777216(domain users) groups=16777216(domain users),16777225(unix-test-group)

Passed 3/5 RHCA: EX333 Security Network Services

I would probably rate EX333 as the most difficult exam among the 3 exams I have passed, Why  is it so difficult?

1. The exam objectives are related: for example, Kerberos depends on NIS, without a working NIS, you are doomed.

2. The exam is broken into morning section and afternoon section, You won’t pass the exam  if either section fails. but  you are still allowed to sit for the afternoon section despite  the result of morning section.

My blog post for EX333 study notes:

Authenticate BIND zone transfer with TSIG key

Setup Postfix SMTP password authentication with SASL