UNIX/LINUX TECH NOTES

Tune nscd(name service cache daemon)

2012-01-24T21:18:00.002+11:00

nscd provides caching for the passwd,group,and hosts tables, it can boot performance for situations, in which the tables need to be serviced remotely e.g. LDAP authentication and DNS.
However, sometimes, it cause trouble.In Red Hat Linux 5 , nscd always return the old entry until the TTL(default is 1hour) is reached, even restarting nscd won’t flush the cache.
There are two solutions:
1.Disable persistent caching

#Persistent caching is enabled by default
$cat /etc/nscd.conf
persistent              passwd          yes
persistent              group           yes
persistent              hosts           yes
positive-time-to-live   hosts           3600
#So the entries are saved to relative tables
$rpm -ql nscd
..
/var/db/nscd/group
/var/db/nscd/hosts
/var/db/nscd/passwd…
#change them to no
persistent              passwd          no
persistent              group           no
persistent              hosts           no

With persistent caching disabled, restart nscd will discard the entries in memory.
2.Flush entries by invalidating the table
The entries in tables (group/passwd/hosts) can manually flushed by the ‘invalidate ‘parameter.

$nscd --invalidate=hosts

Since it is natural for anyone to try restart nscd to resolve the issue and the operation of rebuilding cache is not expensive, I think option 1 is better.

Force puppet agent to regenerate certificate request

2012-01-10T22:53:00.003+11:00

If puppet agent’s certificate is accidentally revoked or deleted, you can force agent to regenerate certificate request.

In general, it is impossible un-revoke a certificate unless the revoke reason is certificateHold, But puppet can hack it. The solution is to recover all revoked certificates then revoke other certificates which don’t need to be recovered

$rm /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem
$rm /etc/puppetlabs/puppet/ssl/crl.pem
#At this point, all revoked certificates become valid certificates.
#So you need to revoke all certificates which don’t need to be recovered
$puppet cert --revoke foo

The following method of regenerating new certificate seems to be a better.

The following is tested in Puppet Enterprise 2, but it should work for puppet open source as well.
$ puppet --version
2.7.6 (Puppet Enterprise 2.0.0)
Force agent to regenerate certificate request by generate command

[puppet agent]$ puppet  certificate   generate    web1  --ca-location  remote
warning: peer certificate won't be verified in this SSL session
err: Error 400 on SERVER: web1 already has a revoked certificate; ignoring certificate request
err: Try 'puppet help certificate generate' for usage
#It because the revoked certificate still exist in the server, it need to be deleted
[puppet master]$ puppet cert list –all
- web1                                     (BA:18:D1:86:D6:5E:9E:99:55:39:3D:67:79:BF:BD:D0) (certificate revoked)
[puppet master]$ puppet cert clean web1
#re-run the command, the warning is expected because the request hasn’t been signed by master yet
[puppet agent]$puppet   certificate   generate    web1  --ca-location  remote
warning: peer certificate won't be verified in this SSL session
true
#The pending request appears in master 
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
[puppet master]# puppet cert sign web1

If the above doesn’t work for you, the last resort is to clean agent’s ssl files

[puppet agent]$ puppet --genconfig | grep certdir
certdir = /etc/puppetlabs/puppet/ssl/certs
$cd /etc/puppetlabs/puppet/ssl/
$find . –type f –exec rm {} \;
$service pe-puppet restart
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
puppet master]# puppet cert sign web1

Augeas quickstart

2012-01-10T22:40:00.004+11:00

Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.
Configuration files support is provided by “lenses”. The default lenses are provided in augeas-lib package

$ rpm -ql pe-augeas-libs | grep lenses
/opt/puppet/share/augeas/lenses/dist
/opt/puppet/share/augeas/lenses/dist/aliases.aug
…

If your configuration is not listed, you can write your own lenses definition file.
Language bindings are available:
Ruby, Python, OCaml, Perl, Haskell, PHP, Lua, and Java
Augeas has two main directories:
- augeas: augeas configuration directory. e.g control loading lenses
- files: user data directory tree. e.g /etc/hosts is mapped to /files/etc/hosts

$augtool
augtool> ls /
augeas/ = (none)
files/ = (none)

Sample hosts file to be tested

$cat /etc/hosts
# comment test1
# comment test2
172.16.1.11 web1 web1.example.com server1.example.com
#######Augeas translate above hosts file to 
augtool>  print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

Adding a new comment

augtool> set /files/etc/hosts/#comment[last()+1] "comment test3"
augtool> print  /files/etc/hosts/
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/#comment[3] = "comment test3"

########New comment will be added to last line.
########If you want to append it next to #comment[2], use insert
augtool> ins #comment after /files/etc/hosts/#comment[last()]
augtool> 
augtool> set  /files/etc/hosts/#comment[last()] "comment test3"
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

Adding a host entry.
You can’t use last() to auto increase it from hosts/1 to hosts/2, because “1” or “2” is label name not an index number like #comment[1]/#comment[2].
The trick is to use “01,02,03” etc.

augtool> set /files/etc/hosts/01/ipaddr 172.16.1.12
augtool>set /files/etc/hosts/01/canonical web2
##at this point, it is hosts/01 added
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "coment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/01
/files/etc/hosts/01/ipaddr = "172.16.1.12"
/files/etc/hosts/01/canonical = "web2"

####after save and load, hosts/2 is added
augtool> save
Saved 1 file(s)
augtool> load
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "coment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/2
/files/etc/hosts/2/ipaddr = "172.16.1.12"
/files/etc/hosts/2/canonical = "web2"

Change an entry by referencing to itself
Change the comment “comment test1” to “newcomment test1”

#Change the comment “comment test1” to “newcomment test1” 
augtool> /set files/etc/hosts/#comment[.='comment test1'] 'newcomment test1'
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "newcomment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3

#Change any comment starting with comment to ‘oldcomment’ by regex match
augtool>setm /files/etc/hosts  #comment[.=~regexp('comment.*')] 'oldcomment'
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "newcomment test1"
/files/etc/hosts/#comment[2] = "oldcomment"
/files/etc/hosts/#comment[3] = "oldcomment"

#Usually, regex match is used to delete entries. e.g delete comment test[23]
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3"
augtool> rm /files/etc/hosts/#comment[.=~regexp('comment test[23]')]
rm : /files/etc/hosts/#comment[.=~regexp('comment test[23]')] 2
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment = "comment test1"
augtool>

Change an entry by referencing to relative nodes
e.g change ip of a host ‘web1’ to 172.16.11.11

augtool> print /files/etc/hosts/1
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
augtool> set /files/etc/hosts/*/ipaddr[../canonical = 'web1'] 172.16.11.11
augtool> print /files/etc/hosts/1
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.11.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

#multiple conditions can be joined together 
augtool> match /files/etc/hosts/*/ipaddr[../canonical = 'web1' and ../alias = 'server1.example.com']
/files/etc/hosts/1/ipaddr = 172.16.11.11
augtool> match /files/etc/hosts/*[canonical = 'web1' and alias = 'server1.example.com']
/files/etc/hosts/1 = (none)

Comment or uncomment a line
Comment or uncomment is difficult for Augeas to implement, it can only be achieved by inserting a new line after the comment line then removing the old line.
You can extend Augeas to include ‘comment’ and ‘uncomment’ functions. The following code is an example for puppet.

##Define a function 
define augeasnew ($file,$line){
$exp=regsubst($line[0], '^(un)?comment *(.*)' , '\2')
case $line[0] {
/^uncomment/: {
exec {"/bin/sed -i -e '/${exp}/s/#//g' $file":
onlyif => "/bin/grep '${exp}' ${file} | /bin/grep '#' ",}
}
/^comment/: {
exec {"/bin/sed -i -e '/${exp}/ s/^/#/' $file":
onlyif => "/bin/grep '${exp}' ${file} | /bin/grep -v '#' ",}
}
default: {
augeas {'augeas-chg-any':
context => "/files/${file}",
changes => $line, }
}
}
}
##execute the function
augeasnew {'chg-hosts-file':
file => '/etc/hosts',
#Mulitple Augeas commands can be stored in an array
#However,none-Augeas command (uncomment or comment) can only be stored in first element of an array.
#line => ["set  1/ipaddr '10.1.1.1'" , "set 2/ipaddr '10.1.1.2'",],
line => ["comment  c line1",],
}

Troubleshooting.
You can check /augeas//error for detailed errors, for example, Augeas can’t load /etc/sudoers

augtool> print /augeas//error
/augeas/files/etc/sudoers/error = "parse_failed"
/augeas/files/etc/sudoers/error/pos = "1998"
/augeas/files/etc/sudoers/error/line = "62"
/augeas/files/etc/sudoers/error/char = "0"
/augeas/files/etc/sudoers/error/lens = "/opt/puppet/share/augeas/lenses/dist/sudoers.aug:478.10-.57:"
/augeas/files/etc/sudoers/error/message = "Iterated lens matched less than it should"
In this case, error is encountered in line 62. 
$sed -n '62p' /etc/sudoers
Defaults   !visiblepw
#You can upgrade Augeas or comment out the line to resolve the issue

Reference:
http://augeas.net/tour.html
http://augeas.net/page/Path_expressions
http://docs.puppetlabs.com/references/2.6.8/function.html
http://docs.puppetlabs.com/guides/language_guide.html

Vsphere PowerCLI script to clone and customize Windows guest OS

2012-01-05T23:52:00.009+11:00

VMware can also customize Windows guest OS by Windows sysprep tool, though the process is more complex than Linux guest OS

There are two options to clone and “sysprep” VMware Windows guest OS:
1.Install sysprep tools in Windows guest OS and  run sysprep.exe  in guest OS command line, then clone it by VMware
2.Install sysprep tools in Virutal Center and let VMware tools in Windows guest in to control sysprep process either by GUI or script.  (It seems sysprep rely on VMware tools, so the VMware tools must be installed in guest OS)

Option #2 is the preferred method, because you can use script to easily customize unique information e.g computer name, ip addresses etc.


Install sysprep tools in Virtual Center
VMware KB: Sysprep file locations and versions 
Windows Vista onwards(vista/2008/7/2008R2/) don’t need this step, because its sysprep is built-in.
Create guest customization by GUI 
 Virtual Center -> view-> management -> “customization specification manager”. Create a new customization and select it when asked for guest customization information in GUI clone action. 
Create guest customization by script. 
Customization by GUI is not flexible, customization by script can be created on the fly. The following is enhancement to the clonevm.ps1 in Vsphere PowerCLI script to clone and customize Linux guest OS, just replace the “Identity for Linux” part with following code block


$s_ostype=Retrieve-values $lines  "ostype"
if ( $s_ostype -eq "linux" ) { 
## Identity for Linux 
$vmclonespec_os.identity= New-Object VMware.Vim.CustomizationLinuxPrep
$vmclonespec_os.identity.hostname= New-Object VMware.Vim.CustomizationFixedName
$vmclonespec_os.identity.hostname.name= $s_dstname
$vmclonespec_os.identity.domain=$s_domain
}
elseif ( $s_ostype -eq "windows" ) {
# WinOptions
$vmclonespec_os.Options = New-Object  VMware.Vim.CustomizationWinOptions
$vmclonespec_os.Options.ChangeSID = 1
#sysprep
$vmclonespec_os.identity = New-Object VMware.Vim.CustomizationSysprep
# GUIUnattended
$vmclonespec_os.Identity.GuiUnattended = New-Object VMware.Vim.CustomizationGuiUnattended
$vmclonespec_os.Identity.GuiUnattended.AutoLogon = 0
#timezone codes: http://msdn.microsoft.com/en-us/library/ms145276(v=sql.90).aspx
$vmclonespec_os.Identity.GuiUnattended.TimeZone  = 255
$vmclonespec_os.Identity.GuiUnattended.Password = New-Object VMware.Vim.CustomizationPassword
$vmclonespec_os.Identity.GuiUnattended.Password.PlainText = 1
$vmclonespec_os.Identity.GuiUnattended.Password.Value = "Secret01"
# Identification
$vmclonespec_os.Identity.Identification = New-Object VMware.Vim.CustomizationIdentification
$vmclonespec_os.Identity.Identification.joinWorkgroup = "workgroup2"
## Userdata
$vmclonespec_os.identity.userData = New-Object VMware.Vim.CustomizationUserData
$vmclonespec_os.identity.userData.computerName = New-Object VMware.Vim.CustomizationFixedName
$vmclonespec_os.identity.userData.computerName.name = $s_dstname
$vmclonespec_os.Identity.UserData.FullName = "Administrator"
$vmclonespec_os.Identity.UserData.OrgName="myOrg"
$vmclonespec_os.Identity.UserData.Productid=""
}
else {
write-host "Unknown ostype: $s_ostype. Please set it to linux or windows"
exit
}
}

NOTES:
- The guest OS(Windows/Linux) will be forcefully rebooted by Vmware tools again in ~1 minute after you power it on, so don’t login in a hurry to check the result.
- The script only works in a live session to Virtualcenter, it doesn’t work in a direct login session to ESX host.

Bootstrap Puppet Enterprise (PE) Puppet Client for RHEL

2012-01-05T23:30:00.002+11:00

Puppet Enterprise(PE) packs all required packages, modules into a tar file.
The file also includes a shell installation script to install packages and modules and do other customizations tasks. So only installing agent packages “pe-puppet pe-mcollective” can’t setup agent station properly.
It is ideal to have PE agent installed automatically along with OS,the installer script can use answer file to automate the process.
The following codes can be put in RHEL kickstart postrun section to bootstrap PE agent.

#Download Puppet Enterprise(PE) (manage up to 10 nodes free)
#http://puppetlabs.com/puppet/puppet-enterprise/
#e.g  puppet-enterprise-2.0.0-el-5-x86_64.tar.gz
...
%post
PKG=puppet-enterprise-2.0.0-el-5-x86_64
cd /tmp && wget -O ${PKG}.tar.gz http://172.16.1.1/boot/os/rhel-5.6-x64/${PKG}.tar.gz  && tar -zxf ${PKG}.tar.gz  && cd /tmp/${PKG} && \
cat >answerfile<<END
q_install=y
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_install=n
q_puppet_symlinks_install=y
q_puppetagent_certname=$(hostname -s)
q_puppetagent_install=y
q_puppetagent_server=puppet
q_puppetmaster_install=n
q_vendor_packages_install=n
q_install=y
END
[ -e answerfile ] && ./puppet-enterprise-installer -a answerfile  2>&1 | tee  /tmp/install_${PKG}.log
rm -rf /tmp/${PKG}*
sync
exit 0

Connect to puppet server after installation finished

#Verify puppet agent is running
[Puppet Client]$/etc/init.d/pe-puppet status
#Puppet server should be able to see the client’s pending certificate request
[Puppet Master]$puppet cert list
web2 (A1:08:15:EF:1F:5D:F9:C5:D9:A0:F3:F2:FD:FF:CE:09)
#Approve the client by signing the cert request
[Puppet Master]$puppet cert sign web2

Build a Linux PXE server to provision Linux and Windows Servers

2011-12-07T20:58:00.012+11:00

Windows Deployment Service (WDS)/SCCM is the traditional way to provision Windows servers, However if you need to provision both Linux and Windows Servers, it would be nice to have a all-in-one server. Before Windows Preinstallation Environment (WinPE) was developed, disk imaging tool like ghost is used to deploy syspreped windows OS as non-windows based provision solution. WinPE is part of Windows Automated Installation Kit(WAIK), it includes, imagex, a tool to capture image on file system level rather than raw sector level.

The PXE Server presented here has the following features:

- Support any PXE capable Linux distribution and most Windows versions: 2000/2003/XP/Vista/7/2008/2008 R2. Potentially, support any other PXE capable server e.g Solaris
- Zero touch installation/full automation
- Dynamic loading configuration using CGI script of your choice PHP/Perl ...
- Downloading kernel/images over http which is more reliable and faster than tftp, additionally, http makes it possible to provision over remote site

Environment setup

7 VMS in Virtualbox 4( for testing provision Centos 5.5, Windows 2003, and Windows 2008 R2)
- PXE server: Centos 5.5
- PXE Linux Client: Centos 5.5
- WAIK tools computer: Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2003 R2
- Imagex/sysprep source: Windows 2008 R2
- Imagex/sysprep source: Windows 2003 R2
Notes:
- Select PCnet-fast III network adapter to support PXE boot
- Windows VM must have at least 1G RAM, because WinPE ISO memdisk will consume few hundreds of MB
- Setup samba share in PXEServer to store Windows imagex(.wim ) files,autoscript for disk partitioning/apply .wim image etc(autorun.bat).
Install TFTP/DHCP
Refer to http://honglus.blogspot.com/2011/06/setup-pxe-boot-server-for-linux-server.html
Just the filename parameter need to be set to: filename "gpxelinux.0";
Install pxeserver(syslinux)
The original syslinux package in Centos 5.5 doesn't support memdisk, I use syslinux-4.03
Upgrade to syslinux-4.03 with RPM binary package, but the gpxelinux.0 need to customized and compiled from source.
gPXE/pxelinux backgroud
gPXE(http://etherboot.org/wiki/index.php) or it is successor iPXE (http://www.ipxe.org/) support dynamic configuration using any CGI script and support kernel/images loading over http.
Because gPXE/iPXE doesn't support memdisk, which is essential for loading winPE ISO image, we have to use pxelinux in syslinux. Fortunately, syslinux integrates gPXE, so we can switch between gPXE and pxelinux freely using CGI script. That is, use gPXE for Linux, use pxelinux for Windows
Compile customized gPXE in syslinux source file

$cd ${syslinux source}/gpxe/ 
$vi pxelinux.gpxe
#!gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}
# "!gpxe" is interpreter, not a comment line 
# chain http://pxe-server/boot/home.php?mac=${net0/mac}?will pass the MAC address to a CGI script, which can generate individual configuration based on MAC address
#To use the DNS name, pxe-server, a working DNS server need to be advertised to DHCP client.
$make
$cp gpxelinux.0 /tftpboot/

How does the CGI script work?

A CGI script, php in this example, retrieve the MAC address passed by gPXE request, lookup host-os mapping table, then generate PXE configuration dynamically.

#The configuration file to define which OS to apply based on MAC address
$cat ${wwwroot}/boot/hosts/host-os.conf 
#Mac               os
08:00:27:4d:4d:aa, centos-5.5-i386
08:00:27:b2:40:3e, windows-2008r2-x64

#Linux Server can use gPXE directly
$curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
#!gpxe
imgfree
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img 
boot

#Windows server has to rely on pxelinux to support memdisk(part of syslinux package)
$curl http://pxe-server/boot/home.php?mac=08:00:27:b2:40:3e
#!gpxe
imgload pxelinux.0
boot pxelinux.0

#A pxelinux config is generated in the mean time 
$cat  /tftpboot/pxelinux.cfg/01-08-00-27-b2-40-3e 
default pe
label  pe
kernel http://pxe-server/boot/os/memdisk
append iso
initrd http://pxe-server/boot/os/w2k8_pe_x64.iso

Implementing Zero touch installation/full automation

Booting up the server via PXE is the first phrase of zero touch installation, OS level automation is done by Linux unattended install: e.g. Kickstart for Redhat Linux family or Windows unattended install:syspreped windows, startnet.cmd in winPE, autoscript in a network share
The following focus on Windows unattended install.
Download WAIK for Windows 7/Windows 2008 R2
http://www.microsoft.com/download/en/details.aspx?id=5753
How can you use WAIK for Windows 2008 R2 to deploy Windows2000/xp/2003
It is true that you can't run setup.exe of Windows 2003 in 2008 R2 winPE, but we use imagex tool to capture files, so it works on most Windows OS.
Create winPE ISO
Install the WAIK in WAIK tools computer
Steps to create winPE ISO

#Open the deployment tools command prompt
#amd64 is for 64bit, use x86 for 32bit
$\>copype.cmd amd64 C:\winpe_amd64
####Optional:Mount the wim file in order to add auto deploy script, you can skip this step to make it easier to get it going.
####BEGIN script customization
$\>imagex /mountrw c:\winpe_amd64\winpe.wim 1 c:\winpe_amd64\mount
#modify C:\winpe_amd64\mount\Windows\System32\starnet.cmd, add following lines below 'wpeinit'
REM -- Start
REM Mount Linux samba share
net use J: \\pxe-server\windows-media
REM go to sub-dir(dirname is the mac address) of the samba share and run the auto script
J:
for /F  "usebackq tokens=2 delims=:"  %%i IN (`ipconfig /all ^| find /i "physical address" `) DO cd %%i
autorun.bat
REM -- end
$\>imagex /unmount c:\winpe_amd64\mount /commit
#####END script customization
$\>copy "C:\Program Files\Windows AIK\Tools\amd64\imagex.exe"  C:\winpe_amd64\ISO\
$\>copy C:\winpe_amd64\winpe.wim C:\winpe_amd64\ISO\sources\boot.wim
$\>Oscdimg -n -bC:\winpe_amd64\Etfsboot.com C:\winpe_amd64\ISO C:\winpe_amd64\winpe_amd64.iso

How to do sysprep in Windows 2003?
On imagex/sysprep source computer:
Download the latest sysprep tool: http://www.microsoft.com/download/en/details.aspx?id=14830
WindowsServer2003-KB926028-v2-x86-ENU.exe /x to extract the deploy.cab file
Extract files in deploy.cab to c:\sysprep
Run setupmgr.exe to create answer file c:\sysprep\Sysprep.inf
Run sysprep /reseal to prepare the OS for imaging
How to do sysprep in Windows 2008 R2?
On WAIK tool computer:
Open Windows system image manager?in WAIK to create unattended answer file
Click New answer file?to select a windows catalogue file (.clg) in installation media \Deploy\Operating Systems\Windows Server 2008 R2 x64\sources\.
In the Windows images?panel, right click various parameters to add to answer file
Detailed instruction:
http://briandesmond.com/blog/how-to-sysprep-in-windows-server-2008-r2-and-windows-7/
On imagex/sysprep source computer:
Retrieve above new answer file *.xml, Run sysprep /generalize /oobe /shutdown /unattend:unattend.xml?
Capture image in imagex/sysprep source computer
This is manual process, because it is once-off job
Once the imagex/sysprep source computer is shutdown, Reboot it with winpe_amd64.iso by mounting to CDROM or loading via PXE
#The files source is the partition with windows folder, It could be d:, because W2k8 assign c: to the small system partition by default, which doesn't need to be captured.
$\>imagex /compress fast /capture [c:|d:] $sambashare:\boot.wim "w2k8 R2 64bit"
Copy the boot.wim to samba share, which will be used by imagex /apply.
Apply image in imagex/sysprep target computer
This need to be full automatic process, boot winpe_amd64.iso via PXE. Once winPE is initialized, it will mount the sambashare, cd to its own directory and run auorun.bat, which will partition disk and apply the image.

It is zero touch installation, no human intervention is needed, however, unlike kickstart, computer name/IP address etc can't be configured automatically. Well, in theory, you can specify the information in sysprep then re-capture image for each unique configuration, but it defeats the purpose of saving time in provision. The other way is fresh installation by giving answer file in samba share to setup.exe.
But it can only be used for Visa/2008/7/2008 R2, because there is no winPE which support running setup.exe of pre-Vista Windows

$ ls $sambashare\08-00-27-b2-40-3e
autorun.bat  boot.wim  diskpart.txt
$ cat $sambashare\08-00-27-b2-40-3e\autorun.bat 
diskpart /s diskpart.txt
PATH=%PATH%;c:;d:;e:;f:
imagex /apply boot.wim 1 c: && bcdboot c:\windows /s c: && exit
#bcdboot c:\windows is not needed for pre-Vista Windows, but it does no harm.

It is done, if you have setup answer file properly, the imagex/sysprep target computer should be able to boot smoothly without any human intervention.
Issues encountered:
Q: DNS name is not resolvable in gPXE.
A: Make sure set use-cached 0 is set in pxelinux.gpxe, then recompile
$vi pxelinux.gpxe
#!gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}

Q: gPXE couldn't boot Linux kernel with error:Kernel panic not syncing: VFS: unable to mount root fs
A:Make sure imgfree is the first statement in Linux pxe config
$curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
#!gpxe
imgfree
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img
boot

Q:Error in imagex /apply after booting winPE via PXE: Not Enough Server Storage is available to process is command
A: Make sure the Windows VM have at least 1G RAM, because winPE ISO will consume a few hundreds MB as memdisk

Q:What if the winPE doesn't recognize the device (RAID/NIC) with default drivers.
A:Add customize driver to winPE, refer to http://technet.microsoft.com/en-us/library/dd744371%28WS.10%29.aspx

Collectl, an all-in-one tool for collecting Linux statistical data

2011-10-13T15:09:00.006+11:00

Collectl,collect for Linux, is a single tool which integrates functions of various tools:sar,iostat,mpstat,top,slaptop,netstat,nfstat,ps .. .
- Supported: Linux
- Requirement: Perl
Collectl features:
- run in command line or run as daemon
- Various output formats: raw,gunplot,gexprt(ganglia),sexpr,lexpr,csv(--sep ,)
- Send data to other programs (ganglia) remotely via socket instead of writing to a file
- IPMI monitoring for fans and temperature sensors
- Support module (Perl scripts) for customized checks
- Monitor process’s disk read/write, find the top processes keeping disk busy
The last one is the most impressive feature, I haven’t found other Linux tools can do it. (DTtrace can in Solaris)
collectl examples

#help, all options 
$collect –x
#-s?, what to monitor:c – cpu  d – disk “collectl   --showsubsys”
#-c 5 : collect 5 samples and exit
#-oT:  T - preface output with time only ; “collectl   --showoptions”
$collectl   -sc -c5 -i2 --verbose -oT
waiting for 2 second sample...
# CPU SUMMARY (INTR, CTXSW & PROC /sec)
#Time      User  Nice   Sys  Wait   IRQ  Soft Steal  Idle  CPUs  Intr  Ctxsw  Proc  RunQ   Run   Avg1  Avg5 Avg15
12:39:34      0     0     0     0     0     1     0    97     1  1082     23     0    76     1   0.42  0.42  0.44
12:39:36      0     0     0     0     0     1     0    97     1  1088     24     0    76     1   0.42  0.42  0.44

The following demonstrates how collectl identify the process reading/writing most data to disk
#Hammer disk by writing 50mb data with dd

$dd if=/dev/urandom of=test bs=1k count=50000
#collectl identifies the “dd” process
#in top mode, sort by  “iokb   total I/O KB” ; “collectl –showtopopts”
$collectl -i2  --top iokb
TOP PROCESSES sorted by iokb (counters are /sec) 12:50:31
# PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct  AccuTime  RKB  WKB MajF MinF Command
6861  root     18  6784    0 R    3M  572K  0  0.91  0.00  45   0:00.91    0 3680    0   97 dd
1  root     15     0    0 S    2M  632K  0  0.00  0.00   0   0:28.21    0    0    0    0 init
2  root     RT     1    0 S     0     0  0  0.00  0.00   0   0:00.00    0    0    0    0 migration/0

Understanding Red Hat Linux recovery runlevels

2011-10-11T17:15:00.003+11:00

If Linux system can boot but hang during starting a service, booting to “recovery runlevels” can skip the service and gain shell to troubleshoot.
If Linux system can’t boot at all, booting from rescue CD (first installation media) and type “linux rescue” to gain shell to troubleshoot
Red Hat Linux boot order

The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/inittab->
/etc/rc.d/rc.sysinit->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S

Recovery runlevels
- runlevel 1
Execute up to /etc/rc.d/rc.sysinit and /etc/rc.d/rc1.d/
Runlevel 1 is identical to singleuser mode. It is switched to singleuser mode in last step, just a number of trivial scripts executed before that.
$ls /etc/rc.d/rc1.d/S*
/etc/rc.d/rc1.d/S02lvm2-monitor /etc/rc.d/rc1.d/S13cpuspeed /etc/rc.d/rc1.d/S99singlesingleuser

- single
Execute up to /etc/rc.d/rc.sysinit

- Emergency
Does not execute /etc/rc.d/rc.sysinit.
Because rc.sysinit is not executed, file system is mounted in read-only mode. You need run “mount –o rw,remount /” to remount it in read-write mode.
emergency runlevel is Red Hat term, it is identical to “init=/bin/sh” in any Linux distribution
How to go to a runlevel
In the grub menu, type “a” to append one of following options to boot line.
1
single
emergency
init=/bin/sh
When Centos hung on starting up boot services, how to get to shell without rescue CD
RHCE Notes - Troubleshooting booting issue

Advanced RPM topics

2011-10-11T12:33:00.002+11:00

Query
“queryformat” option can query every piece information of a rpm package, the information tags (macros ) are returned by “rpm –querytags” command

#list top 2 rpm packages sorted by installation time
$rpm -qa  | xargs -I{} rpm -q --queryformat "{}        %{installtime}\n" {} | sort -rn -k2 | head -2
collectl-3.5.1-1        1317864013
git-1.7.4.1-1.el5        1316484590
#unfortunately the time returned is unixtime.  You can convert it to human readable format by  “date –d @timestring” e.g 
$date -d @1317864013
Thu Oct  6 12:20:13 EST 2011
#but there is a shortcut  “--last”
$ rpm -qa --last  | head -2
collectl-3.5.1-1                              Thu 06 Oct 2011 12:20:13 PM EST
git-1.7.4.1-1.el5                             Tue 20 Sep 2011 12:09:50 PM EST

"rpm -qa" supports regular expression itself, rather than pipe to grep e.g “rpm -qa | grep perl”
“rpm –qa perl\*” also works. There is no improvement on speed but typing become lesser.
requires and provides
#You can check the package dependency before install the package

$ rpm -qp --requires git-1.7.6-1.el5.rf.i386.rpm
..
libssl.so.6 
…
#To meet the dependency, you want to check who provides libssl.so.6 
$yum whatprovides libssl.so.6
openssl-0.9.8e-20.el5.i686 : The OpenSSL toolkit
Repo        : base
Matched from:
Other       : libssl.so.6
#if openssl has been installed, “rpm -q –whatprovides” can also provide the answer
$rpm -q --whatprovides libssl.so.6
openssl-0.9.8e-12.el5_4.6

rpm scriptlets

#query all nopre|nopost|nopreun|nopostun  scripts
$rpm -q --scripts xinetd
postinstall scriptlet (using /bin/sh):
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
fi
preuninstall scriptlet (using /bin/sh):
if [ $1 = 0 ]; then
/sbin/service xinetd stop > /dev/null 2>&1
/sbin/chkconfig --del xinetd
fi
postuninstall scriptlet (using /bin/sh):
if [ $1 -ge 1 ]; then
/sbin/service xinetd condrestart >/dev/null 2>&1
Fi
#query postinstall script only
$ rpm -q --queryformat "%{POSTIN}"  xinetd
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
#Don’t run the scripts during install/remove
rpm –i –noscripts|nopre|nopost|nopreun|nopostun   pkgname
rpm –e –noscripts|nopre|nopost|nopreun|nopostun   pkgname

Extract rpm contents without install

#use rpm2cpio to extract everything
$mkdir /tmp/epel
$ cd /tmp/epel
$ rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd
./etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
..
#use rpm2cpio to extract particular file
$rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd  ./usr/share/doc/epel-release-5
#another way is to use rpm install with alternative root
$ rpm --root /tmp/epel/ -ivh  --nodeps /root/epel-release-5-4.noarch.rpm

Recover corrupted rpm database
Build RPM from source file

Get information for inode or block of a file and vice versa

2011-09-15T16:17:00.002+10:00

Debugfs can retrieve a file’s inode or block information and vice versa, it is provided by e2fsprogs package, which should be installed by default.
Find a file’s information for inode or block

#find a file’s inode id by ls, the sample file’s inode number is 12
$ls -li /boot/message
12 -rw-r--r-- 1 root root 80032 Mar 13  2009 /boot/message
#find a file’s inode or block info by debugfs
$echo "stat message" | debugfs /dev/sda1
debugfs 1.39 (29-May-2006)
debugfs:  Inode: 12   Type: regular    Mode:  0644   Flags: 0x0   Generation: 3354433043
User:     0   Group:     0   Size: 80032
File ACL: 4640    Directory ACL: 0
Links: 1   Blockcount: 162
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4e2371f6 -- Mon Jul 18 09:36:22 2011
atime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
mtime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
BLOCKS:
(0-11):7681-7692, (IND):7693, (12-78):7694-7760
TOTAL: 80

Find a file based on inode or block info
#find a file based on inode number by find, but it may take long time to search

$find /boot/ -inum 12
/boot/message
#find a file based on inode number by debugfs
$echo -e "ncheck 12" | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message
##find a file based on block number by debugfs
#use icheck to find inode number based on block number first
$echo -e "icheck 7693 " | debugfs /dev/sda1
debugfs:  Block Inode number
7693  12
#then use ncheck find the file by inode number
$echo -e "ncheck 12 " | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message

Recover corrupted RPM database

2011-09-15T11:30:00.003+10:00

RPM database consists of a number Berkeley DB files in /var/lib/rpm, the exception is __db.* files, which like cache files are updated for every rpm operation and they can be safely deleted.

#tested in Centos 5.5 
$ ls /var/lib/rpm
Basenames     __db.001  __db.003  Filemd5s  Installtid  Packages      Provideversion  Requireversion  Sigmd5
Conflictname  __db.002  Dirnames  Group     Name         Providename   Pubkeys         Requirename  Sha1header      Triggername
$ file /var/lib/rpm/Packages
/var/lib/rpm/Packages: Berkeley DB (Hash, version 8, native byte-order)

If one of the DB files is partially corrupted and it is still readable by /usr/lib/rpm/rpmdb_dump, you can reload the DB file and rebuild db.
$cd /var/lib/rpm

$rm -f __db*      
$mv Packages Packages.orig
$/usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages
$/usr/lib/rpm/rpmdb_verify Packages
#if you got this error: db_verify: PANIC: fatal region error detected; run recovery
#make sure /var/lib/rpm/__db.* are cleaned
#It is unlikely to rebuilddb if rpmdb_verify fails
$rpm -v –rebuilddb

If one of the DB files is completely corrupted and it is not readable by rpmdb_dump, you have to restore from backup,
$cd /var/lib/rpm

$cp Packages Packages.bak
#simulate a damaged RPM DB file
$ >Packages
$ cp Packages.bak  Packages
# Simply restoring  from backup file won’t work
#file verification is successful 
$ /usr/lib/rpm/rpmdb_verify Packages
#but any rpm operation fails
$rpm -qa 
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Even “rpm –rebuilddb” fails
$rm -f __db.*
$rpm –rebuilddb
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Notice the error about signature: BAD? The Pubkeys have to be cleaned as well.
$ mv Pubkeys Pubkeys.bak
#all good after removing Pubkeys file, a new Pubkeys is generated automatically on “rpm –rebuilddb”
$ rm -f __db.*
$ rpm –rebuilddb
$ rpm -qa | head -2
man-pages-2.39-15.el5_4
bash-3.2-24.el5

Authenticate Linux users by Windows AD: LDAP+Kerberos or Winbind method

2011-08-04T21:35:00.008+10:00

Authenticating Linux users by Windows AD has become popular in many organizations for the convenience of centralized account management. It makes sense, if you need existing Windows account to login to Linux servers, but it doesn’t make sense to install a new Windows AD for the sole purpose of authenticating Linux users, just because Windows has a nice GUI. Redhat Directory Server/Centos Directory Server/389 Directory Server all are capable of doing such task beautifully by ldap+StartTLS or ldap+Kerberos.
At least, there are 4 ways to authenticate Linux users by Windows AD:
1.    Linux Clients use Linux LDAP server synchronized with Windows AD.
     Replicate a subtree of Windows AD accounts to Linux Ldap Server (Redhat Directory Server/Centos Directory Server/389 Directory Server), Linux users authenticate against the   Linux LDAP Server either by Ldap+StartTLS or Ldap+Kerberos
A more complex approach, it is for those who don’t like a Windows AD sitting in datacenter.
2.    Linux Client use LDAP or LDAP+SSL(ldaps) or LDAP+StartTLS
User information store: LDAP
Authentication: LDAP or LDAP+SSL(ldaps) or LDAP+StartTLS
Limitation: Client can login but can’t change its password
3.    Linux Client use ldap+Kerberos
User information store: LDAP
Authentication: Kerberos
The most popular approach, work out of the box, no need to install additional software. User can change its password.
A vbscript to add unix user to windows AD.
http://honglus.blogspot.com/2009/04/add-unix-user-to-windows-ad-by-vbscript.html
4.    Linux Client use samba Winbind
User information store: Winbind local file or winbind ldap
Authentication: Winbind
Cons to solution #3 ldap+Kerberos approach:
- /etc/init.d/windbind dameon need to be running
- The Linux client has to be joined to the domain, it is showed in computers container.
- The UID/GID is derived from Windows SID(ID = RID - BASE_RID + LOW RANGE ID, "man 8 idmap_rid"), you don’t seem to have a choice to set to a specific number.
Pros to solution #3 ldap+Kerberos approach:
- No need to insall Service for Unix in Windows
This post demonstrate method #3 ldap+kerberos and #4 Winbind.
Linux OS: Centos 5.5
Windows OS: Window 2003
FQDN: server1.ad.example.com
Domainname/Realm: ad.example.com

Linux Clients use ldap+Kerberos

Setup Windows Server
- Install “Windows Services for UNIX Version 3.5” for Windows 2003
Windows 2003 R2 or Windows 2008 has built-in component “Server for NIS Service”
- Create a generic user account for Linux nss-ldap client to run ldap query with the credentials.
e.g username=Linux-bind-user password=Redhat123
- Create a test group, go to “Unix Attribute tab” to set GID etc
- Create a test user, go to “Unix Attribute tab”, to set uid/homedir/shell/gid etc
- Modify default domain security group policy to allow users change password immediately, by default it is 1 day, change it to 0 day.
Set up Linux client

#make sure the following pkgs are installed 
krb5-workstation
openldap-clients
nss_ldap
pam_krb5
#Edit configuration files: /etc/nsswitch.conf; /etc/ldap.conf; /etc/krb5.conf; /etc/pam.d/system-auth
#The easiest way is to run authconfig-tui.
#Before run the command, empty config files for easy reading
$cat /dev/null >/etc/ldap.conf ; cat /dev/null > /etc/krb5.conf
$authconfig-tui       #select use ldap for “User information” and “use md5 passwords, use shadow passwords, use Kerberos,local authorization is sufficient” for authentication.
$cat /etc/ldap.conf
uri ldap://server1.ad.example.com/
base cn=Users,dc=ad,dc=example,dc=com
ssl no
binddn cn=linux-bind-user,cn=users,dc=ad,dc=example,dc=com
bindpw Redhat123
##Map Unix account attributes to Winnows SFU 3.5 attributes
##NOTE: Windows 2003 R2/Windows 2008 use different name
##In either scheme, the attribute can be retrieved by ldapsearch command
#$ldapsearch –x –W –b ”cn=Users,dc=ad,dc=example,dc=com” –D “cn=linux-bind-user,cn=users,dc=ad,dc=example,dc=com” 
#....
#msSFU30Name: jsmith
#msSFU30UidNumber: 10001
#...
nss_map_objectclass posixAccount User
nss_map_objectclass shadowAccount User
nss_map_objectclass posixGroup Group
nss_map_attribute uid sAMAccountName
nss_map_attribute uidNumber msSFU30uidNumber
nss_map_attribute gidNumber msSFU30gidNumber
nss_map_attribute gecos         name
nss_map_attribute homeDirectory msSFU30HomeDirectory
nss_map_attribute loginShell msSFU30loginShell
pam_login_attribute sAMAccountName
pam_filter objectclass=User
nss_base_password cn=Users,dc=ad,dc=example,dc=com
nss_base_shadow cn=Users,dc=ad,dc=example,dc=com
nss_base_group cn=Users,dc=ad,dc=example,dc=com
pam_password ad
#Optional, "referrals yes" try to connect both ldap://server1.ad.example.com and ldap://ad.example.com
referrals       no
$cat /etc/krb5.conf
##[logging] section is not needed, I think, it is  for KDC daemon running locally only
[libdefaults]
default_realm = AD.EXAMPLE.COM                    #realm name MUST be in capitals 
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
AD.EXAMPLE.COM = {
kdc = server1.ad.example.com
admin_server = server1.ad.example.com
}
##domain to realm mapping is also optional for user login, it is for the host added to KDC as principal to provide service
[domain_realm]
.example.com = AD.EXAMPLE.COM
example.com = AD.EXAMPLE.COM

#nsswitch.conf system-auth don’t need to be further customization after “authconfig-tui” is run, just post them for reference
$cat /etc/nsswitch.conf
passwd:     files ldap
shadow:     files ldap
group:      files ldap
$grep krb5  /etc/pam.d/system-auth
auth        sufficient    pam_krb5.so use_first_pass
account     [default=bad success=ok user_unknown=ignore] pam_krb5.so
password    sufficient    pam_krb5.so use_authtok
session     optional      pam_krb5.so

Test LDAP+Kerberos

#Query LDAP user information or run getent password|shadow|group to list all 
$id jsmith 
uid=10001(jsmith) gid=10000(unix-test-group) groups=10000(unix-test-group)

Troubleshooting
Error:
pam_krb5[5330]: password change failed for jsmith@AD.EXAMPLE.COM: Password change rejected, Password change rejected
Solution:
test changing password in Windows GUI, Make sure the password meet password complexity policy and minimum change day is 0

Error:
Authentication failure (Cannot find KDC for requested realm)
$ kinit
kinit(v5): Cannot find KDC for requested realm while getting initial credentials
Solution:
check the relam spelling is correct and all in UPPER case, e.g AD.EXAMPLE.COM

ERROR:
Authentication failure (Clock skew too great)
Solution:
synchronize datetime with AD and enable ntpd
$ntpdate -s server1.ad.example.com

Enable DEBUG:
$cat /etc/ldap.conf
debug 7
Linux Clients use samba Winbind

#Windows required software
Windows Services for UNIX is not needed
#Linux required software
samba-client
samba-common
krb5-workstation
#Clear /etc/ldap.conf and /etc/krb5.conf in last test
$cat /dev/null >/etc/ldap.conf; $cat /dev/null >/etc/krb5.conf; 
$authconfig-tui
#select use winbind for “User information” and “use md5 passwords, use shadow passwords, use winbind,local authorization is sufficient” for authentication.
#supply following information and  type in "domain admin"'s username password to join domain
$cat /etc/samba/smb.conf
[global]
#--authconfig--start-line--
workgroup = AD
password server = server1.ad.example.com
realm = AD.EXAMPLE.COM
security = ads
idmap uid = 16777216-33554431
idmap gid = 16777216-33554431
template shell = /bin/bash
winbind offline logon = false
##above lines are generated by authconfig-tui
##following 3 parameters need mannual customization
winbind use default domain = true
winbind enum users = yes
winbind enum groups = yes
#--authconfig--end-line--
##Other files don't need further customization, just paste here for reference
$cat /etc/krb5.conf
[libdefaults]
default_realm = AD.EXAMPLE.COM
dns_lookup_realm = false
dns_lookup_kdc = false
[realms]
AD.EXAMPLE.COM = {
kdc = server1.ad.example.com
}
$grep winbind /etc/pam.d/system-auth
auth        sufficient    pam_winbind.so use_first_pass
account     [default=bad success=ok user_unknown=ignore] pam_winbind.so
password    sufficient    pam_winbind.so use_authtok
$grep winbind /etc/nsswitch.conf
passwd:     files winbind
shadow:     files winbind
group:      files winbind

Test winbind

$wbinfo -u
jsmith
$gettent passwd
jsmith:*:16777216:16777216:John smith:/home/AD/jsmith:/bin/bash
##Group membership is added by opening the group, then click add member button in Windows GUI
$id jsmith
uid=16777216(jsmith) gid=16777216(domain users) groups=16777216(domain users),16777225(unix-test-group)

Passed 3/5 RHCA: EX333 Security Network Services

2011-07-31T10:50:00.001+10:00

I would probably rate EX333 as the most difficult exam among the 3 exams I have passed, Why is it so difficult?

1. The exam objectives are related: for example, Kerberos depends on NIS, without a working NIS, you are doomed.

2. The exam is broken into morning section and afternoon section, You won’t pass the exam if either section fails. but you are still allowed to sit for the afternoon section despite the result of morning section.

My blog post for EX333 study notes:

Authenticate BIND zone transfer with TSIG key

Setup Postfix SMTP password authentication with SASL

CentOS Directory Server 8 Quickstart

2011-07-06T16:50:00.002+10:00

CentOS Directory Server is a rebuild of the Red Hat Directory Server. Red Hat Directory Server, Fedora 389 Directory server, and Sun One Directory Server are similar, because they all originated from Netscape Directory Server (NDS). OpenLDAP is also a member of the extended family , whose root is University of Michigan slapd project, the parent of Netscape Directory Server. The obvious difference is that OpenLDAP doesn't have built-in management console.

Install CentOS Directory Server (CDS) 8.1.0 on Centos 5.5

#CDS requires Centos 5.3 or newer
#install openldap-clients, as CDS ldap clients are not very friendly
$yum install java-1.6.0-openjdk openldap-clients centos-ds
#Link /usr/bin/java to the java 1.6 binary
$alternatives --config java

Setup CentOS Directory Server

#Create ldap user/group for ldap daemon
$groupadd ldap; useradd -g ldap -s /sbin/nologin ldap
#Start installation wizard 
$setup-ds-admin.pl
#start management console
$centos-idm-console

CentOS Directory Server directory structure

/etc/init.d/dirsrv    #server startup script
/etc/init.d/dirsrv-admin   #admin server startup script
/etc/dirsrv/slapd-$instance/   #server config
/etc/dirsrv/slapd-$instance/dse.ldif   #server config for "cn=config"
/etc/dirsrv/slapd-$instance/scheme/99user.ldif #user defined scheme
/etc/dirsrv/admin-serv     #admin server config
/usr/lib/dirsrv/slapd-$instance/    #useful scripts: start&stop; backup&restore ...
/var/lib/dirsrv/slapd-$instance/db/      #database 
/var/lib/dirsrv/slapd-$instance/bak    #default backup dir
/var/log/dirsrv/slapd-$instance/    #logs
$ldapsearch -x -s base -b ""  # Root DSE; Show version, supported plugin etc

CentOS Directory Server backup and restore

##Backup
#/etc/dirsrv/slapd-$instance/dse.ldif needs to be backup manually.
1) in GUI, select backup Directory Server
2) in CLI, /usr/lib/dirsrv/slapd-$instance/db2bak
##Restore
#stop ldap server
$service dirsrv stop
#Restore using CLI, Usage: bak2db archivedir [-n backendname]
$/usr/lib/dirsrv/slapd-station08/bak2db /var/lib/dirsrv/slapd-station08/bak/station08-2011_06_30_15_11_51 -n userRoot
#By default,  backend instance name is  userRoot or NetscapeRoot
$grep  nsslapd-backend /etc/dirsrv/slapd-station08/dse.ldif
nsslapd-backend: userRoot
nsslapd-backend: NetscapeRoot

CentOS Directory Server export and import
##---------------------Export to ldif

1) in GUI, tasks -> export databases. 
2) in CLI, db2ldif
#Find out instance name
$ /usr/lib/dirsrv/slapd-$instance/suffix2instance -s "dc=stationn08, dc=example, dc=com"
Suffix, Instance name pair(s) under "dc=stationn08,dc=example,dc=com":
suffix "dc=station08, dc=example, dc=com"; instance name "userRoot"
$grep  nsslapd-backend /etc/dirsrv/slapd-station08/dse.ldif
nsslapd-backend: userRoot
nsslapd-backend: NetscapeRoot
#Export using backend instance name
$/usr/lib/dirsrv/slapd-$instance/db2ldif -n userRoot -a /tmp/all-userroot.ldif
#Export using suffix name
$/usr/lib/dirsrv/slapd-$instance/db2ldif -s  'dc=example,dc=com'  -a /tmp/all.ldif
##---------------------Import from ldif
#if Server is live
in GUI, Tasks->Import databases; 
in CLI, ldif2db.pl (It is recommended to use GUI for import due to the complexity of the script).
#If server is offline, use ldif2db script
$service dirsrv stop
$/usr/lib/dirsrv/slapd-$instance/ldif2db -n userRoot -i /tmp/all.ldif

LDAP command lines
CDS has built-in “mozldap-tools”, which have similar commands suites to openldap-clients, but Openldap-clients is easier to use because it support client configuration.
OpenLDAP client configuration files and command line options
##OpenLDAP client configuration files

$/etc/openldap/ldap.conf    #Global client conf, but BINDDN (Authenticated user)  is ignored  in Global conf
$HOME/ldaprc, $HOME/.ldaprc       #user ldap configuration file, set BINDDN here
$CWD/ldaprc                #local ldap configuration file
#Typical configuration
$cat /etc/openldap/ldap.conf
BASE    dc=station08, dc=example, dc=com
URI  ldap://station08.example.com
$cat /root/.ldaprc
BINDDN  cn=Directory Manager
##common command line options
-x         Simple authentication, not SASL bind
-W         prompt for bind password
-w passwd  bind password (for simple authentication)
-D binddn  bind DN    #username to authenticate
-b basedn  base dn for search 
-h host    LDAP server
-H URI     LDAP Uniform Resource Identifier(s)  #ldap://station08.example.com:389
-c         continuous operation mode (do not stop on errors), useful for skipping entries already exist when importing from ldif.
-Z         try to start TLS request (-ZZ to require successful response)  or -H ldaps://

Command line Search Operation

# by default, search filter is  (objectclass=*) and display  ALL attributes. search is allowed for anonymous user, no password prompted
$ldapsearch -x
#
$ldapsearch -x -LLL    #less verbose, -LLL stripeout all comments
$ldapsearch -x  -s base    #(search scope). one of base, one(one-level sub), sub (whole subtree) or children,  default is sub
#Search filters, RFC 2254
#NO  ">" or" < "operator
= Exact match; >= greater than or equal; <= less than or equal; ~= aproximate match   
*  wildcard
#Logical operators
NOT   !   #( ! (uidNumber=500) )
OR   |   # (| (uidNumber>=502)(uid~=jim))
AND   &  # (& (uidNumber>=502)(uid~=jim))
#Escaped chars 
Character       ASCII value
---------------------------
*               0x2a
(               0x28
)               0x29
\               0x5c
NUL             0x00
Escaped "(" can be expressed as  "\(" or "\28"
e.g search telephoneNumber: (02)98660000 use ldapsearch -x  '(telephonenumber=\(02\)98660000)'

Command line change operation
Change operation need admin privilege which is specified by BINDDN in /$HOME/.ldaprc or "-D" in CLI

##types of changing whole record, 
changetype: add
changetype: delete
changetype: moddn
changetype: modify
##----------------------changetype: add
#default change type is add, no need to declare changetype: add,  as long as “-a” is specified in ldapmodify 
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
uid: jsmith
givenName: john
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetorgperson
sn: smith
cn: john smith
$ldapmodify -x -W -a -f /tmp/f1.ldif
Enter LDAP Password:
adding new entry "uid=jsmith,ou=People, dc=station08, dc=example, dc=com"
##----------------------changetype: delete
#Option 1, use ldapmodify command
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
changetype: delete
$ldapmodify -x -W -f /tmp/f1.ldif
Enter LDAP Password:
deleting entry "uid=jsmith,ou=People, dc=station08, dc=example, dc=com"
#Option 2, use ldapdelete command, note the ldif format difference
#ldapdelete support recursive delete with "-r"
$cat /tmp/f1.ldif
uid=jsmith,ou=People, dc=station08, dc=example, dc=com
$ldapdelete -x -W -f /tmp/f1.ldif
##----------------------changetype: moddn
#change RDN relative distingished name, the first part of DN
#Option 1, use ldapmodify command
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
changetype: modrdn
newrdn: uid=jsmith2
deleteoldrdn: 1
$ldapmodify -x -W -f /tmp/f1.ldif
Enter LDAP Password:
modifying rdn of entry "uid=jsmith,ou=People, dc=station08, dc=example, dc=com"
rename completed
#Option 2, use ldapmodrn to achive same result
$ cat /tmp/f1.ldif
uid=jsmith,ou=People, dc=station08, dc=example, dc=com
uid=jsmith2
$ldapmodrdn -x -r -W -f /tmp/f1.ldif
##----------------------changetype: modify
#add, replace,delete attributes of a record, not record itself
#multile actions separated by "-"
$cat /tmp/f1.ldif
dn: uid=jsmith,ou=People, dc=station08, dc=example, dc=com
changetype: modify
add: mail
mail: jsmith@example.com
-
delete: facsimileTelephoneNumber
-
replace: telephonenumber
telephonenumber: +1 408 555 1234
$ldapmodify -x -W -f /tmp/f1.ldif

Spacewalk provision Red Hat Linux by PXE kickstart

2011-06-26T13:32:00.003+10:00

Spacewalk is an open source (GPLv2) Linux systems management solution. It is the upstream community project from which the Red Hat Network Satellite product is derived.
What can Spacewalk do:
- YUM repository Server, which is connected by client via yum-rhn-plugin
- Provision, kickstart, physical or virtual systems (using cobbler)
- Manage and deploy configuration files, software to a group of servers
- Monitor your systems (CPU, disk space etc .. ), Inventory your systems (hardware and software information)
OS supported by Spacewalk:
- Red Hat Linux derivatives (Centos, Fedora, Scientific Linux) and Solaris
- Limited support for SUSE Linux (AutoYaST Support is planned in V1.5, https://fedorahosted.org/spacewalk/roadmap )
- Experimental support for Debian (https://fedorahosted.org/spacewalk/wiki/Deb_support_in_spacewalk )
This post is not a complete guide for spacewalk install and administartion, its goal is to PXE kickstart a pysical server and have it registered to spacewalk server when kickstart complete. Kickstarting a server is easy but having it registered to spacewalk server needs tweaking.
Useful document for Spacewalk:
http://wiki.centos.org/HowTos/PackageManagement/Spacewalk
https://fedorahosted.org/spacewalk/wiki/UserDocs
http://docs.redhat.com/docs/en-US/Red_Hat_Network_Satellite/index.html
This post demonstrate spacewalk 1.4 kickstarting Centos 5.5 i386
Steps Summary:
- Setup PXE boot server environment
- Create OS base channel
- Create child channel(tools channel)
- Create distribution tree
- Create activation key
- Create kickstart profile
Setup PXE boot server environment
Setup tftp and dhcpd, refer to http://honglus.blogspot.com/2011/06/setup-pxe-boot-server-for-linux-server.html
However, you don't bother editing pxelinux.cfg/default, which will be managed by spacewalk.
Create OS base channel
Navigate to: Channels | Manage software channels | Create new channel
Channel label is significant, It is channel label, not channel name, is referred for channel operations.
For GPG key section, refer to “GPG Sign RPM file” http://honglus.blogspot.com/2011/05/build-rpm-from-source-file.html
The GPG pub key need to copied to “/var/www/html/pub”, which can be downloaded by http://Server/pub/MY-GPG-FILE-NAME

#Import OS rpms to OS Base channel
#Before importing rpms, it is recommended to resign all rpms with your own GPG key “ rpm –resign *.rpm”, otherwise you need to import the rpm's original GPG key to all  clients.
$spacewalk-repo-sync  -c channel-label –-url  http://mirror.centos.org/centos/5/os/i386/
# You can also import rpms in installation media by –url file:///media/cdrom

Create child channel(tools channel)
Create a child channel for the Base channel created in last step, using same GPG information

#import spacewalk client rpms to the child channel
$spacewalk-repo-sync  -c child-channel-label  –-url  http://spacewalk.redhat.com/yum/1.4-client/RHEL/5/i386/
#python-ethtool doesn't exist in above sites, you need to download it manually from EPEL repository http://fedoraproject.org/wiki/EPEL  
#import a single rpm  to the child channel
$rhnpush  -c  child-channel-label  -u satadmin python-ethtool*.rpm

Create distribution tree
Distribution hold installation files e.g. “images/stage2.img”, which can't be imported to spacewalk channel

#Create distribution tree path
$mkdir -p /var/distro-trees/centos-32-5.5
#Copy everything in installation media except for rpm files to the dir
#rpm files will be retrieved from channels
$cd /media/cdrom; find . ! -path "./CentOS/*"   | cpio -pvd /var/distro-trees/centos-32-5.5

Navigate to: Systems | Kickstart | Distributions | Create new distribution

Create activation key
Activation key is bound to base channel and entitlements, it is used by client to register to spacewalk without password authentication.
Navigate to: Systems | Activation Keys | Create new Key
Select the base channel and enable provisioning add-on entitlements
In child channels, select the child channel.
Optionally, if you want to pull configuration file e.g /etc/ntp.conf during kickstart, you need to create configuration channel and bind the activation key
Create kickstart profile
Navigate to: Systems | Kickstart | Create new kickstart profile

In operating systems, select base channel and child channel

In Software,enter the following packages in addition to @ Base

rhn-check
rhn-setup
yum-rhn-plugin
python-ethtool
python-dmidecode
rhncfg-client
rhncfg-actions
#Above packages provide rhnreg_ks and rhn_check to register to spacewalk during kickstart, otherwise kickstart postscript will encounter errors:
/tmp/ks-script-KOlpXy: line 128: rhnreg_ks: command not found
/tmp/ks-script-KOlpXy: line 134: rhn_check: command not found
#You can also write your own snipplets in “/var/lib/cobbler/snippets” to add packages dynamically.

In Activation Keys, select the activation key
Once kickstart profile is created, some entries are added to pxe configuration file
/tftpboot/pxelinux.cfg/default
Power on the server to be provisioned, when kickstart completed, It should be registered and appeared in spacewalk.

Setup PXE Boot Server for Linux Server Provisioning

2011-06-18T17:58:00.010+10:00

PXE is Preboot eXecution Environment, for PXE to work, NIC and BIOS must both support PXE (Virtualbox pcnet type adapter supports pxe boot)
PXE boot server components
- DHCP Server    #assign ip address and redirect to tftp Server
- tftp Server         #download boot loaders and configuration file
- syslinux      #provides stage1 boot loader pxelinux.0, which is installed in boot server, independent of the OS to be provisioned
The PXE boot process
1. NIC requests DHCP information (DHCP DHCPDISCOVER to port 67/UDP)
2. DHCP server provides bootloader name and IP of tftp server
#relevant DHCP config

nextsever "172.16.1.10";  
filename "pxelinux.0";

3. NIC uses tftp to fetch bootloader into RAM(tftp tftp-server -c get pxelinux.0)
4. BIOS executes bootloader
5. Bootloader uses tftp to find and retrieve configuration file in following order:
[5.1] MAC address using hex and dashes, prefaced with ARP type code
[5.2] IP address expressed in hex

#Convert decimal to hex by gethostip command
$gethostip 192.0.2.91
192.0.2.91 192.0.2.91 C000025B
[5.3]Strips one digit of hex IP at a time from the right-hand side until file is found
[5.4]Last attempt is default

As an example, if the boot file name is /tftpboot/pxelinux.0, the Ethernet MAC address is 88:99:AA:BB:CC:DD and the IP address 192.0.2.91, it will try:
/tftpboot/pxelinux.cfg/01-88-99-aa-bb-cc-dd
/tftpboot/pxelinux.cfg/C000025B

/tftpboot/pxelinux.cfg/C000025
... 
/tftpboot/pxelinux.cfg/C
/tftpboot/pxelinux.cfg/default

6. Bootloader load kernel: vmlinuz and initrd.img defined in the configuration file retrieved.
Install PXE Boot Server components
The setup procedure is demonstrated in Centos 5
$yum install tftp dhcp syslinux
tftp configuration is /etc/xinetd.d/tftp and controlled by /etc/init.d/xinetd
Prepare tftp directory structure and populate initial files

$mkdir -p /tftpboot/{pxelinux.cfg,centos-i686-5.5}
pxelinux.cfg                #The directory for client OS configuration files
centos-i686-5.5           #An optional directory to hold vmlinuz, initrd.img specific to a Linux release 

#find pxelinux.0 on PXE boot Server and copy it to tftpboot
$rpm -ql syslinux | grep pxelinux.0
/usr/lib/syslinux/pxelinux.0
$cp /usr/lib/syslinux/pxelinux.0 /tftpboot/
$cp /usr/lib/syslinux/menu.c32   /tftpboot/

#Copy vmlinuz and initrd.img in installation media for the client OS to be provisioned
$cp /media/cdrom/images/pxeboot/{initrd.img,vmlinuz} /tftpboot/centos-i686-5.5

Create PXE configuration file for client OS

#Derive the configuration file name from the ip to be assigned to client OS
$gethostip 172.16.1.128
172.16.1.128 172.16.1.128 AC100180

#Edit config file
#reference: /usr/share/doc/syslinux*/syslinux.doc
#sample config: /media/cdrom/isolinux/isolinux.cfg
$vi /tftpboot/pxelinux.cfg/AC100180
default linux
prompt 1
#timeout in units of 1/10 s.
timeout 20
#dsplay boot.msg
label linux
kernel centos-i686-5.5/vmlinuz
append initrd=centos-i686-5.5/initrd.img ks=http://172.16.1.10/pxe/centos.ks ksdevice=link

#if no config for the host defined, default to boot from none-pxe media
$vi /tftpboot/pxelinux.cfg/default
default normal
prompt 0
label normal
localboot 0
##instead of above method,loading specific kernel based on individual config, You can have only one default config, let user choose which kernel to load.
$ cat /tftpboot/pxelinux.cfg/default
DEFAULT menu
PROMPT 0
MENU TITLE Select a boot option
TIMEOUT 200
TOTALTIMEOUT 6000
ONTIMEOUT local

LABEL local
MENU LABEL (local)
MENU DEFAULT
LOCALBOOT 0

LABEL centos-i686-5.5
kernel /centos-i686-5.5/vmlinuz
MENU LABEL centos-i686-5.5
append initrd /centos-i686-5.5/initrd.img ks=http://172.16.1.10/pxe/centos.ks ksdevice=link

LABEL centos-x86_64-5.5
kernel /centos-x86_64-5.5/vmlinuz
MENU LABEL centos-x86_64-5.5
append initrd=/centos-x86_64-5.5/initrd.img ks=http://172.16.1.10/pxe/centos.ks ksdevice=link
MENU end

Setup DHCP Server

#Activate dhcpd  on specific NIC only.
$vi /etc/sysconfig/dhcpd
DHCPDARGS=eth1

#Edit dhcpd configuration file
#The client OS is assigned  an fixed IP “172.16.1.128” based on mac address, which can be #retrieved in /var/log/messages when client  boot  from pxe the first time.
$cat /etc/dhcpd.conf
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample
ddns-update-style interim;
ignore client-updates;
subnet 172.16.1.0 netmask 255.255.255.0 {
# --- default gateway
option routers                  172.16.1.254;
option subnet-mask              255.255.255.0;
option domain-name              "example.com";
option domain-name-servers      172.16.1.10;
range dynamic-bootp 172.16.1.128 172.16.1.200;
#time unit is 1 sec
default-lease-time 21600;
max-lease-time 43200;
next-server 172.16.1.10;
filename "pxelinux.0";
host host1 {
hardware ethernet 08:00:27:9b:ac:9b;
fixed-address 172.16.1.128;
}
}

Start dhcp server
$service dhcp start
Boot Client
Change boot order in BIOS to prefer network boot and power on the Server to be provisioned
- Client boots up automatically after finding configuration file “AC100180”

- Client without its configuration file found, waiting for user’s input

Red Hat Enterprise Virtualization(RHEV) Notes

2011-06-15T16:35:00.007+10:00

The post only highlight some useful notes, for step-by-step instructions, refer to Red Hat RHEV document

RHEV has two components: Red Hat enterprise Virtualization manager(RHEV-M) and managed hypervisor,which could be RHEV-H(RHEV hypervisor, a trim down version of RHEL) or full-blown RHEL 5.5 (64bit) or newer.

Download RHEV
Red Hat doesn’t publish public available evaluation copy, contact sales to get a evaluation copy of RHEV
RHEV-M notes
- RHEV-M 2.2 support Windows 2003 SP2 or Windows 2008 R2, although the RHEV 2.2 document only mentions Windows 2008 R2. Windows 2003 SP2 need some hostfix, just run update all after installing .NET 3.5.1/IIS/PowerShell 2.2.
Windows 2008 is NOT supported.
- RHEV-M can use hosts file instead of DNS, but the “Do not validate fully qualified computer name checkbox” need to be select when install RHEV-M
- RHEV-M login rely on Windows account, which can be a generic local account or AD account.
- RHEV-M's backend DB is SQL Server 2005, by default, it installs “SQL Server 2005 express” locally, there is an option to connect to external DB.
- If the RHEV manager login URL is not redirected after installing trusted certificate and adding trusted website, point URL directly to Https://FQDN/RHEVmanager/WPFclient.xbap
RHEV-H notes

#RHEV-H boot prompt options
:     #Just press enter to start installation.
:linux rescue     #same as RHEL rescue mode
:linux firstboot   #invoke interactive installation menu
:linux upgrade   #upgrade hypervisor
:linux nocheck   #disable installation media check
#Hypervisor Configuration Menu
Red Hat Enterprise Virtualization Hypervisor release 5.5-2.2
Hypervisor Configuration Menu
1) Configure storage partitions    6) Configure the host for Red Hat Enterprise
Virtualization
2) Configure authentication        7) View logs
3) Set the hostname                8) Install locally and reboot
4) Networking setup                9) Support Menu
5) Register Host to RHN
#options notes
“5) Register Host to RHN” is optional, just configure 1,2,3,4,6 then choose 8
“9) Support Menu” has an option to uninstall  existing RHEV-H

Troubleshoot after RHEV-H has been installed.
If RHEV-H is successfully connected to RHEV-M, it should be appeared in RHEV-M hosts tab with status “Pending Approval”, click “approve” button will finalize the installation. (“Add host” option only works for RHEL host used as hypervisor host . RHEV-H,a trim down version of RHEL, has to use registration flow)
If for some reason, RHEV-H doesn't appear in RHEV-M, check following first
- RHEV-M Windows 2003 SP2 has all latest update
- RHEV-M host name is resolvable, and telnet to the host on 80,443 works.
- Datetime matched in RHEV-H and RHEV-M, /etc/init.d/ntpd is working
then try to re-register RHEV-H to RHEV-M

#re-invoke the Hypervisor Configuration Menu
$setup                      #select option 6 to re-configure hostname for RHEV-M
#restart registration process
/etc/init.d/vdsm-reg restart
#check registration log
/var/log/vdsm-reg/vdsm-reg.log

#Configure files in RHEV-H 
#vdsm registration script
#register itself to RHEV-M, it seems it doesn't need to be running once registration is successful
/etc/init.d/vdsm-reg                 #start-up script, 
/etc/vdsm-reg/vdsm-reg.conf     #configuration file
/var/log/vdsm-reg/vdsm-reg.log    #log file
#Management agent
#by default, listening on port 54321 to communicate with RHEV-M
/etc/init.d/vdsmd
/etc/vdsm/vdsm.conf
/var/log/vdsm/vdsm.log
You are not supposed to create new configuration files  in RHEV-H, any new files in  /etc/ will be lost after reboot. To survive reboot, you need copy your customization files, e.g /etc/hosts, /etc/resolv.conf, to “/config/etc/” once. Next time RHEV-H boots up, it will synchronize all files in /config/etc/* to /etc

NFS store
- The NFS export must writable by vdsm:kvm, (uid:gid 36:36)
- RHEV-M has a windows tool to upload ISO files to ISO domain, The tool go through 2 steps:first upload to SPM(Storage Pool Manager) host, then move from SPM host to NFS. You can actually winscp to NFS directly, then change file ownership to vdsm:kvm.
Guest OS notes
- RHEV 2.2 doesn't support auto-start guest OS, which means if RHEV-M and RHEV-H are rebooted, someone has to login RHEV-M to click “run” for each VM
- RHEL 5.x has built-in VirtIO driver for harddisk and network
- Windows Guest need the virtual floppy file virtio*.vfd copied to ISO domain and mount the floppy (select “run once” select the file as floppy drive) in order for Windows to recognize VirtIO harddisk. Once Windows boots up, install “Guest tools” for VirtIO NIC driver.

Red Hat RHEV vs Vmware ESX

2011-06-11T22:08:00.016+10:00

In 2009, Red Hat launched Red Hat enterprise Virtualization (RHEV) to compete in commercial virtualization market dominated by VMware. RHEV has two components: Red Hat enterprise Virtualization manager(RHEV-M) and managed hypervisor,which could be RHEV-H(RHEV hypervisor, a trim down version of RHEL) or full-blown RHEL 5.5 (64bit) or newer
Feature wise, in paper, RHEV looks not too bad, However what will be revealed if dug further into technical details and compared with VMware?

	RHEV 2.2	ESX 4
Manager
Name	RHEV-M	vCenter
Compatible OS	Windows 2003 Windows 20008 R2	Windows XP Windows 2003 Windows 2008 Windows 2008 R2
Backend DB	Microsoft SQL Server	Microsoft SQL server Oracle
Application Type	Web application (WPF .xbap application)	Windows native application
User Interface	Web UI	Web UI Windows native application
CLI [1]	Powershell	Powershell(PowerCLI) vCLI
SDK&API	Powershell	Powershell, Perl,C#, Java
Hypervisor
Type	Linux kernel (KVM)	Proprietary
Manager Agent	Python script	Binary daemon
HA/Migration [2]	YES	YES
Manager independent [3]	NO	YES
CLI [4]	NO	esxcfg-*/vimsh commands
SDK&API	NO	Powershell, Perl,C#, Java
Storage Type [5]	NFS/iSCSI/FC	local disk/NFS/iSCSI/FC
Guest OS
supported OS [6]	Red Hat Enterprise Linux Windows	All major Linux distributions Windows Solaris Mac OS/BSD
Clone [7]	Supported	supported
Snapshot [8]	limited support	supported
Supported Hard disk [9]	IDE, VirtIO	IDE,SCSI
Cost	~2/3 of VMware cost	expensive

NOTES:
[1] Manager CLI: RHEV-M PowerShell has fewer number of cmdlets compared to PowerCLI

[2] Manager independent: In my opinion, it is RHEV’s biggest mistake in design. RHEV-M is the central brain, the hypervisor is dummy host, which means you are NOT supposed to login to hypervisor to do configuration or VM operation, e.g. add virtual network or start/stop vms. All must be done in RHEV-M. On the other hand, each VMware ESX host is intelligent by design, you can perform almost anything by esxcfg*/vimsh commands. ESX host just rely manager for HA and Distributed Resource Scheduling.(if RHEV-M fails, VMs in RHEV-H will not be interrupted, but don’t touch them, because you can’t restart them without RHEV-M)

[3] Hypervisor HA: RHEV requires a form of fencing method for HA, e.g smart power switch or LOM card to shoot hypervisor in the head.

[4] Hypervisor CLI: libvirt CLI tools are supported in KVM, but RHEV doesn’t use libvirt.

[5] Storage Type: You can’t utilize RHEV-H local storage, it is not visible in manager.RHEV datacenter has a "storage type" (NFS/iSCSI/FC) attribute, only single storage domain with the same type can be attached to datacenter.

[6] Supported guest OS: In paper, RHEL and Windows are the only supported OS, but you can install almost any x86 OS, because RHEV-H is based on KVM not para-virtualization

[7] Clone: RHEV doesn’t call it clone, You have to choose a template when creating new VM. VMware support clone from template or VM.

[8] Snapshot: You have to shutdown RHEV VM to snapshot it.

[9] VirtIO: RHEL 5.x has built-in VirtIO driver, Other Linux should also has VirtIO driver. for windows, RHEV provide Virtual floppy file, virtio*.vfd, to be used during installation. Any other OS without VirtIO has to use IDE (SCSI is not supported, VirtIO is supposed to deliver better performance than SCSI)

Conclusion:
In my opinion, so far, RHEV Server is not enterprise ready due to limitations of [3] , [4], and [8]. RHEV Server lose to VMware ESX in almost every feature compared, However, RHEV does a better job in desktop virtualization thanks to Qumranet, whose root was desktop virtualization. (In 2008, Red Hat acquired Qumranet, from which the RHEV-M originated).

It is reported that Red Hat is developing RHEV 3, which will be based on Jboss (Java) in Linux with PostgreSQL DB backend. Hopefully, RHEV 3 can redesign RHEV-H to make it “intelligent” by integrating libvirt for CLI ability in hypervisor.

Create a shell script to display progress meter like wget's meter style

2011-06-08T13:05:00.003+10:00

The following is a shell script to display progress meter like wget's meter style
The shell script output

$./meter.sh 
27 % |======================>                                                                   |

The shell script source code
$cat ./meter.sh

#!/bin/ksh
#Given start number and end number, display  progress meter and percentage like wget's style
start=1
#end=333
end=33
scale=$(($end/100.0))
for i in {$start..$end}
do
m1=$(($i / $scale)); m2=$(( ($end - $i ) / $scale ))
integer m1; integer m2
#fill 2 segments of variable length  with zeros
str=$( printf "%0${m1}d %s %0${m2}d\n"   0 ">" 0  ) 
str="|$str|"
#replace first segment zeros with '=' 2nd segment zeros with space then re-join
str1=$(echo $str | awk -F' > ' '{ print $1 }'); str1=${str1//0/=}
str2=$(echo $str | awk -F' > ' '{ print $2 }'); str2=${str2//0/' '}
str="${str1}>${str2}"
pct=$(($i * 100 / $end ))
#Beautify the final loop 
[ $i -eq $end ] && str=$(echo $str | sed -e 's/ /=/g' -e 's/>/=/g' -e 's/0/=/g' )
print -n " \\r ${pct} % $str "
sleep 1
done
printf "\n"

Build RPM from source file

2011-05-30T16:15:00.008+10:00

Traditionally, Installing from source file need to go through several steps: ./configure;make;make install;make clean. RPM can automate the process by SPEC file. Once binary RPM package is generated, it can be easily distributed to other servers.
This article use hping3 source file as an example to demonstrate the basics to build RPM. For further information, please refer to http://rpm5.org/docs/rpm-guide.html
Install rpmbuild

$yum install rpm-build

RPM Macros

#Various RPM Macros locations
/usr/lib/rpm/macros #Global default macros
/etc/rpm/micros   #Global user defined macros
~/.rpmmacros  #per-user defined  macros
rpmbuild --define 'macro_name value '   #define at run time
#display a macro
$ rpm --eval %{_vendor}
redhat
#display all macros
rpm --showrc

Setup build environment

#It is preferred to use a non-root user  to to control build
$useradd builder
$ echo '%_topdir    /home/builder/redhat'  > .rpmmacros
$ mkdir -p /home/builder/redhat/{BUILD,RPMS,SOURCES,SPECS,SRPMS}

Building RPM involves following steps:
1. Preparing for building, including unpacking the sources
2. Building (compiling)
3. Installing the application or library
4. Cleaning up
5. Customized scripts for pre-install,post-install, pre-uninstall, post-uninstall
6. List files to be packaged into RPM
7. Add changelog
8. GPG sign package
The first 7 steps are controlled by SPEC file

##This spec file use hping3 source file as an example 
[builder]$ cat  /home/builder/redhat/SPECS/hping3.spec
%define name hping3
%define version 3.0
Name: %{name}
Version: %{version}
Release: 0
License: GPL
##Pick a name in  /usr/share/doc/rpm-*/GROUPS
Group: Applications/System
URL: http://www.hping.org
##All  source files should be packed  under a dir named:    %{name}-%{version}   e.g. ./hping3-3.0/*
##Packed file name should be  %{name}-%{version}.XX  e.g. hping3-3.0.tar.gz
Source: hping3-3.0.tar.gz
Patch0: hping3.patch
#Patch1: 2.patch
#PreReq: unzip
##libpcap is required package for hping to work
Requires: libpcap
##gcc and libpcap-devel are required duing compling
BuildPreReq: gcc libpcap-devel
BuildArch:x86_64
##BuildRoot is staging area that looks like the final installation directory 
##all final files are copied to BuildRoot
BuildRoot: %{_tmppath}/%{name}-root
Summary: hping3 is a network tool.
%Description
hping3 is a network tool able to send custom TCP/IP
packets and to display target replies like ping do with
ICMP replies.
##1. Prepare
%prep
####%setup will go to ~/redhat/BUILD dir and unpack soure files
%setup -q
%patch0
##2. Build
%build
%configure --no-tcl
make
##3. Install
%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT{/usr/sbin,/usr/share/man/man8}
install -m 755 hping3   $RPM_BUILD_ROOT/usr/sbin/
(cd $RPM_BUILD_ROOT/usr/sbin; ln -s hping3 hping2 ; ln -s hping3 hping )
%{__gzip}  ./docs/hping3.8&& \
install -m 644 ./docs/hping3.8.gz $RPM_BUILD_ROOT/usr/share/man/man8
##4. Clean up
%clean
rm -rf $RPM_BUILD_ROOT
make clean
##-5. customized scripts; view all scripts of a rpm file "rpm -q --scripts file.rpm"
####user is not needed, demonstration purpose only
%pre
useradd hping
%post
chage -M -1 hping
#### $1=0 remove; $1=1 first install; $1>=2 upgrade
%postun
if [ $1 = 0 ]; then
userdel -r hping
fi
##6. list files to be packed to RPM
%files
%defattr(-,root,root)
%attr(755,root,root) /usr/sbin/hping*
%doc /usr/share/man/man8/hping3.8.gz
##7. changlog
%changelog
#### date Format:  date +'%a %b %d %Y'
* Mon May 30 2004   antirez <email@com>
- First public release of hping3

Test each stage by rpmbuild

$rpmbuild --help
Build options with [ <specfile> | <tarball> | <source package> ]:
-bp                           build through %prep (unpack sources and apply
patches) from <specfile>
-bc                           build through %build (%prep, then compile)
from <specfile>
-bi                           build through %install (%prep, %build, then
install) from <specfile>
-bl                           verify %files section from <specfile>
-ba                           build source and binary packages from
<specfile>
-bb                           build binary package only from <specfile>

GPG Sign RPM file
Sign a package to prove source identity of the file

#Create gpg key pair,remmber the keypass for private key, it will be asked when signing package
$gpg --gen-key
#Tell rpm which gpg key to use
$ cat ~/.rpmmacros
%_topdir    /home/builder/redhat
%_signature gpg
%_gpg_name rpm test <rpm.test@com>
#Sign RPM with GPG private key
#Before RPM created, use rpmbuid --sign spec-file
#After RPM created, use rpm --resign
$rpm --resign /home/builder/redhat/RPMS/x86_64/hping3-3.0-0.x86_64.rpm
#Export GPG pulic key
$gpg --export -a > /tmp/my-gpg.pub
#Before import, signature "NOT OK"
$rpm --checksig hping3-3.0-0.x86_64.rpm
hping3-3.0-0.x86_64.rpm: (SHA1) DSA sha1 md5 (GPG) NOT OK (MISSING KEYS: GPG#31f8d18a)
#Import GPG pub key
$rpm --import /tmp/my-gpg.pub
#after import,  signature "OK"
$ rpm --checksig hping3-3.0-0.x86_64.rpm
hping3-3.0-0.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK
#list all imported GPG keys
$ rpm -qa gpg*
gpg-pubkey-32a349c9-493c185a
gpg-pubkey-31f8d18a-4de2fc7b
gpg-pubkey-e8562897-459f07a4

Passed 2/5 RHCA: EX436 Clustering and Storage Management

2011-05-28T15:44:00.004+10:00

EX436 is easier than EX442(System Monitoring and Performance Tuning), because testing subjects are less and the method of testing is just straight forward setup and configuration, unlike EX442, which requires extensive analysis and calculation.
I didn't pay attention to RHEL release during exam, But,RHEL 5.4,is showed in my exam result. Although GFS2 is default starting from RHEL 5.3, GFS is the subject to be tested. I think it won't be changed until RHEL 6 courseware comes out.

My blog post for EX436 study notes

GFS(Global File System) quickstart

RHCS(Red Hat Cluster Suite) quorum disk

RHCS(Red Hat Cluster Suite) I/O fencing using SNMP IFMIB

Do we really need to set partition type to fd(Linux auto raid) for Linux software RAID?

Subversion Quickstart

2011-05-22T17:43:00.005+10:00

This short tutorial is intended for new users to grasp subversion quickly.
Subversion is a open source version control system based on Copy-Modify-Merge model rather than lock-Modify-Unlock model.
It is primarily used for software development, which allows developers to modify files and directories concurrently (no locking) and switch between versions easily. In system administration world, it could be used to track system changes and roll back changes.
Fundamental Concepts(don't skip):
- The Repository
Repository is a central store for all versions of data, subversion server configuration files are also located in the repository.
Once repository is created, you are NOT supposed to visit the repository directory other than changing subversion server configuration
You should modify versions of data in a “working copy” of the repository data.
The repository can be accessed in a number ways:

file:/// Direct repository access (on local disk)
http:// Access via WebDAV protocol to Subversion-aware Apache server
https:// Same as http://, but with SSL encryption.
svn:// Access via custom protocol to an svnserve server
svn+ssh:// Same as svn://, but through an SSH tunnel

To setup svnserv server to offer svn:// access over network, you need to enable authentication and authorization by modifying repository-path/conf/{svnserve.conf,passwd,authz} then start “svnserve -d -r repository-path”
- The Working copy directory
A working copy is a subset of repository data. To creating a working copy, use “svn checkout” to checkout the root or sub directory of repository.
You modify data in “working copy” NOT in repository directory
Install subversion
Most Linux distros include subversion by default. to Install in Centos:
$yum install subversion

$rpm -qa | grep subv
subversion-1.6.11-7.el5_6.3

Create a subversion repository

#It is where all data are saved, you should have enough space
$svnadmin create /var/svn
#svnadmin populated the directory with following structure
#conf is the location of server configuration files
#db is the location of your versions of data
$ls /var/svn
conf  db  format  hooks  locks  README.txt
#It is ideal to create individual directory for different project.
#-m is to give a description of this operation, later, it can be view with “svn log”
#This transaction is recorded as revision 1
#the command is svn not svnadmin
#svnadmin and svnlook are server side commands, They always action on a PATH  NOT a URL like file:///”
$svn mkdir file:///var/svn/proj_1 -m "test mkdir"
Committed revision 1.
#Verify the sub dir is created
$svn list -v  file:///var/svn
3 root                  May 22 11:33 ./
3 root                  May 22 11:33 proj_1/

Import data into the repository

#let's import /etc/sysconfig  into the repository
#import is used to to populate repository for the first time
#adding new files later need “svn add” command in a “working copy”
$ svn import /etc/sysconfig/  file:///var/svn/proj_1 -m "test import"
Adding         /etc/sysconfig/irda
Adding         /etc/sysconfig/kernel
Adding         /etc/sysconfig/syslog
..
Adding         /etc/sysconfig/snmpd.options
Committed revision 2.
#let's view the imported files in repository
#something wrong? Where are those files? even dir “proj_1” doesn't exist
#let me repeat, you are supposed to modify data in repository directly,  Do this in a “working copy”
$ls  /var/svn
conf  db  format  hooks  locks  README.txt
# if you are curious about where the data is stored, all data are “packed” in a binary file
$ strings /var/svn/db/revs/0/2 | grep $(hostname)
HOSTNAME=filer.example.com
# or view “svn ls”  and “svn cat”
svn cat file:///var/svn/proj_1/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=filer.example.com

Create a working copy

#create a working copy by checkout proj_1,  The target dir proj_1 will be automatically created, of course, you can name it differently 
$cd /root/svn
$svn checkout file:///var/svn/proj_1 proj_1
A    proj_1/irda
A    proj_1/kernel
A    proj_1/syslog
#I want to add /etc/hosts to repository
#any operations in “working copy” should use subversion-aware commands e.g “svn mkdir, svn add, svn mv, svn cp”
$ cd  /root/svn/proj_1
$svn mkdir  ./etc
$ cp /etc/hosts ./etc
$ svn add ./etc/hosts
A         etc/hosts
#commit the changes to repository
$svn commit -m "added hosts file"
Adding         etc
Adding         etc/hosts
Transmitting file data .
Committed revision 3.
#svnlook shows the latest version is 3
$svnlook  youngest  /var/svn
3
$svn log /root/svn/proj_1/
------------------------------------------------------------------------
r3 | root | 2011-05-22 11:33:03 +1000 (Sun, 22 May 2011) | 1 line
added hosts file
------------------------------------------------------------------------
r2 | root | 2011-05-22 11:29:17 +1000 (Sun, 22 May 2011) | 1 line
test import
------------------------------------------------------------------------
r1 | root | 2011-05-22 11:29:05 +1000 (Sun, 22 May 2011) | 1 line
test mkdir
------------------------------------------------------------------------
$svn diff -r 2:3  /root/svn/proj_1/
Index: /root/svn/proj_1/etc/hosts
===================================================================
--- /root/svn/proj_1/etc/hosts  (revision 0)
+++ /root/svn/proj_1/etc/hosts  (revision 3)
@@ -0,0 +1,8 @@

Rollback to previous versions
That is where subversion shines, no matter how many changes you have made, one simple command can switch versions

$ svn update -r 2 /root/svn/proj_1/
D    /root/svn/proj_1/etc
Updated to revision 2.
$ ls ./etc
ls: ./etc: No such file or directory
$svn update -r 3 /root/svn/proj_1/
A    /root/svn/proj_1/etc
A    /root/svn/proj_1/etc/hosts
Updated to revision 3.
$ ls ./etc
hosts

GFS (Global File System) quickstart

2011-05-17T21:59:00.010+10:00

What is GFS?
GFS allow all nodes to have direct CONCURRENT write access to the same shared BLOCK storage.
For local file system e.g ext3, A shared BLOCK storage can be mounted in multiple nodes, but CONCURRENT write access is not allowed
For NFS, the CONCURRENT write access is allowed, but it is not direct BLOCK device, which introduce delay and another layer of failure.
GFS requirements:
- A shared block storage (iSCSI, FC SAN etc.. )
- RHCS (Red hat Cluster suite) (although GFS can be mounted in standalone server without cluster, it is primarily used for testing purpose or recovering data when cluster fails)
- RHEl 3.x onwards (RHEL derivatives: Centos/Fedora), it should work in other Linux distributions, since GFS and RHCS have been open sourced.
GFS specifications:
- RHEL 5.3 onwards use GFS2
- RHEl 5/6.1 supports maximum 16 nodes
- RHEL 5/6.1 64 bit supports maximum file system size of 100TB (8 EB in theory)
- Supports: data and metadata journaling, quota, acl, Direct I/O, growing file system online, dynamic inodes (convert inode block to data block)
- LVM snapshot of CLVM under GFS is NOT yet supported.
GFS components:
RHCS components: OpenAIS, CCS, fenced, CMAN and CLVMD (Clustered LVM)
GFS specific component: Distributed Lock Manager (DLM)
Install RHCS and GFS rpms
Luci (Conga project) is the easiest way to install and configure RHCs and GFS.

#GFS specific packages:
#RHEL 5.2 or lower versions 
$yum install gfs-utils    kmod-gfs 
#RHEL 5.3 onwards, gfs2 module is part of kernel 
$yum install gfs2-utils

Create GFS on LVM
You can create GFS on raw device, but LVM is recommended for consistent device names and the ability to extend device

#Assume you have setup and tested a working RHCS
#Edit cluster lock type in /etc/lvm/lvm.conf on ALL nodes
locking_type=3 

#Create PV/VG/LV as if in standalone system ONCE in any ONE of the nodes

#Start Cluster and clvmd on ALL nodes 
#Better use luci GUI interface to start whole cluster 
$Service cman start
$Service rgmanager start
$servcie clvmd start

#Create GFS ONCE in any ONE of the nodes
# -p lock_dlm is required in cluster mode. Lock_nolock is for standalone system
# -t cluster1:gfslv      ( Real cluster-name: arbitrary  GFS name )
# Above information is stored in GFS superblock, which can be changed with “gfs_tool sb” without re-initializing GFS e.g change lock type: "gfs_tool sb /device proto lock_nolock" 
#-j 2: the number of journals, minimum 1 for each node. The default journal size is 128Mib, can be overridden by -J
#additional journal can be added with gfs_jadd
gfs_mkfs -p lock_dlm -t cluser1:gfslv -j 2 /dev/vg01/lv01

#Mount GFS in cluster member by /etc/fstab
#put GFS mount in /etc/fstab in ALL nodes
#NOTE:
#Cluster service can mount GFS without /etc/fstab after adding GFS as resource, but It can only mount on one node (the active node).  Since GFS is supposed to be mounted on all nodes at the same time. /etc/fstab is a must, GFS resource is optional.
#GFS mount options: lockproto, locktable are optional, mount can obtain the information from superblock automatically
$cat /etc/fstab
/dev/vg01/lv01          /mnt/gfs                gfs     defaults 0 0

#Mount all GFS mounts
service gfs start

GFS command lines

####Check GFS super block 
#some values can be changed by “gfs_tool sb”
$gfs_tool sb /dev/vg01/lv01 all
sb_bsize = 4096
sb_lockproto = lock_dlm
sb_locktable = cluster1:gfslv01
.. 
####GFS tunable parameters 
#view parameters
gfs_tool gettune <mountpoint>
#set parameters 
#The parameters don’t persist after re-mount, You can customize /etc/init.d/gfs to set tunable parameters on mounting
gfs_tool settune <mountpoint>

####Performance related parameters
#like other file system, you can disable access time update by mount option “noatime”
#GFS can also allow you to control how often to update access time
$gfs_tool gettune /mnt/gfs | grep atime_quantum   
atime_quantum=3660          #in secs

#Disable quota, if not needed
#GFS2 remove the parameter and implement it in mount option “quota=off”
$gfs_tool settune /mnt/gfs quota_enforce 0

#GFS direct I/O
#Enable directI/O for database files, if DB has its own buffering mechanism to avoid “double” buffering 
$gfs_tool setflag directio /mnt/gfs/test.1     #file attribute
$gfs_tool setflag inherit_directio /mnt/gfs/db/     #DIR attribute
$gfs_tool clearflag directio /mnt/gfs/test.1              #remove attribute
$gfs_tool stat  inherit_directio /mnt/gfs/file     # view attribute

#enable data journal for very small files
#disable data journal for large files
$gfs_tool setflag inherit_jdata  /mnt/gfs/db/     #Enable  data journal (only metadata  has journal  by default) on a dir. (if operate on a file, the file must be zero size)

###GFS backup, CLVM doesn't support snapshot
$gfs_tool freeze /mnt/gfs          #change GFS to read-only (done once in any one of the nodes)
$gfs_tool unfreeze /mnt/gfs

###GFS repair 
#after unmount GFS on all nodes
$gfs_fsck  -v /dev/vg01/lv01         # gfs_fsck -v -n /dev/vg01/lv01 : -n answer no to all questions, inspect gfs only without making changes

GFS implementation scenarios:

GFS’s strength is the ability to do concurrent write to the same block device, It make it possible for Active-Active cluster nodes to write to the same block device, but there are few such cases in real life.

In Active-Active cluster nodes (all nodes perform the same task), RHCS can’t do load balancing itself, it requires external load balancer

- Database server cluster: In theory, all nodes can write to the same DB file concurrently, However, the performance will be degraded, because all nodes try to lock the file via Distributed Lock Manager. You can assign different task to cluster nodes to write to different DB file, e.g. node-A run DB-A and node-B run DB-B, but this can be done, without GFS, by mounting ext3 on individual iSCSI/FC disk.

GFS doesn’t lose to ext3 in above scenario, but its lack of LVM snapshot of in GFS‘s CLVMD kills my inspiration of using DB on GFS

- Application server cluster: e.g. Apache, Jboss server cluster. It is the true that GFS can simply application package deployment because all nodes can share the same application package binaries. But if you only use two nodes cluster, deploying application twice is not big hassle. Maintaining single copy of application binaries is convenient, but at risk of single point of failure.

- NFS Cluster: Because NFS is I/O bound, Why would you run Active-Active NFS cluster with CPU/memory resource in nodes are not being fully utilized?

LVM2: device filter and LVM metadata restore

2011-05-10T21:00:00.004+10:00

Customize LVM device filter to get rid of the annoying “/dev/cdrom: open failed” warning

##/dev/cdrom: open failed warning
$pvcreate /dev/sdb1
/dev/cdrom: open failed: Read-only file system
$ vgcreate vg01 /dev/sdb1
/dev/cdrom: open failed: Read-only file system
##The error because LVM scan all device files by default, you can exclude some device files by device filters
##File /etc/lvm/cache/.cache contains the device file names scanned by LVM
$ cat /etc/lvm/cache/.cache
persistent_filter_cache {
valid_devices=[
"/dev/ram11",
"/dev/cdrom",
##Edit /etc/lvm/lvm.conf, Change default filter  
filter = [ "a/.*/" ]
#to
filter = [ "r|/dev/cdrom|","r|/dev/ram*|" ]
##You need to delete the cache file or ran vgscan to regenerate the file
$rm /etc/lvm/cache/.cache   OR vgscan

LVM metadata backup and restore
LVM record every LVM VG and LV metadata operation and save it to /etc/lvm/backup automatically, old version backup files are archived to /etc/lvm/archive.
The backup file can be used to rollback LVM metadata changes, for example, if you have removed the VG/PV or even re-initialize disk with pvcreate, Don't panic,as long as file system was not re-created, you can use vgcfgrestore to restore all the data.
The following is to demonstrate how to recover a LV after it is completed destroyed from PV level (pvremove)
1.Create test LV and write some data

$pvcreate  /dev/sdb1 /dev/sdb2
Physical volume "/dev/sdb1" successfully created
Physical volume "/dev/sdb2" successfully created
$vgcreate vg01  /dev/sdb1 /dev/sdb2
Volume group "vg01" successfully created
$ lvcreate -L100M -n lv01 vg01
Logical volume "lv01" created
$ mkfs.ext3 /dev/vg01/lv01
$ mount /dev/vg01/lv01 /mnt/
$cp /etc/hosts /mnt/
$ ls /mnt/
hosts  lost+found

2.Destroy LV,VG,and PV

$vgremove vg01
Do you really want to remove volume group "vg01" containing 1 logical volumes? [y/n]: y
Do you really want to remove active logical volume lv01? [y/n]: y
Logical volume "lv01" successfully removed
Volume group "vg01" successfully removed
#VG is removed and PV was also wiped out
$ pvcreate /dev/sdb1 /dev/sdb2
Physical volume "/dev/sdb1" successfully created
Physical volume "/dev/sdb2" successfully created

3.Lets recover the LV and the data

##Find out the backup file to restore from
$vgcfgrestore -l vg01
..
file:         /etc/lvm/archive/vg01_00002.vg
VG name:      vg01
Description:  Created *before* executing 'vgremove vg01'
Backup time:  Tue May 10 15:41:31 2011
##first attempt failed, because PV UUID is changed
$ vgcfgrestore -f /etc/lvm/archive/vg01_00002.vg vg01
Couldn't find device with uuid 'pVf1J2-rAsd-eWkD-mCJc-S0pc-47zc-ImjXSB'.
Couldn't find device with uuid 'J14aVl-mbuj-k9MM-63Ad-TBAa-S0xF-VElV2W'.
Cannot restore Volume Group vg01 with 2 PVs marked as missing.
Restore failed.
##Find old UUID
$ grep -B 2 /dev/sdb /etc/lvm/archive/vg01_00002.vg
pv0 {
id = "pVf1J2-rAsd-eWkD-mCJc-S0pc-47zc-ImjXSB"
device = "/dev/sdb1"    # Hint only
--
pv1 {
id = "J14aVl-mbuj-k9MM-63Ad-TBAa-S0xF-VElV2W"
device = "/dev/sdb2"    # Hint only
$
##Recreate PV with the old UUID
$ pvcreate -u pVf1J2-rAsd-eWkD-mCJc-S0pc-47zc-ImjXSB /dev/sdb1
Physical volume "/dev/sdb1" successfully created
$ pvcreate -u J14aVl-mbuj-k9MM-63Ad-TBAa-S0xF-VElV2W  /dev/sdb2
Physical volume "/dev/sdb2" successfully created
##run vgcfgrestore again
$ vgcfgrestore -f /etc/lvm/archive/vg01_00002.vg vg01
Restored volume group vg01
##data was also recovered
$ mount /dev/vg01/lv01 /mnt/
mount: special device /dev/vg01/lv01 does not exist
$ lvchange -a y vg01/lv01
$ mount /dev/vg01/lv01 /mnt/
$ cat /mnt/hosts
127.0.0.1       localhost
..

RHCS(Red Hat Cluster Suite) quorum disk

2011-05-07T17:12:00.004+10:00

The last post "RHCS I/O fencing" is about dealing with split-brain situation, in which cluster members lost heartbeat communication and each believe it is legitimate to write data to the shared storage.
Methods to deal with split-brain situation:
1. Redundant heartbeat path
network port communication plus serial port communication
2. I/O fencing
Remaining nodes separate failed node from its storage either by shutdown/reboot power port or storage port
3. Quorum disk
Quorum disk is a kind of I/O fencing, but the reboot action is executed by failed node's own quorum daemon. It also has additional feature: contributing vote to cluster. if you want the last standing node to keep the multiple-nodes cluster running, quorum disk appears to be the only solution.
RHCS (Red Hat Cluster Suite) Quorum disk facts
- A shared block device (SCSI/iSCSI/FC..), Device size requirement is approximately 10MiB
- Supports maximum 16 nodes, nodes id must be sequentially ordered
- Quorum disk can contribute votes. In multiple nodes cluster, together with quorum vote, the last standing node can still keep the cluster running
- single node votes+1 <=Quorum's disk vote < nodes total votes
- The failure of the shared quorum disk won’t result in cluster failure, as long as Quorum's disk vote < nodes total votes
- each node write its own health information in its own region, the health is determined by external checking program such as "ping"
Setup Quorum disk

#initialise quorum disk once in any node 
mkqdisk -c /dev/sdx -l myqdisk

Add quorum disk to cluster
Use luci or system-config-cluster to add quorum disk, following is the result xml file

<clusternodes>
<clusternode name="station1.example.com" nodeid="1" votes="2">
<fence/>
</clusternode>
<clusternode name="station2.example.com" nodeid="2" votes="2">
<fence/>
</clusternode>
<clusternode name="station3.example.com" nodeid="3" votes="2">
<fence/>
</clusternode>
</clusternodes>
#expected votes =9=(nodes total votes + quorum disk votes) = (2+2+2+3)       
<cman expected_votes="9"/> 
#Health check result is writen to quorum disk every 2 secs
#if health check fails over 5 tko, 10 (2*5) secs, the node is rebooted by quorum daemon
#Each heuristic check is run very 2 secs and earn 1 score,if shell script return is 0
<quorumd interval="2" label="myqdisk" min_score="2" tko="5" votes="3">
<heuristic interval="2" program="ping -c1 -t1 192.168.1.60" score="1"/>
<heuristic interval="2" program="ping -c1 -t1 192.168.1.254" score="1"/>
</quorumd>

Start quorum disk daemon
The daemon is also one of daemons automatically started by cman
service qdiskd start
Check quorum disk information

$ mkqdisk -L -d
mkqdisk v0.6.0
/dev/disk/by-id/scsi-1IET_00010002:
/dev/disk/by-uuid/55fbf858-df75-493b-a764-5640be5a9b46:
/dev/sdc:
Magic:                eb7a62c2
Label:                myqdisk
Created:              Sat May  7 05:56:35 2011
Host:                 station2.example.com
Kernel Sector Size:   512
Recorded Sector Size: 512
Status block for node 1
Last updated by node 1
Last updated on Sat May  7 15:09:37 2011
State: Master
Flags: 0000
Score: 0/0
Average Cycle speed: 0.001500 seconds
Last Cycle speed: 0.000000 seconds
Incarnation: 4dc4d1764dc4d176
Status block for node 2
Last updated by node 2
Last updated on Sun May  8 01:09:38 2011
State: Running
Flags: 0000
Score: 0/0
Average Cycle speed: 0.001000 seconds
Last Cycle speed: 0.000000 seconds
Incarnation: 4dc55e164dc55e16
Status block for node 3
Last updated by node 3
Last updated on Sat May  7 15:09:38 2011
State: Running
Flags: 0000
Score: 0/0
Average Cycle speed: 0.001500 seconds
Last Cycle speed: 0.000000 seconds
Incarnation: 4dc4d2f04dc4d2f0

The cluster is still running with last node standing
Please note Total votes=quorum votes=5=2+3, if quorum disk vote is less than (node votes+1), the cluster wouldn’t have survived

$cman_tool status
..
Nodes: 1
Expected votes: 9
Quorum device votes: 3
Total votes: 5
Quorum: 5  
..

RHCS(Red Hat Cluster Suite) I/O fencing using SNMP IFMIB

2011-05-02T23:05:00.005+10:00

RHCS is comparable to HP MC/Service Guard, IBM HACMP, SUN Cluster etc. The free open source version is available in Centos/Fedora
Fencing is the act of isolating a cluster node from its storage when the node is not responding,
otherwise, when the node is recovered, the shared file system may be corrupted when written by more than 1 node at the same time.
RHCS supports following Fencing methods:
Power fencing:
Forcefully Power Cycle, just like pull out the power Cord
- Internal Power fencing
    Lights-out management Card ( HP iLO, IBM RSA, SUN iLOM, DELL DRAC, IPMI etc)
- External Power fencing
    SMART Power Switch (APC Switched Rack PDU etc)
Network port fencing:
Shutdown the storage network port
    - IP Switch for iSCSI
    - SAN Switch for FC SAN
SCSI 3 Persistent Reservation
Virtual Guest
shutdown VM guest by VM host
- Xen, Vmware or any guest managed by libvirt tools
full list :
http://www.redhat.com/cluster_suite/hardware/

#Each fence type above has its own fence agent, which is Python/Perl script.
$ls /sbin/fence*
/sbin/fence_ack_manual   /sbin/fence_drac     /sbin/fence_mcdata     /sbin/fence_tool
/sbin/fence_apc          /sbin/fence_drac5    /sbin/fence_node       /sbin/fence_virsh
/sbin/fence_apc_snmp     /sbin/fence_egenera  /sbin/fence_rhevm      /sbin/fence_vixel
/sbin/fence_bladecenter  /sbin/fence_ifmib    /sbin/fence_rps10      /sbin/fence_vmware
/sbin/fence_brocade      /sbin/fence_ilo      /sbin/fence_rsa        /sbin/fence_vmware_helper
/sbin/fence_bullpap      /sbin/fence_ilo_mp   /sbin/fence_rsb        /sbin/fence_wti
/sbin/fence_cisco_mds    /sbin/fence_ipmilan  /sbin/fence_sanbox2    /sbin/fence_xvm
/sbin/fence_cisco_ucs    /sbin/fence_lpar     /sbin/fence_scsi       /sbin/fence_xvmd
/sbin/fenced             /sbin/fence_manual   /sbin/fence_scsi_test
#Power fencing agent (fence_ilo,fence_apc etc) typically use telnet/ssh to IP of LOM card or external Powerswitch to turn off the power.

fence_ifmib is today's subject, it use SNMP write permission to shutdown a network port of iSCSI storage Server
LAB Setup
Virtual Box with 3 VMS
filer: Cluster management Server Luci and iSCSI filer.
node1: Cluster node1 with Luci agent ricci
node2: Cluster node2 with Luci agent ricci
filer, node1, and node2 are connected in separate LAN, LAN1 is for application traffic, LAN2 is for iSCSI stroage traffic, So filer has 3 NICs, each Cluster node has 2 NICs.

Using fence_ifmib, the node1 can fence off node2 by issuing snmpset via LAN1 interface to shutdown filer's one of LAN2 NICs, where node2 is connected and vice versa.
Setup Cluster
on each node, yum install ricci; service ricci restart
on filer, yum install luci; "luci_admin init" to set admin password; service luci resart
login to Luci web interface on port 8084
create a new cluster and add cluster node members
Setup fence_ifmib fence device
- on filer, set up snmpd

$ cat /etc/snmp/snmpd.conf 
createUser fence_admin   MD5 "Pass1234"
authuser   read,write -s usm  fence_admin  authnopriv .1.3.6.1.2.1.2.2.1

I use SNMP V3 USM because Luci requires SNMP username/password. I think, other Cluster management tools: system-config-cluster, ccs_tool accept SNMP V2c and V1
further info on setup snmpV1/V2c/V3
http://honglus.blogspot.com/2009/03/setup-net-snmp-on-linux-centos-52.html
http://honglus.blogspot.com/2011/03/setup-snmp-v3-usm-with-encryption.html
- on cluster node , install snmpagent and assign a shared fence device for each node
yum install netsnmp-utils
The port is the interface index numer in filer, since interface starts with loopback, eth0 index num=2, eth1 index num=3. It is not hardcoded in fence_device, so each node can share the same fence device but different port

Test
On any node, run "fence_node the-other-node-name", then go to filer to check if one of the filer's storage NICs is down

Do we really need to set partition type to fd(Linux auto raid) for Linux software RAID?

2011-04-26T17:54:00.003+10:00

Almost all Linux RAID documents mandate that partition type must be fd(Linux auto raid) before building Linux software RAID. Actually, this step is optional, it helps a little if your RAID device is /dev/md0 in Centos.
What is fd(Linux auto raid)?
As the name implies, it is for auto detection of raid when OS boots. If you have created /dev/md0 but didn't put it in configuration file /etc/mdadm.conf, OS is able to detect the partitions and assemble /dev/md0.
But, this way of assembling RAID device only works for /dev/md0 in Centos by default.
It is because Centos only enable raidautorun for /dev/md0 by default. Any other md will be assembled by reading /etc/mdadm.conf

[Centos 5 ] $grep -A 3 raidautorun  /etc/rc.sysinit 
[ -x /sbin/nash ] && echo "raidautorun /dev/md0" | nash --quiet
if [ -f /etc/mdadm.conf ]; then
/sbin/mdadm -A -s
fi
#The auto detecting behavior is logged in kernel buffer
$ dmesg | grep -i auto
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.

fd VS RAID superblock
Don't confuse fd with RAID superblock, fd is an optional flag recognized by nash raidautorun command. But RAID superblock is, in every RAID device member, an essential piece of information, which contains RAID level, state and parent MD device UUID (man 4 md).

#Examine superblock on logical device will encounter an error
#It is expected because superblock only exist in RAID member device
 $ mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.

#Examine  superblock on RAID member
$ mdadm --examine /dev/sdb2
/dev/sdb2:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a31e6699:4360a3b7:38c544fa:f4e6faa9
  Creation Time : Wed Apr 27 11:19:34 2011
     Raid Level : raid1
  Used Dev Size : 104320 (101.89 MiB 106.82 MB)
     Array Size : 104320 (101.89 MiB 106.82 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Wed Apr 27 12:51:58 2011
          State : clean
Internal Bitmap : present
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 58c72673 - correct
         Events : 20

#Scan  partitions superblock to find existing raid device.
$ mdadm --examine --brief --scan --config=partitions
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=da55e1e2:c781a461:73d6dfa6:8c7cf6d6
##The above output can be saved to /etc/mdadm.conf; then mdadm -A -s will activate the RAID device.
##DEVICE member list is optional, because default is “DEVICE partitions”.

Conclusion
Partition type FD is a way of assembling raid used by nash raidautorun command and it only works for /dev/md0 in Centos by default.
If you use /etc/mdadm.conf to assemble RAID, the FD flag is optional. But setting this flag can help you to recognize RAID members from “fdisk -l”.