Augeas quickstart

Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.
Configuration files support is provided by “lenses”. The default lenses are provided in augeas-lib package

$ rpm -ql pe-augeas-libs | grep lenses
/opt/puppet/share/augeas/lenses/dist
/opt/puppet/share/augeas/lenses/dist/aliases.aug
…

If your configuration is not listed, you can write your own lenses definition file.
Language bindings are available:
Ruby, Python, OCaml, Perl, Haskell, PHP, Lua, and Java
Augeas has two main directories:
- augeas: augeas configuration directory. e.g control loading lenses
- files: user data directory tree. e.g /etc/hosts is mapped to /files/etc/hosts

$augtool
augtool> ls /
augeas/ = (none)
files/ = (none)

Sample hosts file to be tested

$cat /etc/hosts
# comment test1
# comment test2
172.16.1.11 web1 web1.example.com server1.example.com
#######Augeas translate above hosts file to 
augtool>  print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

Adding a new comment

augtool> set /files/etc/hosts/#comment[last()+1] "comment test3"
augtool> print  /files/etc/hosts/
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/#comment[3] = "comment test3"

########New comment will be added to last line.
########If you want to append it next to #comment[2], use insert
augtool> ins #comment after /files/etc/hosts/#comment[last()]
augtool> 
augtool> set  /files/etc/hosts/#comment[last()] "comment test3"
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

Adding a host entry.
You can’t use last() to auto increase it from hosts/1 to hosts/2, because  “1” or “2” is label name not an index number like #comment[1]/#comment[2].
The trick is to use “01,02,03” etc.

augtool> set /files/etc/hosts/01/ipaddr 172.16.1.12
augtool>set /files/etc/hosts/01/canonical web2
##at this point, it is hosts/01 added
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "coment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/01
/files/etc/hosts/01/ipaddr = "172.16.1.12"
/files/etc/hosts/01/canonical = "web2"

####after save and load, hosts/2 is added
augtool> save
Saved 1 file(s)
augtool> load
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "coment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/2
/files/etc/hosts/2/ipaddr = "172.16.1.12"
/files/etc/hosts/2/canonical = "web2"

Change an entry by referencing to itself
Change the comment “comment test1” to “newcomment test1”

#Change the comment “comment test1” to “newcomment test1” 
augtool> /set files/etc/hosts/#comment[.='comment test1'] 'newcomment test1'
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "newcomment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3

#Change any comment starting with comment to ‘oldcomment’ by regex match
augtool>setm /files/etc/hosts  #comment[.=~regexp('comment.*')] 'oldcomment'
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "newcomment test1"
/files/etc/hosts/#comment[2] = "oldcomment"
/files/etc/hosts/#comment[3] = "oldcomment"

#Usually, regex match is used to delete entries. e.g delete comment test[23]
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3"
augtool> rm /files/etc/hosts/#comment[.=~regexp('comment test[23]')]
rm : /files/etc/hosts/#comment[.=~regexp('comment test[23]')] 2
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment = "comment test1"
augtool>

Change an entry by referencing to relative nodes
e.g change ip of  a host ‘web1’  to 172.16.11.11

augtool> print /files/etc/hosts/1
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
augtool> set /files/etc/hosts/*/ipaddr[../canonical = 'web1'] 172.16.11.11
augtool> print /files/etc/hosts/1
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.11.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

#multiple conditions can be joined together 
augtool> match /files/etc/hosts/*/ipaddr[../canonical = 'web1' and ../alias = 'server1.example.com']
/files/etc/hosts/1/ipaddr = 172.16.11.11
augtool> match /files/etc/hosts/*[canonical = 'web1' and alias = 'server1.example.com']
/files/etc/hosts/1 = (none)

Comment or uncomment a line
Comment or uncomment is difficult for Augeas to implement, it can only be achieved by inserting a new line after the comment line then removing the old line.
You can extend Augeas to include ‘comment’ and ‘uncomment’ functions. The following code is an example for puppet.

##Define a function 
define augeasnew ($file,$line){
$exp=regsubst($line[0], '^(un)?comment *(.*)' , '\2')
case $line[0] {
/^uncomment/: {
exec {"/bin/sed -i -e '/${exp}/s/#//g' $file":
onlyif => "/bin/grep '${exp}' ${file} | /bin/grep '#' ",}
}
/^comment/: {
exec {"/bin/sed -i -e '/${exp}/ s/^/#/' $file":
onlyif => "/bin/grep '${exp}' ${file} | /bin/grep -v '#' ",}
}
default: {
augeas {'augeas-chg-any':
context => "/files/${file}",
changes => $line, }
}
}
}
##execute the function
augeasnew {'chg-hosts-file':
file => '/etc/hosts',
#Mulitple Augeas commands can be stored in an array
#However,none-Augeas command (uncomment or comment) can only be stored in first element of an array.
#line => ["set  1/ipaddr '10.1.1.1'" , "set 2/ipaddr '10.1.1.2'",],
line => ["comment  c line1",],
}

Troubleshooting.
You can check /augeas//error for detailed errors, for example, Augeas can’t load /etc/sudoers

augtool> print /augeas//error
/augeas/files/etc/sudoers/error = "parse_failed"
/augeas/files/etc/sudoers/error/pos = "1998"
/augeas/files/etc/sudoers/error/line = "62"
/augeas/files/etc/sudoers/error/char = "0"
/augeas/files/etc/sudoers/error/lens = "/opt/puppet/share/augeas/lenses/dist/sudoers.aug:478.10-.57:"
/augeas/files/etc/sudoers/error/message = "Iterated lens matched less than it should"
In this case, error is encountered in line 62. 
$sed -n '62p' /etc/sudoers
Defaults   !visiblepw
#You can upgrade Augeas or comment out the line to resolve the issue

Reference:
http://augeas.net/tour.html
http://augeas.net/page/Path_expressions
http://docs.puppetlabs.com/references/2.6.8/function.html
http://docs.puppetlabs.com/guides/language_guide.html

Vsphere PowerCLI script to clone and customize Windows guest OS

VMware can also customize Windows guest OS by Windows sysprep tool, though the process is more complex than Linux guest OS

There are two options to clone and “sysprep” VMware Windows guest OS:
1.Install sysprep tools in Windows guest OS and  run sysprep.exe  in guest OS command line, then clone it by VMware
2.Install sysprep tools in Virutal Center and let VMware tools in Windows guest in to control sysprep process either by GUI or script.  (It seems sysprep rely on VMware tools, so the VMware tools must be installed in guest OS)

Option #2 is the preferred method, because you can use script to easily customize unique information e.g computer name, ip addresses etc.

Install sysprep tools in Virtual Center
VMware KB: Sysprep file locations and versions 
Windows Vista onwards(vista/2008/7/2008R2/) don’t need this step, because its sysprep is built-in.
Create guest customization by GUI 
 Virtual Center -> view-> management -> “customization specification manager”. Create a new customization and select it when asked for guest customization information in GUI clone action. 
Create guest customization by script. 
Customization by GUI is not flexible, customization by script can be created on the fly. The following is enhancement to the clonevm.ps1 in Vsphere PowerCLI script to clone and customize Linux guest OS, just replace the “Identity for Linux” part with following code block 

$s_ostype=Retrieve-values $lines  "ostype"
if ( $s_ostype -eq "linux" ) { 
## Identity for Linux 
$vmclonespec_os.identity= New-Object VMware.Vim.CustomizationLinuxPrep
$vmclonespec_os.identity.hostname= New-Object VMware.Vim.CustomizationFixedName
$vmclonespec_os.identity.hostname.name= $s_dstname
$vmclonespec_os.identity.domain=$s_domain
}
elseif ( $s_ostype -eq "windows" ) {
# WinOptions
$vmclonespec_os.Options = New-Object  VMware.Vim.CustomizationWinOptions
$vmclonespec_os.Options.ChangeSID = 1
#sysprep
$vmclonespec_os.identity = New-Object VMware.Vim.CustomizationSysprep
# GUIUnattended
$vmclonespec_os.Identity.GuiUnattended = New-Object VMware.Vim.CustomizationGuiUnattended
$vmclonespec_os.Identity.GuiUnattended.AutoLogon = 0
#timezone codes: http://msdn.microsoft.com/en-us/library/ms145276(v=sql.90).aspx
$vmclonespec_os.Identity.GuiUnattended.TimeZone  = 255
$vmclonespec_os.Identity.GuiUnattended.Password = New-Object VMware.Vim.CustomizationPassword
$vmclonespec_os.Identity.GuiUnattended.Password.PlainText = 1
$vmclonespec_os.Identity.GuiUnattended.Password.Value = "Secret01"
# Identification
$vmclonespec_os.Identity.Identification = New-Object VMware.Vim.CustomizationIdentification
$vmclonespec_os.Identity.Identification.joinWorkgroup = "workgroup2"
## Userdata
$vmclonespec_os.identity.userData = New-Object VMware.Vim.CustomizationUserData
$vmclonespec_os.identity.userData.computerName = New-Object VMware.Vim.CustomizationFixedName
$vmclonespec_os.identity.userData.computerName.name = $s_dstname
$vmclonespec_os.Identity.UserData.FullName = "Administrator"
$vmclonespec_os.Identity.UserData.OrgName="myOrg"
$vmclonespec_os.Identity.UserData.Productid=""
}
else {
write-host "Unknown ostype: $s_ostype. Please set it to linux or windows"
exit
}
}

NOTES: 
 - The guest OS(Windows/Linux) will be forcefully rebooted by Vmware tools again in ~1 minute after you power it on, so don’t login in a hurry to check the result.
 - The script only works in a live session to Virtualcenter, it doesn’t work in a direct login session to ESX host.

Bootstrap Puppet Enterprise (PE) Puppet Client for RHEL

Puppet Enterprise(PE) packs all required packages, modules into a tar file.
The file also includes a shell installation script to install packages and modules and do other customizations tasks. So only installing agent packages “pe-puppet pe-mcollective” can’t setup agent station properly.
It is ideal to have PE agent installed automatically along with OS,the installer script can use answer file to automate the process.
The following codes can be put in RHEL kickstart postrun section to bootstrap PE agent.

#Download Puppet Enterprise(PE) (manage up to 10 nodes free)
#http://puppetlabs.com/puppet/puppet-enterprise/
#e.g  puppet-enterprise-2.0.0-el-5-x86_64.tar.gz
...
%post
PKG=puppet-enterprise-2.0.0-el-5-x86_64
cd /tmp && wget -O ${PKG}.tar.gz http://172.16.1.1/boot/os/rhel-5.6-x64/${PKG}.tar.gz  && tar -zxf ${PKG}.tar.gz  && cd /tmp/${PKG} && \
cat >answerfile<<END
q_install=y
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_install=n
q_puppet_symlinks_install=y
q_puppetagent_certname=$(hostname -s)
q_puppetagent_install=y
q_puppetagent_server=puppet
q_puppetmaster_install=n
q_vendor_packages_install=n
q_install=y
END
[ -e answerfile ] && ./puppet-enterprise-installer -a answerfile  2>&1 | tee  /tmp/install_${PKG}.log
rm -rf /tmp/${PKG}*
sync
exit 0

Connect to puppet server after installation finished

#Verify puppet agent is running
[Puppet Client]$/etc/init.d/pe-puppet status
#Puppet server should be able to see the client’s pending certificate request
[Puppet Master]$puppet cert list
web2 (A1:08:15:EF:1F:5D:F9:C5:D9:A0:F3:F2:FD:FF:CE:09)
#Approve the client by signing the cert request
[Puppet Master]$puppet cert sign web2

Build a Linux PXE server to provision Linux and Windows Servers

Windows Deployment Service (WDS)/SCCM is the traditional way to provision  Windows servers, However if you need to provision both Linux and Windows Servers, it would be nice to have a all-in-one server. Before Windows Preinstallation Environment (WinPE) was developed, disk imaging tool like ghost is used to deploy syspreped windows OS as non-windows based provision solution. WinPE is part of Windows Automated Installation Kit(WAIK), it includes, imagex, a tool to capture image on file system level rather than raw sector level.

The PXE Server presented here has the following features:

- Support any PXE capable Linux distribution and most Windows versions: 2000/2003/XP/Vista/7/2008/2008 R2. Potentially, support any other PXE capable server e.g Solaris
- Zero touch installation/full automation
- Dynamic loading configuration using CGI script of your choice PHP/Perl ...
- Downloading kernel/images over http which is more reliable and faster than tftp, additionally, http makes it possible to provision over remote site

Environment setup

7 VMS in Virtualbox 4( for testing provision Centos 5.5, Windows 2003, and Windows 2008 R2)
- PXE server: Centos 5.5
- PXE Linux Client: Centos 5.5
- WAIK tools computer: Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2003 R2
- Imagex/sysprep source: Windows 2008 R2
- Imagex/sysprep source: Windows 2003 R2
Notes:
- Select PCnet-fast III network adapter to support PXE boot
- Windows VM must have at least 1G RAM, because WinPE ISO memdisk will consume few hundreds of MB
- Setup samba share in PXEServer to store Windows imagex(.wim ) files,autoscript for disk partitioning/apply .wim image etc(autorun.bat).
Install TFTP/DHCP
Refer to http://honglus.blogspot.com/2011/06/setup-pxe-boot-server-for-linux-server.html
Just the filename parameter need to be set to:  filename "gpxelinux.0";
Install pxeserver(syslinux)
The original syslinux package in Centos 5.5 doesn't support memdisk, I use syslinux-4.03
Upgrade to syslinux-4.03 with RPM binary package, but the gpxelinux.0 need to customized and compiled from source.
gPXE/pxelinux backgroud
gPXE(http://etherboot.org/wiki/index.php) or it is successor iPXE (http://www.ipxe.org/) support dynamic configuration using any CGI script and support kernel/images loading over http.
Because gPXE/iPXE doesn't support memdisk, which is essential for loading winPE ISO image, we have to use pxelinux in syslinux. Fortunately, syslinux integrates gPXE, so we can switch between gPXE and pxelinux freely using CGI script. That is, use gPXE for Linux, use pxelinux for Windows
Compile customized gPXE in syslinux source file

$cd ${syslinux source}/gpxe/ 
$vi pxelinux.gpxe
#!gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}
# "!gpxe" is interpreter, not a comment line 
# chain http://pxe-server/boot/home.php?mac=${net0/mac}?will pass the MAC address to a CGI script, which can generate individual configuration based on MAC address
#To use the DNS name, pxe-server, a working DNS server need to be advertised to DHCP client.
$make
$cp gpxelinux.0 /tftpboot/

How does the CGI script work?

A CGI script, php in this example, retrieve the MAC address passed by gPXE request, lookup host-os mapping table, then generate PXE configuration dynamically.

#The configuration file to define which OS to apply based on MAC address
$cat ${wwwroot}/boot/hosts/host-os.conf 
#Mac               os
08:00:27:4d:4d:aa, centos-5.5-i386
08:00:27:b2:40:3e, windows-2008r2-x64

#Linux Server can use gPXE directly
$curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
#!gpxe
imgfree
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img 
boot

#Windows server has to rely on pxelinux to support memdisk(part of syslinux package)
$curl http://pxe-server/boot/home.php?mac=08:00:27:b2:40:3e
#!gpxe
imgload pxelinux.0
boot pxelinux.0

#A pxelinux config is generated in the mean time 
$cat  /tftpboot/pxelinux.cfg/01-08-00-27-b2-40-3e 
default pe
label  pe
kernel http://pxe-server/boot/os/memdisk
append iso
initrd http://pxe-server/boot/os/w2k8_pe_x64.iso

Implementing Zero touch installation/full automation

Booting up the server via PXE is the first phrase of zero touch installation, OS level automation is done by Linux unattended install: e.g. Kickstart for Redhat Linux family or Windows unattended install:syspreped windows, startnet.cmd in winPE, autoscript in a network share
The following focus on Windows unattended install.
Download WAIK for Windows 7/Windows 2008 R2
http://www.microsoft.com/download/en/details.aspx?id=5753
How can you use WAIK for Windows 2008 R2 to deploy Windows2000/xp/2003
 It is true that you can't run setup.exe of Windows 2003 in 2008 R2 winPE, but we use imagex tool to capture files, so it works on most Windows OS.
Create winPE ISO
Install the WAIK  in WAIK tools computer
Steps to create winPE ISO

#Open the deployment tools command prompt
#amd64 is for 64bit, use x86 for 32bit
$\>copype.cmd amd64 C:\winpe_amd64
####Optional:Mount the wim file in order to add auto deploy script, you can skip this step to make it easier to get it going.
####BEGIN script customization
$\>imagex /mountrw c:\winpe_amd64\winpe.wim 1 c:\winpe_amd64\mount
#modify C:\winpe_amd64\mount\Windows\System32\starnet.cmd, add following lines below 'wpeinit'
REM -- Start
REM Mount Linux samba share
net use J: \\pxe-server\windows-media
REM go to sub-dir(dirname is the mac address) of the samba share and run the auto script
J:
for /F  "usebackq tokens=2 delims=:"  %%i IN (`ipconfig /all ^| find /i "physical address" `) DO cd %%i
autorun.bat
REM -- end
$\>imagex /unmount c:\winpe_amd64\mount /commit
#####END script customization
$\>copy "C:\Program Files\Windows AIK\Tools\amd64\imagex.exe"  C:\winpe_amd64\ISO\
$\>copy C:\winpe_amd64\winpe.wim C:\winpe_amd64\ISO\sources\boot.wim
$\>Oscdimg -n -bC:\winpe_amd64\Etfsboot.com C:\winpe_amd64\ISO C:\winpe_amd64\winpe_amd64.iso

How to do sysprep in Windows 2003?
On imagex/sysprep source computer:
Download the latest sysprep tool: http://www.microsoft.com/download/en/details.aspx?id=14830
WindowsServer2003-KB926028-v2-x86-ENU.exe /x to extract the deploy.cab file
Extract files in deploy.cab to c:\sysprep
Run setupmgr.exe to create answer file c:\sysprep\Sysprep.inf
Run sysprep /reseal to prepare the OS for imaging
How to do sysprep in Windows 2008 R2?
On WAIK tool computer:
Open Windows system image manager?in WAIK to create unattended answer file
Click New answer file?to select a windows catalogue file (.clg) in installation media \Deploy\Operating Systems\Windows Server 2008 R2 x64\sources\.
In the Windows images?panel, right click various parameters to add to answer file
Detailed instruction:
http://briandesmond.com/blog/how-to-sysprep-in-windows-server-2008-r2-and-windows-7/
On imagex/sysprep source computer:
Retrieve above new answer file *.xml, Run sysprep /generalize /oobe /shutdown /unattend:unattend.xml?
Capture image in imagex/sysprep source computer
This is manual process, because it is once-off job
Once the imagex/sysprep source computer is shutdown, Reboot it with winpe_amd64.iso by mounting to CDROM or loading via PXE
#The files source is the partition with windows folder, It could be d:, because W2k8 assign c: to the small system partition by default, which  doesn't need to be captured.
$\>imagex /compress fast  /capture [c:|d:]  $sambashare:\boot.wim "w2k8 R2 64bit"
Copy the boot.wim to samba share, which will be used by imagex /apply.
Apply image in imagex/sysprep target computer
This need to be full automatic process, boot winpe_amd64.iso via PXE. Once winPE is initialized, it will mount the sambashare, cd to its own directory and run auorun.bat, which will partition disk and apply the image.

It is zero touch installation, no human intervention is needed, however, unlike kickstart, computer name/IP address etc can't be configured automatically. Well, in theory, you can specify the information in sysprep then re-capture image for each unique configuration, but it defeats the purpose of saving time in provision. The other way is fresh installation by giving answer file in samba share to setup.exe.
 But it can only be used for Visa/2008/7/2008 R2, because there is no winPE which support running setup.exe of pre-Vista Windows

$ ls $sambashare\08-00-27-b2-40-3e
autorun.bat  boot.wim  diskpart.txt
$ cat $sambashare\08-00-27-b2-40-3e\autorun.bat 
diskpart /s diskpart.txt
PATH=%PATH%;c:;d:;e:;f:
imagex /apply boot.wim 1 c: && bcdboot c:\windows /s c: && exit
#bcdboot c:\windows is not needed for pre-Vista Windows, but it does no harm.

It is done, if you have setup answer file properly, the imagex/sysprep target computer should be able to boot smoothly without any human intervention.
Issues  encountered:
Q: DNS name is not resolvable in gPXE.
A: Make sure set use-cached 0 is set in pxelinux.gpxe, then recompile
$vi pxelinux.gpxe
#!gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}

Q: gPXE couldn't boot Linux kernel with error:Kernel panic not syncing: VFS: unable to mount root fs
A:Make sure imgfree is the first statement in Linux pxe config
 $curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
#!gpxe
imgfree
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img
boot

Q:Error in imagex /apply after booting winPE via PXE: Not Enough Server Storage is available to process is command
A: Make sure the Windows VM have at least 1G RAM, because winPE ISO will consume a few hundreds MB as memdisk

Q:What if the winPE doesn't recognize the device (RAID/NIC) with default drivers.
A:Add customize driver to winPE, refer to http://technet.microsoft.com/en-us/library/dd744371%28WS.10%29.aspx

Collectl, an all-in-one tool for collecting Linux statistical data

Collectl,collect for Linux, is a single tool which integrates functions of various tools:sar,iostat,mpstat,top,slaptop,netstat,nfstat,ps .. .
- Supported: Linux
- Requirement: Perl
Collectl features:
- run in command line or run as daemon
- Various output formats: raw,gunplot,gexprt(ganglia),sexpr,lexpr,csv(--sep ,)
- Send data to other programs (ganglia) remotely via socket instead of writing to a file
- IPMI monitoring for fans and temperature sensors
- Support module (Perl scripts)  for customized checks
- Monitor process’s disk read/write, find the top processes keeping disk busy
The last one is the most impressive feature, I haven’t found other Linux tools can do it. (DTtrace can in Solaris)
collectl  examples

#help, all options 
$collect –x
#-s?, what to monitor:c – cpu  d – disk “collectl   --showsubsys”
#-c 5 : collect 5 samples and exit
#-oT:  T - preface output with time only ; “collectl   --showoptions”
$collectl   -sc -c5 -i2 --verbose -oT
waiting for 2 second sample...
# CPU SUMMARY (INTR, CTXSW & PROC /sec)
#Time      User  Nice   Sys  Wait   IRQ  Soft Steal  Idle  CPUs  Intr  Ctxsw  Proc  RunQ   Run   Avg1  Avg5 Avg15
12:39:34      0     0     0     0     0     1     0    97     1  1082     23     0    76     1   0.42  0.42  0.44
12:39:36      0     0     0     0     0     1     0    97     1  1088     24     0    76     1   0.42  0.42  0.44

The following demonstrates how collectl identify the process reading/writing most data to disk
#Hammer disk by writing 50mb data with dd

$dd if=/dev/urandom of=test bs=1k count=50000
#collectl identifies the “dd” process
#in top mode, sort by  “iokb   total I/O KB” ; “collectl –showtopopts”
$collectl -i2  --top iokb
TOP PROCESSES sorted by iokb (counters are /sec) 12:50:31
# PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct  AccuTime  RKB  WKB MajF MinF Command
6861  root     18  6784    0 R    3M  572K  0  0.91  0.00  45   0:00.91    0 3680    0   97 dd
1  root     15     0    0 S    2M  632K  0  0.00  0.00   0   0:28.21    0    0    0    0 init
2  root     RT     1    0 S     0     0  0  0.00  0.00   0   0:00.00    0    0    0    0 migration/0

Understanding Red Hat Linux recovery runlevels

If Linux system can boot but hang during starting a service, booting to “recovery runlevels” can skip the service and gain shell to troubleshoot.
If Linux system can’t boot at all,  booting from rescue CD (first installation media) and type “linux rescue” to gain shell to troubleshoot
Red Hat Linux boot order

The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/inittab->
/etc/rc.d/rc.sysinit->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S

Recovery runlevels
- runlevel  1
Execute up to /etc/rc.d/rc.sysinit and /etc/rc.d/rc1.d/
Runlevel 1 is identical to singleuser mode. It is switched to singleuser mode in last step, just a number of trivial scripts executed before that.
 $ls  /etc/rc.d/rc1.d/S*
 /etc/rc.d/rc1.d/S02lvm2-monitor  /etc/rc.d/rc1.d/S13cpuspeed  /etc/rc.d/rc1.d/S99singlesingleuser

- single
Execute up to /etc/rc.d/rc.sysinit

- Emergency
Does not execute /etc/rc.d/rc.sysinit.
 Because rc.sysinit is not executed, file system is mounted in read-only mode. You need run “mount –o rw,remount /” to remount it in read-write mode.
emergency runlevel is Red Hat term, it is identical to  “init=/bin/sh” in any Linux distribution
How to go to a  runlevel
In the grub menu, type “a” to append one of following options to boot line.
1  
single  
emergency   
init=/bin/sh
When Centos hung on starting up boot services, how to get to shell without rescue CD
RHCE Notes - Troubleshooting booting issue

Advanced RPM topics

Query
“queryformat”  option can query every piece information of a rpm package, the  information tags (macros ) are returned  by “rpm –querytags” command

#list top 2 rpm packages sorted by installation time
$rpm -qa  | xargs -I{} rpm -q --queryformat "{}        %{installtime}\n" {} | sort -rn -k2 | head -2
collectl-3.5.1-1        1317864013
git-1.7.4.1-1.el5        1316484590
#unfortunately the time returned is unixtime.  You can convert it to human readable format by  “date –d @timestring” e.g 
$date -d @1317864013
Thu Oct  6 12:20:13 EST 2011
#but there is a shortcut  “--last”
$ rpm -qa --last  | head -2
collectl-3.5.1-1                              Thu 06 Oct 2011 12:20:13 PM EST
git-1.7.4.1-1.el5                             Tue 20 Sep 2011 12:09:50 PM EST

"rpm -qa" supports regular expression itself, rather than pipe to grep e.g “rpm -qa | grep perl”
“rpm –qa perl\*” also works. There is no improvement on speed but typing become lesser.
requires and provides
#You can check the package dependency before install the package

$ rpm -qp --requires git-1.7.6-1.el5.rf.i386.rpm
..
libssl.so.6 
…
#To meet the dependency, you want to check who provides libssl.so.6 
$yum whatprovides libssl.so.6
openssl-0.9.8e-20.el5.i686 : The OpenSSL toolkit
Repo        : base
Matched from:
Other       : libssl.so.6
#if openssl has been installed, “rpm -q –whatprovides” can also provide the answer
$rpm -q --whatprovides libssl.so.6
openssl-0.9.8e-12.el5_4.6

rpm scriptlets

#query all nopre|nopost|nopreun|nopostun  scripts
$rpm -q --scripts xinetd
postinstall scriptlet (using /bin/sh):
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
fi
preuninstall scriptlet (using /bin/sh):
if [ $1 = 0 ]; then
/sbin/service xinetd stop > /dev/null 2>&1
/sbin/chkconfig --del xinetd
fi
postuninstall scriptlet (using /bin/sh):
if [ $1 -ge 1 ]; then
/sbin/service xinetd condrestart >/dev/null 2>&1
Fi
#query postinstall script only
$ rpm -q --queryformat "%{POSTIN}"  xinetd
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
#Don’t run the scripts during install/remove
rpm –i –noscripts|nopre|nopost|nopreun|nopostun   pkgname
rpm –e –noscripts|nopre|nopost|nopreun|nopostun   pkgname

Extract rpm contents without install

#use rpm2cpio to extract everything
$mkdir /tmp/epel
$ cd /tmp/epel
$ rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd
./etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
..
#use rpm2cpio to extract particular file
$rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd  ./usr/share/doc/epel-release-5
#another way is to use rpm install with alternative root
$ rpm --root /tmp/epel/ -ivh  --nodeps /root/epel-release-5-4.noarch.rpm

Recover corrupted rpm database
Build RPM from source file