Thursday, January 5, 2012

Bootstrap Puppet Enterprise (PE) Puppet Client for RHEL

Puppet Enterprise(PE) packs all required packages, modules into a tar file.
The file also includes a shell installation script to install packages and modules and do other customizations tasks. So only installing agent packages “pe-puppet pe-mcollective” can’t setup agent station properly.
It is ideal to have PE agent installed automatically along with OS,the installer script can use answer file to automate the process.
The following codes can be put in RHEL kickstart postrun section to bootstrap PE agent.
#Download Puppet Enterprise(PE) (manage up to 10 nodes free)
#http://puppetlabs.com/puppet/puppet-enterprise/
#e.g  puppet-enterprise-2.0.0-el-5-x86_64.tar.gz
...
%post
PKG=puppet-enterprise-2.0.0-el-5-x86_64
cd /tmp && wget -O ${PKG}.tar.gz http://172.16.1.1/boot/os/rhel-5.6-x64/${PKG}.tar.gz  && tar -zxf ${PKG}.tar.gz  && cd /tmp/${PKG} && \
cat >answerfile<<END
q_install=y
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_install=n
q_puppet_symlinks_install=y
q_puppetagent_certname=$(hostname -s)
q_puppetagent_install=y
q_puppetagent_server=puppet
q_puppetmaster_install=n
q_vendor_packages_install=n
q_install=y
END
[ -e answerfile ] && ./puppet-enterprise-installer -a answerfile  2>&1 | tee  /tmp/install_${PKG}.log
rm -rf /tmp/${PKG}*
sync
exit 0

Connect to puppet server after installation finished

#Verify puppet agent is running
[Puppet Client]$/etc/init.d/pe-puppet status
#Puppet server should be able to see the client’s pending certificate request
[Puppet Master]$puppet cert list
web2 (A1:08:15:EF:1F:5D:F9:C5:D9:A0:F3:F2:FD:FF:CE:09)
#Approve the client by signing the cert request
[Puppet Master]$puppet cert sign web2

Wednesday, December 7, 2011

Build a Linux PXE server to provision Linux and Windows Servers

Windows Deployment Service (WDS)/SCCM is the traditional way to provision  Windows servers, However if you need to provision both Linux and Windows Servers, it would be nice to have a all-in-one server. Before Windows Preinstallation Environment (WinPE) was developed, disk imaging tool like ghost is used to deploy syspreped windows OS as non-windows based provision solution. WinPE is part of Windows Automated Installation Kit(WAIK), it includes, imagex, a tool to capture image on file system level rather than raw sector level.

The PXE Server presented here has the following features:

- Support any PXE capable Linux distribution and most Windows versions: 2000/2003/XP/Vista/7/2008/2008 R2. Potentially, support any other PXE capable server e.g Solaris
- Zero touch installation/full automation
- Dynamic loading configuration using CGI script of your choice PHP/Perl ...
- Downloading kernel/images over http which is more reliable and faster than tftp, additionally, http makes it possible to provision over remote site

Environment setup

7 VMS in Virtualbox 4( for testing provision Centos 5.5, Windows 2003, and Windows 2008 R2)
- PXE server: Centos 5.5
- PXE Linux Client: Centos 5.5
- WAIK tools computer: Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2008 R2
- PXE Windows Client(imagex/sysprep target): Windows 2003 R2
- Imagex/sysprep source: Windows 2008 R2
- Imagex/sysprep source: Windows 2003 R2
Notes:
- Select PCnet-fast III network adapter to support PXE boot
- Windows VM must have at least 1G RAM, because WinPE ISO memdisk will consume few hundreds of MB
- Setup samba share in PXEServer to store Windows imagex(.wim ) files,autoscript for disk partitioning/apply .wim image etc(autorun.bat).
Install TFTP/DHCP
Refer to http://honglus.blogspot.com/2011/06/setup-pxe-boot-server-for-linux-server.html
Just the filename parameter need to be set to:  filename "gpxelinux.0";
Install pxeserver(syslinux)
The original syslinux package in Centos 5.5 doesn't support memdisk, I use syslinux-4.03
Upgrade to syslinux-4.03 with RPM binary package, but the gpxelinux.0 need to customized and compiled from source.
gPXE/pxelinux backgroud
gPXE(http://etherboot.org/wiki/index.php) or it is successor iPXE (http://www.ipxe.org/) support dynamic configuration using any CGI script and support kernel/images loading over http.
Because gPXE/iPXE doesn't support memdisk, which is essential for loading winPE ISO image, we have to use pxelinux in syslinux. Fortunately, syslinux integrates gPXE, so we can switch between gPXE and pxelinux freely using CGI script. That is, use gPXE for Linux, use pxelinux for Windows
Compile customized gPXE in syslinux source file
$cd ${syslinux source}/gpxe/ 
$vi pxelinux.gpxe
#!gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}
# "!gpxe" is interpreter, not a comment line 
# chain http://pxe-server/boot/home.php?mac=${net0/mac}?will pass the MAC address to a CGI script, which can generate individual configuration based on MAC address
#To use the DNS name, pxe-server, a working DNS server need to be advertised to DHCP client.
$make
$cp gpxelinux.0 /tftpboot/
How does the CGI script work?

A CGI script, php in this example, retrieve the MAC address passed by gPXE request, lookup host-os mapping table, then generate PXE configuration dynamically.

#The configuration file to define which OS to apply based on MAC address
$cat ${wwwroot}/boot/hosts/host-os.conf 
#Mac               os
08:00:27:4d:4d:aa, centos-5.5-i386
08:00:27:b2:40:3e, windows-2008r2-x64

#Linux Server can use gPXE directly
$curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
#!gpxe
imgfree
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img 
boot

#Windows server has to rely on pxelinux to support memdisk(part of syslinux package)
$curl http://pxe-server/boot/home.php?mac=08:00:27:b2:40:3e
#!gpxe
imgload pxelinux.0
boot pxelinux.0

#A pxelinux config is generated in the mean time 
$cat  /tftpboot/pxelinux.cfg/01-08-00-27-b2-40-3e 
default pe
label  pe
kernel http://pxe-server/boot/os/memdisk
append iso
initrd http://pxe-server/boot/os/w2k8_pe_x64.iso
Implementing Zero touch installation/full automation

Booting up the server via PXE is the first phrase of zero touch installation, OS level automation is done by Linux unattended install: e.g. Kickstart for Redhat Linux family or Windows unattended install:syspreped windows, startnet.cmd in winPE, autoscript in a network share
The following focus on Windows unattended install.
Download WAIK for Windows 7/Windows 2008 R2
http://www.microsoft.com/download/en/details.aspx?id=5753
How can you use WAIK for Windows 2008 R2 to deploy Windows2000/xp/2003
 It is true that you can't run setup.exe of Windows 2003 in 2008 R2 winPE, but we use imagex tool to capture files, so it works on most Windows OS.
Create winPE ISO
Install the WAIK  in WAIK tools computer
Steps to create winPE ISO
#Open the deployment tools command prompt
#amd64 is for 64bit, use x86 for 32bit
$\>copype.cmd amd64 C:\winpe_amd64
####Optional:Mount the wim file in order to add auto deploy script, you can skip this step to make it easier to get it going.
####BEGIN script customization
$\>imagex /mountrw c:\winpe_amd64\winpe.wim 1 c:\winpe_amd64\mount
#modify C:\winpe_amd64\mount\Windows\System32\starnet.cmd, add following lines below 'wpeinit'
REM -- Start
REM Mount Linux samba share
net use J: \\pxe-server\windows-media
REM go to sub-dir(dirname is the mac address) of the samba share and run the auto script
J:
for /F  "usebackq tokens=2 delims=:"  %%i IN (`ipconfig /all ^| find /i "physical address" `) DO cd %%i
autorun.bat
REM -- end
$\>imagex /unmount c:\winpe_amd64\mount /commit
#####END script customization
$\>copy "C:\Program Files\Windows AIK\Tools\amd64\imagex.exe"  C:\winpe_amd64\ISO\
$\>copy C:\winpe_amd64\winpe.wim C:\winpe_amd64\ISO\sources\boot.wim
$\>Oscdimg -n -bC:\winpe_amd64\Etfsboot.com C:\winpe_amd64\ISO C:\winpe_amd64\winpe_amd64.iso

How to do sysprep in Windows 2003?
On imagex/sysprep source computer:
Download the latest sysprep tool: http://www.microsoft.com/download/en/details.aspx?id=14830
WindowsServer2003-KB926028-v2-x86-ENU.exe /x to extract the deploy.cab file
Extract files in deploy.cab to c:\sysprep
Run setupmgr.exe to create answer file c:\sysprep\Sysprep.inf
Run sysprep /reseal to prepare the OS for imaging
How to do sysprep in Windows 2008 R2?
On WAIK tool computer:
Open Windows system image manager?in WAIK to create unattended answer file
Click New answer file?to select a windows catalogue file (.clg) in installation media \Deploy\Operating Systems\Windows Server 2008 R2 x64\sources\.
In the Windows images?panel, right click various parameters to add to answer file
Detailed instruction:
http://briandesmond.com/blog/how-to-sysprep-in-windows-server-2008-r2-and-windows-7/
On imagex/sysprep source computer:
Retrieve above new answer file *.xml, Run sysprep /generalize /oobe /shutdown /unattend:unattend.xml?
Capture image in imagex/sysprep source computer
This is manual process, because it is once-off job
Once the imagex/sysprep source computer is shutdown, Reboot it with winpe_amd64.iso by mounting to CDROM or loading via PXE
#The files source is the partition with windows folder, It could be d:, because W2k8 assign c: to the small system partition by default, which  doesn't need to be captured.
$\>imagex /compress fast  /capture [c:|d:]  $sambashare:\boot.wim "w2k8 R2 64bit"
Copy the boot.wim to samba share, which will be used by imagex /apply.
Apply image in imagex/sysprep target computer
This need to be full automatic process, boot winpe_amd64.iso via PXE. Once winPE is initialized, it will mount the sambashare, cd to its own directory and run auorun.bat, which will partition disk and apply the image.

It is zero touch installation, no human intervention is needed, however, unlike kickstart, computer name/IP address etc can't be configured automatically. Well, in theory, you can specify the information in sysprep then re-capture image for each unique configuration, but it defeats the purpose of saving time in provision. The other way is fresh installation by giving answer file in samba share to setup.exe.
 But it can only be used for Visa/2008/7/2008 R2, because there is no winPE which support running setup.exe of pre-Vista Windows

$ ls $sambashare\08-00-27-b2-40-3e
autorun.bat  boot.wim  diskpart.txt
$ cat $sambashare\08-00-27-b2-40-3e\autorun.bat 
diskpart /s diskpart.txt
PATH=%PATH%;c:;d:;e:;f:
imagex /apply boot.wim 1 c: && bcdboot c:\windows /s c: && exit
#bcdboot c:\windows is not needed for pre-Vista Windows, but it does no harm.

It is done, if you have setup answer file properly, the imagex/sysprep target computer should be able to boot smoothly without any human intervention.
Issues  encountered:
Q: DNS name is not resolvable in gPXE.
A: Make sure set use-cached 0 is set in pxelinux.gpxe, then recompile
$vi pxelinux.gpxe
#!gpxe
set use-cached 0
dhcp net0
chain http://pxe-server/boot/home.php?mac=${net0/mac}

Q: gPXE couldn't boot Linux kernel with error:Kernel panic not syncing: VFS: unable to mount root fs
A:Make sure imgfree is the first statement in Linux pxe config
 $curl http://pxe-server/boot/home.php?mac=08:00:27:4d:4d:aa
#!gpxe
imgfree
kernel http://pxe-server/boot/os/centos-5.5-i386/pxeboot/vmlinuz ksdevice=link  ks=http://pxe-server/boot/hosts/linux-ks/08-00-27-4d-4d-aa.txt
initrd http://pxe-server/boot/os/centos-5.5-i386/pxeboot/initrd.img
boot

Q:Error in imagex /apply after booting winPE via PXE: Not Enough Server Storage is available to process is command
A: Make sure the Windows VM have at least 1G RAM, because winPE ISO will consume a few hundreds MB as memdisk

Q:What if the winPE doesn't recognize the device (RAID/NIC) with default drivers.
A:Add customize driver to winPE, refer to http://technet.microsoft.com/en-us/library/dd744371%28WS.10%29.aspx

Thursday, October 13, 2011

Collectl, an all-in-one tool for collecting Linux statistical data

Collectl,collect for Linux, is a single tool which integrates functions of various tools:sar,iostat,mpstat,top,slaptop,netstat,nfstat,ps .. .
- Supported: Linux
- Requirement: Perl
Collectl features:
- run in command line or run as daemon
- Various output formats: raw,gunplot,gexprt(ganglia),sexpr,lexpr,csv(--sep ,)
- Send data to other programs (ganglia) remotely via socket instead of writing to a file
- IPMI monitoring for fans and temperature sensors
- Support module (Perl scripts)  for customized checks
- Monitor process’s disk read/write, find the top processes keeping disk busy
The last one is the most impressive feature, I haven’t found other Linux tools can do it. (DTtrace can in Solaris)
collectl  examples
#help, all options 
$collect –x
#-s?, what to monitor:c – cpu  d – disk “collectl   --showsubsys”
#-c 5 : collect 5 samples and exit
#-oT:  T - preface output with time only ; “collectl   --showoptions”
$collectl   -sc -c5 -i2 --verbose -oT
waiting for 2 second sample...
# CPU SUMMARY (INTR, CTXSW & PROC /sec)
#Time      User  Nice   Sys  Wait   IRQ  Soft Steal  Idle  CPUs  Intr  Ctxsw  Proc  RunQ   Run   Avg1  Avg5 Avg15
12:39:34      0     0     0     0     0     1     0    97     1  1082     23     0    76     1   0.42  0.42  0.44
12:39:36      0     0     0     0     0     1     0    97     1  1088     24     0    76     1   0.42  0.42  0.44

The following demonstrates how collectl identify the process reading/writing most data to disk
#Hammer disk by writing 50mb data with dd
$dd if=/dev/urandom of=test bs=1k count=50000
#collectl identifies the “dd” process
#in top mode, sort by  “iokb   total I/O KB” ; “collectl –showtopopts”
$collectl -i2  --top iokb
TOP PROCESSES sorted by iokb (counters are /sec) 12:50:31
# PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct  AccuTime  RKB  WKB MajF MinF Command
6861  root     18  6784    0 R    3M  572K  0  0.91  0.00  45   0:00.91    0 3680    0   97 dd
1  root     15     0    0 S    2M  632K  0  0.00  0.00   0   0:28.21    0    0    0    0 init
2  root     RT     1    0 S     0     0  0  0.00  0.00   0   0:00.00    0    0    0    0 migration/0

Tuesday, October 11, 2011

Understanding Red Hat Linux recovery runlevels

If Linux system can boot but hang during starting a service, booting to “recovery runlevels” can skip the service and gain shell to troubleshoot.
If Linux system can’t boot at all,  booting from rescue CD (first installation media) and type “linux rescue” to gain shell to troubleshoot
Red Hat Linux boot order
The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/inittab->
/etc/rc.d/rc.sysinit->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S
Recovery runlevels
- runlevel  1
Execute up to /etc/rc.d/rc.sysinit and /etc/rc.d/rc1.d/
Runlevel 1 is identical to singleuser mode. It is switched to singleuser mode in last step, just a number of trivial scripts executed before that.
 $ls  /etc/rc.d/rc1.d/S*
 /etc/rc.d/rc1.d/S02lvm2-monitor  /etc/rc.d/rc1.d/S13cpuspeed  /etc/rc.d/rc1.d/S99singlesingleuser

- single
Execute up to /etc/rc.d/rc.sysinit

- Emergency
Does not execute /etc/rc.d/rc.sysinit.
 Because rc.sysinit is not executed, file system is mounted in read-only mode. You need run “mount –o rw,remount /” to remount it in read-write mode.
emergency runlevel is Red Hat term, it is identical to  “init=/bin/sh” in any Linux distribution
How to go to a  runlevel
In the grub menu, type “a” to append one of following options to boot line.
1  
single  
emergency   
init=/bin/sh
When Centos hung on starting up boot services, how to get to shell without rescue CD
RHCE Notes - Troubleshooting booting issue

Advanced RPM topics

Query
“queryformat”  option can query every piece information of a rpm package, the  information tags (macros ) are returned  by “rpm –querytags” command
#list top 2 rpm packages sorted by installation time
$rpm -qa  | xargs -I{} rpm -q --queryformat "{}        %{installtime}\n" {} | sort -rn -k2 | head -2
collectl-3.5.1-1        1317864013
git-1.7.4.1-1.el5        1316484590
#unfortunately the time returned is unixtime.  You can convert it to human readable format by  “date –d @timestring” e.g 
$date -d @1317864013
Thu Oct  6 12:20:13 EST 2011
#but there is a shortcut  “--last”
$ rpm -qa --last  | head -2
collectl-3.5.1-1                              Thu 06 Oct 2011 12:20:13 PM EST
git-1.7.4.1-1.el5                             Tue 20 Sep 2011 12:09:50 PM EST

"rpm -qa" supports regular expression itself, rather than pipe to grep e.g “rpm -qa | grep perl”
“rpm –qa perl\*” also works. There is no improvement on speed but typing become lesser.
requires and provides
#You can check the package dependency before install the package
$ rpm -qp --requires git-1.7.6-1.el5.rf.i386.rpm
..
libssl.so.6 
…
#To meet the dependency, you want to check who provides libssl.so.6 
$yum whatprovides libssl.so.6
openssl-0.9.8e-20.el5.i686 : The OpenSSL toolkit
Repo        : base
Matched from:
Other       : libssl.so.6
#if openssl has been installed, “rpm -q –whatprovides” can also provide the answer
$rpm -q --whatprovides libssl.so.6
openssl-0.9.8e-12.el5_4.6

rpm scriptlets

#query all nopre|nopost|nopreun|nopostun  scripts
$rpm -q --scripts xinetd
postinstall scriptlet (using /bin/sh):
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
fi
preuninstall scriptlet (using /bin/sh):
if [ $1 = 0 ]; then
/sbin/service xinetd stop > /dev/null 2>&1
/sbin/chkconfig --del xinetd
fi
postuninstall scriptlet (using /bin/sh):
if [ $1 -ge 1 ]; then
/sbin/service xinetd condrestart >/dev/null 2>&1
Fi
#query postinstall script only
$ rpm -q --queryformat "%{POSTIN}"  xinetd
if [ $1 = 1 ]; then
/sbin/chkconfig --add xinetd
#Don’t run the scripts during install/remove
rpm –i –noscripts|nopre|nopost|nopreun|nopostun   pkgname
rpm –e –noscripts|nopre|nopost|nopreun|nopostun   pkgname

Extract rpm contents without install

#use rpm2cpio to extract everything
$mkdir /tmp/epel
$ cd /tmp/epel
$ rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd
./etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
..
#use rpm2cpio to extract particular file
$rpm2cpio /root/epel-release-5-4.noarch.rpm | cpio -ivd  ./usr/share/doc/epel-release-5
#another way is to use rpm install with alternative root
$ rpm --root /tmp/epel/ -ivh  --nodeps /root/epel-release-5-4.noarch.rpm
Recover corrupted rpm database
Build RPM from source file

Thursday, September 15, 2011

Get information for inode or block of a file and vice versa

Debugfs can retrieve a file’s inode or block information and vice versa, it is provided by e2fsprogs package, which should be installed by default.
Find a file’s information for inode or block
#find a file’s inode id by ls, the sample file’s inode number is 12
$ls -li /boot/message
12 -rw-r--r-- 1 root root 80032 Mar 13  2009 /boot/message
#find a file’s inode or block info by debugfs
$echo "stat message" | debugfs /dev/sda1
debugfs 1.39 (29-May-2006)
debugfs:  Inode: 12   Type: regular    Mode:  0644   Flags: 0x0   Generation: 3354433043
User:     0   Group:     0   Size: 80032
File ACL: 4640    Directory ACL: 0
Links: 1   Blockcount: 162
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4e2371f6 -- Mon Jul 18 09:36:22 2011
atime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
mtime: 0x49b95d4d -- Fri Mar 13 06:06:53 2009
BLOCKS:
(0-11):7681-7692, (IND):7693, (12-78):7694-7760
TOTAL: 80
Find a file based on inode or block info
#find a file based on inode number by find, but it may take long time to search
$find /boot/ -inum 12
/boot/message
#find a file based on inode number by debugfs
$echo -e "ncheck 12" | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message
##find a file based on block number by debugfs
#use icheck to find inode number based on block number first
$echo -e "icheck 7693 " | debugfs /dev/sda1
debugfs:  Block Inode number
7693  12
#then use ncheck find the file by inode number
$echo -e "ncheck 12 " | debugfs /dev/sda1
debugfs:  Inode Pathname
12    /message

Recover corrupted RPM database

RPM database consists of a number Berkeley DB   files in  /var/lib/rpm, the exception  is  __db.* files, which like cache files are updated for every rpm operation and they can be safely deleted.
#tested in Centos 5.5 
$ ls /var/lib/rpm
Basenames     __db.001  __db.003  Filemd5s  Installtid  Packages      Provideversion  Requireversion  Sigmd5
Conflictname  __db.002  Dirnames  Group     Name         Providename   Pubkeys         Requirename  Sha1header      Triggername
$ file /var/lib/rpm/Packages
/var/lib/rpm/Packages: Berkeley DB (Hash, version 8, native byte-order)
If one of the DB files is partially corrupted and it is still readable by /usr/lib/rpm/rpmdb_dump, you can reload the DB file and rebuild db.
$cd /var/lib/rpm
$rm -f __db*      
$mv Packages Packages.orig
$/usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages
$/usr/lib/rpm/rpmdb_verify Packages
#if you got this error: db_verify: PANIC: fatal region error detected; run recovery
#make sure /var/lib/rpm/__db.* are cleaned
#It is unlikely to rebuilddb if rpmdb_verify fails
$rpm -v –rebuilddb
If one of the DB files is completely corrupted and it is not readable by rpmdb_dump, you have to restore from backup,
$cd /var/lib/rpm
$cp Packages Packages.bak
#simulate a damaged RPM DB file
$ >Packages
$ cp Packages.bak  Packages
# Simply restoring  from backup file won’t work
#file verification is successful 
$ /usr/lib/rpm/rpmdb_verify Packages
#but any rpm operation fails
$rpm -qa 
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Even “rpm –rebuilddb” fails
$rm -f __db.*
$rpm –rebuilddb
error: rpmdbNextIterator: skipping h#     294 Header V3 DSA signature: BAD, key ID e8562897
#Notice the error about signature: BAD? The Pubkeys have to be cleaned as well.
$ mv Pubkeys Pubkeys.bak
#all good after removing Pubkeys file, a new Pubkeys is generated automatically on “rpm –rebuilddb”
$ rm -f __db.*
$ rpm –rebuilddb
$ rpm -qa | head -2
man-pages-2.39-15.el5_4
bash-3.2-24.el5