Tune nscd(name service cache daemon)

nscd provides caching for the passwd,group,and hosts tables, it can boot performance for situations, in which the tables need to be serviced remotely  e.g. LDAP authentication and DNS.
However, sometimes, it cause trouble.In Red Hat Linux 5 , nscd always return the old entry until the TTL(default is 1hour) is reached, even restarting nscd won’t flush the cache.
There are two solutions:
1.Disable persistent caching

#Persistent caching is enabled by default
$cat /etc/nscd.conf
persistent              passwd          yes
persistent              group           yes
persistent              hosts           yes
positive-time-to-live   hosts           3600
#So the entries are saved to relative tables
$rpm -ql nscd
..
/var/db/nscd/group
/var/db/nscd/hosts
/var/db/nscd/passwd…
#change them to no
persistent              passwd          no
persistent              group           no
persistent              hosts           no

With persistent caching disabled, restart nscd will discard the entries in memory.
2.Flush entries by invalidating the  table 
The entries in tables (group/passwd/hosts) can manually flushed by the ‘invalidate ‘parameter.

$nscd --invalidate=hosts

Since it is natural for anyone to try restart nscd to resolve the issue and the operation of rebuilding cache is not expensive, I think option 1 is better.

Force puppet agent to regenerate certificate request

If puppet agent’s certificate is accidentally revoked or deleted, you can force agent to regenerate certificate request.

In general, it is impossible un-revoke a certificate unless the revoke reason is certificateHold, But puppet can hack it. The solution is to recover all revoked certificates then revoke other certificates which don’t need to be recovered

$rm /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem
$rm /etc/puppetlabs/puppet/ssl/crl.pem
#At this point, all revoked certificates become valid certificates.
#So you need to revoke all certificates which don’t need to be recovered
$puppet cert --revoke foo

The following method of regenerating new certificate seems to be a better.

The following is tested in Puppet Enterprise 2, but it should work for puppet open source as well.
$ puppet --version
2.7.6 (Puppet Enterprise 2.0.0)
Force agent to regenerate certificate request by generate command

[puppet agent]$ puppet  certificate   generate    web1  --ca-location  remote
warning: peer certificate won't be verified in this SSL session
err: Error 400 on SERVER: web1 already has a revoked certificate; ignoring certificate request
err: Try 'puppet help certificate generate' for usage
#It because the revoked certificate still exist in the server, it need to be deleted
[puppet master]$ puppet cert list –all
- web1                                     (BA:18:D1:86:D6:5E:9E:99:55:39:3D:67:79:BF:BD:D0) (certificate revoked)
[puppet master]$ puppet cert clean web1
#re-run the command, the warning is expected because the request hasn’t been signed by master yet
[puppet agent]$puppet   certificate   generate    web1  --ca-location  remote
warning: peer certificate won't be verified in this SSL session
true
#The pending request appears in master 
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
[puppet master]# puppet cert sign web1

If the above doesn’t work for you, the last resort is to clean agent’s ssl files

[puppet agent]$ puppet --genconfig | grep certdir
certdir = /etc/puppetlabs/puppet/ssl/certs
$cd /etc/puppetlabs/puppet/ssl/
$find . –type f –exec rm {} \;
$service pe-puppet restart
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
puppet master]# puppet cert sign web1

Augeas quickstart

Augeas is a configuration editing tool. It parses configuration files in their native formats and transforms them into a tree. Configuration changes are made by manipulating this tree and saving it back into native config files.
Configuration files support is provided by “lenses”. The default lenses are provided in augeas-lib package

$ rpm -ql pe-augeas-libs | grep lenses
/opt/puppet/share/augeas/lenses/dist
/opt/puppet/share/augeas/lenses/dist/aliases.aug
…

If your configuration is not listed, you can write your own lenses definition file.
Language bindings are available:
Ruby, Python, OCaml, Perl, Haskell, PHP, Lua, and Java
Augeas has two main directories:
- augeas: augeas configuration directory. e.g control loading lenses
- files: user data directory tree. e.g /etc/hosts is mapped to /files/etc/hosts

$augtool
augtool> ls /
augeas/ = (none)
files/ = (none)

Sample hosts file to be tested

$cat /etc/hosts
# comment test1
# comment test2
172.16.1.11 web1 web1.example.com server1.example.com
#######Augeas translate above hosts file to 
augtool>  print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

Adding a new comment

augtool> set /files/etc/hosts/#comment[last()+1] "comment test3"
augtool> print  /files/etc/hosts/
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/#comment[3] = "comment test3"

########New comment will be added to last line.
########If you want to append it next to #comment[2], use insert
augtool> ins #comment after /files/etc/hosts/#comment[last()]
augtool> 
augtool> set  /files/etc/hosts/#comment[last()] "comment test3"
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

Adding a host entry.
You can’t use last() to auto increase it from hosts/1 to hosts/2, because  “1” or “2” is label name not an index number like #comment[1]/#comment[2].
The trick is to use “01,02,03” etc.

augtool> set /files/etc/hosts/01/ipaddr 172.16.1.12
augtool>set /files/etc/hosts/01/canonical web2
##at this point, it is hosts/01 added
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "coment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/01
/files/etc/hosts/01/ipaddr = "172.16.1.12"
/files/etc/hosts/01/canonical = "web2"

####after save and load, hosts/2 is added
augtool> save
Saved 1 file(s)
augtool> load
augtool> print /files/etc/hosts
/files/etc/hosts
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "coment test3"
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
/files/etc/hosts/2
/files/etc/hosts/2/ipaddr = "172.16.1.12"
/files/etc/hosts/2/canonical = "web2"

Change an entry by referencing to itself
Change the comment “comment test1” to “newcomment test1”

#Change the comment “comment test1” to “newcomment test1” 
augtool> /set files/etc/hosts/#comment[.='comment test1'] 'newcomment test1'
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "newcomment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3

#Change any comment starting with comment to ‘oldcomment’ by regex match
augtool>setm /files/etc/hosts  #comment[.=~regexp('comment.*')] 'oldcomment'
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "newcomment test1"
/files/etc/hosts/#comment[2] = "oldcomment"
/files/etc/hosts/#comment[3] = "oldcomment"

#Usually, regex match is used to delete entries. e.g delete comment test[23]
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment[1] = "comment test1"
/files/etc/hosts/#comment[2] = "comment test2"
/files/etc/hosts/#comment[3] = "comment test3"
augtool> rm /files/etc/hosts/#comment[.=~regexp('comment test[23]')]
rm : /files/etc/hosts/#comment[.=~regexp('comment test[23]')] 2
augtool> print /files/etc/hosts/#comment
/files/etc/hosts/#comment = "comment test1"
augtool>

Change an entry by referencing to relative nodes
e.g change ip of  a host ‘web1’  to 172.16.11.11

augtool> print /files/etc/hosts/1
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.1.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"
augtool> set /files/etc/hosts/*/ipaddr[../canonical = 'web1'] 172.16.11.11
augtool> print /files/etc/hosts/1
/files/etc/hosts/1
/files/etc/hosts/1/ipaddr = "172.16.11.11"
/files/etc/hosts/1/canonical = "web1"
/files/etc/hosts/1/alias[1] = "web1.example.com"
/files/etc/hosts/1/alias[2] = "server1.example.com"

#multiple conditions can be joined together 
augtool> match /files/etc/hosts/*/ipaddr[../canonical = 'web1' and ../alias = 'server1.example.com']
/files/etc/hosts/1/ipaddr = 172.16.11.11
augtool> match /files/etc/hosts/*[canonical = 'web1' and alias = 'server1.example.com']
/files/etc/hosts/1 = (none)

Comment or uncomment a line
Comment or uncomment is difficult for Augeas to implement, it can only be achieved by inserting a new line after the comment line then removing the old line.
You can extend Augeas to include ‘comment’ and ‘uncomment’ functions. The following code is an example for puppet.

##Define a function 
define augeasnew ($file,$line){
$exp=regsubst($line[0], '^(un)?comment *(.*)' , '\2')
case $line[0] {
/^uncomment/: {
exec {"/bin/sed -i -e '/${exp}/s/#//g' $file":
onlyif => "/bin/grep '${exp}' ${file} | /bin/grep '#' ",}
}
/^comment/: {
exec {"/bin/sed -i -e '/${exp}/ s/^/#/' $file":
onlyif => "/bin/grep '${exp}' ${file} | /bin/grep -v '#' ",}
}
default: {
augeas {'augeas-chg-any':
context => "/files/${file}",
changes => $line, }
}
}
}
##execute the function
augeasnew {'chg-hosts-file':
file => '/etc/hosts',
#Mulitple Augeas commands can be stored in an array
#However,none-Augeas command (uncomment or comment) can only be stored in first element of an array.
#line => ["set  1/ipaddr '10.1.1.1'" , "set 2/ipaddr '10.1.1.2'",],
line => ["comment  c line1",],
}

Troubleshooting.
You can check /augeas//error for detailed errors, for example, Augeas can’t load /etc/sudoers

augtool> print /augeas//error
/augeas/files/etc/sudoers/error = "parse_failed"
/augeas/files/etc/sudoers/error/pos = "1998"
/augeas/files/etc/sudoers/error/line = "62"
/augeas/files/etc/sudoers/error/char = "0"
/augeas/files/etc/sudoers/error/lens = "/opt/puppet/share/augeas/lenses/dist/sudoers.aug:478.10-.57:"
/augeas/files/etc/sudoers/error/message = "Iterated lens matched less than it should"
In this case, error is encountered in line 62. 
$sed -n '62p' /etc/sudoers
Defaults   !visiblepw
#You can upgrade Augeas or comment out the line to resolve the issue

Reference:
http://augeas.net/tour.html
http://augeas.net/page/Path_expressions
http://docs.puppetlabs.com/references/2.6.8/function.html
http://docs.puppetlabs.com/guides/language_guide.html

Vsphere PowerCLI script to clone and customize Windows guest OS

VMware can also customize Windows guest OS by Windows sysprep tool, though the process is more complex than Linux guest OS

There are two options to clone and “sysprep” VMware Windows guest OS:
1.Install sysprep tools in Windows guest OS and  run sysprep.exe  in guest OS command line, then clone it by VMware
2.Install sysprep tools in Virutal Center and let VMware tools in Windows guest in to control sysprep process either by GUI or script.  (It seems sysprep rely on VMware tools, so the VMware tools must be installed in guest OS)

Option #2 is the preferred method, because you can use script to easily customize unique information e.g computer name, ip addresses etc.

Install sysprep tools in Virtual Center
VMware KB: Sysprep file locations and versions 
Windows Vista onwards(vista/2008/7/2008R2/) don’t need this step, because its sysprep is built-in.
Create guest customization by GUI 
 Virtual Center -> view-> management -> “customization specification manager”. Create a new customization and select it when asked for guest customization information in GUI clone action. 
Create guest customization by script. 
Customization by GUI is not flexible, customization by script can be created on the fly. The following is enhancement to the clonevm.ps1 in Vsphere PowerCLI script to clone and customize Linux guest OS, just replace the “Identity for Linux” part with following code block 

$s_ostype=Retrieve-values $lines  "ostype"
if ( $s_ostype -eq "linux" ) { 
## Identity for Linux 
$vmclonespec_os.identity= New-Object VMware.Vim.CustomizationLinuxPrep
$vmclonespec_os.identity.hostname= New-Object VMware.Vim.CustomizationFixedName
$vmclonespec_os.identity.hostname.name= $s_dstname
$vmclonespec_os.identity.domain=$s_domain
}
elseif ( $s_ostype -eq "windows" ) {
# WinOptions
$vmclonespec_os.Options = New-Object  VMware.Vim.CustomizationWinOptions
$vmclonespec_os.Options.ChangeSID = 1
#sysprep
$vmclonespec_os.identity = New-Object VMware.Vim.CustomizationSysprep
# GUIUnattended
$vmclonespec_os.Identity.GuiUnattended = New-Object VMware.Vim.CustomizationGuiUnattended
$vmclonespec_os.Identity.GuiUnattended.AutoLogon = 0
#timezone codes: http://msdn.microsoft.com/en-us/library/ms145276(v=sql.90).aspx
$vmclonespec_os.Identity.GuiUnattended.TimeZone  = 255
$vmclonespec_os.Identity.GuiUnattended.Password = New-Object VMware.Vim.CustomizationPassword
$vmclonespec_os.Identity.GuiUnattended.Password.PlainText = 1
$vmclonespec_os.Identity.GuiUnattended.Password.Value = "Secret01"
# Identification
$vmclonespec_os.Identity.Identification = New-Object VMware.Vim.CustomizationIdentification
$vmclonespec_os.Identity.Identification.joinWorkgroup = "workgroup2"
## Userdata
$vmclonespec_os.identity.userData = New-Object VMware.Vim.CustomizationUserData
$vmclonespec_os.identity.userData.computerName = New-Object VMware.Vim.CustomizationFixedName
$vmclonespec_os.identity.userData.computerName.name = $s_dstname
$vmclonespec_os.Identity.UserData.FullName = "Administrator"
$vmclonespec_os.Identity.UserData.OrgName="myOrg"
$vmclonespec_os.Identity.UserData.Productid=""
}
else {
write-host "Unknown ostype: $s_ostype. Please set it to linux or windows"
exit
}
}

NOTES: 
 - The guest OS(Windows/Linux) will be forcefully rebooted by Vmware tools again in ~1 minute after you power it on, so don’t login in a hurry to check the result.
 - The script only works in a live session to Virtualcenter, it doesn’t work in a direct login session to ESX host.

Bootstrap Puppet Enterprise (PE) Puppet Client for RHEL

Puppet Enterprise(PE) packs all required packages, modules into a tar file.
The file also includes a shell installation script to install packages and modules and do other customizations tasks. So only installing agent packages “pe-puppet pe-mcollective” can’t setup agent station properly.
It is ideal to have PE agent installed automatically along with OS,the installer script can use answer file to automate the process.
The following codes can be put in RHEL kickstart postrun section to bootstrap PE agent.

#Download Puppet Enterprise(PE) (manage up to 10 nodes free)
#http://puppetlabs.com/puppet/puppet-enterprise/
#e.g  puppet-enterprise-2.0.0-el-5-x86_64.tar.gz
...
%post
PKG=puppet-enterprise-2.0.0-el-5-x86_64
cd /tmp && wget -O ${PKG}.tar.gz http://172.16.1.1/boot/os/rhel-5.6-x64/${PKG}.tar.gz  && tar -zxf ${PKG}.tar.gz  && cd /tmp/${PKG} && \
cat >answerfile<<END
q_install=y
q_puppet_cloud_install=n
q_puppet_enterpriseconsole_install=n
q_puppet_symlinks_install=y
q_puppetagent_certname=$(hostname -s)
q_puppetagent_install=y
q_puppetagent_server=puppet
q_puppetmaster_install=n
q_vendor_packages_install=n
q_install=y
END
[ -e answerfile ] && ./puppet-enterprise-installer -a answerfile  2>&1 | tee  /tmp/install_${PKG}.log
rm -rf /tmp/${PKG}*
sync
exit 0

Connect to puppet server after installation finished

#Verify puppet agent is running
[Puppet Client]$/etc/init.d/pe-puppet status
#Puppet server should be able to see the client’s pending certificate request
[Puppet Master]$puppet cert list
web2 (A1:08:15:EF:1F:5D:F9:C5:D9:A0:F3:F2:FD:FF:CE:09)
#Approve the client by signing the cert request
[Puppet Master]$puppet cert sign web2