ESXi 4 kickstart and ESXi 5 kickstart examples

What is changed in ESXi 5 kickstart:
1. ESXi 5 supports EFI PXE boot(the file is under install media\EFI\boot)
2. ESXi 5 loads all needed packages into memory, there is no need to specify installation media in http/nfs etc
3. ESXi 5 supports vlan tagging by –vlanid parameter, previously PXE boot NIC has to be native vlan (VLAN 1)
4. ESXi 5 no longer supports set license data in kickstart file

ESXi 4 kickstart file example

vmaccepteula
#create encrypted password with command “openssl passwd -1”
rootpw --iscrypted $1$XXXXXXXXXXXXXXXX
autopart --firstdisk=mpx.vmhba32:C0:T0:L0 --overwritevmfs
install url http://172.16.1.18/boot/XXXX
network --device=vmnic0   --hostname=esxi1  --bootproto=static  --ip=172.16.1.4 --netmask=255.255.255.224 --gateway=172.16.1.30 --nameserver="172.16.1.18,172.16.1.19"
serialnum --esx=4H42M-XXXX-XXXXX-XXXXX-XXXXX
%post --interpreter=busybox --ignorefailure=true
reboot
#enable local console and remote ssh login
%firstboot --interpreter=busybox
/etc/init.d/TSM start
/etc/init.d/TSM-SSH  start
/sbin/chkconfig TSM on
/sbin/chkconfig TSM-SSH on

ESXi 5 kickstart file example

vmaccepteula
#create encrypted password with command “openssl passwd -1”
rootpw --iscrypted $1$XXXXXXXXXXXXXXXX
clearpart --firstdisk=local --overwritevmfs
install --firstdisk --overwritevmfs
#ESXi 5 retrieve software from the image loaded into memory by PXE, so there is no need to specify software repository location
network --device=vmnic0  --vlanid=3 --hostname=esxi2  --bootproto=static  --ip=172.16.1.5 --netmask=255.255.255.224 --gateway=172.16.1.30 --nameserver="172.16.1.18,172.16.1.19"
#loading license data in kickstart file not longer supported in ESXi 5
#serialnum --esx=4H42M-XXXX-XXXXX-XXXXX-XXXXX
%post --interpreter=busybox --ignorefailure=true
reboot
#enable local console and remote ssh login
%firstboot --interpreter=busybox
/etc/init.d/SSH start
/etc/init.d/ESXShell start
/sbin/chkconfig SSH on
/sbin/chkconfig ESXShell  on

PXE kickstart VMware ESXi 5 on a trunked interface (Tagged VLAN)

PXE Kickstart VMware ESX host was usually performed on the native VLAN, It was not possible in a trunked interface (Tagged VLAN) due to two obstacles:

The above have been overcome by the following technologies:
1. Mutiple boot agent(MBA):  NIC supports VLAN tag in BIOS, such as MBA of Broadcom NICs (otherwise PXE client couldn’t get IP address from DHCP in the first place)
2. VMware ESXi 5: A new parameter, vlanid, is introduced as a boot option.(not  the same as vlanid in kickstart configuration file)

Configure MBA for Broadcom NetXtreme II BCM5709 in IBM X3850 server
F1 go to BIOS->System Setting->Network->select the network adapter in device list -> configure mutiple boot agent(MBA) and iSCSI parameter -> MBA configuration menu-> vlan mode/vlan id

Add VLANID in ESXi 5 boot option

A sample boot.cfg for gPXE

bootstate=0
title=Loading ESXi installer
kernel=http://pxe-server.example.com/boot/os/esxi-5.0-x64/pxeboot/tboot.b00
#BOOTIF is mandatory for gPXE, it seems ESXi 5 couldn’t get  name server and domain name and from DHCP, So the name server is assigned manually.
kernelopt=BOOTIF=01-5c-f3-fc-94-e4-18 vlanid=3 nameserver=172.16.1.1  ks=http://pxe-server.example.com/boot/hosts/linux-ks/5c-f3-fc-94-e4-18.txt
modules=http://pxe-server.example.com//boot/os/esxi-5.0-x64/pxeboot/b.b00  ... < ..omitted ..>

Force legacy mode PXE in IBM system X(xSeries) server

The Unified Extensible Firmware Interface (UEFI) tends to replace the basic input/output system (BIOS) of x86 personal computer.
Intel Corporation developed EFI in the mid 1990s to enable the Itanium class of processors without having to implement legacy processor operating modes that are required by BIOS
Later, the UEFI Forum Inc. was formed by Intel, IBM, Hewlett-Packard, Dell, Apple, and other
companies to turn EFI into a industry standard that is appropriate for broader use on other
architectures, including x86 and x64.
Today, UEFI is becoming widely accepted throughout the industry, and many systems are
making the transition to UEFI-compliant firmware.  The list of Vendors using UEFI:
http://www.uefi.org/news/uefi_industry/UEFIEvaluationPlatforms_2011_05.pdf
PXE on UEFI
UEFI PXE uses a special boot file (install media/boot/efi/EFI/redhat/grub.efi) rather than the traditional boot file “pxelinux.0”. If you have an existing PXE provision system and don’t want to change it to meet UEFI PXE requirement, you can force legacy mode PXE.
Force legacy mode PXE in IBM system X(xSeries) server
System Setting -> Boot manager-> add boot option
add "legacy only".
System Setting -> Boot manager-> change boot order
Move "legacy only" option above the device (PXE network):
  legacy only
  PXE network
The system will boot PXE in legacy mode automatically, If you want to boot from PXE netowrk manually after pressing F12 , select legacy only not PXE network.
Each network adapter also has PXE mode switch, the options are UEFI, Legacy, Both, or None. But forcing  legacy mode here has doesn’t seem to work.

Fix curl client certificate error: curl: (58) unable to set private key file

I used curl to post xml payload to a web service by client certificate authentication.
It worked fine Linux(curl 7.15.5 (x86_64-redhat-linux-gnu))
But the same command with the same certificate failed in Windows(curl 7.23.1 (x86_64-pc-win32)).

$curl -k -s -o /dev/null  --connect-timeout 5 -w "\ntime_total=%{time_total}\nhttp_code=%{http_code}\n" --cert cert2.pem: Pass123 --key cert2.pem    -d @data.xml https://server1/
curl: (58) unable to set private key file: 'cert2.pem' type PEM

This error could be due to invalid certificate password or invalid certificate format,  It seems the curl in windows is picker on certificate format than the curl in Linux.
In general, you can combine both certificate and key into one file by copy&paste the text in a text editor. But it seems the curl for Windows only accept the “formal format”, which has additional lines of “Bag Attributes” ( or maybe just different behaviour in different versions)
General  certificate

#cat cert1.pem
-----BEGIN CERTIFICATE-----
………………..
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,D49E12E8D074DA59
…………….
-----END RSA PRIVATE KEY-----

”Formal” certificate with “Bag Attributes”

Bag Attributes
localKeyID: 92 47 FA AC C8 D4 94 D8 0C B0 D0 1D 9F D7 43 0B B5 D8 23 AD
subject=/C=AU/ST=NSW/L=Sydney/O=ORG1/CN=client1
issuer=/C=AU/ST=NSW/O=ORG1/CN=CA
-----BEGIN CERTIFICATE-----
………………….
-----END CERTIFICATE-----
Bag Attributes
localKeyID: 92 47 FA AC C8 D4 94 D8 0C B0 D0 1D 9F D7 43 0B B5 D8 23 AD
Key Attributes: <No Attributes>
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,6F6BA23BB03B5049
……………..
-----END RSA PRIVATE KEY-----

How to add the “Bag Attributes”?

#I found the “bag Attributes” can be added by converting PEM to pkcs12 format then convert back to PEM.
#Convert  from PEM to PKCS12
$openssl pkcs12 -export -in cert1.pem  -out cert1.pfx
#Convert back from  PKCS12 to PEM
$ openssl pkcs12 -in cert1.pfx  -out cert2.pem -clcerts
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
#It worked with the new certificate
C:\temp\curl\curl -k -s -o /dev/null  --connect-timeout 5 -w "\ntime_total=%{time_total}\nhttp_code=%{http_code}\n" --cert cert2.pem:Pass123  --key cert2.pem  -d @data.xml https://server1/
time_total=2.812
http_code=200

Script automation with SSH in Windows

I am testing a monitoring software, like the most monitoring software, it  relies on SSH to call remote customized scripts. But it is running on Windows. To get SSH command line working nicely in Windows is not quite easy.
There are two options to use SSH to do script automation in Windows.
1.Use SSH command line
2.Use SSH API for a scripting language i.e. Perl
 Use SSH command line
PuTTY is a well-known SSH GUI client,the non-interactive, command line interface is plink
The other option of SSH command line is ssh in cygwin, this ssh works exactly the same way as ssh in Linux.
SSH in cygwin support more features than plink, but it requires installing the whole cygwin environment, while plink is just a single binary file.
However plink has a draw back, it seems there is no way to suppress the “accept remote host key prompt” and its host key cache is stored in Windows registry.
SSH in cygwin can suppress the prompt by setting “-o stricthostkeychecking=no” and the host key cache is stored in .ssh in user's home folder. If the monitoring software is running as a service under 'local system account', where is the .ssh located? The answer for Windows 2008 R2 is “c:\Windows\SysWOW64\config\systemprofile”, you have to create “.ssh”  folder manually, otherwise the service can't save the ssh host key cache.
 Use SSH API for a scripting language i.e. Perl
The above method of using SSH command line works, but is not as flexible as using SSH API.
The SSH API is from libssh2, Perl has a module,NET::SSH2,to utilize libssh2 to provide the same function of ssh in cygwin. The benefits of NET::SSH2  are: better error handling and no ssh host key prompt issue.
Install Perl for Windows
Activeperl is probably the most popular choice in Windows, but it has limited number of pre-built Perl Modules, it is very hard to compile from source for new modules. Activeperl itself doesn't have compiler or make utility, you can get MinGW compiler or dmake, but the chance of success build is very slim.
Strawberryperl,http://strawberryperl.com/  is an opensource project for Perl in Windows, it has built-in compiler and related tools. It can download and install modules in www.cpan.org
As of Strawberry Perl 5.12, net::ssh2 is built-in, the following is simple example of of net:ssh2 usage.

##Note: if you encounter an error "libssh2-1_.dll was not found", you need to add the path of the dll, "C:\strawberry\c\bin", into %PATH% env variable
use  warnings;
use strict;
use Net::SSH2;
use constant BUFLEN => 10_0000 ;
my $ssh2 = Net::SSH2->new();
$ssh2->connect('server1.example.com',22) or die "Unable to connect Host $@ \n";
if ($ssh2->auth_password('user1','Pass1')) {
#if ( $ssh2->auth_publickey ( 'user1', 'id_rsa.pub', 'id_rsa')) { 
my $chan2 = $ssh2->channel();
#$chan2-&gtblocking(0);
$chan2->exec ("uname -a\n");
$chan2->read($buf, BUFLEN );
print $buf;
$chan2->close;
} else {
warn "auth failed.\n";
}

Control ISC DHCP to allocate IP address based on vendor ID

ISC DHCP supports conditional evaluation(man dhcp-eval), one of the evaluation expressions is hardware, the MAC  address of the network card.
Conditional evaluation make it possible to allocate different IP pool for clients or even allow multiple DHCPD daemons running in the same broadcasts domain.
My challenge is to setup a new DHCP server to PXE boot VMware Vms without affecting an existing DHCP server.
The solution is to create a new class which only response DHCP request  from Vmware NICs, the key is to use expression binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) to get the vendor ID.

$cat /etc/dhcpd.conf
ddns-update-style none;
ignore client-updates;
#log-facility local7;
#log (debug, binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)));
class "vmware-nics"
{
match if  ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:05:69") or ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:0c:29") or ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:1c:14") or ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:50:56");
}
subnet 192.168.100.0 netmask 255.255.255.0 {
pool {
allow members of "vmware-nics";
option routers                  192.168.100.254;
option subnet-mask              255.255.255.0;
option domain-name              "example.com";
option domain-name-servers      192.168.100.1;
range dynamic-bootp 192.168.100.1 192.168.100.200;
#time unit is 1 sec
default-lease-time 3000;
max-lease-time 6000;
next-server 192.168.100.1;
filename "gpxelinux.0";
}
}

NOTE: the double 0  in MAC address  will be translated to single 0. e.g "00:05:69" = "0:05:69"
Troubleshooting:
If the expression doesn't work, you can check the expression by logging it to a file.You might need to disable "allow members" restriction in order for the expression to be logged.

log-facility local7;
log (debug, binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)));

By default, DHCP log is directed to /var/log/message, it seems, in order to log expression, the DHCP must use separate log file. Because “local7.* /var/log/boot.log” is configured in /etc/syslogd.conf  by default, “log-facility local7” in dhcpd.conf will direct messages to /var/log/boot.log

Enable timestamp logging in bash commands history

Bash has the built-in feature to support timestamp in history command , but it is not enabled by default in Red Hat Linux(ksh doesn’t have the feature at all)
It is recommended to tune two environment variables for history command to tighten audit.
1.Enable timestamp in command history
2.Increase the default  history file size
Enable timestamp in command history
Just need to set HISTTIMEFORMAT env variable, e.g.  HISTTIMEFORMAT="[%F %T %Z] "
The time format is the same as that of date command.
Increase the default history file size
There are two environment variables  for history size.
HISTSIZE: How many lines to keep in memory for current bash session, commands in current session are appended to ~/.bash_history on quitting.
HISTFILESIZE: How many lines to keep in history file ~/.bash_history
Both HISTSIZE and HISTFILESIZE seem to be 1000 by default, HISTSIZE=1000 is ok for current session, but HISTFILESIZE=1000 is too small to log enough commands for any past login session.
Both environment variables  are to be enabled for all users,  they can be put in global profile /etc/profile , but user defined global profile in /etc/profile.d/ is a better option.
For example, create a user global profile in /etc/profile.d

$cat /etc/profile.d/user.sh 
HISTSIZE=1000
HISTFILESIZE=40000
HISTTIMEFORMAT="[%F %T %Z] "
export HISTSIZE HISTFILESIZE HISTTIMEFORMAT
#With timestamp enabled, the history command output looks like this
$history 
19  [2012-02-03 13:41:56 EST] history
20  [2012-02-03 13:42:58 EST] date
22  [2012-02-03 13:42:55 EST] pwd

You may notice that the commands was not sorted by the time order, it is because, even “pwd” was executed  before date, but  it is was in different session and the “pwd session” was terminated after the “date session”, so the “pwd” command was appended after “date”.

To view all 40000 lines in ~/.bash_history for tracing very old commands, you need to temporarily increase HISTSIZE to 40000 before run “history”.  You can set the default HISTSIZE is 4000, but it takes too long to process for history command used in daily basis.

Other useful history related commands

#clear commands history in current session 
$history -c
#disable logging history commands
$set +o history 
#enable logging history commands
$set –o history 
#fc (fix command) 
#open last 4 commands in an editor and re-execute them after closing the editor
#it is useful if you need to re-execute more than one command
$fc -4 -1