VMware ESX command line essentials PART #2

Following on PART #1, PART #2 is focusing on exploring storage Information.

Display currently mounted storage

####vdf for ESX host  is equivalent to  df  command for Linux, it shows any type (SCSI/FC/NFS/ISCSI)
$ vdf -h
Filesystem            Size  Used Avail Use% Mounted on
..
/vmfs/volumes/4c5be33b-3a356289-ea3f-0024c1a35c5c
135G   16G  118G  12% /vmfs/volumes/esx01:stor1
/vmfs/volumes/7292e403-4f367279
1.7T  1.5T  206G  88% /vmfs/volumes/nfs_stor 
####Check detailed file system information on the storage
$ vmkfstools -P -h /vmfs/volumes/esx01:stor1
VMFS-3.33 file system spanning 1 partitions.
File system label (if any): esx01:stor1
Mode: public
Capacity 135.2 GB, 118.5 GB available, file block size 1 MB
UUID: 4c5be33b-3a356289-ea3f-0024c1a35c5c
Partitions spanned (on "lvm"):
mpx.vmhba0:C0:T0:L0:5 
$ vmkfstools -P -h /vmfs/volumes/nfs_stor
NFS-1.00 file system spanning 1 partitions.
File system label (if any): nfs_stor
Mode: public
Capacity 1.7 TB, 206.5 GB available, file block size 4 KB
UUID: 7292e403-4f367279-0000-000000000000
Partitions spanned (on "notDCS"):
nfs:nfs_stor 

Find out storage type (SCSI/FC/NFS/ISCSI) 

####SCSI/FC TYPE
[ESX4]$esxcfg-scsidevs -m
mpx.vmhba0:C0:T0:L0:5   /dev/cciss/c0d0p5    4c5be33b-3a356289-ea3f-0024c1a35c5c  0  esx01:stor1
[ESX3]$ esxcfg-vmhbadevs  -m
vmhba0:0:0:9    /dev/cciss/c0d0p9                4951068b-1dabf5db-74c8-0022640e9ea6
####NFS TYPE
$esxcfg-nas -l
nfs_stor is /vol/vmdk_vol1 from 10.1.1.1.1  mounted
####ISCSI TYPE
vmkiscsi-tool ?

Find out adapter Vendor/ Device name

$ esxcfg-info | grep -B 12 vmhba0  
|----Vendor Name........................................Hewlett-Packard Company
|----Device Name........................................Smart Array E200i
|----Device Class.......................................260
|----PIC Line...........................................11
|----Service Console IRQ................................25
|----Vector.............................................162
|----PCI Pin............................................0
|----Spawned Bus........................................0
|----Flags..............................................513
|----Module Id..........................................34
|----Chassis............................................0
|----Physical Slot......................................255
|----VmKernel Device Name...............................vmhba0

How to find out storage not mounted

[ESX3]$esxcfg-mpath -l
Disk vmhba0:0:0 /dev/cciss/c0d0 (139979MB) has 1 paths and policy of Fixed
Local 66:8.0 vmhba0:0:0 On active preferred
[ESX4]$  esxcfg-scsidevs -l
mpx.vmhba0:C0:T0:L0
Device Type: Direct-Access
Size: 139979 MB
Display Name: Local VMware Disk (mpx.vmhba0:C0:T0:L0)
Plugin: NMP
Console Device: /dev/cciss/c0d0
Devfs Path: /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0
Vendor: VMware    Model: Block device      Revis: 1.0
SCSI Level: 2  Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true
Other Names:
vml.0000000000566d676261303a303b30
[ESX4]$ esxcfg-mpath -l device=vml.0000000000566d676261303a303b30
block.cciss/c0d0:0-block.0:0-mpx.vmhba0:C0:T0:L0
Runtime Name: vmhba0:C0:T0:L0
Device: mpx.vmhba0:C0:T0:L0
Device Display Name: Local VMware Disk (mpx.vmhba0:C0:T0:L0)
Adapter: vmhba0 Channel: 0 Target: 0 LUN: 0
Adapter Identifier: block.cciss/c0d0:0
Target Identifier: block.0:0
Plugin: NMP
State: active
Transport: block

Misc

####show vmware crash dump partition 
esxcfg-dumppart -l 
####clone disk/convert thick disk to thin disk 
vmkfstools   -i /vmfs/volumes/storage1/vm01.vmdk  -d thin    /vmfs/volumes/storage2/vm02.vmdk 
Thick Disk: size of ls output = size of du output 
Thin Disk: size of ls output > size of du output

VMware ESX command line essentials PART #1

[1] Overview of the methods to manage Vmware ESX/ESXi Host

1.1 ESX host native commands

1) Basic commands:
/usr/sbin/esxcfg-* commands ,  vmware-cmd, vmkfstools etc
2) Advanced command:
/usr/bin/vimsh:  an interactive shell to peform almost any task that can be done in vCenter

1.2 External management tools to manage ESX/ESXi Host

VMware vSphere PowerCLI
"SDK"  for  Windows  to perform any task that can be done in vCenter.
Interactive shell with 200+ PowerShell cmdlets, easy to use
Supported OS: Windows
Supported Target: ESX/ESXi or vCenter Server
http://honglus.blogspot.com/2009/12/vsphere-powercli-quick-start.html
Vmware  vSphere SDK for Perl SDK for Perl to perform any task that can be done in vCenter
Supported OS: Windows/Linux
Supported Target: ESX/ESXi host or vCenter Server
VI Perl Toolkit
Based on Perl SDK, Some scripts  to perform common administration tasks (vmclone.pl vmcontrol.pl ... etc)
VI Perl Tookit only has 10+ utilities compared to PowerCLI  200+  cmdlets.
VMware vSphere  vCLI
Based on Perl SDK, port most ESX host native commands (esxcfg*,vmware-cmd...) to external management host. It is commonly used for ESXi host which doesn't have service console.
Supported OS: Windows/Linux
Supported Target: ESX/ESXi host or vCenter Server

[2] VMware basic concepts

Memory management

[Memory Size] This is the amount of memory we assign to the vm and is also the amount of memory the guest OS will see as its physical memory. This is a HARD LIMIT  and the vm cannot exceed this limit if it demands more memory.
[Reservation]  A guaranteed amount of memory assigned to the vm. This is a way of ensuring that the vm gets a minimal amount of memory assigned
[Limit] SOFT LIMIT  controlled by VMkernel,  SOFT LIMIT<=HARD LIMIT (Memory size),  The default value is unlimited which is VM Configured Size
Maximum allocation target = min{ VM’s memory size, VM’s limit }
[Share]  the priority/weight when requesting resource, High:normal:low = 4:2:1

Memory management key points

1) Allocation rule: Reservation  <=the amount  dynamically allocated  based on shares <=limits <= memory size
2) It is important to set reservation  for mission critical VMs, but limit is not very important.
3) It is important to have VM tools running inside VM. Because ballooning driver (vmmemctl) in VM tools is more efficient than host swapping under memory overcommitment.
(Under memory overcommitment, reclamation techniques techniques are transparent page sharing, ballooning, and host swapping)
http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
Resource pool
Cluster is a big resource pool by default
You can create sub pool under the Cluster and drag vm to the sub pool in order to fine tune VM resource.

VMware virtual machine files

location: /vmfs/volumes/$DATASTORE/$VIRTUALMACHINE
.vmx This is the primary configuration file
.nvram  This is the file that stores the state of the virtual machine's BIOS.
.vmdk   Disk DescriptorFile (ASCII text, less than 1M )
*-flat.vmdk Binary file stores the contents of the virtual machine's hard disk drive.
.vmem  The virtual machine's paging file
.vmsn VM snapshot file

Vmware configuration files in /etc/vmware/

/etc/vmware/esx.conf  is the ESX host configuration file.

Vmware runtime information in /proc/vmware/

/proc/vmware/version /proc/vmware/cpuinfo … etc.

VMware services control scripts  in /etc/init.d/

vmware : Vmware basic service supporting VMS, restarting vmware will affect running VMs
mgmt-vmware: Vmware Host Agent (/usr/lib/vmware/bin/vmware-hostd). Vmware management service, restart won't affect running vms
vmware-aam  :Vmware HA agent
vmware-vmkauthd: VMware VMkernel authorization daemon
vmware-webAccess: Vmware Web management interface 
vmware-autostart: auto start/stop VMs configured to do so
vmware-late: VMware late init tools (start kernel logger, nfs/iscsi volumes .. )
vmware-vpxa VirtualCenter agents

[3] VMware basic administrative tasks:

####Get ESX host Vmware version
$cat /proc/vmware/version
VMware ESX 4.0.0 [Releasebuild-208167], built on Nov  8 2009
…
$/usr/bin/vmware -v
VMware ESX 4.0.0 build-208167
####List running VMS
$/usr/bin/vmware-cmd -l
/vmfs/volumes/8292a402-3f357289/vm-01/vm-01.vmx
..
$/usr/bin/vm-support -x
..
vmid=4331       vm-01
..
$vimsh -n -e vmsvc/getallvms
Vmid         Name                                    File                                Guest OS      Version
16     vm01   [stor1] vm01/vm01.vmx   sles10_64Guest   vmx-07
####Manage VMs
/usr/bin/vmware-cmd <cfg file ( .vmx)>  command
VM command:
getstate
start [soft|hard|trysoft]
stop [soft|hard|trysoft]
reset [soft|hard|trysoft]
suspend [soft|hard|trysoft]
getconfig <variable>
setguestinfo <variable> <value>
getguestinfo <variable>
getproductinfo <product_info>
connectdevice <device_name>
disconnectdevice <device_name>
getid
getconfigfile
getheartbeat
getuptime
gettoolslastactive
hassnapshot
createsnapshot <name> <description> <quiesce> <memory>
revertsnapshot
removesnapshots
answer
####register and unregister VMs
/usr/bin/vmware-cmd -s register <config_file_path>
/usr/bin/vmware-cmd -s unregister <config_file_path>
####vmkping 
ping from vmkernel interface instead of service console interface. e.g checking connectivity to NFS server where VMS reside.
####esxcfg-advcfg   -g
-g Get/set  the value of the VMkernel advanced configuration option listed in esxcfg-info -o 
$ esxcfg-advcfg -g /Irq/RoutingPolicy
Value of RoutingPolicy is 2
####esxcfg-advcfg   -c
- c|--get-config         Get the value of a specific line in esx.conf.
esxcfg-advcfg -c /adv/Misc/HostName
####vimsh
An interactive shell on ESX host to provide almost any operation available in virtual centre
$ /usr/bin/vimsh
-n          Non-interactive mode.  Just quit after processing command line.
-e <cmd>    Run a series of commands specified by <cmd>.
$ vimsh -n -e vmsvc/getallvms
To find out the command name, press tab key to list commands in interactive shell(start vimsh without any argument)

PART #2 will be focusing on VMware command line storage management

Setup Perl in Solaris 10 to talk to Microsoft SQL Server 2000.

I tested following solution in Solaris 10 to connect to Microsoft SQL Server 2000, But it should work with MS SQL 2005. (Maybe with MS SQL 2008 as well)
In Unix/Linux, there are two typical ways to have Perl to talk to Microsoft SQL Server.
1)  Perl DBI + Perl DBD::ODBC + ODBC Driver
2)  Perl DBI + Perl DBD::Sybase + Freetds driver
I choose Freetds solution  because Unix ODBC driver is not free (http://www.unixodbc.org/).

Install complier and build tools

Compiler choices for Solaris: 1) GCC For Sun Systems: Compatible with gcc and optimised for Sun hardware
free download from Oracle.
2) Solaris Studio: Solaris Native complier. free download from Oracle 
3) gcc GNU compiler
Free download from  http://www.sunfreeware.com/  or retrieve from Solaris 10 CD (SUNWgcc)
My Personal opinion: “GCC For Sun Systems” is the best choice for compatibility  and performance,GNU gcc is the last choice. Build tools choices for Solaris: 1) GNU build tools
NAME: SUNWgmake  SUNWbinutils SUNWgm4 
DOWNLOAD: Free download from http://www.sunfreeware.com/ or retrieve from Solaris 10 CD
PATH: /usr/sfw/bin/{gmake,gld,gnm,gar,gas,ggprof,gm4,granlib,gsize,gstrip}
2) SUN build tools
NAME:SUNWbtool
DOWNLOAD: Retrieve from Solaris 10 CD
PATH:/usr/ccs/bin/{make,ld,nm,ar,as,gprof,m4,ranlib,size,strip}
I recommend to start with SUN build tools, if it doesn’t work try GNU build tools.
If you choose GNU build tools, you may need to create links to remove prefix “g” and rename dir /usr/ccs/bin (in case SUN build tools are picked up first)
 for i in   {gmake,gld,gnm,gar,gas,ggprof,gm4,granlib,gsize,gstrip}; do echo ln -s $i ${i#g}; done

Setup Freetds, Perl DBI and Perl DBD::sybase

Install Freetds
##Download
  Download source code from: http://www.freetds.org/
##Install
./configure --with-tdsver=8.0; make ; make install
tdsver=8.0 is compatible with MS SQL 2000, but it has been reported that SQL 2005 works with it as well.
By default, freetds binaries are installed in /usr/local/bin and libraries are installed in /usr/local/lib. you can override this with --prefix.
##add /usr/local/lib to search path.
If your crle output doesn't include /usr/local/lib, you can append it current lib dirs e.g 
crle -u -l /lib:/usr/lib:/usr/local/lib
(the equivalent Linux command is ldconfig)

##Test Login
   /usr/local/bin/tsql -H 172.1.1.1 -p 1433 -U 'username' -P password -D DB-NAME
（tsql is just for testing connection, The interactive shell working with freetds is http://shellsql.sourceforge.net/, so you can run sql command in shell script with shellsql!）

make sure above command works before continue to next step

Install Perl DBI
download from http://search.cpan.org/
perl Makefile.PL; make ; make install
Install Perl DBD::Sybase
download from http://search.cpan.org/
export SYBASE=/usr/local;
perl Makefile.PL; make ; make install
Sample Perl Script

#!/usr/local/bin/perl -w
use DBI;
my $dbh = DBI->connect('DBI:Sybase:server=172.1.1.1;port=1433','username,'password') or die $DBI::errstr;
$dbh->do("use DB-NAME");
$SQL= "SELECT uid,name from employee;";
my $ids = $dbh->selectall_arrayref( "$SQL", { Slice => {} } );
foreach my $id ( @$ids ) {
print "$id->{uid} | $id->{name} \n";
}

Configure Jumbo frame for VMware ESX 3.5 and ESX 4.x

Vmware KB Article: 1007654 iSCSI and Jumbo Frames configuration on ESX 3.x and 4.x  may lead you to think that enabling jumbo frame on vmkernel interface is required. Actually, it depends on what you want to achieve. 
1) If you want to place VMs on NFS/ISCSI data-store, enable jumbo frame for ESX Host's vmkernel interface and vSwitch, where vmkernel port group reside.
2) if you want  guest OS to connect NFS/ISCSI server, enable jumbo frame on guest OS's network interface and vSwitch, where guest OS port group reside.
Please notes the following:
    Jumbo frames need to be enabled end to end [OS level - ESX Virtual switch - physical switch - peer host ]
    For ESX 3.5/4.x , jumbo frames are supported in both the guest operating system and in the ESX kernel TCP/IP stack.  (Jumbo Frames on software iSCSI  for ESX 3.x is only experimental)
    ESX 3.5 doesn't allow update MTU on vSwitch, the vSwitch has to be re-created.
    ESX 4 allow update MTU on vSwitch on the fly.

Example #1 for ESX 4: changing MTU to 9000 for vmkernel interface, to which NFS/ISCSI datastore port group linked.

####Shutdown VMs in current ESX host or Migrating running VM to other hosts 
####Write down current information: port group name/ip address/network mask 
$ esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type
vmk1       esx_nfs      IPv4      172.16.1.88                           255.255.255.0 172.16.1.255   00:50:56:71:48:18 1500    65535     true    STATIC
#### Delete current port group (Can't change MTU on existing vmkernel interface )
esxcfg-vmknic -d esx_nfs
####Re-create Vmkernel interface with mtu 9000 and update Vswitch, where esx_nfs resides
esxcfg-vmknic -a -i 172.16.1.88 -n 255.255.255.0 -m 9000  esx_nfs  
esxcfg-vswitch -m 9000  vSwitch2
####Test by vmkping
vmkping -s 8900  NFS-Server-IP 
variable overhead is added to payload, it is safe to start with 8900 byte
normal "ping" command is for testing connectivity for ESX console network
if vmkping fails,reboot ESX host and check NFS server has jumbo frame enabled as well.

Example #2 for ESX 4: Changing MTU to 9000 for Linux Guest OS on normal data networking.

####Find the vSwitch name
esxcfg-vswitch -l 
####Change MTU for the switch, the vmnicX on ESX host will be changed automatically
esxcfg-vswitch -m 9000  vSwitch3
####Change MTU on Linux Guest OS
Append  "MTU=9000"  to Linux NIC configure file: /etc/sysconfig/network-scripts/ifcfg-ethX
restart the interface ifdown ethX  ; ifup ethX
####Test by ping with don't fragment flag (-M do ) set, Without it, "ping -s 8900" might be successful even on MTU=1500.  
$ping -M do -s 8900 NFS-Server-IP
variable overhead is added to payload, it is safe to start with 8900 byte