Monday, December 20, 2010

Why 32bit Linux only see 3GB memory in a machine with 4GB RAM.

The Symptom:
OS detected  4194304k but only 3105024k is available.

[32bit Linux]$ dmesg | grep -i mem
3200MB HIGHMEM available.
896MB LOWMEM available.
HighMem zone: 819200 pages, LIFO batch:31
ACPI: SRAT (v002 VMWARE MEMPLUG  0x06040000 VMW  0x00000001) @ 0xbfeef3e1
Memory: 3105024k/4194304k available (1584k kernel code, 39372k reserved, 622k data, 168k init, 2228096k highmem)
[32bit Linux]$free 
total       used       free     shared    buffers     cached
Mem:       3108956      73076    3035880          0       8000      37932
-/+ buffers/cache:      27144    3081812
Swap:       779112          0     779112
The Theory: 
hugemem/PAE enabled kernel  is ONLY needed for RAM from 4GB up to 64GB, The generic 32bit Linux Kernel can see 4GB without hugemem/PAE enabled kernel.
The Cause: 
Sometimes, why 32bit Linux can't see all the 4GB ram? The answer is from “physical RAM map” provided by BIOS.
The Analysis:
[32bit Linux]$dmesg | less
Linux version (geeko@buildhost) (gcc version 4.1.2 20070115 (SUSE Linux)) #1 Tue May 6 12:41:02 UTC 2008
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 00000000000e4000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bfee0000 (usable)
BIOS-e820: 00000000bfee0000 - 00000000bfeff000 (ACPI data)
BIOS-e820: 00000000bfeff000 - 00000000bff00000 (ACPI NVS)
BIOS-e820: 00000000bff00000 - 00000000c0000000 (usable)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
Warning only 4GB will be used.
Use a PAE enabled kernel.
3200MB HIGHMEM available.
896MB LOWMEM available.
The interesting part of  physical RAM map is the last line, and interesting number is  0000000100000000 which is hex value of 4GB. 
Converting them decimal values to be more readable.

[32bit Linux]$ dmesg | awk  --re-interval   --non-decimal-data '/[0-9a-z]{16}/  {  x1=sprintf ("%d", "0x"$2) ; x2=sprintf ("%d", "0x"$4);  printf "%d %s %d %s %d %s\n",  x1/1024/1024, " - ", x2/1024/1024, "=", (x1-x2)/1024/1024,$NF }' 
0  -  0 = -0 (usable)
0  -  0 = -0 (reserved)
0  -  0 = -0 (reserved)
0  -  0 = -0 (reserved)
0  -  1 = -0 (reserved)
1  -  3070 = -3069 (usable)
3070  -  3070 = -0 data)
3070  -  3071 = -0 NVS)
3071  -  3072 = -1 (usable)
3584  -  3840 = -256 (reserved)
4076  -  4076 = -0 (reserved)
4078  -  4078 = -0 (reserved)
4095  -  4096 = -0 (reserved)
4096  -  5120 = -1024 (usable)

It is clear that BIOS has too many reserved parts  in lower address space and push the one trunk of 1GB over 4GB address space.
The Solution:
 1)To release some reserved space and bring all usable spaces below 4GB, you might try to disable some devices in BIOS. It is the best option, but might not be achievable , consult with your hardware vendor.
 2)Reinstall system with 64bit Kernel
 3)Install hugemem/PAE kernel on current 32 Bit system. It is the last option because hugemem/PAE  kernel hurt performance due to the “dynamic rempapping with three-level paging model.

Tuesday, December 14, 2010

Zabbix VS Zenoss

Zenoss was my favourite network monitoring tool, But it is Zabbix now. The conclusion was made after comparing Zenoss 3.0.3 to Zabbix 1.8.3 in Lab environment.

Features Zenoss Zabbix
Server Zope application server (in Phyton) Server Daemon (Written in C)
Frontend Phyton PHP
Backend Mysql MySQL, PostgreSQL, SQLite or Oracle
Client None Linux,Window,BSD,AIX,HP-UX,Solaris
Monitoring methods SNMP/SSH/External command SNMP/Agent/SSH/External command
Graphing Supported Supported
Cost Zenoss Core and community contributed zenpacks are free All features are free
Others Latest package in RPMS/DEB, very easy to install
Heavy weight due to Phyton and Zope, requires higher hardware specs
Latest package has to be compiled from source
Light weight, the Server daemon has very small footprint, the other components are just web server(e.g. Apache) and DB server.
As shown in the above table, Zabbix has more features than Zenoss's  

Zenoss Strength

Better looking GUI Zabbix's  PHP can't beat Zenoss's Phyton
Better support for SNMP
Zenoss has built-in snmp template for monitoring servers, Server auto discovery is smooth
Because Zabbix is more agent focused, Zabbix doesn't include snmp template  for monitoring servers (at least in 1.8.3), You have to create your own.
Lots of community contributed zenpacks
Better support for agent-less type of  Windows  Servers

Actually Zabbix has better support for  Windows servers, if you don't mind installing monitoring agent.
For agent-less monitoring, Zenoss is easier than Zabbix. Because Zenoss has  zenpacks for Windows.
Zabbix users have to write their own WMI scripts using  WMI tool, which can be downloaded and complied from
or use SNMP Provider to link SNMP to WMI

Zabbix Strength

You don't need to create graph definition for a single data source, the graph can be dynamically generated on demand. For multi-data source graph (e.g. SYS CPU/USR CPU in one graph) , the graph definition is easier than Zenoss (changing display order is just a matter of clicking up/down button.
Alarm expression It seems Zenoss only supports alarm value ranging  from min to max in numeric values.
Zabbix supports dozens of expressions: avg,min,max,last,diff,regexp …
for example you can use regexp ('ERROR') to check if the response contains string “ERROR”
Web application monitoring
Zabbix supports  complex scenarios consisting of many steps (HTTP requests) .e.g login, execute some tasks, logout 
Unique Agent checks Search Windows Event log for a string: eventlog[logtype,<pattern>,<severity>,<source>,<eventid>,<maxlines>]   
Search any text file for a string:
Check if a file has been modified:
vfs.file.cksum [file]:
Zabbix Proxy
Zabbix Proxy is a process which collects performance and availability data from one or more monitored devices and sends the information to a Zabbix Server.  It is very useful for monitoring device which can't be reached by  Zabbix Server
---Update Zabbix 2.0 Low Level Discovery
Zabbix 2.0 has new feature called Low Level Discovery,It can do SNMP dynamic discovery and any customized application discovery based on JSON format. In order to monitor an item of a script output, other monitoring system  I know of requires item name has to be typed in manually through UI, but Zabbix can discover the item automatically, if there dozens of items to be monitored, huge  amount of maintenance effort will be saved. check out this example.

Thursday, December 2, 2010

Floating point mathematical operations in Shell Script

Ksh supports floating point natively but bash doesn't, bash has to rely on external command to support floating point

Bash doesn't support floating point .
[bash]$printf "%.2f\n", $((10.0/3.0))
-bash: 10.0/3.0: syntax error in expression (error token is ".0/3.0") 
Ksh supports floating point  perfectly 
[ksh]$ printf "%.2f\n", $((10.0/3.0))
Because the input (10.0, 3.0) are floating point, the output will be floating point by default, but you can force to treat it as integer.
[ksh]$ integer x
[ksh]$ ((x=10.0/3.0))
[ksh]$printf "%.2f\n", $x
External commands supporting  floating point
the following example shows bc and awk, which are available in any shell
$echo 'scale = 2 ;10/3' | bc
$awk 'BEGIN {printf "%.2f\n",10/3 ;}'

Tuesday, October 12, 2010

Vsphere PowerCLI script to clone and customize Linux guest OS

PowerShell is very powerful object-orientated programming language inWindows, the comparable counterpart in Unix/Linux is Python.
I couldn’t find a PowerCLI script to clone Linux and assign ip addresses to multiple adapters so I decided to write my own.
If you are new to Vsphere PowerCLI, you may need to check my previous post to learn how to connect to vi-server via PowerCLI at first.

There are 2 scripts:
discover-info.ps1  # Discovery information of target viserver/ESX host and generate template file to be used for clonevm.ps1
vmclone.ps1 #clone after read information from template file generated by discover-info.ps1 script
The scripts has been tested in VMware vSphere PowerCLI 4.0 for VMs in ESX 3.5/ESX4.0 managed by Virtual Center Server.
Scripts in action
[vSphere PowerCLI] D:\Temp> .\discover-info.ps1
Discovered ESX hosts:esx01 esx02 
Please enter target ESX hostname: esx01
Template file: vmdata.txt has been generated. Edit the file then run 'clonevm.ps1 -f file-name' to start cloning
[vSphere PowerCLI] D:\Temp> cat vmdata.txt (after customization)
####---------------- Discovered information
#PortGroups='Service Console' 'vlan-01' 'vlan-02'
#Folders='host' 'Discovered Virtual Machine' 'Datacenters' 'vm'
#DataStores= 'nfs_storage' 'esx01:storage1'
#resourcepools=Name: Resources ID:ResourcePool-resgroup-7 Name: Resources ID:ResourcePool-resgroup-65
####----------------Customize your VM values using the template 
##Resource pool id is  NOT required for using virtual machine as source, but IS required for  using template as source
#vm means Virtual Machine; vt means template (  resourcepoolid  is required for vt)
##Use vm as folder if no user folders defined
##Gateway is optional, enter - to skip this value
##Networkadapter type  is optional (Auto choosen for guest OS) , enter - to skip this value
##Supported adapter types: e.g e1000, Flexible, Vmxnet, EnhancedVmxnet
Network=EnhancedVmxnet,vlan-01 ,,, ; -,vlan-02,,,-
[vSphere PowerCLI] D:\Temp> .\clonevm.ps1 -f .\vmdata.txt
==> Retrieving info for source VM: linuxtemplate
==> Starting clone process ...
4%  10%  16%  21%  27%  33%  38%  45%  51%  57%  62%  68%  74%  79%  85%  91%  %
Completed Task: Task-task-2366
==> Checking new VM ...
Name                 PowerState Num CPUs Memory (MB)
----                 ---------- -------- -----------
pstest1              PoweredOff 1        4096
==> Removing existing network adapters ...
==> Adding  2 new  network adapters ...
==> Customizing guest OS ...
Completed Task: Task-task-2370
==>  Checking new VM ...
pstest1              PoweredOff 1        4096

Discover-info.ps1    download

# *Execute the script in a live PowerCLI VIsserver session*
# Generate template file to be used for clonevm.ps1
# DATE: 12 Oct 2010 
# Author:     
$hosts=get-vmhost | foreach { $ }
write-host "Discovered ESX hosts: $hosts"
switch  ($hosts.length) {
0 {  write-host "no ESX host found!  exiting.. " ; exit}
1 {  $esxhost=$hosts[0]  }
default { 
$esxhost=read-host "Please enter target ESX hostname"
if ( -not $($hosts -contains "$esxhost")  ) { write-host "Unknown host: '$esxhost', exiting.. " ; exit }
$pgs=Get-VirtualPortGroup -vmhost $esxhost |  foreach  { "'$_'" }
$folders=get-folder |  foreach  { "'$_'" }
$dss=get-datastore -vmhost $esxhost | foreach { "'$_'"}
$rspool=get-resourcepool |  foreach  { "Name: " + $ + " ID:" + $ }
write-output   " 
####---------------- Discovered information
#DataStores= $dss
####----------------Customize your VM values using the template 
##Resource pool id is  NOT required for using virtual machine as source, but IS required for  using template as source
#vm means Virtual Machine; vt means template (  resourcepoolid  is required for vt)
##Use vm as folder if no user folders defined
##Gateway is optional, enter - to skip this value
##Networkadapter type  is optional (Auto choosen for guest OS) , enter - to skip this value
##Supported adapter types: e.g e1000, Flexible, Vmxnet, EnhancedVmxnet
Network=adapter-type,port-group1,ip-address1,net-mask1,gateway1 ; EnhancedVmxnet,port-group2,ip-address2,net-mask2,gateway2 ; ...
"  | tee $template_file
write-host "Template file: $template_file has been generated. Edit the file then run 'clonevm.ps1 -f file-name' to start cloning"

clonevm.ps1 download

# *Execute the script in a live PowerCLI VIsserver session*
##### Clone VM for Linux guest
#Read information from template file generated by discover-info.ps1 script
#Set target host/datastore for Clone ;Start Clone
#Delete existing adapters; add new adapters; Customize guest OS; powerone VM (Optional)
#DATE: 12 Oct 2010 
if ( ( $args.count -ne 2 ) -or   (  $args[0] -ne "-f" ) ) { write-host "Usage: .\program -f template-filename" ; exit}
function Retrieve-Values {
$line=[string] ($strings |  select-string -pattern "$item")
return $line.Substring( $line.LastIndexOf("=")+1 )
function wait-for-task {
if ( $args[0] -eq $null ) {write-host "failed " ;exit }
$progress=get-task  -status running | foreach { if ( $ -eq  $tskid ) {  $_.PercentComplete } }
while ( $progress -ne $null ) {
sleep 30
$progress=get-task  -status running | foreach { if ( $ -eq  $tskid ) {  $_.PercentComplete} }
write-host "$progress%  " -nonewline
write-host  ""
write-host "Completed Task:" $tskid
$lines=get-content $file1 -ErrorAction stop |  select-string -notmatch "(^#|^$)"   -ErrorAction stop 
$s_esxhost=Retrieve-values $lines  "esxhost"
$s_rspoolid=Retrieve-values $lines  "resourcepoolid"
$s_srcname=Retrieve-values $lines  "SrcName"
$s_dstname=Retrieve-values $lines  "Dstname"
$s_folder=Retrieve-values $lines  "Folder"
$s_datastore=Retrieve-values $lines  "DataStore"
$s_domain=Retrieve-values $lines  "Domain"
$s_network=Retrieve-values $lines  "Network"
$s_poweron=Retrieve-values $lines  "Poweron"
$s_dns=Retrieve-values $lines  "DNS"
#Initialize hash table
$h_network=@{atype=(1..$nic_cnt); pgroup=(1..$nic_cnt); ip=(1..$nic_cnt); mask=(1..$nic_cnt); gateway=(1..$nic_cnt) }
For ($i=0; $i -lt $nic_cnt ; $i++) {
if ( $array_temp2.count -ne  5 ) { write-host "Expected the number of values for network is 5, given"  $array_temp2.count ; exit}
$vmclonespec_host = New-Object VMware.Vim.VirtualMachineCloneSpec
#### Customize  Target host /Target datastore
$vmclonespec_host.location = New-Object  VMware.Vim.VirtualMachineRelocateSpec
$ = (Get-vmHost -Name $s_esxhost | Get-View).MoRef
$vmclonespec_host.location.datastore = (Get-Datastore -Name $s_datastore| Get-View).MoRef
if ( $s_rspoolid.length -gt 0 )  {
$vmclonespec_host.location.pool= (Get-ResourcePool -id $s_rspoolid  -ErrorAction stop|get-view).MoRef 
####  Customize Guest OS
$vmclonespec_os= New-Object  VMware.Vim.CustomizationSpec
## Identity for Linux 
$vmclonespec_os.identity= New-Object VMware.Vim.CustomizationLinuxPrep
$vmclonespec_os.identity.hostname= New-Object VMware.Vim.CustomizationFixedName
$ $s_dstname
$vmclonespec_os.GlobalIPSettings = New-Object VMware.Vim.CustomizationGlobalIPSettings
# adapter mapping
for ( $i=0;  $i -lt $nic_cnt; $i++ )  {
$vmclonespec_os.NicSettingMap += @(New-Object VMware.Vim.CustomizationAdapterMapping) 
$vmclonespec_os.NicSettingMap[$i].Adapter = New-Object VMware.Vim.CustomizationIPSettings
$vmclonespec_os.NicSettingMap[$i].Adapter = New-Object VMware.Vim.CustomizationIPSettings
# FixedIP
$vmclonespec_os.NicSettingMap[$i].Adapter.Ip = New-Object VMware.Vim.CustomizationFixedIp
$vmclonespec_os.NicSettingMap[$i].Adapter.Ip.IpAddress = $h_network.ip[$i]
$vmclonespec_os.NicSettingMap[$i].Adapter.SubnetMask = $h_network.mask[$i]
if ( $h_network.gateway[$i] -ne "-" ) {
$vmclonespec_os.NicSettingMap[$i].Adapter.Gateway = $h_network.gateway[$i] 
$s_srcname=$s_srcname.Substring( $s_srcname.LastIndexOf(",")+1 )
# Template  or Virtual Machine from which the guest will be cloned
write-host "==> Retrieving info for source VM: $s_srcname"
if ( $s_srcname_pre -eq "vm," ) {
$svm = Get-View (get-vm  $s_srcname -ErrorAction stop ).ID 
else {
$svm = Get-View (get-template  $s_srcname -ErrorAction stop ).ID  
$tfld = Get-Folder -Name $s_folder -ErrorAction stop  | Get-View
write-host "==> Starting clone process ... "
$taskid=$svm.CloneVM_Task($tfld.moref,$s_dstname,$vmclonespec_host)   | foreach {$_.value}
Wait-for-Task  "Task-$taskid"
write-host "==> Checking new VM ..." 
get-vm $s_dstname -ErrorAction stop 
write-host "==> Removing existing network adapters ...  "
$adapter = Get-NetworkAdapter -VM "$s_dstname"   -ErrorAction stop
if ( $adapter -ne $null ) { Remove-NetworkAdapter -NetworkAdapter $adapter -confirm:$false }
write-host "==> Adding "  $nic_cnt "new  network adapters ...  "
for ( $i=0;  $i -lt $nic_cnt; $i++ )  {
if ( $h_network.atype[$i]  -ne "-" ) {
New-NetworkAdapter -Type  $h_network.atype[$i]   -NetworkName $h_network.pgroup[$i] -VM $s_dstname  -StartConnected -ErrorAction stop
else {
New-NetworkAdapter     -NetworkName $h_network.pgroup[$i] -VM $s_dstname  -StartConnected -ErrorAction stop
write-host "==> Customizing guest OS ...  "
$newvm = Get-View (get-vm  $s_dstname -ErrorAction stop ).ID 
$taskid=$newvm.CustomizeVM_Task($vmclonespec_os) | foreach {$_.value}
Wait-for-Task  "Task-$taskid"
write-host "==>  Checking new VM ..." 
sleep 4
get-vm $s_dstname -ErrorAction stop
if ( $S_poweron -eq "yes" ) { start-vm $s_dstname }

Related Documents:

1. vSphere PowerCLI Quick Start

2. Master-PowerShell 
3. VMware vSphere Web Services SDK 4.0 (Supported languages: PowerShell  C# JAVA Perl )

Thursday, September 30, 2010

VMware ESX command line essentials PART #2

Following on PART #1, PART #2 is focusing on exploring storage Information.

Display currently mounted storage

####vdf for ESX host  is equivalent to  df  command for Linux, it shows any type (SCSI/FC/NFS/ISCSI)
$ vdf -h
Filesystem            Size  Used Avail Use% Mounted on
135G   16G  118G  12% /vmfs/volumes/esx01:stor1
1.7T  1.5T  206G  88% /vmfs/volumes/nfs_stor 
####Check detailed file system information on the storage
$ vmkfstools -P -h /vmfs/volumes/esx01:stor1
VMFS-3.33 file system spanning 1 partitions.
File system label (if any): esx01:stor1
Mode: public
Capacity 135.2 GB, 118.5 GB available, file block size 1 MB
UUID: 4c5be33b-3a356289-ea3f-0024c1a35c5c
Partitions spanned (on "lvm"):
$ vmkfstools -P -h /vmfs/volumes/nfs_stor
NFS-1.00 file system spanning 1 partitions.
File system label (if any): nfs_stor
Mode: public
Capacity 1.7 TB, 206.5 GB available, file block size 4 KB
UUID: 7292e403-4f367279-0000-000000000000
Partitions spanned (on "notDCS"):

Find out storage type (SCSI/FC/NFS/ISCSI) 
[ESX4]$esxcfg-scsidevs -m
mpx.vmhba0:C0:T0:L0:5   /dev/cciss/c0d0p5    4c5be33b-3a356289-ea3f-0024c1a35c5c  0  esx01:stor1
[ESX3]$ esxcfg-vmhbadevs  -m
vmhba0:0:0:9    /dev/cciss/c0d0p9                4951068b-1dabf5db-74c8-0022640e9ea6
$esxcfg-nas -l
nfs_stor is /vol/vmdk_vol1 from  mounted
vmkiscsi-tool ?

Find out adapter Vendor/ Device name

$ esxcfg-info | grep -B 12 vmhba0  
|----Vendor Name........................................Hewlett-Packard Company
|----Device Name........................................Smart Array E200i
|----Device Class.......................................260
|----PIC Line...........................................11
|----Service Console IRQ................................25
|----PCI Pin............................................0
|----Spawned Bus........................................0
|----Module Id..........................................34
|----Physical Slot......................................255
|----VmKernel Device Name...............................vmhba0

How to find out storage not mounted

[ESX3]$esxcfg-mpath -l
Disk vmhba0:0:0 /dev/cciss/c0d0 (139979MB) has 1 paths and policy of Fixed
Local 66:8.0 vmhba0:0:0 On active preferred
[ESX4]$  esxcfg-scsidevs -l
Device Type: Direct-Access
Size: 139979 MB
Display Name: Local VMware Disk (mpx.vmhba0:C0:T0:L0)
Plugin: NMP
Console Device: /dev/cciss/c0d0
Devfs Path: /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0
Vendor: VMware    Model: Block device      Revis: 1.0
SCSI Level: 2  Is Pseudo: false Status: on
Is RDM Capable: false Is Removable: false
Is Local: true
Other Names:
[ESX4]$ esxcfg-mpath -l device=vml.0000000000566d676261303a303b30
Runtime Name: vmhba0:C0:T0:L0
Device: mpx.vmhba0:C0:T0:L0
Device Display Name: Local VMware Disk (mpx.vmhba0:C0:T0:L0)
Adapter: vmhba0 Channel: 0 Target: 0 LUN: 0
Adapter Identifier: block.cciss/c0d0:0
Target Identifier: block.0:0
Plugin: NMP
State: active
Transport: block


####show vmware crash dump partition 
esxcfg-dumppart -l 
####clone disk/convert thick disk to thin disk 
vmkfstools   -i /vmfs/volumes/storage1/vm01.vmdk  -d thin    /vmfs/volumes/storage2/vm02.vmdk 
Thick Disk: size of ls output = size of du output 
Thin Disk: size of ls output > size of du output

Wednesday, September 29, 2010

VMware ESX command line essentials PART #1

[1] Overview of the methods to manage Vmware ESX/ESXi Host

1.1 ESX host native commands
1) Basic commands:
/usr/sbin/esxcfg-* commands ,  vmware-cmd, vmkfstools etc
2) Advanced command:
/usr/bin/vimsh:  an interactive shell to peform almost any task that can be done in vCenter

1.2 External management tools to manage ESX/ESXi Host
VMware vSphere PowerCLI
"SDK"  for  Windows  to perform any task that can be done in vCenter.
Interactive shell with 200+ PowerShell cmdlets, easy to use
Supported OS: Windows
Supported Target: ESX/ESXi or vCenter Server
Vmware  vSphere SDK for Perl SDK for Perl to perform any task that can be done in vCenter
Supported OS: Windows/Linux
Supported Target: ESX/ESXi host or vCenter Server

VI Perl Toolkit
Based on Perl SDK, Some scripts  to perform common administration tasks ( ... etc)
VI Perl Tookit only has 10+ utilities compared to PowerCLI  200+  cmdlets.

VMware vSphere  vCLI
Based on Perl SDK, port most ESX host native commands (esxcfg*,vmware-cmd...) to external management host. It is commonly used for ESXi host which doesn't have service console.
Supported OS: Windows/Linux
Supported Target: ESX/ESXi host or vCenter Server

[2] VMware basic concepts

Memory management
[Memory Size] This is the amount of memory we assign to the vm and is also the amount of memory the guest OS will see as its physical memory. This is a HARD LIMIT  and the vm cannot exceed this limit if it demands more memory.
[Reservation]  A guaranteed amount of memory assigned to the vm. This is a way of ensuring that the vm gets a minimal amount of memory assigned
[Limit] SOFT LIMIT  controlled by VMkernel,  SOFT LIMIT<=HARD LIMIT (Memory size),  The default value is unlimited which is VM Configured Size
Maximum allocation target = min{ VM’s memory size, VM’s limit }
[Share]  the priority/weight when requesting resource, High:normal:low = 4:2:1

Memory management key points
1) Allocation rule: Reservation  <=the amount  dynamically allocated  based on shares <=limits <= memory size
2) It is important to set reservation  for mission critical VMs, but limit is not very important.
3) It is important to have VM tools running inside VM. Because ballooning driver (vmmemctl) in VM tools is more efficient than host swapping under memory overcommitment.
(Under memory overcommitment, reclamation techniques techniques are transparent page sharing, ballooning, and host swapping)
Resource pool
Cluster is a big resource pool by default
You can create sub pool under the Cluster and drag vm to the sub pool in order to fine tune VM resource.

VMware virtual machine files
location: /vmfs/volumes/$DATASTORE/$VIRTUALMACHINE
.vmx This is the primary configuration file
.nvram  This is the file that stores the state of the virtual machine's BIOS.
.vmdk   Disk DescriptorFile (ASCII text, less than 1M )
*-flat.vmdk Binary file stores the contents of the virtual machine's hard disk drive.
.vmem  The virtual machine's paging file
.vmsn VM snapshot file

Vmware configuration files in /etc/vmware/
/etc/vmware/esx.conf  is the ESX host configuration file.
Vmware runtime information in /proc/vmware/
/proc/vmware/version /proc/vmware/cpuinfo … etc.
VMware services control scripts  in /etc/init.d/
vmware : Vmware basic service supporting VMS, restarting vmware will affect running VMs
mgmt-vmware: Vmware Host Agent (/usr/lib/vmware/bin/vmware-hostd). Vmware management service, restart won't affect running vms
vmware-aam  :Vmware HA agent
vmware-vmkauthd: VMware VMkernel authorization daemon
vmware-webAccess: Vmware Web management interface 
vmware-autostart: auto start/stop VMs configured to do so
vmware-late: VMware late init tools (start kernel logger, nfs/iscsi volumes .. )
vmware-vpxa VirtualCenter agents

[3] VMware basic administrative tasks:

####Get ESX host Vmware version
$cat /proc/vmware/version
VMware ESX 4.0.0 [Releasebuild-208167], built on Nov  8 2009
$/usr/bin/vmware -v
VMware ESX 4.0.0 build-208167
####List running VMS
$/usr/bin/vmware-cmd -l
$/usr/bin/vm-support -x
vmid=4331       vm-01
$vimsh -n -e vmsvc/getallvms
Vmid         Name                                    File                                Guest OS      Version
16     vm01   [stor1] vm01/vm01.vmx   sles10_64Guest   vmx-07
####Manage VMs
/usr/bin/vmware-cmd <cfg file ( .vmx)>  command
VM command:
start [soft|hard|trysoft]
stop [soft|hard|trysoft]
reset [soft|hard|trysoft]
suspend [soft|hard|trysoft]
getconfig <variable>
setguestinfo <variable> <value>
getguestinfo <variable>
getproductinfo <product_info>
connectdevice <device_name>
disconnectdevice <device_name>
createsnapshot <name> <description> <quiesce> <memory>
####register and unregister VMs
/usr/bin/vmware-cmd -s register <config_file_path>
/usr/bin/vmware-cmd -s unregister <config_file_path>
ping from vmkernel interface instead of service console interface. e.g checking connectivity to NFS server where VMS reside.
####esxcfg-advcfg   -g
-g Get/set  the value of the VMkernel advanced configuration option listed in esxcfg-info -o 
$ esxcfg-advcfg -g /Irq/RoutingPolicy
Value of RoutingPolicy is 2
####esxcfg-advcfg   -c
- c|--get-config         Get the value of a specific line in esx.conf.
esxcfg-advcfg -c /adv/Misc/HostName
An interactive shell on ESX host to provide almost any operation available in virtual centre
$ /usr/bin/vimsh
-n          Non-interactive mode.  Just quit after processing command line.
-e <cmd>    Run a series of commands specified by <cmd>.
$ vimsh -n -e vmsvc/getallvms
To find out the command name, press tab key to list commands in interactive shell(start vimsh without any argument)

PART #2 will be focusing on VMware command line storage management