Monday, December 20, 2010

Why 32bit Linux only see 3GB memory in a machine with 4GB RAM.

The Symptom:
OS detected  4194304k but only 3105024k is available.

[32bit Linux]$ dmesg | grep -i mem
3200MB HIGHMEM available.
896MB LOWMEM available.
HighMem zone: 819200 pages, LIFO batch:31
ACPI: SRAT (v002 VMWARE MEMPLUG  0x06040000 VMW  0x00000001) @ 0xbfeef3e1
Memory: 3105024k/4194304k available (1584k kernel code, 39372k reserved, 622k data, 168k init, 2228096k highmem)
[32bit Linux]$free 
total       used       free     shared    buffers     cached
Mem:       3108956      73076    3035880          0       8000      37932
-/+ buffers/cache:      27144    3081812
Swap:       779112          0     779112
The Theory: 
hugemem/PAE enabled kernel  is ONLY needed for RAM from 4GB up to 64GB, The generic 32bit Linux Kernel can see 4GB without hugemem/PAE enabled kernel.
The Cause: 
Sometimes, why 32bit Linux can't see all the 4GB ram? The answer is from “physical RAM map” provided by BIOS.
The Analysis:
[32bit Linux]$dmesg | less
Linux version (geeko@buildhost) (gcc version 4.1.2 20070115 (SUSE Linux)) #1 Tue May 6 12:41:02 UTC 2008
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 00000000000e4000 (reserved)
BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 00000000bfee0000 (usable)
BIOS-e820: 00000000bfee0000 - 00000000bfeff000 (ACPI data)
BIOS-e820: 00000000bfeff000 - 00000000bff00000 (ACPI NVS)
BIOS-e820: 00000000bff00000 - 00000000c0000000 (usable)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
BIOS-e820: 0000000100000000 - 0000000140000000 (usable)
Warning only 4GB will be used.
Use a PAE enabled kernel.
3200MB HIGHMEM available.
896MB LOWMEM available.
The interesting part of  physical RAM map is the last line, and interesting number is  0000000100000000 which is hex value of 4GB. 
Converting them decimal values to be more readable.

[32bit Linux]$ dmesg | awk  --re-interval   --non-decimal-data '/[0-9a-z]{16}/  {  x1=sprintf ("%d", "0x"$2) ; x2=sprintf ("%d", "0x"$4);  printf "%d %s %d %s %d %s\n",  x1/1024/1024, " - ", x2/1024/1024, "=", (x1-x2)/1024/1024,$NF }' 
0  -  0 = -0 (usable)
0  -  0 = -0 (reserved)
0  -  0 = -0 (reserved)
0  -  0 = -0 (reserved)
0  -  1 = -0 (reserved)
1  -  3070 = -3069 (usable)
3070  -  3070 = -0 data)
3070  -  3071 = -0 NVS)
3071  -  3072 = -1 (usable)
3584  -  3840 = -256 (reserved)
4076  -  4076 = -0 (reserved)
4078  -  4078 = -0 (reserved)
4095  -  4096 = -0 (reserved)
4096  -  5120 = -1024 (usable)

It is clear that BIOS has too many reserved parts  in lower address space and push the one trunk of 1GB over 4GB address space.
The Solution:
 1)To release some reserved space and bring all usable spaces below 4GB, you might try to disable some devices in BIOS. It is the best option, but might not be achievable , consult with your hardware vendor.
 2)Reinstall system with 64bit Kernel
 3)Install hugemem/PAE kernel on current 32 Bit system. It is the last option because hugemem/PAE  kernel hurt performance due to the “dynamic rempapping with three-level paging model.

Tuesday, December 14, 2010

Zabbix VS Zenoss

Zenoss was my favourite network monitoring tool, But it is Zabbix now. The conclusion was made after comparing Zenoss 3.0.3 to Zabbix 1.8.3 in Lab environment.

Features Zenoss Zabbix
Server Zope application server (in Phyton) Server Daemon (Written in C)
Frontend Phyton PHP
Backend Mysql MySQL, PostgreSQL, SQLite or Oracle
Client None Linux,Window,BSD,AIX,HP-UX,Solaris
Monitoring methods SNMP/SSH/External command SNMP/Agent/SSH/External command
Graphing Supported Supported
Cost Zenoss Core and community contributed zenpacks are free All features are free
Others Latest package in RPMS/DEB, very easy to install
Heavy weight due to Phyton and Zope, requires higher hardware specs
Latest package has to be compiled from source
Light weight, the Server daemon has very small footprint, the other components are just web server(e.g. Apache) and DB server.
As shown in the above table, Zabbix has more features than Zenoss's  

Zenoss Strength

Better looking GUI Zabbix's  PHP can't beat Zenoss's Phyton
Better support for SNMP
Zenoss has built-in snmp template for monitoring servers, Server auto discovery is smooth
Because Zabbix is more agent focused, Zabbix doesn't include snmp template  for monitoring servers (at least in 1.8.3), You have to create your own.
Lots of community contributed zenpacks
Better support for agent-less type of  Windows  Servers

Actually Zabbix has better support for  Windows servers, if you don't mind installing monitoring agent.
For agent-less monitoring, Zenoss is easier than Zabbix. Because Zenoss has  zenpacks for Windows.
Zabbix users have to write their own WMI scripts using  WMI tool, which can be downloaded and complied from
or use SNMP Provider to link SNMP to WMI

Zabbix Strength

You don't need to create graph definition for a single data source, the graph can be dynamically generated on demand. For multi-data source graph (e.g. SYS CPU/USR CPU in one graph) , the graph definition is easier than Zenoss (changing display order is just a matter of clicking up/down button.
Alarm expression It seems Zenoss only supports alarm value ranging  from min to max in numeric values.
Zabbix supports dozens of expressions: avg,min,max,last,diff,regexp …
for example you can use regexp ('ERROR') to check if the response contains string “ERROR”
Web application monitoring
Zabbix supports  complex scenarios consisting of many steps (HTTP requests) .e.g login, execute some tasks, logout 
Unique Agent checks Search Windows Event log for a string: eventlog[logtype,<pattern>,<severity>,<source>,<eventid>,<maxlines>]   
Search any text file for a string:
Check if a file has been modified:
vfs.file.cksum [file]:
Zabbix Proxy
Zabbix Proxy is a process which collects performance and availability data from one or more monitored devices and sends the information to a Zabbix Server.  It is very useful for monitoring device which can't be reached by  Zabbix Server
---Update Zabbix 2.0 Low Level Discovery
Zabbix 2.0 has new feature called Low Level Discovery,It can do SNMP dynamic discovery and any customized application discovery based on JSON format. In order to monitor an item of a script output, other monitoring system  I know of requires item name has to be typed in manually through UI, but Zabbix can discover the item automatically, if there dozens of items to be monitored, huge  amount of maintenance effort will be saved. check out this example.

Thursday, December 2, 2010

Floating point mathematical operations in Shell Script

Ksh supports floating point natively but bash doesn't, bash has to rely on external command to support floating point

Bash doesn't support floating point .
[bash]$printf "%.2f\n", $((10.0/3.0))
-bash: 10.0/3.0: syntax error in expression (error token is ".0/3.0") 
Ksh supports floating point  perfectly 
[ksh]$ printf "%.2f\n", $((10.0/3.0))
Because the input (10.0, 3.0) are floating point, the output will be floating point by default, but you can force to treat it as integer.
[ksh]$ integer x
[ksh]$ ((x=10.0/3.0))
[ksh]$printf "%.2f\n", $x
External commands supporting  floating point
the following example shows bc and awk, which are available in any shell
$echo 'scale = 2 ;10/3' | bc
$awk 'BEGIN {printf "%.2f\n",10/3 ;}'