Setup SAN Boot for RHEL 6.x using native multipath on EMC storage

Requirements:
1) RHEL 6.x (most apply to RHEL 5.x too, RHEL 5.x use mkinitrd instead of Dracut and the /etc/multipath.conf is slightly different refer to Red Hat KB in reference section)
2) EMC storage was setup with Active/Active (ALUA)
3) Boot LUN was presented with single path for initial install


Procedures:

1. Server boots up after initial install
2. Login to server as root to enable multipath
[root@server1]#mpathconf --enable –-with_multipathd y
3. Edit /etc/multipath.conf and make sure it only contains following valid parameters

blacklist {
}


defaults {
 user_friendly_names yes
}
devices {
  device {
    vendor "DGC"
    product ".*"
    product_blacklist "LUNZ"
    hardware_handler "1 alua"   
    path_checker directio    
    prio alua                
  }
}

4.Find out the logical path the root disk is mapped
[root@server1]#multipath –v3
It should be /dev/mapper/mpatha

5. Create initramfs with multipath module
[root@server1]#dracut --force -–add multipath

6. Make sure multipath.conf is included in initrd image

[root@server1]#lsinitrd /boot/initramfs-*.x86_64.img | grep multipath.conf
-rw-r--r--   1 root     root         2525 Feb 27 13:31 etc/multipath.conf

7. Modify the /boot/grub/device.map and change

 (hd0) /dev/sda    to
 (hd0) /dev/mapper/mpatha

This is assuming the boot disk is on /dev/mapper/mpatha as verified in step 2 above.

8. Reboot the server.

9. Verify multipath, check hwhandler='1 alua' and member disk sda for mpatha

[root@server1]#multipath –ll 
mpatha (3600601609973310067eb1e1ed69ae311) dm-0 DGC,VRAID
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:0 sda 8:0   active ready running
  

10. Ask storage administrator to enable other paths for boot LUN.
11. Reboot server again after multipath is aenabled in storage too
12. Login server to verify all paths, check hwhandler='1 alua' prio>0
if hwhandler='1 emc' or prio=0 means PNR mode

[root@server1]#multipath -ll
mpatha (3600601609973310067eb1e1ed69ae311) dm-0 DGC,VRAID
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 1:0:1:0 sdd 8:48  active ready running
| `- 2:0:1:0 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:0 sda 8:0   active ready running
  `- 2:0:0:0 sdg 8:96  active ready running
mpathb (360060160997331009fd6e124d69ae311) dm-1 DGC,VRAID
size=800G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 1:0:0:1 sdb 8:16  active ready running
| `- 2:0:0:1 sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:1:1 sde 8:64  active ready running
  `- 2:0:1:1 sdk 8:160 active ready running

13. Partition other LUNS using fdisk command as normal, but use logical path /dev/mapper/mpathb etc (partition will be created as /dev/mapper/mpathbp1 instead of /dev/mapper/mpathb1
NOTE: any change to /etc/multipath.conf requires re-create initramfs ( dracut --force -–add multipath) and a reboot, because the boot LUN is on SAN, if boot LUN is local disk, change to /etc/multipath.conf only requires multipathd restart

Script to automatically partition a new disk and create LVM PV

It is a very common task to create a single partition on  whole disk  and create LVM PV, How to automate it?

fdisk doesn't support making partition in script mode, sfdisk can, but it is not as good as the powerful parted tool. parted can also optimize partition alignment automatically(parted -a optimal).

#!/bin/ksh
#Create a single primary partiton with whole disk size and create LVM PV on it
disk=$1
partno=1
if [[ -z $disk ]]; then
 echo "Usage: $0 disk device name: e.g $0 /dev/sdb"
 exit
fi


if [[ -e ${disk}${partno} ]]; then
 echo "==> ${disk}${partno} already exist"
 exit
fi

echo "==> Create MBR label"
parted -s $disk  mklabel msdos
ncyl=$(parted $disk unit cyl print  | sed -n 's/.*: \([0-9]*\)cyl/\1/p')

if [[ $ncyl != [0-9]* ]]; then
 echo "disk $disk has invalid cylinders number: $ncyl"
 exit
fi

echo "==> create primary parition  $partno with $ncyl cylinders"
parted -a optimal $disk mkpart primary 0cyl ${ncyl}cyl
echo "==> set partition $partno to type: lvm "
parted $disk set $partno lvm on
partprobe > /dev/null 2>&1
echo "==> create PV ${disk}${partno} "
pvcreate ${disk}${partno}

Understanding SysV-style Initscripts in Red Hat Linux

It is often needed to write your own init start/stop script, the following is the minimum requirement for your script to behave as expected. The discussion is based on Red Hat Linux family, other distributions like Debian use LSB (Linux Standard Base Specification) Init Scripts.

Location of the script

/etc/init.d is the well known location, but actually /etc/rc.d/init.d is the real original location. Since /etc/init.d is a hard link to /etc/rc.d/init.d, it makes no difference.

Header of the script

It needs at least 3 lines.  The shell script interpreter (/bin/sh, /bin/bash .. etc), the chkconfig header and script description

#!/bin/sh
#   chkconfig: 345 56 10
#   description: Startup/shutdown script for the Common UNIX 

Body of the script

Obviously, it need  to accept parameter “start”, which  /etc/rc3.d/S* will call  on OS startup and accept parameter “stop”, which /etc/rc0.d/K* script will call on OS shutdown.
The lockfile is often overlooked, it is used to check the existence of the daemon on OS shutdown, otherwise the stop action won’t be called. If you found an issue that a script started on OS startup but never stop properly on shutdown, you need to create lockfile. note: lockfile is not pidfile which contains PID of the process, lockfile is usually a blank file.

lockfile=/var/lock/subsys/$(basename $0)
case $1 in
 start)
  start
  [ $? = 0 ] && touch $lockfile
 ;;
 stop)
  stop
  [ $? = 0 ] && rf –f  $lockfile

Others

It is recommended to import functions in /etc/rc.d/init.d/functions to use ‘daemon’ to startup your application or killproc to shutdown your application instead of reinventing the wheel.

LSB headers

You may see something like this in an init script.

 # Provides: boot_facility_1 [ boot_facility_2 ...]
 # Required-Start: boot_facility_1 [ boot_facility_2 ...]
 # Required-Stop: boot_facility_1 [ boot_facility_2 ...]
 # Should-Start: boot_facility_1 [ boot_facility_2 ...]
 # Should-Stop: boot_facility_1 [ boot_facility_2 ...]
 # Default-Start: run_level_1 [ run_level_2 ...]
 # Default-Stop: run_level_1 [ run_level_2 ...]
 # Short-Description: short_description
 # Description: multiline_description

They are LSB(Linux Standard Base) headers, they are supported by default in Debian and SUSE Linux.
Red Hat Linux supports this by additional package “redhat-lsb” and it is not installed by default, Be warned,50+ dependences need to installed as well.

Reference

http://refspecs.linuxfoundation.org/LSB_2.1.0/LSB-generic/LSB-generic/initscrcomconv.html

https://fedoraproject.org/wiki/Packaging:SysVInitScript?rd=Packaging/SysVInitScript

Shell script to check Oracle Tablespace usage

I searched a shell script to check Oracle Tablespace usage, most scripts returned use complex SQL statements and they don’t report usage accurately, because auto-extend or multiple data files was not taken into account for calculation. Actually, there is a built-in view “dba_tablespace_usage_metrics” for the purpose starting from Oracle 10g. 
The following script check the Oracle database availability or tablespace usage and measure the response time.The scripts output “key=value” format, which can be easily discovered by LLD in Zabbix.(with LLD, Zabbix can dynamically discover any number of items to monitor without adding the items manually )

Script sample output

db-time= 71
db-status=[OK]: Name:SYSAUX SizeMB:1024 Used%: 73 ; Name:SYSTEM SizeMB:1024 Used%: 72 ; Name:USERS SizeMB:5 Used%: 20 ; Name:TEMP SizeMB:2048 Used%: 2 ; Name:UNDOTBS1 SizeMB:2048 Used%: 1 ;  8 rows selected.

The Oracle login in the script should have permission to read the view or have “select_catalog_role” role granted.

Script detail

#!/bin/ksh
function checkdb {
TNSNAME=$1
OUSER=$2
OPASS=$3

ORACLE_HOME=${ORACLE_HOME:=/u01/app/oracle/product/11.2.0/client_1}
export ORACLE_HOME
t1="$(date +%s%N)"

rt=$($ORACLE_HOME/bin/sqlplus -S ${OUSER}/${OPASS}@${TNSNAME}<< _END
set heading off
set linesize 200
select
   'Name:'|| tablespace_name,
   'SizeMB:'||round(TABLESPACE_SIZE*8/1024)||' Used%:',
   round(used_percent),
   ';'
from
   dba_tablespace_usage_metrics
order by 3 desc;
exit;
_END)

t2="$(date +%s%N)"
echo "db-time= $((($t2 - $t1)/1000000))"
#remove blank lines,ignore UNDOTBS,get the numeric value by removing tab and spaces
tbpct=$(echo "$rt" | egrep -v '^$|UNDOTBS' | head -1 | sed 's/.*Used%:\(.*\);/\1/'  |  sed 's/[ \t]*//g')
#Critical condition: thresh-hold > 95 or non-numeric value returned
if [ $tbpct -gt 95 ] || [[ "$tbpct" != +(\d) ]] ; then
 echo "db-status=[CRITICAL]:" $rt
else
 echo "db-status=[OK]:" $rt
fi
}

Zabbix: Monitor customized applications by SNMP and JSON Low-level discovery

Low-level discovery (LLD) is an amazing feature in Zabbix, once you have defined the template you can start to monitor hundreds of hosts in few minutes by importing hosts definition in XML file.
Low-level discovery is used to discover dynamic items, for example, you can’t monitor disk space usage of a volume by static binding, because the index number of the volume is dynamic.
The customized application in this example, is a shell script to check database and web service, it outputs to stdout, which follows “key=value” format and it contains with string and numeric values.

db-status= [OK]
web-status=[OK]: http-code: 200
web-time=42
db-time=40

The goal is to create monitoring entry and trigger for each item automatically


Execute the script by SNMP


Net-snmp allows you to execute any arbitrary command by “exec” parameter. Edit /etc/snmp/snmpd.conf and map OID to the script. exec .1.3.6.1.4.1.2021.200 testscript1 /usr/local/bin/script1.sh The lines of output are in MIBIOD.101.x ( .1.3.6.1.4.1.2021.200.101), which can be retrieved by snmpwalk on .1.3.6.1.4.1.2021.200.101


What is Low Level discovery?


https://www.zabbix.com/documentation/2.0/manual/discovery/low_level_discovery


Discovery of SNMP OIDs


It is possible to automatically add the script by SNMP OID LLD, but it has drawbacks: once LLD is completed, the 4 lines of output will be added as 4 items, each value of item is retrieved by snmpget, which means the script will be executed 4 times.


Discovery item JSON format


https://www.zabbix.com/documentation/2.0/manual/discovery/low_level_discovery


Zabbix also supports LLD by parsing output in JSON format. This method can overcome the drawbacks of SNMP OID LLD, the idea is to run an external script to do snmpwalk on the OID then save outputs to a text file on local server, the other script retrieve value of each item by simply reading the text file. This script will be still executed 4 times, but it is executed locally and doesn’t do remote connections.


There are two scripts to execute snmpwalk for different output values, one for string and the other for numeric. If you don’t need to do graphing of the numeric value, one script is enough.

#retrieve string values with “-s” and output in JSON format
#The macro name KEY/VALUE is arbitrary
[root@zabbix:externalscripts]# ./snmpwalk.pl  -s server1
{
 "data":[
 {
  "{#KEY}":"db-status",
  "{#VALUE}":" [OK]
"
 }
 ,
 {
  "{#KEY}":"web-status",
  "{#VALUE}":"[OK]: http-code: 200
"
 }
 ]
}

#Item prototype details for scripts-getstringoutput

Column “Name” is _{#KEY}, {#KEY} is macro in JSON output, _ is just simply prefixed to make it a valid name. Column “Key” (actually, it is value) is retrieved by getvalue.pl

[root@zabbix:externalscripts]#./getvalue.pl -s server1 web-status
[OK]: http-code: 200

Just link the template to a host, the keys and values of the items will be discovered automatically.

LLD is such an amazing feature, even there are hundreds of items, they can be automatically discovered in few minutes.

Some tips in Zabbix implementation:


1) Define global macro for SNMP community string in administration->General->Macros, the macro can also be defined in template or host level.


2) Items  won’t be updated  straightaway after being added, it has to wait for next update specified in update interval


3) Define regular expressions in administration->General->Expressions


For example there are some virtual interfaces you want to exclude create regular expression called


“Real Adapters” with expression: (^lo$|^Microsoft|^RAS|^WAN|-0000$|^Teredo Tunneling|^Software Loopback|^sit)


And refer to the macro by @Real Adapters in the default “Template SNMP Interfaces”

The snmpwalk.pl script:

#!/usr/bin/perl -w
use warnings;
use FileHandle;
###---------- main ----------------------
my ($cdir)=$0=~m|(.*)/|;
my $type=$ARGV[0];
my $host=$ARGV[1];
my $oid='200.101';
my $fname="${cdir}/logs/${host}";
my $afname="${fname}a.out";  $ifname="${fname}i.out";  $sfname="${fname}s.out"; 
my $timegap=120;
if ( ! defined $host ) {
 print "hostname is required\n Usage: $0 [-i|-s] hostname \n";
 exit 1;
}
my $mtime = (stat($afname))[9];
my $ctime=time;
#no need to do snmpwalk, if the file is recent 
if ( ( ($ctime - $mtime) > $timegap ) or (! defined $mtime ) ) {
 
 @output=&snmprun ("$host", "$oid");
 map (s/^STRING: "//,@output);
 map (s/\"$//,@output);
 #all values
 open(OUTFILE, ">$afname") or die "Can't write to $fname: $!";
 print OUTFILE @output;
 close (OUTFILE);
 
#numberic values
 open(OUTFILE, ">$ifname") or die "Can't write to $fname: $!";
 print OUTFILE grep(/.*=\s*\d+\.*\d+$/, @output);
 close (OUTFILE);
#string values
 open(OUTFILE, ">$sfname") or die "Can't write to $fname: $!";
 print OUTFILE grep(/.*=.*[a-zA-Z]/, @output); 
 close (OUTFILE);
}
($type)=$type=~m/-(i|s)/;
&printjson ( "${fname}${type}.out" );
sub snmprun {
  my $host=$_[0];
  my $oid=$_[1];
  $ucdprefix='.1.3.6.1.4.1.2021.';
  $snmpwalk='/usr/bin/snmpwalk -v 2c -O v -c public';
  $fulloid=${ucdprefix}.${oid};
  @rt0=`$snmpwalk $host $fulloid 2>&1`;
  return @rt0;
}
#Read a file and print out in JSON format
sub printjson { 
$first = 1;
open (INFILE,"$_[0]") or die "Can't open $_[0] $!"; 
print "{\n";
print "\t\"data\":[\n\n";
 
while (<INFILE>)
{
    ($key, $value) = split (/=/,$_,2);
    print "\t,\n" if not $first;
    $first = 0;
 
    print "\t{\n";
    print "\t\t\"{#KEY}\":\"$key\",\n";
    print "\t\t\"{#VALUE}\":\"$value\"\n";
    print "\t}\n";
}
 
print "\n\t]\n";
print "}\n";
}

The getvalue.pl script:

#!/usr/bin/perl -w
use warnings;
use FileHandle;
my $type=$ARGV[0];
my $host=$ARGV[1];
my $key=$ARGV[2];
my ($cdir)=$0=~m|(.*)/|;
my $timegap=600;
if ( $#ARGV != 2 )  {
 print "\nUsage: [-i|-s} hostname key \n";
 exit 1;
}
($type)=$type=~m/-(i|s)/;
$fname="${cdir}/logs/${host}${type}.out";
open(INFILE, "<$fname") or die "Can't open $fname: $!";
my $mtime = (stat($fname))[9];
my $ctime=time;
if ( ($ctime-$mtime) > $timegap ) {
  print "$key=<CRITICAL>: $fname hasn't been updated for > $timegap seconds, the outdated value will not be retrieved\n";
  exit 1
}
@lines=<INFILE>;
@line1=grep(/$key=/, @lines);
#print "@line1\n";
if ( ! defined ( @line1 )) {
 print "key $key is not found in $fname\n";
 exit 1
}
($value)=$line1[0]=~m/$key=(.*)/;
print "$value \n";