Understanding Linux CPU scheduling priority

Scheduling priority depends on scheduling class.
scheduling classes
- SCHED_FIFO: A First-In, First-Out real-time process
- SCHED_RR: A Round Robin real-time process
- SCHED_NORMAL: A conventional, time-shared process
Most  processes are SCHED_NORMAL
How to find out the scheduling class of a process.

# ps  command with “class” flag
#   TS  SCHED_OTHER (SCHED_NORMAL)
#   FF  SCHED_FIFO
#   RR  SCHED_RR
$  ps -e -o class,cmd | grep sshd
TS  /usr/sbin/sshd
#chrt command
$ chrt -p 1836
pid 1836's current scheduling policy: SCHED_OTHER
pid 1836's current scheduling priority: 0

Scheduling priorities
- Real-time process (SCHED_FIFO/SCHED_RR)  real-time priority , ranging from 1 (lowest priority) to 99 (higest priority).
- Conventional process  static priority(SCHED_NORMAL ),  ranging from 100 (highest priority) to 139 (lowest priority).
Nice value and static priority
Conventional process's  static priority = (120 + Nice value)
So user can use nice/renice command to  change nice value in order to change conventional process's priority.
By default, conventional process  starts with nice value of 0 which equals static priority 120
Checking  Real-time/Conventional process priority.
$ ps -e -o class,rtprio,pri,nice,cmd

CLS RTPRIO PRI  NI CMD
TS       -  21   0 init [3]
FF      99 139   - [watchdog/0]

Watchdog is a real time process (CLASS=SCHED_FIFO), whose real time priority is 99 (I think the PRI column  is irrelevant  for it)
init is  a conventional process (CLASS=SCHED_OTHER), whose nice is 0 and dynamic priority is 121 (100+21)(I think the RTPRIO column  is irrelevant for it  )
Why init's priority is 121 not 120? Please noted I used the term: dynamic priority not static priority.
dynamic priority = max (100, min (  static priority - bonus + 5, 139))
bonus is ranging from 0 to 10,  which is set by scheduler depends on the past history of the process; more precisely, it is related to the average sleep time of the process.
Changing  Real-time/Conventional process priority.
#Real-time process

$chrt 80  ps -e -o class,rtprio,pri,nice,cmd
..
FF      80 120   - ps -e -o class,rtprio,pri,nice,cmd
# Conventional  process
$nice -n 10  ps -e -o class,rtprio,pri,nice,cmd
...
TS       -  12  10 ps -e -o class,rtprio,pri,nice,cmd

Proactive monitoring by snmptrap

Pulling snmp information is used in most monitoring solutions, however pushing information  is  an alternative monitoring solution by snmptrap.
This post demonstrates how to email alarms being pushed to receiver: snmptrapd from snmp agent.
Tested on Centos 5.5 +NET-SNMP  5.3.2.2
Install email daemon and net-snmp

$yum install postfix net-snmp net-snmp-utils
$cat /etc/snmp/snmptrapd.conf
#authCommunity   TYPES COMMUNITY  [SOURCE [OID | -v VIEW ]]
authCommunity  execute public  default  .1
traphandle  default /usr/bin/traptoemail -s localhost -f snmp@localhost root@localhost

start snmptrapd  and start postfix
Test by snmptrap tool
Send email if eth0 operation status is up (1)
(IF-MIB::linkUp is notification object defined in MIB file: IF-MIB.txt)

$snmptrap -v 2c -c public 127.0.0.1 "" IF-MIB::linkUp  .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOperStatus.1 i 1

Sample email received

$mail
…
>N 86 snmp@localhost.local  Mon Mar  7 15:59  18/747   "trap received from localhost: IF-MIB::linkUp"
& 86
Message 86:
From snmp@localhost.local.net  Mon Mar  7 15:59:13 2011
X-Original-To: root@localhost
Delivered-To: root@localhost.local.net
To: root@localhost.local.net
From: snmp@localhost.local.net
Subject: trap received from localhost: IF-MIB::linkUp
Date: Mon,  7 Mar 2011 15:59:13 +1100 (EST)
Host: localhost (UDP: [127.0.0.1]:35453)
DISMAN-EVENT-MIB::sysUpTimeInstance  0:6:08:29.87
SNMPv2-MIB::snmpTrapOID.0  IF-MIB::linkUp
IF-MIB::ifOperStatus.1  up

The above configuration make snmptrapd ready to receive traps, the following steps is to
configure snmp agent to send traps.
A SNMP v3 USM user need to be created, even the trap is intended for snmp v1/v2c only.
Check my previous post for creating and managing SNMP v3 USM users

$ cat /etc/snmp/snmpd.conf
#authuser    read,write [-s secmodel] user [noauth|auth|priv [oid|-V view]]
authuser   read -s v2c guest_user noauth  .1
authuser   read -s usm guest_user noauth  .1
authcommunity read  public  default .1
trap2sink 127.0.0.1 public
iquerySecName guest_user
agentSecName  guest_user
monitor   -u guest_user  -r 60 "interface down" -o ifDescr ifOperStatus != 1
If you shutdown any interface and restart snmpd, following email notification should appear
$mail 
..
N 87 snmp@localhost.local  Mon Mar  7 16:24  23/1030  "trap received from localhost: DISMAN-EVENT-MIB::mteTriggerFired"
& 87
Message 87:
From snmp@localhost.local.net  Mon Mar  7 16:24:28 2011
X-Original-To: root@localhost
Delivered-To: root@localhost.local.net
To: root@localhost.local.net
From: snmp@localhost.local.net
Subject: trap received from localhost: DISMAN-EVENT-MIB::mteTriggerFired
Date: Mon,  7 Mar 2011 16:24:28 +1100 (EST)
Host: localhost (UDP: [127.0.0.1]:46356)
DISMAN-EVENT-MIB::sysUpTimeInstance  0:0:00:00.84
SNMPv2-MIB::snmpTrapOID.0  DISMAN-EVENT-MIB::mteTriggerFired
DISMAN-EVENT-MIB::mteHotTrigger.0  interface down
DISMAN-EVENT-MIB::mteHotTargetName.0
DISMAN-EVENT-MIB::mteHotContextName.0
DISMAN-EVENT-MIB::mteHotOID.0  IF-MIB::ifOperStatus.4
DISMAN-EVENT-MIB::mteHotValue.0  2
IF-MIB::ifDescr.4  eth2

You can enable  “linkUpDownNotifications yes” to track interface status, but I found this type of  notification didn’t have interface name information.
Troubleshooting
1.failed to run mteTrigger query error
- make sure the user has permission in sec mode: usm as well.  “authuser   read -s usm guest_user noauth  .1”
- specifically set user with  “–u guest_user” in monitor command
2.Start snmpd in debugging mode for disman (Distributed Management )

/usr/sbin/snmpd -Ddisman -Lsd -Lf /var/log/snmpd.log -p /var/run/snmpd.pid -a

Setup SNMP V3 USM with encryption.

SNMP v3 introduces advanced security which support USM(user-based security model) and data encryption,  SNMPv1 and SNMPv2 only support access control  based on community string and  send data in clear text. SNMP V3 on longer has the term: community string and (it seems) the ability to control access based on source network.
The following instructions are based on Centos 5.5 + NET-SNMP   5.3.2.2
Create user
Create user guest_user whose password is "Pass0001" and shared key for encryption is "sharedkey001"
 Put create user command into file /var/net-snmp/snmpd.conf, once snmpd restarted, the line will be deleted for security reason and the user will be created in usmUsertable

$cat /var/net-snmp/snmpd.conf
createUser guest_user     MD5 "Pass0001" DES "sharedkey001"

Grant user permission to all OIDs (.1) 
$ cat /etc/snmp/snmpd.conf

##authuser    read,write [-s secmodel] user [noauth|auth|priv [oid|-V view]]
#auth=authentication no privacy (encryption)
#priv=authentication plus privacy (encryption)
authuser   read -s usm  guest_user priv  .1

Restart snmpd
service snmpd restart
Test  by snmpget
$snmpget -v 3 -u guest_user -l Priv -a MD5 -A Pass0001 -x DES -X sharedkey001 192.168.56.31 sysName.0

NMPv2-MIB::sysName.0 = STRING: centos64.local.net
List users
$ snmptable -v 3 -u guest_user   -l Priv  -a MD5 -A Pass0001 -x DES -X sharedkey001  192.168.56.31 usmUsertable
SNMP table: SNMP-USER-BASED-SM-MIB::usmUserTable
guest_user
Add  user 
#add  user guest_user2  by cloning guest_user
#The connecting user must be given write access (authuser read,write …. )  in order to add/delete users
$snmpusm -v 3 -u guest_user   -l Priv  -a MD5 -A Pass0001 -x DES -X sharedkey001  192.168.56.31 create  guest_user2  guest_user
User successfully created
Delete user
$snmpusm -v 3 -u guest_user   -l Priv  -a MD5 -A Pass0001 -x DES -X sharedkey001  192.168.56.31 delete  me2
Client configuration file snmp.conf
 You can put most command options in client config file: /etc/snmp/snmp.conf or  ~/.snmp/snmp.conf
$cat ~/.snmp/snmp.conf
defVersion 3
defSecurityName guest_user
defAuthType MD5
defSecurityLevel authPriv
defAuthPassphrase Pass0001
defPrivType  DES
defPrivPassphrase sharedkey001
#the long command can be simplified to
$snmpget  192.168.56.31 sysName.0
SNMPv2-MIB::sysName.0 = STRING: centos64.local.net

When Centos hung on starting up boot services, how to get to shell without rescue CD

Centos 5.5 hung on starting up udev service.  My first instinct was to try to go to interactive startup  mode to skip udev, as message hints  “press i to enter interactive startup”.
I later discovered “ interactive startup  mode” is almost useless, firstly, it is hard to active this mode by press “I” key, secondly not all services observe this mode. Network service seems to be the only one.
A flag file: /var/run/confirm will be created,  when key “I” (case insensitive) is pressed. It seems only network service check this file.

[root@centos64 init.d]# grep -C 2 /var/run/confirm /etc/init.d/*
/etc/init.d/network-            fi
/etc/init.d/network-            # If we're in confirmation mode, get user confirmation.
/etc/init.d/network:            if [ -f /var/run/confirm ]; then
/etc/init.d/network-                    confirm $i
/etc/init.d/network-                    test $? = 1 && continue

So how can you gain shell access without rescue CD?  The answer is to append “init=/bin/sh” to kernel line in grub boot loader.
Let’s  review the Linux boot order
The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->

/etc/inittab->
/etc/rc.d/rc.sysinit->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S

By default “init=/sbin/init”, which will transfer control in above order.
If you set “init=/bin/sh”, it will stop there and give login shell.
Booting to single user mode won’t fix udev startup issue, because udev starts before single user mode (udev is in /etc/rc.d/rc.sysinit , single user mode is in /etc/rc.d/rc1.d)
Instructions:
In Grub menu, select the kernel,  press “a” to edit boot option, then append “init=/bin/sh”, then press enter to boot
After gaining the login shell, the fs is most likely in Read-only file system state.
 Remount partitions to rewrite mode by  “mount –o rw,remount / “

Graphing sar output

In Linux, sysstat package installs tools: sar, iostat .. , in the mean time, setups  a cron job to run sar periodically. The sar binary output  files are in /var/log/sa or /var/log/sysstat.
The files are very useful  for troubleshooting performance issues, if you don’t have monitoring solution in place.
To visualize the data into graph, you can use generic plotting tool: gnuplot or special tool designed for sar: ksar.

Visualize sar output  by  gnuplot

gnuplot can be directly installed online  in most Linux distributions.
file saved by sar cron job is binary, convert it to ascii format. The following example output CPU usage

$LC_ALL=C;sar -u -f /var/log/sa/sa27  | egrep  '[0-9][0-9]:[0-9][0-9]:[0-9][0-9]'    | sed  '1s/^/#/' >sar-cpu.log

LC_ALL=C to ensure time format is H:M:S
Sed  is used to add comment line to the first line: the header.
Create gnuplot script to show user CPU (3th column)  and system cpu  (5th column) usage
$cat cpu.p

set title 'HOST  CPU usage'
set xdata time
set timefmt '%H:%M:%S'
set xlabel 'time'
set ylabel ' CPU Usage'
set style data lines
plot 'sar-cpu.log' using 1:3 title 'User' ,\
'sar-cpu.log' using 1:5 title 'System'

Type ‘gnuplot’ to enter interactive shell then run the script.
gnuplot>

gnuplot> load ‘cpu.p’
or  
$gnuplot -persist cpu.p

#Other advanced operations
#Zoom in, display a set period of data only

gnuplot> set xrange ['01:51:01':'03:51:01']
gnuplot> replot
#Save the output to image 
gnuplot> set terminal png         
gnuplot> set output "cpu.png"  
gnuplot> replot

Visualize sar output by ksar
The generic graphing tool, gnuplot, can process any data, it is not designed for sar. As a trade-off, it needs lots of customization.
ksar is specifically desgined for sar and understand Linux, Mac and Solaris sar output.
Kar can be downloaded at http://sourceforge.net/projects/ksar/, Ksar is written in Java, so Java executables are prerequisite for ksar.
#Convert sar binary output to ascii for ksar, “-A” means include all counters
$LC_ALL=C;sar -A -f /var/log/sa/sa27 >sar-all.log
It is very easy to view any counter, once sar output files are imported into ksar.
[root@ kSar-5.0.6]$./run.sh -help

[root@ kSar-5.0.6]$./run.sh  -input  /tmp/sar-all.log

Manage Xen by libvirt tools

libvirt is an open source API, daemon and management tool for managing platform virtualization
It is much easier to use than the xen native tools for VM creation, network management and storage management
Pros:
- Standard, easy and neat commands to manage VM creation, network management and storage management.
- Support all well known hypervisors (Linux KVM, Xen, VMware ESX, OpenVZ..), so the knowledge is transferable.
- Remote management with TLS encryption and Kerberos authentication.
- API bindings for multiple languages: Python,Perl,Ruby, Java, OCaml , C#, and PHP
- Operation isolation: Stopping libvirt (version> 0.6.0) daemon won't affect VM
Cons:
- Libvirt couldn’t keep up with the development of the underlying hypervisor, so it might not be able understand new features in hypervisor.
- An additional layer of management introduces availability and security concerns. Although stopping libvirt daemon won’t affect VM, but if libvrit daemon fails upon hypervisor reboot. The network bridge managed by libvirt won’t be created. But it can be quickly remedied by simple command:

$brctl addbr br-name; ifconfig br-name IP up

Where does libvirt save the VM configuration file?
It depends on the hypervisor. For Xen, libvirt use Xen API to save it to xenstore(/var/lib/xenstored). Because xenstore is Xen component, that is why VM native tools can start VM without libvirt daemon.
The following stript can be used to examine the VM configuration.
#!/bin/sh

function dumpkey() {
local param=${1}
local key
local result
result=$(xenstore-list ${param})
if [ "${result}" != "" ] ; then
for key in ${result} ; do dumpkey ${param}/${key} ; done
else
echo -n ${param}'='
xenstore-read ${param}
fi
}
for key in /vm /local/domain /tool ; do dumpkey ${key} ; done

Install libvirt on Debian
$apt-get install libvirt-bin virtinst
Enable xend-unix-server in xend to talk to libvirt
$ grep xend-unix-server /etc/xen/xend-config.sxp

(xend-unix-server yes)

$/etc/init.d/xend restart

Define new network bridge
root@xen4:/etc/xen# cat /tmp/net.xml

<network>
<name>private</name>
<bridge name="virbr2" />
<ip address="192.168.152.1" netmask="255.255.255.0">
</ip>
</network>

Type “virsh” to enter an virsh interactive prompt
virsh # net-define /tmp/net.xml

Network private defined from /tmp/net.xml
virsh # net-autostart  private
Network private marked as autostarted
virsh # net-start  private
Network private started
virsh # net-list
Name                 State      Autostart
-----------------------------------------
private              active     yes
root@xen4:/# ifconfig  virbr2
virbr2    Link encap:Ethernet  HWaddr de:49:4e:43:c5:5d
inet addr:192.168.152.1  Bcast:192.168.152.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Install Centos para-virtualization guest.
#Prepare sparse disk file with qemu-img tool
$qemu-img create -f raw /data/pv2.raw 2G
Para virtualization guest can’t use cdrom as install source, In this example, I mount the ISO file to a web server directory.
$virt-install \

--paravirt \
--name pv2 \
--ram 256 \
--disk path=/data/pv2.raw,size=2,format=raw \
--os-type=linux --os-variant=rhel5.4 \
--nographics \
--network network=private \
--location http://192.168.152.1/pkgs/

After the VM has been created, you can use xen native tool /usr/sbin/xm or libvirt virsh command to start/stop VM. But any configuration change require the virsh edit commands (edit, net-edit, pool-edit vol-edit, iface-edit)

Access guest VM console via text mode VNC.

In my previous post: Access Linux console via text mode VNC. The technique is not very useful, other than running VNC on non-X window system. It is not real lights out management (LOM), because the VNC service lives on OS. How can you access the console before OS boots up? It is possible for guest VM by directing VNC input/out in host hypervisor to guest VM console. The following is to realize the technique used in virtualization vendors to access guest VM console.
The linuxvnc tool, mentioned in my last post, is actually from libvncserver, there is sister tool called vncommand for executing <command> redirecting stdin from a vncviewer and stdout & stderr to the vnc clients). It doesn’t work well, fortunately, Proxmox fixed some issues and renamed it to vncterm.
Install the library: libvncsever.

$libvncserver0  0.9.7-2+b1    API to write one's own vnc server 

Install vncterm.
Download vncterm
Proxmox only packaged it for Debian, but I found, as long as you have libvncserver libraries, The extracted file: vncterm from DEB pkg works for other distributions. You can also compile from source.
Basic setup: input/output on VNC port 5910 are directed to a command (xen server for example)
-c is the command to be executed. –rfbport is the listening port. All options in x11vnc are supported
$vncterm –timeout 0 –rfbport 5910 –c /usr/sbin/xm console pv2
Advanced setup: enable httpsever and vnc password authentication (check my previous post for instructions)
$vncterm –timeout 0 -httpport 8080 -httpdir /usr/share/x11vnc/classes -rfbauth /root/linuxvncpass -rfbport 5910 -c /usr/sbin/xm console pv2
 
Connect: Use VNC client (Realvnc viewer, Tightvnc client or Java VNC viewer) to connect to VM hypervisor host on VNC port 5910, the VM console is accessible before VM OS boots up.