Graphing sar output

In Linux, sysstat package installs tools: sar, iostat .. , in the mean time, setups  a cron job to run sar periodically. The sar binary output  files are in /var/log/sa or /var/log/sysstat.
The files are very useful  for troubleshooting performance issues, if you don’t have monitoring solution in place.
To visualize the data into graph, you can use generic plotting tool: gnuplot or special tool designed for sar: ksar.

Visualize sar output  by  gnuplot

gnuplot can be directly installed online  in most Linux distributions.
file saved by sar cron job is binary, convert it to ascii format. The following example output CPU usage

$LC_ALL=C;sar -u -f /var/log/sa/sa27  | egrep  '[0-9][0-9]:[0-9][0-9]:[0-9][0-9]'    | sed  '1s/^/#/' >sar-cpu.log

LC_ALL=C to ensure time format is H:M:S
Sed  is used to add comment line to the first line: the header.
Create gnuplot script to show user CPU (3th column)  and system cpu  (5th column) usage
$cat cpu.p

set title 'HOST  CPU usage'
set xdata time
set timefmt '%H:%M:%S'
set xlabel 'time'
set ylabel ' CPU Usage'
set style data lines
plot 'sar-cpu.log' using 1:3 title 'User' ,\
'sar-cpu.log' using 1:5 title 'System'

Type ‘gnuplot’ to enter interactive shell then run the script.
gnuplot>

gnuplot> load ‘cpu.p’
or  
$gnuplot -persist cpu.p

#Other advanced operations
#Zoom in, display a set period of data only

gnuplot> set xrange ['01:51:01':'03:51:01']
gnuplot> replot
#Save the output to image 
gnuplot> set terminal png         
gnuplot> set output "cpu.png"  
gnuplot> replot

Visualize sar output by ksar
The generic graphing tool, gnuplot, can process any data, it is not designed for sar. As a trade-off, it needs lots of customization.
ksar is specifically desgined for sar and understand Linux, Mac and Solaris sar output.
Kar can be downloaded at http://sourceforge.net/projects/ksar/, Ksar is written in Java, so Java executables are prerequisite for ksar.
#Convert sar binary output to ascii for ksar, “-A” means include all counters
$LC_ALL=C;sar -A -f /var/log/sa/sa27 >sar-all.log
It is very easy to view any counter, once sar output files are imported into ksar.
[root@ kSar-5.0.6]$./run.sh -help

[root@ kSar-5.0.6]$./run.sh  -input  /tmp/sar-all.log

Manage Xen by libvirt tools

libvirt is an open source API, daemon and management tool for managing platform virtualization
It is much easier to use than the xen native tools for VM creation, network management and storage management
Pros:
- Standard, easy and neat commands to manage VM creation, network management and storage management.
- Support all well known hypervisors (Linux KVM, Xen, VMware ESX, OpenVZ..), so the knowledge is transferable.
- Remote management with TLS encryption and Kerberos authentication.
- API bindings for multiple languages: Python,Perl,Ruby, Java, OCaml , C#, and PHP
- Operation isolation: Stopping libvirt (version> 0.6.0) daemon won't affect VM
Cons:
- Libvirt couldn’t keep up with the development of the underlying hypervisor, so it might not be able understand new features in hypervisor.
- An additional layer of management introduces availability and security concerns. Although stopping libvirt daemon won’t affect VM, but if libvrit daemon fails upon hypervisor reboot. The network bridge managed by libvirt won’t be created. But it can be quickly remedied by simple command:

$brctl addbr br-name; ifconfig br-name IP up

Where does libvirt save the VM configuration file?
It depends on the hypervisor. For Xen, libvirt use Xen API to save it to xenstore(/var/lib/xenstored). Because xenstore is Xen component, that is why VM native tools can start VM without libvirt daemon.
The following stript can be used to examine the VM configuration.
#!/bin/sh

function dumpkey() {
local param=${1}
local key
local result
result=$(xenstore-list ${param})
if [ "${result}" != "" ] ; then
for key in ${result} ; do dumpkey ${param}/${key} ; done
else
echo -n ${param}'='
xenstore-read ${param}
fi
}
for key in /vm /local/domain /tool ; do dumpkey ${key} ; done

Install libvirt on Debian
$apt-get install libvirt-bin virtinst
Enable xend-unix-server in xend to talk to libvirt
$ grep xend-unix-server /etc/xen/xend-config.sxp

(xend-unix-server yes)

$/etc/init.d/xend restart

Define new network bridge
root@xen4:/etc/xen# cat /tmp/net.xml

<network>
<name>private</name>
<bridge name="virbr2" />
<ip address="192.168.152.1" netmask="255.255.255.0">
</ip>
</network>

Type “virsh” to enter an virsh interactive prompt
virsh # net-define /tmp/net.xml

Network private defined from /tmp/net.xml
virsh # net-autostart  private
Network private marked as autostarted
virsh # net-start  private
Network private started
virsh # net-list
Name                 State      Autostart
-----------------------------------------
private              active     yes
root@xen4:/# ifconfig  virbr2
virbr2    Link encap:Ethernet  HWaddr de:49:4e:43:c5:5d
inet addr:192.168.152.1  Bcast:192.168.152.255  Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Install Centos para-virtualization guest.
#Prepare sparse disk file with qemu-img tool
$qemu-img create -f raw /data/pv2.raw 2G
Para virtualization guest can’t use cdrom as install source, In this example, I mount the ISO file to a web server directory.
$virt-install \

--paravirt \
--name pv2 \
--ram 256 \
--disk path=/data/pv2.raw,size=2,format=raw \
--os-type=linux --os-variant=rhel5.4 \
--nographics \
--network network=private \
--location http://192.168.152.1/pkgs/

After the VM has been created, you can use xen native tool /usr/sbin/xm or libvirt virsh command to start/stop VM. But any configuration change require the virsh edit commands (edit, net-edit, pool-edit vol-edit, iface-edit)

Access guest VM console via text mode VNC.

In my previous post: Access Linux console via text mode VNC. The technique is not very useful, other than running VNC on non-X window system. It is not real lights out management (LOM), because the VNC service lives on OS. How can you access the console before OS boots up? It is possible for guest VM by directing VNC input/out in host hypervisor to guest VM console. The following is to realize the technique used in virtualization vendors to access guest VM console.
The linuxvnc tool, mentioned in my last post, is actually from libvncserver, there is sister tool called vncommand for executing <command> redirecting stdin from a vncviewer and stdout & stderr to the vnc clients). It doesn’t work well, fortunately, Proxmox fixed some issues and renamed it to vncterm.
Install the library: libvncsever.

$libvncserver0  0.9.7-2+b1    API to write one's own vnc server 

Install vncterm.
Download vncterm
Proxmox only packaged it for Debian, but I found, as long as you have libvncserver libraries, The extracted file: vncterm from DEB pkg works for other distributions. You can also compile from source.
Basic setup: input/output on VNC port 5910 are directed to a command (xen server for example)
-c is the command to be executed. –rfbport is the listening port. All options in x11vnc are supported
$vncterm –timeout 0 –rfbport 5910 –c /usr/sbin/xm console pv2
Advanced setup: enable httpsever and vnc password authentication (check my previous post for instructions)
$vncterm –timeout 0 -httpport 8080 -httpdir /usr/share/x11vnc/classes -rfbauth /root/linuxvncpass -rfbport 5910 -c /usr/sbin/xm console pv2
 
Connect: Use VNC client (Realvnc viewer, Tightvnc client or Java VNC viewer) to connect to VM hypervisor host on VNC port 5910, the VM console is accessible before VM OS boots up.