Solaris/Linux: find port number for a program and vice-versa

#== Find port number for a program
- lsof tool(Platform independent)

$lsof -nc | sshd grep TCP

sshd 1962 root 3u IPv6 6137 TCP *:ssh (LISTEN)
sshd 2104 root 3u IPv6 7425 TCP 172.16.31.3:ssh->172.16.31.2:cs-services (ESTABLISHED
- Linux

$netstat -anp |grep sshd

tcp 0 0 :::22 :::* LISTEN 1962/sshd
- Solaris
$ pfiles 16976
...
sockname: AF_INET 172.18.126.148 port: 22
..

#==Find program name for port number
- lsof tool(Platform independent)
$lsof -i TCP:22
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
sshd 1962 root 3u IPv6 6137 TCP *:ssh (LISTEN)
sshd 2104 root 3u IPv6 7425 TCP 172.16.31.3:ssh->172.16.31.2:cs-services (ESTABLISHED)
- Linux
$netstat -anp grep 22
tcp 0 0 :::22 :::* LISTEN 1962/sshd
- Solaris
list open files for all process,then search the file for "port: 22"

$ ps -e -o pid | xargs pfiles > /tmp/pfiles.log 

RHCE NOTES - SElinux

Quiick SElinux notes for the impatient, read full document at

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/selinux-guide/


Selinux has 2 levels access control:
1) File context, Daemon can only access file with particular file context
2) Boolean Value: enable/disalbe a feature
for example: By default SElinux does not allow users to login and read their home directories, turn it on by "setsebool -P ftp_home_dir 1"

#==Confined and Unconfined Process
Confined process enter paritcular domain after started, only particular domain has access to particular TYPE files
SElinux has no effect for Unconfined Processes (apps doen's support SElinux)

==Example
$ ls -Z /usr/sbin/httpd
-rwxr-xr-x root root system_u:object_r:httpd_exec_t /usr/sbin/httpd #httpd is confined by default
$chcon -Rt unconfined_exec_t /usr/sbin/httpd #change httpd to unconfied_exec_t, it will enter unconfied domain, so it can access any file as long as OS level file permission allowed
$ restorecon -Rv /usr/sbin/httpd #restore default type

#== SELinux: File context
user:role:type:sensitivity:category
for example: system_u:object_r:httpd_sys_content_t :s0:c0
Not all systems will display s0:c0

==example
# ls -aZ /var/www/html/
drwxr-xr-x root root system_u:object_r:httpd_sys_content_t .
drwxr-xr-x root root system_u:object_r:httpd_sys_content_t ..
# ls -aZd /home
drwxr-xr-x root root system_u:object_r:home_root_t /home
httpd_exec_t can access httpd_sys_content_t not home_root_t

#==SElinux managment
SELINUX=permissive #in /etc/selinux/config. if it changed from disabled . it needs reboot to lable files
getenforce or sestatus #get current status
setenforce 0 # set to permissive mode
setenforce 1 #set to enforce mode
getsebool -a #list booleans and its value , no desc
setsebool httpd_can_network_connect_db on #change current boolean
setsebool -P httpd_can_network_connect_db on #change permanent boolean with -P

- Temparary change context
chcon -R -t httpd_sys_content_t /web/ #change context type dir/file
# it will survive reboot, but not relabel. To relabel, touch /.autorelabel reboot

- Persistent Changes: semanage fcontext
/etc/selinux/targeted/contexts/files/file_contexts #saved to orginal context
/etc/selinux/targeted/contexts/files/ file_contexts.local #saved to new user context
semanage fcontext -a -t samba_share_t /etc/file1 #-a add new context, the file doesn't need to exist.
restorecon -Rv /etc/file1 #read the new customized context and apply it

- Restore default context
semanage fcontext -d /etc/file1 #remove context,the file doesn't need to exist
restorecon -RFv /etc/file1 #apply the change, -F is needed you to restore from customized to default.

#==Troubleshooting
/var/log/messages.X
/var/log/audit/audit.log #enable auditd daemon first
chkconfig --levels 345 setroubleshoot on #enable troubleshoot daemon
sealert -a /var/log/messages #analyse log
sealert -l \* #show all alert
grep "SELinux is preventing" /var/log/messages
grep "denied" /var/log/audit/audit.log
Port Numbers # services are allowed to run on some defined ports
/usr/sbin/semanage port -l grep http_port_t
ttp_port_t tcp 80, 443, 488, 8008, 8009, 8443
semanage port -a -t http_port_t -p tcp 9876 #add the new port to allowed range

#==== document
selinux-policy-2.4.6-137.el5#man pages for ftpd_selinux, samba_selinux ...etc

OpenNMS monitor disk space usage by SNMP

#==Overview
This post demonstrates two ways to monitor disk space usage by 2 different SNMP MIBS
1) .iso.org.dod.internet.private.enterprises.ucdavis.dskTable
2) .iso.org.dod.internet.mgmt.mib-2.host.hrStorage

What is the diffrence? Option #1 requires disk path to be hardcoded in snmpd.conf at target system, But Option #2 can monitor all partions by default, even there is need to monitor specific partions, the filter is set on OpenNMS, not target system, So Option #2 is more flexible

The alarm is triggered by threshold in SNMP, so you don't need to setup monitors to trigger alarm. Firstly, Please make sure you have basic snmp working, refer to my post Set up Net-snmp on CentOS

#==ENV
OpenNMS 1.6.2 + Centos 5.2 +net-snmp 5.3.1

#==(1) Monitor disk space usage by dskTable MIB
add the parttiton to be monitored to snmpd.conf
$vi /etc/snmp/snmpd.conf
disk /opt2

#====Test by snmpwalk first

$snmpwalk -v2c 172.16.31.3 -c public .iso.org.dod.internet.private.enterprises.ucdavis.dskTable

UCD-SNMP-MIB::dskPath.1 = STRING: /opt2 

OpenNMS 1.6.2 has set default threshold, so you don't need to config any thing in openNMS. doublecheck the threshold by
GUI->Admin->Manage Thresholds->netsnmp

#====Sample alarm appeared
High threshold exceeded for SNMP datasource ns-dskPercent on interface 172.16.31.3, parms: ds="ns-dskPercent" value="100.0" threshold="90.0" trigger="2" rearm="75.0" label="/opt2" ifIndex="2

#== (2) Monitor disk space usage by hrStorage MIB
no need to add disk path to snmpd.conf, but openNMS needs to be customized.

#===Test by snmpwalk first

$snmpwalk -v2c 172.16.31.3 -c public .iso.org.dod.internet.mgmt.mib-2.host.hrStorage

#==== Find systemOID
This OID is the same to all Net-SNMP agent

$snmptranslate .iso.org.dod.internet.private.enterprises.netSnmp.netSnmpEnumerations.netSnmpAgentOIDs -On

.1.3.6.1.4.1.8072.3.2

#==== Include the SysOID to ./etc/datacollection-config.xml
By default, mib2-host-resources-storage is not included for Net-SNMP
The value in sysoidMask should include your systemOID
For example sysoidMask .1.3.6.1.4.1.8072.3. includes .1.3.6.1.4.1.8072.3.2

systemDef name="Net-SNMP"
        sysoidMask.1.3.6.1.4.1.8072.3./sysoidMask
        collect
          ...
 includeGroup mib2-host-resources-storage/includeGroup

#====Include your sysOID to ./etc/threshd-configuration.xml

package name="hrstorage"
         filterIPADDR != '0.0.0.0' & (nodeSysOID LIKE '.1.3.6.1.4.1.8072.3.2.%'  nodeSysOID LIKE '.1.3.6.1.4.1.311.%'   nodeSysOID LIKE '.1.3.6.1.4.1.2.3.1.2.1.1.3.%')/filter
..
  /package

OpenNMS 1.6.2 has set default threshold, doublecheck the threshold by
GUI->Admin->Manage Thresholds->hrStorage

By default, it monitors all partions. You can create filter to monitor specific partions only

#===Sample alarm appeared
High threshold exceeded for SNMP datasource hrStorageUsed / hrStorageSize * 100.0 on interface 172.16.31.3, parms: ds="hrStorageUsed / hrStorageSize * 100.0" value="94.88754412506304" threshold="90.0" trigger="2" rearm="75.0" label="/opt2" ifIndex="2"

Setup net-snmp on Linux (CentOS 5.2)

The default configuration on net-snmp is very secure, it allows public access to system OID only, If you try access any other OID, it give erorr:No Such Object available on this agent at this OID. This article show how to setup a basic net-snmp with access control ability.

$snmpwalk -v 2c localhost -c public system
SNMPv2-MIB::sysDescr.0 = STRING: Linux centos-ks 2.6.18-92.el5 #1 SMP Tue Jun 10 18:49:47 EDT 2008 i686

$ snmpwalk -v 2c localhost -c public interfaces
IF-MIB::interfaces = No Such Object available on this agent at this OID



#==Env
NET-SNMP version 5.3.1 Centos 5.2

#=== sample /etc/snmpd/snmpd.conf
- It is important to comment out any default statement above, Because access decision is based on first match.


## sec.name source community
com2sec mynetwork 127.0.0.1 public
com2sec mynetwork 172.16.31.0/24 public

## group.name sec.model sec.name
group MyROGroup v1 mynetwork
group MyROGroup v2c mynetwork

## incl/excl subtree mask
view all included .1

## context sec.model sec.level prefix read write notif
access MyROGroup "" any noauth exact all none none

**Updated:  28 March 2011

The above statements can be simplified as:
rocommunity  public  127.0.0.1  .1

rocommunity  public  172.16.31.0/24  .1

NOTE:rocommunity can't restrict SNMP version, it allows  all versions:v1 and v2c

#== Troubleshooting
- Snmpd still starts despite syntax error, it make troubleshooting difficult, But if you start it with DEBUG it will warn you any errors
/usr/sbin/snmpd -LE 7 -p /var/run/snmpd.pid -a

- By default, SNMPD looks for modules in /usr/share/snmp/mibs, The following command will check the loaded module
snmpd -Dmib_init

- If you don't know the OID of an object, snmptranslate can help, The following demostrate how to find objectname and its OID

$ snmptranslate -Ts  grep interface
.iso.org.dod.internet.mgmt.mib-2.interfaces
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifNumber
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex
.iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifDescr


$ snmpget -v 1 localhost -c public interfaces.ifTable.ifEntry.ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0

$ snmptranslate .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifDescr.2 -On
.1.3.6.1.2.1.2.2.1.2.2

$ snmpget -v 1 localhost -c public .1.3.6.1.2.1.2.2.1.2.2
IF-MIB::ifDescr.2 = STRING: eth0

How to display content of files along with file names?

Sometimes it is useful to display content of files along with file names. egrep or pr can do the trick


$ cat 1.txt
0
1

$ cat 2.txt
2
3

#==cat can't display file name

$ cat *.txt
0
1
2
3

#==display all with wildcard filter *

$ egrep \* *.txt
1.txt:0
1.txt:1
2.txt:2
2.txt:3

#==The header of pr displays filename, sed is used to chop blank lines

$ pr *.txt sed '/^$/d'
2009-03-25 03:17 1.txt Page 1
0
1
2009-03-25 03:17 2.txt Page 1
2
3

When some processes stop Solaris Zone from being shutdown

If Solaris Zone takes long time to shutdown, you may need to examine the process with '*' on the state status.

$ svcs -a grep sendmail
*online Mar_09 svc:/network/smtp:sendmail

#==find the process id of the offending process
$svcs -p sendmail
STATE STIME FMRI
*online Mar_09 svc:/network/smtp:sendmail
Mar_09 309 sendmail
Mar_09 310 sendmail

#==Then kill with kill cmd



#==If it happens quite offen, You may find the following script handy.

#!/bin/ksh
SVCNAMES=$@
CNT=0
DELAY=5
#
getpid () {
           
        SVC=$1
        /usr/bin/svcs $SVC >/dev/null
        if [ $? -ne 0 ];then
                PID=0       
                return 1
        fi
        PID=`/usr/bin/svcs -Hp $SVC|tail +2 | awk '{print $2}'| tail -1`
        if [ -z "$PID" ];then
                PID=1            
        fi                       
        return 0    
}

[ -z $SVCNAMES ] && echo "Usage $0 svcname1 [svcname2] .."

for SVCNAME in $SVCNAMES
do

getpid  $SVCNAME 

if [ $PID -lt 1 ];then
        echo "No pid found for $SVCNAME"
                exit 1
fi

while [ $PID -gt 1 ]
do
        echo "Delay for " $DELAY " secs"
        sleep $DELAY;
        getpid $SVCNAME
        CNT=`expr $CNT + 1 `    
        if [ $CNT -le 7 ] && [ $PID -gt 1 ];then
         echo "Service $SVCNAME is still running after " `expr $CNT \* $DELAY ` "secs, Gracefully  kill it: kill $PID"
           kill $PID
        elif [ $CNT -gt 7 ] && [ $PID -gt 1 ];then
         echo "Service $SVCNAME is still running after " `expr $CNT \* $DELAY ` "secs, Forcefully  kill it: kill -9 $PID"
        kill -9 $PID

        fi  
done
sleep 2;
/usr/bin/svcs $SVCNAME | grep disabled  

if [ $?  -eq 0   ]; then
        echo "Service is stopped"
else
        echo "Service is till running, please kill it mannually"
        exit 1
fi

done

Integrating Nagios plugin with OpenNMS

OpenNMS is highly scalable enterprise level management system. I like its features of versatile built-in monitors, auto-discovery and graphing ability. It can also work with ngaios plugin to use any customized monitor.

Install OpenNMS

http://www.opennms.org/documentation/InstallStable.html

Setup NRPE on client

yum install nrpe nagios-plugins-nrpe nagios-plugins

#==create a test script
$vi /usr/lib/nagios/plugins/check_test.sh

#!/bin/sh
STATE_OK=0
STATE_WARNING=1
STATE_CRITICAL=2
STATE_UNKNOWN=3
echo "check test"
exit $STATE_OK


$ make sure nagios user has rx permission for the script.
chmod +rx check_test.sh


#vi /etc/nagios/nrpe.cfg
allowed_hosts=127.0.0.1,IP of OpenNMS
command[check_test]=/usr/lib/nagios/plugins/check_test.sh

#==start nrpe
service nrpe start

#==Now test it mannually, It is important to run the check as user nagios not root

sudo -u nagios /usr/lib/nagios/plugins/check_nrpe -n -H localhost -c check_test
sudo -u nagios /usr/lib/nagios/plugins/check_nrpe -H localhost -c check_test
-n = Do no use SSL,if nrpe doesn't support both mode, it is important to set usessl value in opennms config file.

Setup NRPE on OpenNMS system:

yum install nagios-plugins-nrpe nagios-plugins
#==no configuration needed here, first run a mannual test
sudo -u nagios /usr/lib/nagios/plugins/check_nrpe -n -H remote-host -c check_test
sudo -u nagios /usr/lib/nagios/plugins/check_nrpe -H remote-host -c check_test
-n = Do no use SSL. if nrpe doesn't support both mode, it is important to set usessl value in opennms config file.


OpenNMS configuration

Two configuartion files need to be modified for new added service.
/opt/opennms/etc/capsd-configuration.xml /* service definition for initial scan */
/opt/opennms/etc/ poller-configuration.xml /* service definition for constant polling */


/opt/opennms/etc/capsd-configuration.xml
protocol-plugin
protocol-plugin protocol="NRPE-test" class-name="org.opennms.netmgt.capsd.plugins.NrpePlugin" scan="on"
property key="banner" value="*"
property key="port" value="5666"
property key="timeout" value="3000"
property key="retry" value="2"
property key="usessl" value="true"
property key="command" value="check_test"
protocol-plugin

- Important Notes:
Set usessl value depending on your nrpe ssl supporting ability
command used for polling, it better to be set to your customized script(or system built-in cmd: _NRPE_CHECK)

/opt/opennms/etc/ poller-configuration.xml

service name="NRPE-test" interval="300000" user-defined="true" status="on"
parameter key="retry" value="3"
parameter key="timeout" value="3000"
parameter key="port" value="5666"
parameter key="command" value="check_test"
parameter key="usessl" value="true"
parameter key="padding" value="2"
parameter key="rrd-repository" value="/opt/opennms/share/rrd/response"
parameter key="ds-name" value="nrpe-test"
service

- Important Notes:

The name attribute of the service in poller-configuration.xml needs to match the protocol attribute of the protocol-plugin in capsd-configuration.xml.
The ds-name attribute also needs to be unique for each service, or you'll find response time from one service overwriting response time from another.
You'll also need a line to map the new service to a monitor class (see at the end of the file)

monitor service="NRPE-test" class-name="org.opennms.netmgt.poller.monitors.NrpeMonitor"
Restart OpenNMS for the changes to take effect
The new service should be discovered by re-scan.

Troubleshooting
if the new added service can't be discovered, turn on debug on for discovery process capsd

/opt/opennms/etc/log4j.properties

# Capsd
log4j.category.OpenNMS.Capsd=DEBUG, CAPSD
log4j.appender.CAPSD.File=/opt/opennms/logs/daemon/capsd.log


if the new added service can be discovered, but having issue with polling,turn on debug on for polling process poller
/opt/opennms/etc/log4j.properties

# Pollers
log4j.category.OpenNMS.Poller=DEBUG, POLLERS