Proactive monitoring by snmptrap

Pulling snmp information is used in most monitoring solutions, however pushing information  is  an alternative monitoring solution by snmptrap.
This post demonstrates how to email alarms being pushed to receiver: snmptrapd from snmp agent.
Tested on Centos 5.5 +NET-SNMP  5.3.2.2
Install email daemon and net-snmp

$yum install postfix net-snmp net-snmp-utils
$cat /etc/snmp/snmptrapd.conf
#authCommunity   TYPES COMMUNITY  [SOURCE [OID | -v VIEW ]]
authCommunity  execute public  default  .1
traphandle  default /usr/bin/traptoemail -s localhost -f snmp@localhost root@localhost

start snmptrapd  and start postfix
Test by snmptrap tool
Send email if eth0 operation status is up (1)
(IF-MIB::linkUp is notification object defined in MIB file: IF-MIB.txt)

$snmptrap -v 2c -c public 127.0.0.1 "" IF-MIB::linkUp  .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOperStatus.1 i 1

Sample email received

$mail
…
>N 86 snmp@localhost.local  Mon Mar  7 15:59  18/747   "trap received from localhost: IF-MIB::linkUp"
& 86
Message 86:
From snmp@localhost.local.net  Mon Mar  7 15:59:13 2011
X-Original-To: root@localhost
Delivered-To: root@localhost.local.net
To: root@localhost.local.net
From: snmp@localhost.local.net
Subject: trap received from localhost: IF-MIB::linkUp
Date: Mon,  7 Mar 2011 15:59:13 +1100 (EST)
Host: localhost (UDP: [127.0.0.1]:35453)
DISMAN-EVENT-MIB::sysUpTimeInstance  0:6:08:29.87
SNMPv2-MIB::snmpTrapOID.0  IF-MIB::linkUp
IF-MIB::ifOperStatus.1  up

The above configuration make snmptrapd ready to receive traps, the following steps is to
configure snmp agent to send traps.
A SNMP v3 USM user need to be created, even the trap is intended for snmp v1/v2c only.
Check my previous post for creating and managing SNMP v3 USM users

$ cat /etc/snmp/snmpd.conf
#authuser    read,write [-s secmodel] user [noauth|auth|priv [oid|-V view]]
authuser   read -s v2c guest_user noauth  .1
authuser   read -s usm guest_user noauth  .1
authcommunity read  public  default .1
trap2sink 127.0.0.1 public
iquerySecName guest_user
agentSecName  guest_user
monitor   -u guest_user  -r 60 "interface down" -o ifDescr ifOperStatus != 1
If you shutdown any interface and restart snmpd, following email notification should appear
$mail 
..
N 87 snmp@localhost.local  Mon Mar  7 16:24  23/1030  "trap received from localhost: DISMAN-EVENT-MIB::mteTriggerFired"
& 87
Message 87:
From snmp@localhost.local.net  Mon Mar  7 16:24:28 2011
X-Original-To: root@localhost
Delivered-To: root@localhost.local.net
To: root@localhost.local.net
From: snmp@localhost.local.net
Subject: trap received from localhost: DISMAN-EVENT-MIB::mteTriggerFired
Date: Mon,  7 Mar 2011 16:24:28 +1100 (EST)
Host: localhost (UDP: [127.0.0.1]:46356)
DISMAN-EVENT-MIB::sysUpTimeInstance  0:0:00:00.84
SNMPv2-MIB::snmpTrapOID.0  DISMAN-EVENT-MIB::mteTriggerFired
DISMAN-EVENT-MIB::mteHotTrigger.0  interface down
DISMAN-EVENT-MIB::mteHotTargetName.0
DISMAN-EVENT-MIB::mteHotContextName.0
DISMAN-EVENT-MIB::mteHotOID.0  IF-MIB::ifOperStatus.4
DISMAN-EVENT-MIB::mteHotValue.0  2
IF-MIB::ifDescr.4  eth2

You can enable  “linkUpDownNotifications yes” to track interface status, but I found this type of  notification didn’t have interface name information.
Troubleshooting
1.failed to run mteTrigger query error
- make sure the user has permission in sec mode: usm as well.  “authuser   read -s usm guest_user noauth  .1”
- specifically set user with  “–u guest_user” in monitor command
2.Start snmpd in debugging mode for disman (Distributed Management )

/usr/sbin/snmpd -Ddisman -Lsd -Lf /var/log/snmpd.log -p /var/run/snmpd.pid -a