Monitor customized application in Windows by SNMP

The native SNMP service  in Windows can provide basic metrics like CPU, memory and disk etc, but it doesn’t have “extend” feature in net-snmp, which allows you run a script for application monitoring. Net-snmp can’t be used as replacement for Windows SNMP service because some SNMP extension agent relies on it and known issue like HOST-RESOURCES MIB doesn’t work in net-snmp. 

 The good news is that you can have net-snmp co-exist with Windows SNMP, you can have nice features like extend ability, in the mean time, pass the other functions to native Windows SNMP service.

As of Net-SNMP 5.4, the Net-SNMP agent is able to load the Windows SNMP service extension DLLs by using the Net-SNMP winExtDLL extension. The extension requires the net-snmp binary to be native (32bit net-snmp extension won’t work in 64bit Windows).

Net-snmp 64bit binary is hard to find, it seems only net-snmp-5.5.0-2 has 64bit binary pre-compiled, you might need to compile yourself for other versions. 

Install net-snmp

Run the net-snmp binary installer select “with Windows Extenstion” instead of standard agent, unselect “net-snmp trap service” and “Perl SNMP modules”, the default path is c:\usr

Configure net-snmp

Register net-snmp as Windows service

Edit c:\usr\registeragent.bat to disable modules conflicting to Windows   by adding parameter.

“-I-udp,udpTable,tcp,tcpTable,icmp,ip,interfaces,snmp_mib”

(Note: if system_mib is also disabled, SNMPv2-MIB::sysuptime won’t report correct time)

Run c:\usr\registeragent.bat

Edit C:\usr\etc\snmp\snmpd.conf

rocommunity public 192.168.1.10
#Test extend feature to execute a script, the script path must use Unix style ‘/’
extend userscript c:/temp/test1.bat

Start Windows service “net-snmp agent”(Native SNMP service must be stopped)

Test

#Test standard SNMP metrics, the HOST-RESOURCES-MIB is provided by native SNMP service, not net-snmp
[root@zabbix]#/usr/bin/snmpwalk -v 2c  -c public 192.168.1.20   HOST-RESOURCES-MIB::hrSystemUptime
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (640892116) 74 days, 4:15:21.16

#The extend feature is provided by net-snmp, Execute the script by snmpwalk
[root@zabbix]#/usr/bin/snmpwalk -v 2c -Ov -c public 192.168.1.20 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."userscript"'
STRING: web-time=80
STRING: web-status=[ok]

Troubleshooting

C:\usr\log\snmpd.log

Check which Windows modules loaded, start snmpd in command line with debugging “WinExtDLL”

Snmpd.exe -I-udp,udpTable,tcp,tcpTable,icmp,ip,interfaces,snmp_mib  -DwinExtDLL 

Reference:

http://net-snmp.sourceforge.net/docs/README.win32.html

Authenticate RHEL 6 Linux users by Windows 2008 R2 AD

The nss_ldap in RHEL 5 for LDAP authentication has been obsolete in RHEL 6, the replacement is nss-pam-ldapd. But the preferred method for LDAP authentication in RHEL 6 is System Security Services Daemon (SSSD) (in fact, RHEL 5.6 or later supports SSSD ).
SSSD unique  features:
- Credentials caching, user can still login when LDAP server is offline.
- Persistent connection, reducing the overhead of opening a new socket for each request
- support for multiple LDAP/NIS domains

Install SSSD packages

$yum install sssd sssd-client

Run the following command, which will make necessary changes in /etc/krb5.conf, /etc/sssd/sssd.conf, /etc/nsswitch.conf, /etc/pam.d/

$authconfig --enablesssd --ldapserver=ldap://adc.ad.example.com --ldapbasedn="OU=_USERS,DC=ad,DC=example,DC=com" --enablerfc2307bis --enablesssdauth --krb5kdc=adc.ad.example.com --krb5realm=AD.EXAMPLE.COM --disableforcelegacy --enablelocauthorize --enablemkhomedir   --updateall

All files should be updated automatically, only /etc/sssd/sssd.conf need to be customized. The following is an example file with minimum parameters needed.

#cat /etc/sssd/sssd.conf
[sssd]
config_file_version = 2
services = nss, pam
domains = default
[nss]
#debug_level=7
[pam]
[domain/default]
ldap_id_use_start_tls = False
cache_credentials = True
#Without enumerate = True,  users won’t be show from ‘getent passwd’ output.
enumerate = True
id_provider = ldap
auth_provider = krb5
chpass_provider = krb5
ldap_schema = rfc2307bis
ldap_force_upper_case_realm = True
ldap_user_object_class = user
ldap_group_object_class = group
ldap_user_gecos = displayName
ldap_user_home_directory = unixHomeDirectory
ldap_uri = ldap://adc.ad.example.com
ldap_search_base = OU=_USERS,DC=ad,DC=example,DC=com
ldap_user_search_base = OU=_USERS,DC=ad,DC=example,DC=com
ldap_group_search_base = OU=_GROUPS,DC=ad,DC=example,DC=com
ldap_default_bind_dn = CN=svc_ldap_client,OU=MGT,OU=_USERS,DC=ad,DC=example,DC=com
ldap_default_authtok_type = password
ldap_default_authtok = P@ss123
ldap_tls_cacertdir = /etc/openldap/cacerts
krb5_server = adc.ad.example.com
krb5_kpasswd = adc.ad.example.com
krb5_realm = AD.EXAMPLE.COM

Authenticate RHEL 5 Linux users by Windows 2008 R2 AD

My previous post was tested on Windows 2003 AD, which use non-RFC compliant scheme, Windows 2003 R2 or later is RFC2307bis compliant, the following is tested on Windows 2008 R2, but it should be working for Windows 2003 R2 and Windows 2008 as well.
The following use nss_ldap to do AD authentication by Kerberos, if you use RHEl 5.6 or later, you may consider  System Security Services Daemon (SSSD), which offer many great features.
 Setup Windows AD
Windows 2008 R2 AD has built-in component to perform the same function of “Windows services for Unix” in Windows 2003, it is named “identity management for Unix”
Install “identity management for Unix” by clicking “Add Role Services” under role of “Active Directory Domain services”. Choose all three sub-components in identity management for Unix.(note: QLOGIC SANsurfer software conflicts with RPC services, remove it before install identity management for Unix)
Setup ldapbind user, create test user, test group and set Unix attributes as the previous post
Setup configuration files
1.    Configure /etc/ldap.conf
nss_map_attribute in Windows 2008 R2 is different to Windows 2003, the following is a sample file

#cat /etc/ldap.conf
base OU=_USERS,DC=AD,DC=example,DC=com
BINDDN CN=svc_ldap_client,OU=MGT,OU=_USERS,DC=AD,DC=example,DC=com
BINDPW Pass123

timelimit 60
bind_timelimit 10
#by default, if ldap server is not reachable, it will retry long time before giving up,
# nss_reconnect_tries 1 limit it to be less than a minute. 
nss_reconnect_tries 1
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_objectclass posixGroup  group
nss_map_attribute uid sAMAccountName
#nss_map_attribute uidNumber uidNumber
#nss_map_attribute gidNumber gidNumber
nss_map_attribute gecos         name
nss_map_attribute homeDirectory unixHomeDirectory
#nss_map_attribute loginShell loginShell
nss_map_attribute shadowLastChange pwdLastSet
nss_base_password OU=_USERS,DC=AD,DC=example,DC=com
nss_base_shadow OU=_USERS,DC=AD,DC=example,DC=com
nss_base_group OU=_GROUPS, DC=AD,DC=example,DC=com
pam_login_attribute sAMAccountName
pam_filter objectclass=User
pam_password ad
nss_initgroups_ignoreusers root,ldap,named,avahi,haldaemon,dbus,radvd,tomcat,radiusd,news,mailman,nscd,gdm
#adc.ad.example.com is alias DNS name load balanced to DCs
uri ldap://adc.ad.example.com
ssl no
tls_cacertdir /etc/openldap/cacerts

2.Configure /etc/krb5.conf, no difference to Windows 2003 AD

#cat /etc/krb5.conf 
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log
[libdefaults]
 default_realm =   AD.EXAMPLE.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 forwardable = yes
[realms]
   AD.EXAMPLE.COM = {
  kdc = adc.ad.example.com
  admin_server=adc.ad.example.com
 }
[domain_realm]
 example.com = AD.EXAMPLE.COM
 .example.com = AD.EXAMPLE.COM
[appdefaults]
 pam = {
   debug = false
   ticket_lifetime = 36000
   renew_lifetime = 36000
   forwardable = true
   krb4_convert = false
 }

3.You also need to modify files in /etc/pam.d/ /etc/nsswitch.conf, once you have copied the ldap.conf and krb5.conf files, run the following command to automate the tasks.

$authconfig --enablecache  --enableldap --usemd5 --useshadow  --enablelocauthorize --enablekrb5  --enablemkhomedir  --update

Clone Windows 2008 R2 on UEFI based Servers

Unified Extensible Firmware Interface (UEFI) technology has been widely adopted by x86 servers hardware manufactures such as IBM and DELL to supersede BIOS, but it presents a challenge for operating system cloning, because the boot code is in efi files not MBR, and even efi files are replicated by file copy or sector copy, the files need to be updated for new hardware, the following example demonstrate cloning Windows 2008 R2 by imagex tool in IBM system X servers.

In source computer, capture the two partitions after booting from winPE

#capture Windows system partition after being syspreped.
$imagex /compress fast  /capture c:  $networkshare:\boot.wim "w2k8 R2 64bit"
#capture EFT system partition
#driver letter S: was assigned by diskpart command: diskpart; select disk 0; list volume; select volume 2; assign letter=s:
$imagex /compress fast  /append  s:  $networkshare:\boot.wim "w2k8 R2 64bit eftsys"

In target computer, create 3 mandatory partitions after booting from winPE

#MSR partition is required for partition operations like converting to dynamic disk, encrypting partition.
#“diskpart  /s diskpart.txt”
$type diskpart.txt
select disk 0
clean
convert gpt
rem == 1. System partition =========================
create partition efi size=100
format quick fs=fat32 label="System"
select partition 1
assign letter="s"
rem == 2. Microsoft Reserved (MSR) partition =======
create partition msr size=128
rem == 3. Microsoft Windows partition =======
create partition primary size=102400
select partition 3
format quick fs=ntfs label="Windows"
assign letter="c"

# In target computer, apply the images

imagex /apply  $networkshare\boot.wim 1 c: && imagex /apply $networkshare boot.wim 2 s:  

In target computer, update BCD store in EFI system partition

bcdedit -store S:\EFI\Microsoft\Boot\BCD /set {bootmgr} device partition=s:
bcdedit -store S:\EFI\Microsoft\Boot\BCD /set {memdiag} device partition=s:
bcdedit -store S:\EFI\Microsoft\Boot\BCD /set {default} device partition=c:
bcdedit -store S:\EFI\Microsoft\Boot\BCD /set {default} osdevice partition=c:
bcdedit -store S:\EFI\Microsoft\Boot\BCD /set {fwbootmgr} displayorder {bootmgr} /addfirst 

Reboot target computer, press F1 to go to BIOS setup

Select the boot file by navigating to


boot manager -> boot from file ->EFI->boot->bootx64.efi


(This file is actually S:\EFI\Microsoft\Boot\bootx64.efi)


After the file(script) is selected, it will call s:\EFI\Microsoft\Boot\bootmgfw.efi which, in turn, will call c:\Windows\system32\winload.efi to boot Windows, Additionally , a new boot entry named “Windows boot manager” will be added as new boot option to the top of boot list in EFI BIOS. So next time, it will boot to Windows automatically without manual intervention.

Script automation with SSH in Windows

I am testing a monitoring software, like the most monitoring software, it  relies on SSH to call remote customized scripts. But it is running on Windows. To get SSH command line working nicely in Windows is not quite easy.
There are two options to use SSH to do script automation in Windows.
1.Use SSH command line
2.Use SSH API for a scripting language i.e. Perl
 Use SSH command line
PuTTY is a well-known SSH GUI client,the non-interactive, command line interface is plink
The other option of SSH command line is ssh in cygwin, this ssh works exactly the same way as ssh in Linux.
SSH in cygwin support more features than plink, but it requires installing the whole cygwin environment, while plink is just a single binary file.
However plink has a draw back, it seems there is no way to suppress the “accept remote host key prompt” and its host key cache is stored in Windows registry.
SSH in cygwin can suppress the prompt by setting “-o stricthostkeychecking=no” and the host key cache is stored in .ssh in user's home folder. If the monitoring software is running as a service under 'local system account', where is the .ssh located? The answer for Windows 2008 R2 is “c:\Windows\SysWOW64\config\systemprofile”, you have to create “.ssh”  folder manually, otherwise the service can't save the ssh host key cache.
 Use SSH API for a scripting language i.e. Perl
The above method of using SSH command line works, but is not as flexible as using SSH API.
The SSH API is from libssh2, Perl has a module,NET::SSH2,to utilize libssh2 to provide the same function of ssh in cygwin. The benefits of NET::SSH2  are: better error handling and no ssh host key prompt issue.
Install Perl for Windows
Activeperl is probably the most popular choice in Windows, but it has limited number of pre-built Perl Modules, it is very hard to compile from source for new modules. Activeperl itself doesn't have compiler or make utility, you can get MinGW compiler or dmake, but the chance of success build is very slim.
Strawberryperl,http://strawberryperl.com/  is an opensource project for Perl in Windows, it has built-in compiler and related tools. It can download and install modules in www.cpan.org
As of Strawberry Perl 5.12, net::ssh2 is built-in, the following is simple example of of net:ssh2 usage.

##Note: if you encounter an error "libssh2-1_.dll was not found", you need to add the path of the dll, "C:\strawberry\c\bin", into %PATH% env variable
use  warnings;
use strict;
use Net::SSH2;
use constant BUFLEN => 10_0000 ;
my $ssh2 = Net::SSH2->new();
$ssh2->connect('server1.example.com',22) or die "Unable to connect Host $@ \n";
if ($ssh2->auth_password('user1','Pass1')) {
#if ( $ssh2->auth_publickey ( 'user1', 'id_rsa.pub', 'id_rsa')) { 
my $chan2 = $ssh2->channel();
#$chan2-&gtblocking(0);
$chan2->exec ("uname -a\n");
$chan2->read($buf, BUFLEN );
print $buf;
$chan2->close;
} else {
warn "auth failed.\n";
}