Build Custom AWS Config rules using Guard DSL

I want to use AWS config to report S3 buckets without public blocking enabled except for some buckets with prefix like www, there is a AWS managed rule which can achieve this with a limitation: the excluded buckets must be provided as exact names separated by comma, regex match is not supported.

 

Previously, to create a custom rule, you would have to define an AWS Lambda function, from June 2022, you can author AWS Config custom rules using Guard DSL without needing to develop AWS Lambda functions.

 

What is Guard DSL?

Guard DSL is open-source policy-as-code domain-specific language (DSL) to write rules and validate JSON- and YAML-formatted data such as CloudFormation Templates, K8s configurations, and Terraform JSON plans/configurations against those rules. It is language developed by CloudFormation Guard project.

 Ignore the word CloudFormation here, Guard DSL is mainly used for validating CloudFormation template, but in the AWS config context, it has nothing to do with CloudFormation.

 

Get started with Guard DSL

Learn Guard DSL syntax

Ignore all object reference name from above document, the names are CloudFormation Objects. For the purpose of AWS config, refer to aws-config-resource-schema 

Sample AWS Config Rule with Guard DSL.

Create AWS Config Rule->Create custom rule using Guard

Scope of changes ->Resource Type(S3 Buckets)

Rule Content: 

rule s3_bucket_is_public when
    resourceType == "AWS::S3::Bucket" configuration.name != /^www/ {
        supplementaryConfiguration.PublicAccessBlockConfiguration.blockPublicPolicy exists
        supplementaryConfiguration.PublicAccessBlockConfiguration.blockPublicAcls exists
        supplementaryConfiguration.PublicAccessBlockConfiguration.restrictPublicBuckets exists
        supplementaryConfiguration.PublicAccessBlockConfiguration.ignorePublicAcls exists
    }

rule s3_bucket_not_public when
    s3_bucket_is_public {
    supplementaryConfiguration.PublicAccessBlockConfiguration {
        blockPublicAcls == true
        blockPublicPolicy == true
        restrictPublicBuckets == true
        ignorePublicAcls == true
    }
}

The rules are evaluated in order, evaluation continues regardless the result of previous rule.
PublicAccessBlockConfiguration block  has  conditions separated by new line, which implies AND
Please note the first rule name is being used as condition in the second rule

- when bucket with www prefix is being evaluated, the result is not_applicable for both conditions.

- when bucket with public blocking enabled is being evaluated, the result is compliant for both conditions

- when bucket without public blocking enabled is being evaluated, 1st rule result is not_compliant, 2nd rule result is not_applicable. effective result is not_compliant

Send Docker container log to Splunk

Docker engine has Splunk logging driver to send container logs to Splunk via HEC(http event collector. It is easy to setup, however it could suffer data loss if HEC is down. That is the advantage of file based data ingestion which can retry and resume even splunk service is down for long period of time. The default Docker logging driver is json-file, there is no problem for Splunk agent to read the file, however how could you set index for different application, since the container id is random so the file path is random. The trick to use jounald driver and use rsyslog rules to read the jounald log and write to different file names based on any docker label.


Overview:
steps included:
- enable persistent journald storage
- create rsyslog rule to read journald and write to json log file,file name depends on docker_appnane

remaining steps to be achieved by other Ansible roles
- docker container need to set log driver to journald and expose label docker-appname e.g
docker run --log-driver=journald \
--log-opt labels=docker-appname \
--label docker-appname=mulesoft \

- Splunk UF read /var/log/docker-{{docker_appname}}.log and forward to splunk cloud
- logrorate rule to rotate /var/log/docker-{{docker_appname}}.log

Rsyslog rule in RHEL7

#imjournal module is loaded in main syslogd.conf

$IMJournalStateFile imjournal.state
$imjournalRatelimitInterval 300
$imjournalRatelimitBurst 30000

module(load="mmjsonparse") 

action(type="mmjsonparse")

#output all json fileds and remove redundant last msg field
#template(name="jsonformat" type="string" string="%$!all-json:R,ERE,1,FIELD:(.*), (\"msg\":.*)--end% }\n")
template(name="jsonformat" type="string" string="%$!all-json%\n")

if ($!DOCKER_APPNAME == "{{docker_appname}}") then {
    action(type="omfile" file="/var/log/docker-{{docker_appname}}.log" template="jsonformat")
    stop
} 

Splunk customized source type to be set in Heavy forwarder or Indexer(universal forwarder doesn't support sourcetype defination)

props.conf

[json_realtime_timestamp]
KV_MODE = json
MAX_TIMESTAMP_LOOKAHEAD = 16
NO_BINARY_CHECK = 1
SHOULD_LINEMERGE = false
TIME_FORMAT = %s%6N
TIME_PREFIX = "_SOURCE_REALTIME_TIMESTAMP":\s*"
pulldown_type = 1
TZ=UTC

inputs.conf in universal forwarder in Docker host

[monitor:///var/log/docker-mulesoft.log]
disabled = true
sourcetype=json_realtime_timestamp
index = mulesoft

Python script to generate Ansible ini inventory file from csv file

Ansible in memory inventory file created by add_host is often used in AWS EC2 provisioning. Inventory file can be generated easily,however it has drawback. Because it is in memory, all server post build tasks have to be in one big playbook. Which means it is not easy to re-run failed tasks if there is failure and existing post build playbooks can't be reused.

I created a Python script to generate a temporary inventory file from csv file used in EC2 provisioning. The inventory file can be used in multiple post build playbooks. The file name is static, however it will not be overwritten,if you set concurrent build limit to 1 in CI/CD server.
Some AWS EC2 instances in my company need static hostname  The ip field will be changed automatically with the EC2 private IP return right after provisioning and and there is a playbook to create host record in infoblox. the group are Ansible group vars,multipe groups are separated by semicolon and the order is important,vars in last group will take precedence
The csv file

name,ip,group,zone,env
awselk1,,elasticsearch;elasticsearch-master,2a,prod
awselk2,,elasticsearch;elasticsearch-data,2a,prod

The script

#!/usr/bin/python
# Takes a file CSV file "xxx.csv" and outputs xxx.ini for Ansible host inventory data
import csv
import sys
import os
 
if len(sys.argv) <= 1:
   print "Usage:" +sys.argv[0]+" input-filename"
   sys.exit(1) 
net_dn = {'prod':'prod.example.com', 'preprod':'preprod.example.com',
          'test':'test.example.com', 'dev':'dev.example.com'}
groups = []
envs = set()
hosts_ini = {}

csvname = sys.argv[1]
scriptpath = os.path.dirname(sys.argv[0])

ansible_ini = os.path.join(scriptpath, 'hosts-aws-tmp.ini')

lines = []
hosts_text = ''
with open(csvname) as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        domain = net_dn[row['env'].strip()]
        line = row['name'].strip()+'.'+domain
        #lines.append(line)
        envs.add(row['env'])
        # support multiple groups separated by ;
        for g in row['group'].strip().split(';'):
          g = g.strip()
          if (not g in groups):
            groups.append(g)
          hosts_ini.setdefault(g, []).append(line)

#groups=set(groups)
if ( len(envs) !=1 ):
   print "ERROR: only single enviroment is supported!"
   sys.exit(1)
env = list(envs)[0]
env_text = "["+env+":children]"+"\n"+"\n".join(groups)   
vars_text = "\n\n["+env+":vars]"
vars_text += """
ansible_user=ansible
ansible_ssh_private_key_file=~/.ssh/id_rsa
ansible_become=true
ansible_become_user=root
ansible_become_method=sudo
ansible_gather_facts=no
"""
vars_text+="aws_env=aws-"+env+'\n'
#generate groups in order as input
for g in groups:
   hosts_text+='\n['+g+']\n'
   hosts_text+='\n'.join(hosts_ini[g])
   hosts_text+='\n'
 
all_text = env_text+vars_text+hosts_text
print all_text
with open(ansible_ini,'w') as new_ini_file:
    new_ini_file.write(all_text)   
print "INFO:Generated Ansible host inventory file: " + ansible_ini  

The Ansible inventory file generated

[prod:children]
elasticsearch
elasticsearch-master
elasticsearch-data

[prod:vars]
ansible_user=ansible
ansible_ssh_private_key_file=~/.ssh/id_rsa
ansible_become=true
ansible_become_user=root
ansible_become_method=sudo
ansible_gather_facts=no
aws_env=aws-prod

[elasticsearch]
awselk1.prod.example.com
awselk2.prod.example.com

[elasticsearch-master]
awselk1.prod.example.com

[elasticsearch-data]
awselk2.prod.example.com

Ansible - turn values in CSV file into list of dictonary

Ansible can load yaml values into Ansible variable easily, However I am still fan of plain CSV file, because it is easy to edit and has no strict format and repetitive of variable names.
How to convert csv into yaml? 
I used to use Python script to archive this,actually Ansible can do this natively.

Given this csv file for AWS EC2 instances

$ cat vars/aws/ec2data.csv
name,ip,zone,group,env
splunk01,10.1.1.1,2a,splunk,prod
splunk02,10.1.1.2,2b,splunk,prod

The playbook

---
- hosts: localhost
  connection: local
  gather_facts: no
  vars:
    ec2data_file: vars/aws/ec2data.csv

  tasks:
  - name:  reading {{ec2data_file}}
    command: /usr/bin/awk -F',' '!/^#/ && !/^$/ && NR!=1 { print $1,$2,$4,$5}' {{ec2data_file}}
    register: csvout
 #turn ec2_host into list with default filter and append list of dictionary in each loop. 
 #split is Python function to split string,default delimeter is space
  - name: turn csv output to list of dict
    set_fact:
      ec2_host: "{{ ec2_host|default([]) + [ { \
                      'name': item.split().0,  \
                      'ip':   item.split().1,  \
                      'group':item.split().2,  \
                      'env':  item.split().3 } ] }}"
    with_items: "{{csvout.stdout_lines}}"

  - debug: msg="{{item.name}},{{item.ip}}" verbosity=1
    with_items: "{{ ec2_host }}"

The result

skipping: [localhost] => (item={'ip': u'10.1.1.1', 'group': u'splunk', 'name': u'splunk01', 'env': u'prod'})
skipping: [localhost] => (item={'ip': u'10.1.1.2', 'group': u'splunk', 'name': u'splunk02', 'env': u'prod'})

Python script to run remote SSH commands with sudo permission

I created a Python script to run remote SSH command with sudo permission. Linux SSH command doesn’t support password as command option, you have to use expect script to connect to multiple servers for automation. plink tool in Windows support password as command option.
The trick to accept sudo password is ‘-S’ option in sudo, which accept sudo password piped from stdin.It seems to be safe, I turned on debug and I couldn’t see the password recorded in secure/messages logs.
There are two versions of the script: the command line one and the class/module one.

The command line version.

if the clear text password is an concern, you can wrap the script by getpasswd module in Python,which read password from stdin.Read password once and apply the password to multiple servers.

[root@~]# ./pyssh.py  -s server1 -u admin -p Passwd123 date
Thu Oct 30 15:36:27 EST 2014

#'service sshd status' command  ran successfully with sudo enabled '-t'
[root@~]# ./pyssh.py  -t -s server1 -u admin -p Passwd123  'service sshd status'
openssh-daemon (pid  15686) is running...

#!/usr/bin/env python
import sys
import paramiko
import argparse
import socket
parser = argparse.ArgumentParser()
parser.add_argument("-s", "--servername", help="hostname or IP", required=True)
parser.add_argument("-P", "--port", help="ssh port default=22", default=22)
parser.add_argument("-t", "--sudo", help="enable sudo,sudo password will use the value of --password",action='store_true')
parser.add_argument("-u","--username",help="username",required=True)
parser.add_argument("-p","--password",help="password",required=True)
parser.add_argument("cmd",help="command to run")
args=parser.parse_args()

host = args.servername
port = args.port
user = args.username 
password = args.password
cmd = args.cmd
if args.sudo:
    fullcmd="echo " + password + " |   sudo -S -p '' " + cmd
else:
    fullcmd=cmd

#if __name__ == "__main__":
client = paramiko.SSHClient()
#Don't use host key auto add policy for production servers
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.load_system_host_keys()
try: 
    client.connect(host,port,user,password)
    transport=client.get_transport()
except (socket.error,paramiko.AuthenticationException) as message:
    print "ERROR: SSH connection to "+host+" failed: " +str(message)
    sys.exit(1)
session=transport.open_session()
session.set_combine_stderr(True)
if args.sudo: 
    session.get_pty()
session.exec_command(fullcmd)
stdout = session.makefile('rb', -1)
print stdout.read()
transport.close()
client.close() 

The  class version

The class version allow multiple commands to run in an existing SSH transport,which is more efficient.To use the class,copy pyssh.sh to a folder and create a new script to import the class 'from pyssh import PySSH',then reference the code in MAIN section without if statement.

#!/usr/bin/env python
import sys
import socket
import paramiko
#=================================
# Class: PySSH
#=================================
class PySSH(object):
  
  
    def __init__ (self):
        self.ssh = None
        self.transport = None  

    def disconnect (self):
        if self.transport is not None:
           self.transport.close()
        if self.ssh is not None:
           self.ssh.close()

    def connect(self,hostname,username,password,port=22):
        self.hostname = hostname
        self.username = username
        self.password = password

        self.ssh = paramiko.SSHClient()
        #Don't use host key auto add policy for production servers
        self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.ssh.load_system_host_keys()
        try:
            self.ssh.connect(hostname,port,username,password)
            self.transport=self.ssh.get_transport()
        except (socket.error,paramiko.AuthenticationException) as message:
            print "ERROR: SSH connection to "+self.hostname+" failed: " +str(message)
            sys.exit(1)
        return  self.transport is not None

    def runcmd(self,cmd,sudoenabled=False):
        if sudoenabled:
            fullcmd="echo " + self.password + " |   sudo -S -p '' " + cmd
        else:
            fullcmd=cmd
        if self.transport is None:
            return "ERROR: connection was not established"
        session=self.transport.open_session()
        session.set_combine_stderr(True)
        #print "fullcmd ==== "+fullcmd
        if sudoenabled:
            session.get_pty()
        session.exec_command(fullcmd)
        stdout = session.makefile('rb', -1)
        #print stdout.read()
        output=stdout.read()
        session.close()
        return output

#===========================================
# MAIN
#===========================================        
if __name__ == '__main__':
    hostname = 'server1'
    username = 'admin'
    password = 'password123'
    ssh = PySSH()
    ssh.connect(hostname,username,password)
    output=ssh.runcmd('date')
    print output
    output=ssh.runcmd('service sshd status',True)
    print output
    ssh.disconnect()

Build Puppet module to use Hiera lookup

Puppet can use Hiera to look up data. This helps you disentangle site-specific data from Puppet code, for easier code re-use and easier management of data that needs to differ across your node population

One typical example is IP address and NTP/DNS servers, the IP address is unique for each server and NTP/DNS is global. I built a linux-network test module to demonstrate the usage of Hiera
Hiera supports yaml and Json as backend by default, however you can write your  custom backend using Hiera API.

Define datadir in hiera.yaml

[root@server1 modules]# cat /etc/puppetlabs/puppet/hiera.yaml 
---
:backends:
  - yaml

:hierarchy:
  - defaults
  - "%{clientcert}"
  - "%{environment}"
  - global

:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /var/lib/hiera on *nix
# - %CommonAppData%\PuppetLabs\hiera\var on Windows
# When specifying a datadir, make sure the directory exists.
  :datadir: /etc/puppetlabs/puppet/hieradata

Set all values in YAML file instead of manifest file
You can also add class name in YAML file, then assign class to node with hiera_include

[root@server1 modules]# cat /etc/puppetlabs/puppet/hieradata/global.yaml 
---
#
# ntp.conf
ntpservers: [10.1.1.11, 10.1.1.12]

#resolv.conf
domainname: example.com
searchdomain: [example1.com, example2.com]
nameservers: [10.1.1.13, 10.1.1.14]

[root@server1 modules]# cat /etc/puppetlabs/puppet/hieradata/server1.example.com.yaml 
eth1:
   device: eth1
   ipaddr: 172.16.1.2
   netmask: 255.255.255.0
   routes: ['192.168.1.0/24 via 172.16.1.254', '192.168.2.0/24 via 172.16.1.254']
   gateway: 172.16.1.254
eth3:
   device: eth3
   ipaddr: 172.16.1.3
   netmask: 255.255.255.0
   #routes: ['192.168.1.0/24 via 172.16.1.254', '192.168.2.0/24 via 172.16.1.254']

Execute the whole class or a function of the class in site.pp, the codes in site.pp become universal.
The site.pp manifest file is just generic code

[root@server1 modules]# cat /etc/puppetlabs/puppet/manifests/site.pp

node "server1" {
include "linux-network"
}

node "server2" {
linux-network::setinterface { 'eth1': }
}

linux-network module manifest files

[root@server1 modules]# cat ./linux-network/manifests/init.pp 
class linux-network {
 linux-network::setinterface { 'eth1': ; 'eth3': }
 linux-network::setroute { 'eth1': ; 'eth3':}

 linux-network::setconf_ntp {'ntp.conf':}
 linux-network::setconf_resolv {'resolv.conf':}
}

[root@server1 modules]# cat ./linux-network/manifests/setconf_ntp.pp 
define linux-network::setconf_ntp  ( ) {

$ntpservers=hiera_array('ntpservers')

file {"/etc/ntp.conf":
 ensure => present,
 owner => root,
 mode => 644,
 content => template("${module_name}/ntp.conf.erb")
 }
}

[root@server1 modules]# cat ./linux-network/manifests/setconf_resolv.pp 
define linux-network::setconf_resolv  ( ) {

$domainname=hiera('domainname')
$searchdomain=hiera_array('searchdomain')
$nameservers=hiera_array('nameservers')

file {"/etc/resolv.conf":
ensure => present,
owner => root,
mode => 644,
content => template("${module_name}/resolv.conf.erb")
 }
}

[root@server1 modules]# cat ./linux-network/manifests/setinterface.pp 
define linux-network::setinterface  ( ) {

$device=$title
$eth=hiera($device)
$ipaddr=$eth['ipaddr']
$netmask=$eth['netmask']
$gateway=$eth['gateway']

file {"/etc/sysconfig/network-scripts/ifcfg-$device":
 ensure => present,
 owner => root,
 mode => 644,
 content => template("${module_name}/ifcfg.erb")
 }

}

[root@server1 modules]# cat ./linux-network/manifests/setroute.pp 
define linux-network::setroute  ( ) {

$device=$title
$eth=hiera($device)
$routes=$eth['routes']

file {"/etc/sysconfig/network-scripts/route-$device":
ensure => present,
owner => root,
mode => 644,
content => template("${module_name}/route.erb")
 }
}

linux-network module template files

[root@server1 modules]# cat ./linux-network/templates/ifcfg.erb 
DEVICE=<%=@device %>
BOOTPROTO=static
ONBOOT=yes
USERCTL=no
IPADDR=<%=@ipaddr%>
NETMASK=<%=@netmask%>
<%- if @gateway =~ /(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/   -%>
GATEWAY=<%=@gateway %>
<%- end -%>

[root@server1 modules]# cat ./linux-network/templates/ntp.conf.erb 
 tinker panic 0
 restrict default kod nomodify notrap nopeer noquery
 restrict -6 default kod nomodify notrap nopeer noquery
 restrict 127.0.0.1 
 restrict -6 ::1
#
 <%- @ntpservers.each do |x| -%>
 server <%= x %>
<%- end  -%>
 driftfile /var/lib/ntp/drift

 [root@server1 modules]# cat ./linux-network/templates/resolv.conf.erb
#
# resolver configuration file...
#
options         timeout:1 attempts:8 rotate
domain       <%=@domainname %>
<%-  if !@searchdomain.empty?   -%>
search <%=@domainname  -%> <%=  @searchdomain.join(' ') %>
<%- end -%>
<%-  @nameservers.each do |  x | -%>
nameserver <%= x %>
<%- end -%>

[root@server1 modules]# cat ./linux-network/templates/route.erb 
<%- if defined?(@routes)   -%>
<%- @routes.each do | x | -%>
<%=x %>
<%- end -%>
<%- end -%>

Setup SAN Boot for RHEL 6.x using native multipath on EMC storage

Requirements:
1) RHEL 6.x (most apply to RHEL 5.x too, RHEL 5.x use mkinitrd instead of Dracut and the /etc/multipath.conf is slightly different refer to Red Hat KB in reference section)
2) EMC storage was setup with Active/Active (ALUA)
3) Boot LUN was presented with single path for initial install


Procedures:

1. Server boots up after initial install
2. Login to server as root to enable multipath
[root@server1]#mpathconf --enable –-with_multipathd y
3. Edit /etc/multipath.conf and make sure it only contains following valid parameters

blacklist {
}


defaults {
 user_friendly_names yes
}
devices {
  device {
    vendor "DGC"
    product ".*"
    product_blacklist "LUNZ"
    hardware_handler "1 alua"   
    path_checker directio    
    prio alua                
  }
}

4.Find out the logical path the root disk is mapped
[root@server1]#multipath –v3
It should be /dev/mapper/mpatha

5. Create initramfs with multipath module
[root@server1]#dracut --force -–add multipath

6. Make sure multipath.conf is included in initrd image

[root@server1]#lsinitrd /boot/initramfs-*.x86_64.img | grep multipath.conf
-rw-r--r--   1 root     root         2525 Feb 27 13:31 etc/multipath.conf

7. Modify the /boot/grub/device.map and change

 (hd0) /dev/sda    to
 (hd0) /dev/mapper/mpatha

This is assuming the boot disk is on /dev/mapper/mpatha as verified in step 2 above.

8. Reboot the server.

9. Verify multipath, check hwhandler='1 alua' and member disk sda for mpatha

[root@server1]#multipath –ll 
mpatha (3600601609973310067eb1e1ed69ae311) dm-0 DGC,VRAID
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:0 sda 8:0   active ready running
  

10. Ask storage administrator to enable other paths for boot LUN.
11. Reboot server again after multipath is aenabled in storage too
12. Login server to verify all paths, check hwhandler='1 alua' prio>0
if hwhandler='1 emc' or prio=0 means PNR mode

[root@server1]#multipath -ll
mpatha (3600601609973310067eb1e1ed69ae311) dm-0 DGC,VRAID
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 1:0:1:0 sdd 8:48  active ready running
| `- 2:0:1:0 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:0:0 sda 8:0   active ready running
  `- 2:0:0:0 sdg 8:96  active ready running
mpathb (360060160997331009fd6e124d69ae311) dm-1 DGC,VRAID
size=800G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| |- 1:0:0:1 sdb 8:16  active ready running
| `- 2:0:0:1 sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 1:0:1:1 sde 8:64  active ready running
  `- 2:0:1:1 sdk 8:160 active ready running

13. Partition other LUNS using fdisk command as normal, but use logical path /dev/mapper/mpathb etc (partition will be created as /dev/mapper/mpathbp1 instead of /dev/mapper/mpathb1
NOTE: any change to /etc/multipath.conf requires re-create initramfs ( dracut --force -–add multipath) and a reboot, because the boot LUN is on SAN, if boot LUN is local disk, change to /etc/multipath.conf only requires multipathd restart

Automate Server deployment with Ansible

There are many server automation applications in the market: puppet,chef,cfengine and salt. Ansible is relatively new, but I think it is better than puppet in server deployment automation tasks.
1. Dependency packages
Ansible depends on python, which is installed by default at least for Red Hat alike distributions
Puppet depends on ruby, which is not installed by default.
2. Agent
Ansible is agentless, it rely on SSH
Puppet need agent running in target server as a daemon.
3. Security
Ansible use SSH as transport method, so Username and password are required for each connection.(Ansible is smart enough to cache the SSH and sudo password, so it will be only prompted once for the first server)
Puppet: agent is controlled by master server, if master server is compromised, all hosts can be brought down easily
4. Setup
Ansible is easy to setup, as there is no agent. Ansible server is easy to setup too, there are just python scripts. You can even run it without installing it.
Puppet need packages installed in agent host or server, the agent certificate need to be signed before server can talk agent.
Ansible use SSH TCP port 22, which is standard firewall port already opened in most infrastructure.
Puppet use customized TCP port , typically 8139
5. Command line mode
Ansible supports command line mode for ad-hoc tasks, so you don’t need to write tasks definitions, just pass the command to ansbile such as return date for a number servers.
ansbile myservers –k –K –u admin –m raw –a “date”

The following example show a typical server deployment

[root@centos1 post]# cat setup.yml 
---        #ansible playbook use YAML syntax http://en.wikipedia.org/wiki/YAML
- hosts: server1          #It is a server or server group as defined in /etc/ansbile/hosts
  user: admin 
  sudo: yes
  vars_files:
    - vars/settings.yml   #global variables
    - vars/{{ ansible_hostname }}.yml             #server specific variable . ansible_hostname is variable, it is server1.yml for server1
  tasks:

  - name: yum
    action: yum name=${item}  state=present      #install yum packages
    with_items:
      - kernel-devel-{{ ansible_kernel }}
      - ed
      - ksh
      - ntp
  - script: ./scripts/sshd.sh        #- The script will insert 'UseDNS no' , - script is shorthand  for - name: XX ,action: YY

  - name: users | Delete users       #delete users delusers is list if users defined in setting.yml
    action: user name=$item state=absent
    with_items: delusers

  - name: ifcfg-eth0 | Configuration file      #ansible template engine is Jinja2 http://jinja.pocoo.org/docs/
    action: template src=./templates/ifcfg-eth0.j2 dest=/etc/sysconfig/network-scripts/ifcfg-eth0 owner=root group=root

  - name: route-eth0 | Configuration file, /etc/sysconfig/network-scripts/route-eth0
    action: template src=templates/route-eth0.j2 dest=/etc/sysconfig/network-scripts/route-eth0

  - name: resolv.conf | Configuration file, /etc/resolv.conf
    action: template src=templates/resolv.conf.j2 dest=/etc/resolv.conf

  - name: ntpd | Configuration file, /etc/ntp.conf
    action: template src=templates/ntp.conf.j2 dest=/etc/ntp.conf
    notify:
    - restart ntpd

  - name: snmpd | Configuration file, /etc/snmp/snmpd.conf
    action: copy src=files/snmpd.conf dest=/etc/snmp/snmpd.conf owner=root group=root mode=0644
    notify:
    - restart snmpd


  - copy: src=files/clock dest=/etc/sysconfig/clock owner=root group=root mode=0644
  - command: ln -fs /usr/share/zoneinfo/Australia/Sydney /etc/localtime


  handlers:
  - name: restart sshd
    action: service name=sshd enabled=yes state=restarted
  - name: restart ntpd
    action: service name=ntpd enabled=yes state=restarted
  - name: restart snmpd
    action: service name=snmpd enabled=yes state=restarted

####----Global variables 
[root@centos1 post]# cat vars/settings.yml 
#
# ntp.conf
ntpservers: [10.1.1.1, 10.1.1.2]

#users to delete
delusers: [user1, user2]

#resolv.conf
domainname: .example.com
searchdomain: [example.com]
nameservers: [10.1.1.1, 10.1.1.2]

####----Server specific  variable
[root@centos1 post]# cat vars/server1.yml 
eth1: 
   device: eth1
   ipaddr: 172.16.1.2
   netmask: 255.255.255.0
   routes: ['192.168.1.0/24 via 172.16.1.254', '192.168.2.0/24 via 172.16.1.254']
 
####----How the tempalate reference the variable
[root@centos1 post]# cat templates/resolv.conf.j2 
#
# resolver configuration file...
#
options         timeout:1 attempts:8 rotate
domain          {{domainname}}
search          {{domainname}} {{ searchdomain | join (' ') }}

{% for host in nameservers %}
nameserver {{host}}
{% endfor %}

[root@centos1 post]# cat templates/ifcfg-eth1.j2 
DEVICE={{eth1.device}}
BOOTPROTO=static
ONBOOT=yes
USERCTL=no
IPADDR={{eth1.ipaddr}}
NETMASK={{eth1.netmask}}
{% if eth1.gateway is defined  %} 
GATEWAY={{eth1.gateway}}
{%endif%}



####----a separate playbook to create LVM and file system 
[root@centos1 post]# cat setup-lvm.yml 
---
- hosts: server1
  user: admin
  sudo: yes
  gather_facts: no
  vars:
    mntp:  /opt
    vgname: vg01
    pvname: /dev/sdb1
    lv1: opt
 
  tasks:

  - script: ./scripts/disks.sh $pvname       #a script to create LVM partion and create physical volume
  - name: filesystem | Create pv,vg,lv and file systems
    action: lvg  vg=$vgname pvs=$pvname

  #- name: filesystem | create lv
  - lvol: vg=$vgname lv=$lv1 size=51196

 # - name: filesystem | create fs
  - filesystem: fstype=ext4 dev=/dev/${vgname}/${lv1}

  #- name: filesytem | mount dir
  - mount: name=${mntp} src=/dev/${vgname}/${lv1} dump=1 passno=2 fstype=ext4 state=mounted

How to run the playbook?

[root@centos1 post]# ansible-playbook -k -K setup.yml

  -k, --ask-pass        ask for SSH password
  -K, --ask-sudo-pass   ask for sudo password

Download all the files
https://drive.google.com/file/d/0B-RHmV4ubtk8Y2wyazhZRS1pSVk/edit?usp=sharing

VMware PowerCLI: Map datastore name to LUN devicename.

It is not obvious as it is thought to be to map datastore name to LUN devicename in native PowerCLI codes,
The  esxcli interface exposed to PowerCLI  make it very easy.(Tested in ESXi 5.0)

PowerCLI>$esxcli=get-esxcli -vmhost esx01
PowerCLI>$esxcli.storage.vmfs.extent.list() | ft devicename,volumename -autosize

DeviceName                           VolumeName
----------                           ----------
naa.600601605bc02e00007fb97cacbee211 datastore01
naa.600601605bc02e00fac8cd88acbee211 datastore02

Script to automatically partition a new disk and create LVM PV

It is a very common task to create a single partition on  whole disk  and create LVM PV, How to automate it?

fdisk doesn't support making partition in script mode, sfdisk can, but it is not as good as the powerful parted tool. parted can also optimize partition alignment automatically(parted -a optimal).

#!/bin/ksh
#Create a single primary partiton with whole disk size and create LVM PV on it
disk=$1
partno=1
if [[ -z $disk ]]; then
 echo "Usage: $0 disk device name: e.g $0 /dev/sdb"
 exit
fi


if [[ -e ${disk}${partno} ]]; then
 echo "==> ${disk}${partno} already exist"
 exit
fi

echo "==> Create MBR label"
parted -s $disk  mklabel msdos
ncyl=$(parted $disk unit cyl print  | sed -n 's/.*: \([0-9]*\)cyl/\1/p')

if [[ $ncyl != [0-9]* ]]; then
 echo "disk $disk has invalid cylinders number: $ncyl"
 exit
fi

echo "==> create primary parition  $partno with $ncyl cylinders"
parted -a optimal $disk mkpart primary 0cyl ${ncyl}cyl
echo "==> set partition $partno to type: lvm "
parted $disk set $partno lvm on
partprobe > /dev/null 2>&1
echo "==> create PV ${disk}${partno} "
pvcreate ${disk}${partno}

Understanding SysV-style Initscripts in Red Hat Linux

It is often needed to write your own init start/stop script, the following is the minimum requirement for your script to behave as expected. The discussion is based on Red Hat Linux family, other distributions like Debian use LSB (Linux Standard Base Specification) Init Scripts.

Location of the script

/etc/init.d is the well known location, but actually /etc/rc.d/init.d is the real original location. Since /etc/init.d is a hard link to /etc/rc.d/init.d, it makes no difference.

Header of the script

It needs at least 3 lines.  The shell script interpreter (/bin/sh, /bin/bash .. etc), the chkconfig header and script description

#!/bin/sh
#   chkconfig: 345 56 10
#   description: Startup/shutdown script for the Common UNIX 

Body of the script

Obviously, it need  to accept parameter “start”, which  /etc/rc3.d/S* will call  on OS startup and accept parameter “stop”, which /etc/rc0.d/K* script will call on OS shutdown.
The lockfile is often overlooked, it is used to check the existence of the daemon on OS shutdown, otherwise the stop action won’t be called. If you found an issue that a script started on OS startup but never stop properly on shutdown, you need to create lockfile. note: lockfile is not pidfile which contains PID of the process, lockfile is usually a blank file.

lockfile=/var/lock/subsys/$(basename $0)
case $1 in
 start)
  start
  [ $? = 0 ] && touch $lockfile
 ;;
 stop)
  stop
  [ $? = 0 ] && rf –f  $lockfile

Others

It is recommended to import functions in /etc/rc.d/init.d/functions to use ‘daemon’ to startup your application or killproc to shutdown your application instead of reinventing the wheel.

LSB headers

You may see something like this in an init script.

 # Provides: boot_facility_1 [ boot_facility_2 ...]
 # Required-Start: boot_facility_1 [ boot_facility_2 ...]
 # Required-Stop: boot_facility_1 [ boot_facility_2 ...]
 # Should-Start: boot_facility_1 [ boot_facility_2 ...]
 # Should-Stop: boot_facility_1 [ boot_facility_2 ...]
 # Default-Start: run_level_1 [ run_level_2 ...]
 # Default-Stop: run_level_1 [ run_level_2 ...]
 # Short-Description: short_description
 # Description: multiline_description

They are LSB(Linux Standard Base) headers, they are supported by default in Debian and SUSE Linux.
Red Hat Linux supports this by additional package “redhat-lsb” and it is not installed by default, Be warned,50+ dependences need to installed as well.

Reference

http://refspecs.linuxfoundation.org/LSB_2.1.0/LSB-generic/LSB-generic/initscrcomconv.html

https://fedoraproject.org/wiki/Packaging:SysVInitScript?rd=Packaging/SysVInitScript

Configure multipath on Solaris 11 for IBM V7000 SAN storage

IBM V7000 is not listed from command “mpathadm show mpath-support libmpscsi_vhci.so”, but it is still supported, Solaris 11 mpxio supports any third party storage device that is T10/T11 standards-compliant.
Procedures to setup multipath

#Setup zoning in SAN switch
#Login to V7000 management UI to map Solaris host to the volume, select host type ‘TPGS’
#Rescan new SAN disks without rebooting
$cfgadm -o force_update -c configure cX (X is the port id as shown cfgadm -al)
#verify SAN disks are detected.
$echo|format
#create scsi_vhci.conf
#scsi_vhci.conf doesn’t need to customized, scsi-vhci-failover-override parameter is optional, IBM V7000 is detected as f_tpgs with the standard probe.
$cp /kernel/drv/scsi_vhci.conf /etc/driver/drv/scsi_vhci.conf

#Obtain the device path of the fc ports of a single HBA
$ls -l /dev/cfg
lrwxrwxrwx   1 root     root          60 Feb 28 10:20 c4 -> ../../devices/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0/fp@0,0:fc
lrwxrwxrwx   1 root     root          62 Feb 28 10:20 c5 -> ../../devices/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1/fp@0,0:fc

the path needed is the string between ./devices/ and fp@, so the paths are  
/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0
/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1

#Create fp.conf
cp /kernel/drv/fp.conf /etc/driver/drv/fp.conf
#edit fp.conf to enable multipath for the two fc ports only
mpxio-disable="yes";
name="fp" parent="/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0" port=0 mpxio-disable="no";
name="fp" parent="/pci@400/pci@2/pci@0/pci@8/SUNW,qlc@0,1" port=0 mpxio-disable="no";
#run the command to enable multiple path on fc ports only, server will need to be rebooted.
$stmsboot –u –D fp

#verify 
$ echo| format will show only a single disk
$stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c4t50050768024046D8d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
/dev/rdsk/c4t50050768022046D9d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
/dev/rdsk/c5t50050768022046D8d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
/dev/rdsk/c5t50050768024046D9d0 /dev/rdsk/c0t6005076802830163A000000000000005d0
$mpathadm list lu
        /dev/rdsk/c0t5000CCA0164251B8d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t5000CCA01642CDDCd0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t6005076802830163A000000000000005d0s2
                Total Path Count: 4
                Operational Path Count: 4
$mpathadm show lu /dev/rdsk/c0t6005076802830163A000000000000005d0s2

#the disk is deteced as f_tpgs as shown in messages log
$grep f_ /var/adm/messages
Mar 21 13:56:46 dnmsovm1 scsi: [ID 583861 kern.info] ssd4 at scsi_vhci0: unit-address g6005076802830163a000000000000005: f_tpgs

Monitor customized application in Windows by SNMP

The native SNMP service  in Windows can provide basic metrics like CPU, memory and disk etc, but it doesn’t have “extend” feature in net-snmp, which allows you run a script for application monitoring. Net-snmp can’t be used as replacement for Windows SNMP service because some SNMP extension agent relies on it and known issue like HOST-RESOURCES MIB doesn’t work in net-snmp. 

 The good news is that you can have net-snmp co-exist with Windows SNMP, you can have nice features like extend ability, in the mean time, pass the other functions to native Windows SNMP service.

As of Net-SNMP 5.4, the Net-SNMP agent is able to load the Windows SNMP service extension DLLs by using the Net-SNMP winExtDLL extension. The extension requires the net-snmp binary to be native (32bit net-snmp extension won’t work in 64bit Windows).

Net-snmp 64bit binary is hard to find, it seems only net-snmp-5.5.0-2 has 64bit binary pre-compiled, you might need to compile yourself for other versions. 

Install net-snmp

Run the net-snmp binary installer select “with Windows Extenstion” instead of standard agent, unselect “net-snmp trap service” and “Perl SNMP modules”, the default path is c:\usr

Configure net-snmp

Register net-snmp as Windows service

Edit c:\usr\registeragent.bat to disable modules conflicting to Windows   by adding parameter.

“-I-udp,udpTable,tcp,tcpTable,icmp,ip,interfaces,snmp_mib”

(Note: if system_mib is also disabled, SNMPv2-MIB::sysuptime won’t report correct time)

Run c:\usr\registeragent.bat

Edit C:\usr\etc\snmp\snmpd.conf

rocommunity public 192.168.1.10
#Test extend feature to execute a script, the script path must use Unix style ‘/’
extend userscript c:/temp/test1.bat

Start Windows service “net-snmp agent”(Native SNMP service must be stopped)

Test

#Test standard SNMP metrics, the HOST-RESOURCES-MIB is provided by native SNMP service, not net-snmp
[root@zabbix]#/usr/bin/snmpwalk -v 2c  -c public 192.168.1.20   HOST-RESOURCES-MIB::hrSystemUptime
HOST-RESOURCES-MIB::hrSystemUptime.0 = Timeticks: (640892116) 74 days, 4:15:21.16

#The extend feature is provided by net-snmp, Execute the script by snmpwalk
[root@zabbix]#/usr/bin/snmpwalk -v 2c -Ov -c public 192.168.1.20 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."userscript"'
STRING: web-time=80
STRING: web-status=[ok]

Troubleshooting

C:\usr\log\snmpd.log

Check which Windows modules loaded, start snmpd in command line with debugging “WinExtDLL”

Snmpd.exe -I-udp,udpTable,tcp,tcpTable,icmp,ip,interfaces,snmp_mib  -DwinExtDLL 

Reference:

http://net-snmp.sourceforge.net/docs/README.win32.html