Python script to generate Ansible ini inventory file from csv file

Ansible in memory inventory file created by add_host is often used in AWS EC2 provisioning. Inventory file can be generated easily,however it has drawback. Because it is in memory, all server post build tasks have to be in one big playbook. Which means it is not easy to re-run failed tasks if there is failure and existing post build playbooks can't be reused.

I created a Python script to generate a temporary inventory file from csv file used in EC2 provisioning. The inventory file can be used in multiple post build playbooks. The file name is static, however it will not be overwritten,if you set concurrent build limit to 1 in CI/CD server.
Some AWS EC2 instances in my company need static hostname  The ip field will be changed automatically with the EC2 private IP return right after provisioning and and there is a playbook to create host record in infoblox. the group are Ansible group vars,multipe groups are separated by semicolon and the order is important,vars in last group will take precedence
The csv file

name,ip,group,zone,env
awselk1,,elasticsearch;elasticsearch-master,2a,prod
awselk2,,elasticsearch;elasticsearch-data,2a,prod

The script

#!/usr/bin/python
# Takes a file CSV file "xxx.csv" and outputs xxx.ini for Ansible host inventory data
import csv
import sys
import os
 
if len(sys.argv) <= 1:
   print "Usage:" +sys.argv[0]+" input-filename"
   sys.exit(1) 
net_dn = {'prod':'prod.example.com', 'preprod':'preprod.example.com',
          'test':'test.example.com', 'dev':'dev.example.com'}
groups = []
envs = set()
hosts_ini = {}

csvname = sys.argv[1]
scriptpath = os.path.dirname(sys.argv[0])

ansible_ini = os.path.join(scriptpath, 'hosts-aws-tmp.ini')

lines = []
hosts_text = ''
with open(csvname) as csvfile:
    reader = csv.DictReader(csvfile)
    for row in reader:
        domain = net_dn[row['env'].strip()]
        line = row['name'].strip()+'.'+domain
        #lines.append(line)
        envs.add(row['env'])
        # support multiple groups separated by ;
        for g in row['group'].strip().split(';'):
          g = g.strip()
          if (not g in groups):
            groups.append(g)
          hosts_ini.setdefault(g, []).append(line)

#groups=set(groups)
if ( len(envs) !=1 ):
   print "ERROR: only single enviroment is supported!"
   sys.exit(1)
env = list(envs)[0]
env_text = "["+env+":children]"+"\n"+"\n".join(groups)   
vars_text = "\n\n["+env+":vars]"
vars_text += """
ansible_user=ansible
ansible_ssh_private_key_file=~/.ssh/id_rsa
ansible_become=true
ansible_become_user=root
ansible_become_method=sudo
ansible_gather_facts=no
"""
vars_text+="aws_env=aws-"+env+'\n'
#generate groups in order as input
for g in groups:
   hosts_text+='\n['+g+']\n'
   hosts_text+='\n'.join(hosts_ini[g])
   hosts_text+='\n'
 
all_text = env_text+vars_text+hosts_text
print all_text
with open(ansible_ini,'w') as new_ini_file:
    new_ini_file.write(all_text)   
print "INFO:Generated Ansible host inventory file: " + ansible_ini  

The Ansible inventory file generated

[prod:children]
elasticsearch
elasticsearch-master
elasticsearch-data

[prod:vars]
ansible_user=ansible
ansible_ssh_private_key_file=~/.ssh/id_rsa
ansible_become=true
ansible_become_user=root
ansible_become_method=sudo
ansible_gather_facts=no
aws_env=aws-prod

[elasticsearch]
awselk1.prod.example.com
awselk2.prod.example.com

[elasticsearch-master]
awselk1.prod.example.com

[elasticsearch-data]
awselk2.prod.example.com

Ansible - turn values in CSV file into list of dictonary

Ansible can load yaml values into Ansible variable easily, However I am still fan of plain CSV file, because it is easy to edit and has no strict format and repetitive of variable names.
How to convert csv into yaml? 
I used to use Python script to archive this,actually Ansible can do this natively.

Given this csv file for AWS EC2 instances

$ cat vars/aws/ec2data.csv
name,ip,zone,group,env
splunk01,10.1.1.1,2a,splunk,prod
splunk02,10.1.1.2,2b,splunk,prod

The playbook

---
- hosts: localhost
  connection: local
  gather_facts: no
  vars:
    ec2data_file: vars/aws/ec2data.csv

  tasks:
  - name:  reading {{ec2data_file}}
    command: /usr/bin/awk -F',' '!/^#/ && !/^$/ && NR!=1 { print $1,$2,$4,$5}' {{ec2data_file}}
    register: csvout
 #turn ec2_host into list with default filter and append list of dictionary in each loop. 
 #split is Python function to split string,default delimeter is space
  - name: turn csv output to list of dict
    set_fact:
      ec2_host: "{{ ec2_host|default([]) + [ { \
                      'name': item.split().0,  \
                      'ip':   item.split().1,  \
                      'group':item.split().2,  \
                      'env':  item.split().3 } ] }}"
    with_items: "{{csvout.stdout_lines}}"

  - debug: msg="{{item.name}},{{item.ip}}" verbosity=1
    with_items: "{{ ec2_host }}"

The result

skipping: [localhost] => (item={'ip': u'10.1.1.1', 'group': u'splunk', 'name': u'splunk01', 'env': u'prod'})
skipping: [localhost] => (item={'ip': u'10.1.1.2', 'group': u'splunk', 'name': u'splunk02', 'env': u'prod'})