Fix curl client certificate error: curl: (58) unable to set private key file

I used curl to post xml payload to a web service by client certificate authentication.
It worked fine Linux(curl 7.15.5 (x86_64-redhat-linux-gnu))
But the same command with the same certificate failed in Windows(curl 7.23.1 (x86_64-pc-win32)).

$curl -k -s -o /dev/null  --connect-timeout 5 -w "\ntime_total=%{time_total}\nhttp_code=%{http_code}\n" --cert cert2.pem: Pass123 --key cert2.pem    -d @data.xml https://server1/
curl: (58) unable to set private key file: 'cert2.pem' type PEM

This error could be due to invalid certificate password or invalid certificate format,  It seems the curl in windows is picker on certificate format than the curl in Linux.
In general, you can combine both certificate and key into one file by copy&paste the text in a text editor. But it seems the curl for Windows only accept the “formal format”, which has additional lines of “Bag Attributes” ( or maybe just different behaviour in different versions)
General  certificate

#cat cert1.pem
-----BEGIN CERTIFICATE-----
………………..
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,D49E12E8D074DA59
…………….
-----END RSA PRIVATE KEY-----

”Formal” certificate with “Bag Attributes”

Bag Attributes
localKeyID: 92 47 FA AC C8 D4 94 D8 0C B0 D0 1D 9F D7 43 0B B5 D8 23 AD
subject=/C=AU/ST=NSW/L=Sydney/O=ORG1/CN=client1
issuer=/C=AU/ST=NSW/O=ORG1/CN=CA
-----BEGIN CERTIFICATE-----
………………….
-----END CERTIFICATE-----
Bag Attributes
localKeyID: 92 47 FA AC C8 D4 94 D8 0C B0 D0 1D 9F D7 43 0B B5 D8 23 AD
Key Attributes: <No Attributes>
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,6F6BA23BB03B5049
……………..
-----END RSA PRIVATE KEY-----

How to add the “Bag Attributes”?

#I found the “bag Attributes” can be added by converting PEM to pkcs12 format then convert back to PEM.
#Convert  from PEM to PKCS12
$openssl pkcs12 -export -in cert1.pem  -out cert1.pfx
#Convert back from  PKCS12 to PEM
$ openssl pkcs12 -in cert1.pfx  -out cert2.pem -clcerts
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
#It worked with the new certificate
C:\temp\curl\curl -k -s -o /dev/null  --connect-timeout 5 -w "\ntime_total=%{time_total}\nhttp_code=%{http_code}\n" --cert cert2.pem:Pass123  --key cert2.pem  -d @data.xml https://server1/
time_total=2.812
http_code=200

Script automation with SSH in Windows

I am testing a monitoring software, like the most monitoring software, it  relies on SSH to call remote customized scripts. But it is running on Windows. To get SSH command line working nicely in Windows is not quite easy.
There are two options to use SSH to do script automation in Windows.
1.Use SSH command line
2.Use SSH API for a scripting language i.e. Perl
 Use SSH command line
PuTTY is a well-known SSH GUI client,the non-interactive, command line interface is plink
The other option of SSH command line is ssh in cygwin, this ssh works exactly the same way as ssh in Linux.
SSH in cygwin support more features than plink, but it requires installing the whole cygwin environment, while plink is just a single binary file.
However plink has a draw back, it seems there is no way to suppress the “accept remote host key prompt” and its host key cache is stored in Windows registry.
SSH in cygwin can suppress the prompt by setting “-o stricthostkeychecking=no” and the host key cache is stored in .ssh in user's home folder. If the monitoring software is running as a service under 'local system account', where is the .ssh located? The answer for Windows 2008 R2 is “c:\Windows\SysWOW64\config\systemprofile”, you have to create “.ssh”  folder manually, otherwise the service can't save the ssh host key cache.
 Use SSH API for a scripting language i.e. Perl
The above method of using SSH command line works, but is not as flexible as using SSH API.
The SSH API is from libssh2, Perl has a module,NET::SSH2,to utilize libssh2 to provide the same function of ssh in cygwin. The benefits of NET::SSH2  are: better error handling and no ssh host key prompt issue.
Install Perl for Windows
Activeperl is probably the most popular choice in Windows, but it has limited number of pre-built Perl Modules, it is very hard to compile from source for new modules. Activeperl itself doesn't have compiler or make utility, you can get MinGW compiler or dmake, but the chance of success build is very slim.
Strawberryperl,http://strawberryperl.com/  is an opensource project for Perl in Windows, it has built-in compiler and related tools. It can download and install modules in www.cpan.org
As of Strawberry Perl 5.12, net::ssh2 is built-in, the following is simple example of of net:ssh2 usage.

##Note: if you encounter an error "libssh2-1_.dll was not found", you need to add the path of the dll, "C:\strawberry\c\bin", into %PATH% env variable
use  warnings;
use strict;
use Net::SSH2;
use constant BUFLEN => 10_0000 ;
my $ssh2 = Net::SSH2->new();
$ssh2->connect('server1.example.com',22) or die "Unable to connect Host $@ \n";
if ($ssh2->auth_password('user1','Pass1')) {
#if ( $ssh2->auth_publickey ( 'user1', 'id_rsa.pub', 'id_rsa')) { 
my $chan2 = $ssh2->channel();
#$chan2-&gtblocking(0);
$chan2->exec ("uname -a\n");
$chan2->read($buf, BUFLEN );
print $buf;
$chan2->close;
} else {
warn "auth failed.\n";
}

Control ISC DHCP to allocate IP address based on vendor ID

ISC DHCP supports conditional evaluation(man dhcp-eval), one of the evaluation expressions is hardware, the MAC  address of the network card.
Conditional evaluation make it possible to allocate different IP pool for clients or even allow multiple DHCPD daemons running in the same broadcasts domain.
My challenge is to setup a new DHCP server to PXE boot VMware Vms without affecting an existing DHCP server.
The solution is to create a new class which only response DHCP request  from Vmware NICs, the key is to use expression binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) to get the vendor ID.

$cat /etc/dhcpd.conf
ddns-update-style none;
ignore client-updates;
#log-facility local7;
#log (debug, binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)));
class "vmware-nics"
{
match if  ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:05:69") or ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:0c:29") or ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:1c:14") or ( binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)) = "0:50:56");
}
subnet 192.168.100.0 netmask 255.255.255.0 {
pool {
allow members of "vmware-nics";
option routers                  192.168.100.254;
option subnet-mask              255.255.255.0;
option domain-name              "example.com";
option domain-name-servers      192.168.100.1;
range dynamic-bootp 192.168.100.1 192.168.100.200;
#time unit is 1 sec
default-lease-time 3000;
max-lease-time 6000;
next-server 192.168.100.1;
filename "gpxelinux.0";
}
}

NOTE: the double 0  in MAC address  will be translated to single 0. e.g "00:05:69" = "0:05:69"
Troubleshooting:
If the expression doesn't work, you can check the expression by logging it to a file.You might need to disable "allow members" restriction in order for the expression to be logged.

log-facility local7;
log (debug, binary-to-ascii (16, 8, ":", substring (hardware, 1, 3)));

By default, DHCP log is directed to /var/log/message, it seems, in order to log expression, the DHCP must use separate log file. Because “local7.* /var/log/boot.log” is configured in /etc/syslogd.conf  by default, “log-facility local7” in dhcpd.conf will direct messages to /var/log/boot.log

Enable timestamp logging in bash commands history

Bash has the built-in feature to support timestamp in history command , but it is not enabled by default in Red Hat Linux(ksh doesn’t have the feature at all)
It is recommended to tune two environment variables for history command to tighten audit.
1.Enable timestamp in command history
2.Increase the default  history file size
Enable timestamp in command history
Just need to set HISTTIMEFORMAT env variable, e.g.  HISTTIMEFORMAT="[%F %T %Z] "
The time format is the same as that of date command.
Increase the default history file size
There are two environment variables  for history size.
HISTSIZE: How many lines to keep in memory for current bash session, commands in current session are appended to ~/.bash_history on quitting.
HISTFILESIZE: How many lines to keep in history file ~/.bash_history
Both HISTSIZE and HISTFILESIZE seem to be 1000 by default, HISTSIZE=1000 is ok for current session, but HISTFILESIZE=1000 is too small to log enough commands for any past login session.
Both environment variables  are to be enabled for all users,  they can be put in global profile /etc/profile , but user defined global profile in /etc/profile.d/ is a better option.
For example, create a user global profile in /etc/profile.d

$cat /etc/profile.d/user.sh 
HISTSIZE=1000
HISTFILESIZE=40000
HISTTIMEFORMAT="[%F %T %Z] "
export HISTSIZE HISTFILESIZE HISTTIMEFORMAT
#With timestamp enabled, the history command output looks like this
$history 
19  [2012-02-03 13:41:56 EST] history
20  [2012-02-03 13:42:58 EST] date
22  [2012-02-03 13:42:55 EST] pwd

You may notice that the commands was not sorted by the time order, it is because, even “pwd” was executed  before date, but  it is was in different session and the “pwd session” was terminated after the “date session”, so the “pwd” command was appended after “date”.

To view all 40000 lines in ~/.bash_history for tracing very old commands, you need to temporarily increase HISTSIZE to 40000 before run “history”.  You can set the default HISTSIZE is 4000, but it takes too long to process for history command used in daily basis.

Other useful history related commands

#clear commands history in current session 
$history -c
#disable logging history commands
$set +o history 
#enable logging history commands
$set –o history 
#fc (fix command) 
#open last 4 commands in an editor and re-execute them after closing the editor
#it is useful if you need to re-execute more than one command
$fc -4 -1

nsupdate,the command line tool to manage BIND DNS records

nsupdate can submit Dynamic DNS Update requests as defined in RFC2136 to a name server, it is provided by “bind-utils” package.
Authentication for DNS updates
Transaction signatures (TSIG) can be used to authenticate the Dynamic DNS updates, The only supported encryption algorithm for TSIG is HMAC-MD5.
The TSIG public key is stored in a KEY record in a zone served by the name server, the TSIG private key is used by nsupdate to get authenticated.

$mkdir  /var/named/keys; cd /var/named/keys
$dnssec-keygen -a HMAC-MD5 -b 256 -n USER  server1-ddns-key
Kserver1-ddns-key.+157+00575.key 
Kserver1-ddns-key.+157+00575.private 
#Since HMAC-MD5 is symmetric encryption algorithm, the “key string” in private key and public key are equivalent, the public key can be deleted.
$ rm Kserver1-ddns-key.+157+00575.key 
$ cat Kserver1-ddns-key.+157+00575.private 
Private-key-format: v1.2
Algorithm: 157 (HMAC_MD5)
Key: x3NLKprWMvSMaPx75At6zA+DiWb2SvzpFzPFowc7SBU=

#Create a BIND configuration file to be included in /etc/named.conf (it is optional, the statements can be specified in /etc/named.conf directly)
$cat keys.conf
#the key name is arbitrary , but  must be the same name  specified in dnssec-keygen command line
key server1-ddns-key {
algorithm HMAC-MD5;
secret "x3NLKprWMvSMaPx75At6zA+DiWb2SvzpFzPFowc7SBU=";
};
$cat /etc/named.conf
include "/var/named/keys/keys.conf";
...
#zone example.com allow the key to do update
zone "example.com" IN {
type master;
file "example.zone";
allow-update { key server1-ddns-key ; };
};

The journal file for dynamic Update
All changes made to a zone using dynamic update are stored in the zone's journal file.
The name of the journal file is formed by appending the extension .jnl to the name of the corresponding zone file.
The updated entries are saved in .jnl file, the entries are merged to the zone file on the following conditions
1.Every 15 minutes
2."rndc freeze zone" command is issued
3.Service restart
However, the updated record in memory become effective immediately without waiting to be merged to the zone file.
In order for named process to write the journal file, the zone directory /var/named/ must be writable by named user or group
$chown root:named /var/named; chmod g+rw /var/named
nsupdate sample commands

$nsupdate -k Kserver1-ddns-key.+157+00575.private 
> server localhost
> zone example.com
> update add www1.example.com 86400 A 172.16.1.11
> send

Suspend dynamic DNS update

#sometimes it is necessary to update the zone file manually, follow this order to edit zone file manually
$rndc freeze example.com 
$vi example.com.zone
$rndc thaw example.com 

#if your zone exists in multiple views, you may get this error.
$rndc freeze example.com
rndc: 'freeze' failed: not found
#You have to specify class name (IN) and the view name in full.
$rndc freeze example.com  IN myview-name

A wrapper script to make DNS update even easier

$./dnscmd.sh add example.com www5 172.16.1.11
==> Checking result: www5.example.com. ...
Using domain server:
Name: localhost
Address: 127.0.0.1#53
Aliases: 
www5.example.com has address 172.16.1.11

#script content
$cat dnscmd.sh 
#!/bin/ksh
SERVER='localhost'
TTL='86400'
KEY='/var/named/keys/server1-ddns-key.key'
cmd=$1
zone=$2
node=$3
data=$4
if [[ $# -ne 4  ]]; then
cat <<END 
Usage $0 add|delete zone-name node-name data
For example:
- Add A record: $0  add  example.com web 192.168.100.34
- Add PTR record: $0  add  100.168.192.in-addr.arpa  34 web.example.com.
- Delete record: replace add with delete
*** It only supports A and PTR record, for other records,edit zone file manually: rndc free zone-name; vi zonefile; rndc thaw zone-name
END
exit
fi
if [[ "$node" = {1,3}(\d) ]]; then
class='PTR'
else
class='A'
fi
tmpfile=$(mktemp)
cat >$tmpfile <<END
server $SERVER
update $cmd ${node}.${zone} $TTL $class $data
send
END
nsupdate -k $KEY -v $tmpfile
rm -f $tmpfile
record=${node}.${zone}
[[ ${zone} = *in-addr.arpa ]] && record=$(echo ${node}.${zone}  | awk -F"." '{ print $4"."$3"."$2"."$1} ' )
echo -e "\n ==> Checking result: ${record}. ..."
host $record $SERVER

Tune nscd(name service cache daemon)

nscd provides caching for the passwd,group,and hosts tables, it can boot performance for situations, in which the tables need to be serviced remotely  e.g. LDAP authentication and DNS.
However, sometimes, it cause trouble.In Red Hat Linux 5 , nscd always return the old entry until the TTL(default is 1hour) is reached, even restarting nscd won’t flush the cache.
There are two solutions:
1.Disable persistent caching

#Persistent caching is enabled by default
$cat /etc/nscd.conf
persistent              passwd          yes
persistent              group           yes
persistent              hosts           yes
positive-time-to-live   hosts           3600
#So the entries are saved to relative tables
$rpm -ql nscd
..
/var/db/nscd/group
/var/db/nscd/hosts
/var/db/nscd/passwd…
#change them to no
persistent              passwd          no
persistent              group           no
persistent              hosts           no

With persistent caching disabled, restart nscd will discard the entries in memory.
2.Flush entries by invalidating the  table 
The entries in tables (group/passwd/hosts) can manually flushed by the ‘invalidate ‘parameter.

$nscd --invalidate=hosts

Since it is natural for anyone to try restart nscd to resolve the issue and the operation of rebuilding cache is not expensive, I think option 1 is better.

Force puppet agent to regenerate certificate request

If puppet agent’s certificate is accidentally revoked or deleted, you can force agent to regenerate certificate request.

In general, it is impossible un-revoke a certificate unless the revoke reason is certificateHold, But puppet can hack it. The solution is to recover all revoked certificates then revoke other certificates which don’t need to be recovered

$rm /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem
$rm /etc/puppetlabs/puppet/ssl/crl.pem
#At this point, all revoked certificates become valid certificates.
#So you need to revoke all certificates which don’t need to be recovered
$puppet cert --revoke foo

The following method of regenerating new certificate seems to be a better.

The following is tested in Puppet Enterprise 2, but it should work for puppet open source as well.
$ puppet --version
2.7.6 (Puppet Enterprise 2.0.0)
Force agent to regenerate certificate request by generate command

[puppet agent]$ puppet  certificate   generate    web1  --ca-location  remote
warning: peer certificate won't be verified in this SSL session
err: Error 400 on SERVER: web1 already has a revoked certificate; ignoring certificate request
err: Try 'puppet help certificate generate' for usage
#It because the revoked certificate still exist in the server, it need to be deleted
[puppet master]$ puppet cert list –all
- web1                                     (BA:18:D1:86:D6:5E:9E:99:55:39:3D:67:79:BF:BD:D0) (certificate revoked)
[puppet master]$ puppet cert clean web1
#re-run the command, the warning is expected because the request hasn’t been signed by master yet
[puppet agent]$puppet   certificate   generate    web1  --ca-location  remote
warning: peer certificate won't be verified in this SSL session
true
#The pending request appears in master 
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
[puppet master]# puppet cert sign web1

If the above doesn’t work for you, the last resort is to clean agent’s ssl files

[puppet agent]$ puppet --genconfig | grep certdir
certdir = /etc/puppetlabs/puppet/ssl/certs
$cd /etc/puppetlabs/puppet/ssl/
$find . –type f –exec rm {} \;
$service pe-puppet restart
[puppet master]$ puppet cert list
web1 (3B:ED:D9:8D:2F:C2:A1:D3:89:B4:D0:FD:41:7E:5E:0C)
#Sign the certificate
puppet master]# puppet cert sign web1