Passed 3/5 RHCA: EX333 Security Network Services

I would probably rate EX333 as the most difficult exam among the 3 exams I have passed, Why  is it so difficult?

1. The exam objectives are related: for example, Kerberos depends on NIS, without a working NIS, you are doomed.

2. The exam is broken into morning section and afternoon section, You won’t pass the exam  if either section fails. but  you are still allowed to sit for the afternoon section despite  the result of morning section.

My blog post for EX333 study notes:

Authenticate BIND zone transfer with TSIG key

Setup Postfix SMTP password authentication with SASL

Passed 2/5 RHCA: EX436 Clustering and Storage Management

EX436 is easier than EX442(System Monitoring and Performance Tuning), because testing subjects are less and the method of testing is just straight forward setup and configuration, unlike EX442, which requires extensive analysis and calculation.
I didn't pay attention to RHEL release during exam, But,RHEL 5.4,is showed in my exam result. Although GFS2 is default starting from RHEL 5.3, GFS is the subject to be tested. I think it won't  be changed until RHEL 6 courseware comes out.

My blog post for EX436 study notes

GFS(Global File System) quickstart

RHCS(Red Hat Cluster Suite) quorum disk

RHCS(Red Hat Cluster Suite) I/O fencing using SNMP IFMIB

Do we really need to set partition type to fd(Linux auto raid) for Linux software RAID?

Passed 1/5 RHCA: EX442 System Monitoring and Performance Tuning

I passed the first exam,EX442 System Monitoring and Performance Tuning,in my road to RHCA (Red Hat Certified Architect) today.
Here are some tips I can share, I think, without violating confidential agreement.
Document Source.
EX442 is very heavy on theory, you simply can't remember every bit ( and  its different unit: kb,byte,sector, block, bit,million secs, secs). The key is to  get them in the system documents without Googling.

#Man pages e.g
man 5 proc
#man pages keyword search
$ man -k semget
semget               (2)  - get a semaphore set identifier
semget               (3p)  - get set of XSI semaphores
#man pages full text search
$man -w -K SEMMNI
/usr/share/man/man5/proc.5.gz
/usr/share/man/man2/semget.2.gz
#/usr/share/doc, full text search
$yum install kernel-doc
$grep -ri isolcpus /usr/share/doc/kernel-doc*/Documentation/.* 
/usr/share/doc/kernel-doc-2.6.18/Documentation/kernel-parameters.txt:   isolcpus=       [KNL,SMP] Isolate CPUs from the general scheduler.
#Other documentations in html/pdf/info/ps format
$rpm -ql valgrind | grep /usr/share
..
/usr/share/doc/valgrind-3.5.0/html/index.html
..
$elinks /usr/share/doc/valgrind-3.5.0/html/index.html

Make the change persistent

#always put the change in configuration file first, then run the command to activate the change 
- Startup service: chkconfig svc-name on; service svc-name start
- User auto start script: edit /etc/rc.local;  execute “/etc/rc.local”
- Kernel Parameter: edit /etc/sysctl.conf; execute “sysctl -p”
- File system: edit /etc/fstab ; mount -a
- Kernel modules: edit /etc/sysconfig/modules/XXX.modules;   execute  “/etc/sysconfig/modules/XXX.modules”

Calculator:

#You got to know how to use bc.
#Always start with bc with “-l” option to support floating  point.
#e.g 1Gibit=131072KiByte
$ bc -l
1*2^30/1024/8
131072.00000000000000000000
#if there are too many to be added up, you got to use awk
#e.g  sum and average value calculated from 1 to 1000
$ echo {1..1000} | sed "s/ /\n/g" | awk ' {x=$1+x} END {print $1,x,x/NR } '
1000 500500 500.5
#convert  hexadecimal  value  to decimal  
#the hex letter must be in upper case
$echo -e " obase=10 \n ibase=16 \n 0F0" | bc -l
240

Time is essence, every minute counts, you got to master bash command line.
Check this: http://honglus.blogspot.com/2010/07/master-bash-command-line.html
lastly, the Exam day
 - take your RHCE number with you
 - there are a couple of tricky questions, I finished all, I thought I could get 90, but I only got 80. I don't know what went wrong.
 - Unlike RHCE exam, which has 2 sessions with a break for lunch, the EX442  only has 1 morning  session  last for 4 hours
 - Exam result will be emailed in 2 business days, but I got mine in 2 hours.


My Blog posts for EX442 study notes
Load kernel modules at boot time on Redhat/Centos Linux.
Tune Interrupt and Process CPU affinity
Understanding Linux CPU scheduling priority

Calculate chunk size for RAID device

Setup SNMP V3 USM with encryption.

Graphing sar output

Linux memory management study notes

RHCE NOTES - SElinux

Quiick SElinux notes for the impatient, read full document at

http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/selinux-guide/


Selinux has 2 levels access control:
1) File context, Daemon can only access file with particular file context
2) Boolean Value: enable/disalbe a feature
for example: By default SElinux does not allow users to login and read their home directories, turn it on by "setsebool -P ftp_home_dir 1"

#==Confined and Unconfined Process
Confined process enter paritcular domain after started, only particular domain has access to particular TYPE files
SElinux has no effect for Unconfined Processes (apps doen's support SElinux)

==Example
$ ls -Z /usr/sbin/httpd
-rwxr-xr-x root root system_u:object_r:httpd_exec_t /usr/sbin/httpd #httpd is confined by default
$chcon -Rt unconfined_exec_t /usr/sbin/httpd #change httpd to unconfied_exec_t, it will enter unconfied domain, so it can access any file as long as OS level file permission allowed
$ restorecon -Rv /usr/sbin/httpd #restore default type

#== SELinux: File context
user:role:type:sensitivity:category
for example: system_u:object_r:httpd_sys_content_t :s0:c0
Not all systems will display s0:c0

==example
# ls -aZ /var/www/html/
drwxr-xr-x root root system_u:object_r:httpd_sys_content_t .
drwxr-xr-x root root system_u:object_r:httpd_sys_content_t ..
# ls -aZd /home
drwxr-xr-x root root system_u:object_r:home_root_t /home
httpd_exec_t can access httpd_sys_content_t not home_root_t

#==SElinux managment
SELINUX=permissive #in /etc/selinux/config. if it changed from disabled . it needs reboot to lable files
getenforce or sestatus #get current status
setenforce 0 # set to permissive mode
setenforce 1 #set to enforce mode
getsebool -a #list booleans and its value , no desc
setsebool httpd_can_network_connect_db on #change current boolean
setsebool -P httpd_can_network_connect_db on #change permanent boolean with -P

- Temparary change context
chcon -R -t httpd_sys_content_t /web/ #change context type dir/file
# it will survive reboot, but not relabel. To relabel, touch /.autorelabel reboot

- Persistent Changes: semanage fcontext
/etc/selinux/targeted/contexts/files/file_contexts #saved to orginal context
/etc/selinux/targeted/contexts/files/ file_contexts.local #saved to new user context
semanage fcontext -a -t samba_share_t /etc/file1 #-a add new context, the file doesn't need to exist.
restorecon -Rv /etc/file1 #read the new customized context and apply it

- Restore default context
semanage fcontext -d /etc/file1 #remove context,the file doesn't need to exist
restorecon -RFv /etc/file1 #apply the change, -F is needed you to restore from customized to default.

#==Troubleshooting
/var/log/messages.X
/var/log/audit/audit.log #enable auditd daemon first
chkconfig --levels 345 setroubleshoot on #enable troubleshoot daemon
sealert -a /var/log/messages #analyse log
sealert -l \* #show all alert
grep "SELinux is preventing" /var/log/messages
grep "denied" /var/log/audit/audit.log
Port Numbers # services are allowed to run on some defined ports
/usr/sbin/semanage port -l grep http_port_t
ttp_port_t tcp 80, 443, 488, 8008, 8009, 8443
semanage port -a -t http_port_t -p tcp 9876 #add the new port to allowed range

#==== document
selinux-policy-2.4.6-137.el5#man pages for ftpd_selinux, samba_selinux ...etc

RHCE TIPS - Sitting the test

SECTION I: TROUBLESHOOTING AND SYSTEM MAINTENANCE [Morning Session]
This section is easy and the proctor will let you know the result immediately.

- Compulsory Section I: It tests system maintenance, It counts for 80, which is enough for RHCE. So if you have got 80 here, you don’t need to take next non-compulsory question.
- Non-compulsory Section I: It tests system booting issue, It counts for 20. If you didn’t complete compulsory section or just after perfect sore 100, take the non-compulsory question. The proctor will re-image your PC to introduce the booting issue, So you can’t go back, once you have made the decision. You should be safe , once you have mastered all scenarios in my previous post.

SECTION II: INSTALLATION AND CONFIGURATION [Afternoon Session]
If you have breezed through SECTION I, Don’t be too joyful, the hardest part is here.
It is hard because the time is limited, there are many tasks to complete, if you stuck with one, time is quickly running out. Secondly, no one will verify the result, you have to check by yourself. It is quite tricky, if you misinterpret the requirement, your check method maybe wrong, or service lost function after reboot.

A few tips during the test:
- TIME:
Manage well the time, don’t stuck with one question too long, you don’t need full score to pass RHCE.
- CHECK:
Check the result immediately after complete a task. Don’t expect to check everything at last, you can login to remote Linux to verify your result.

#User and file permission:
‘su – username’ to check permissions creating/listing file
#HTTP:
curl http://ServerName-or-IPAddress or elinks http://server
#Squid
curl –x proxyip:port url
#Send mail:
echo "test" mail –s "subject" user
# Send mail with specific sender:
telnet server 25 \n; mail from: user@x \n; rcpt to: user@y \n;data \n; subject: "subject x" \n; "text body "\n; .
#Receive local email:
mail or mail –f mailbox-filename
#Receive remote email:
mutt –f pop://server or mutt –f imap://server
#samba
smbclient //ip/share -U username /* Because you may not able to access the share ,even smbclient -L IP –U show the share*/
#test firewall
nc -z IP 1-200 /*scan remote hosts opening ports */
nc -v IP 25 or telnet IP 25 /* check availability of 1 port */

- Security tasks:
RHCE tasks are about restricting access for services, There are many options to achieve the result, It is up to you which one to use. Service’s native support, PAM, tcp-wrapper, iptables. Be careful using iptables, you should use open firewall which means accepting everything except specifically denied. You don’t want your firewall deny the services completed earlier.

- Need help?
Unfortunately, you don’t have internet access during test. You can only rely on the local man pages and documents. So during the preparation of the test, avoid finding the answers from internet straight away, try the local man pages and docs first. For example if you forget the format for ifcfg-ethx.cfg, the syntax is documented here:
/usr/share/doc/initscripts-XXX-/sysconfig.txt

- Lastly:
In the last mins, you should reboot your PC, check the services are still running after reboot. As pre-caution, always begin with task with this command chkconfig svcname on

RHCE TIPS - Preparation

Reference book:
RHCE Red Hat Certified Engineer Linux Study Guide (Exam RH302) 5th edition by Michael Jang.
if something is not clear in the book, read official Red Hat Enterprise Linux Documentation


Lab Setup:
Install CentOS on Virutalbox
Virtualbox is free opensource virtualization software alternative to Vmware.You need 2 CentOS instances to prepare for RHCE lab, The networking in Virtualbox is very different to Vmware.


-Virtualbox Networking Type:
--NAT: your guest OS can access outside network through NAT provided by virtualbox, but your host OS can’t access guest OS
--Host interface networking: Host and guest can communicate each other, but guest can’t access outside network unless you setup NAT manually on Host OS
--Internal network: Guest OS can communicate with each other within the SAME network name (something ike VLAN ID), but not Host OS.


-Centos ServerA network setup
1*NAT adapter for internet access to do yum.
1*Host network adapter for your host to ssh to ServerA
1*Internal Network adapter to communicate with ServerB


-Centos ServerB network setup
1* Internal Network adapter to communicate with ServerA (join the SAME network name of ServerA )


How can ServerB access outside network? Point the default GW to serverA, and turn on ip forwarding on ServerA.
How can my Host OS access ServerB?
1. ssh to serverA first then jump from serverA to ServerB
2. -setup porforwarding or 1 to 1 static mapping in ServerA
--Forwarding port 200 to ssh of ServerB

iptables -t nat -A PREROUTING -p tcp -d ServerA-Host-NIC-IP --dport 200 -j DNAT --to-destination ServerB-IP:22 

--Static 1 to 1 mapping
Assign secondary ip to serverA’s host Inc then

iptables -t nat -A PREROUTING -p tcp -d ServerA-SEC-NIC-IP -j DNAT --to-destination ServerB-IP

Last but least, read through each chapter and practice it LAB, you never know if it works until you really do it! RHCE exam is all about security, hence I suggest jumping to security chapter before reading networking services. Then apply your security knowledge (pam/tcp-wrapper/iptables/selinux) to each network services read later.

RHCE Notes - Troubleshooting booting issue

booting issue is optional question in section I,The proctor will re-image your PC to introduce booting issue, You will be given rescue CD to fix it.

It is easy to troubleshoot Linux boot issue, if you break it intentionally at each step, observe the symptom and find the fix.

#==Linux boot order
The BIOS ->MBR->Boot Loader->Kernel->/sbin/init->
/etc/inittab->
/etc/rc.d/rc.sysinit->
/etc/rc.d/rcX.d/ #where X is run level in /etc/inittab
run script with K then script with S

#==Linux rescue env
boot first linux cd then type linux rescue
TIP:
linux rescue will try to mount all partions, however if there is error only some partions are mounted, run choot /mnt/sysimage now will lost /dev /proc mounts, here is how to transfer these mounts.
mount -o bind /dev /mnt/sysimage/dev
mount -o bind /proc /mnt/sysimage/proc
Linux rescue env supports both software RAID and LVM. normal LVM commands e.g vgdisplay are not availiable,but it can be accssed by LVM "master" command e.g "lvm vgdisplay"

#== Grub boot manager
= go to grub cmd prompt by pressing c at boot menu
=find root partition, 2 methords
grub> root
(hd0,0) Filesystem type is ext2fs, partition type 0x83
grub> find /grub/stage1
(hd0,0)
=list files/dirs in current drive
cat / #type cat SPACE / TAB, it will list all fies/dir just like ls
= display contents of the file
cat /grub/grub.conf
= now you can boot interactively by type kernel and initrd commands from grub.conf


#==Restore missed file from RPM
#cd /tmp
#rpm2cpio initscripts-7.93.11.EL-1.i386.rpm cpio -icumvd ./etc/inittab
or
#rpm2cpio initscripts-7.93.11.EL-1.i386.rpm >init.cpio /* file is ./etc/inittab not /etc/initab
List contents: cpio -tv
or

install file to alternative location the copy the file
rpm --root-directory /var/tmp/a X.rpm

#== MBR corrupted.
MBR has 512 byte in total
446 Executable code section
4 Optional Disk signature
2 Usually nulls
64 Partition table #if this is overwritten, no way to recover unless you backuped the partion table or re-partion using #exact same layout
2 MBR signature

Corrupt MBR intentionaly:dd if=/dev/zero of=/dev/hda bs=446 count=1 #MBR should be at the start whole disk(not partition hda1), it has 512, the first 446 byte is exec code. DON'T overwrite whole 512 byte because it has partion table data.
ERR: no bootable media found,Missing operating system" or "Operating System Not Found
boot from cd run "linux rescue", let it mount linux partions automaticlly.
chroot /mnt/sysimage then grub-install /dev/hda
boot from cd run "linux rescue", if linux partions failed to mout
mount mannually. sfdisk -l; e2label find the boot partition
mkdir /a; mount /dev/hda1 /a; ln -s /usr/sbin/grub /sbin/grub; grub-install --root-direcotry=/a /dev/hda #it is hda not hda1


#= root (/)was not mounted
mount couldn't find file system /dev/root
switchroot mount faild...
Error 2 mounting none;exec of init ((null)) failed!!!
kernel /vmlinuz-test ro root=LABEL=/
/* root=LABEL=/ mout using label, or root=/dev/sda3 mount with direct dev-name */


#= not loading initrd image
VFS: Cannot open root device "Label=/1" or unknow-block(0,0)
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount roof fs on unknow-block(0,0)
1) Kernel doesn't Support for the file system .compile kernel with FS support NOT as a module
2) initrd was not loaded. Add initrd=... in grub.conf
linux rescue, then chroot /mnt/syimage and create initrd file
mkinitrd /boot/initrd-filename `uname -r` #make initrd file mannually


#==/sbin/init problem.
Switching to new root
kernel panic -not syncing :Attepmted to kill init
switching to new root
/bin/sh: ro : no such file or directory
/* boot to rescue, check /sbin/init. restore from rpm package*/


#== /etc/inittab not found
"enter run level" prompt enter s. or at grub menu append s or init=/bin/sh or emergency, then restore initab from source RPM

Passed RHCE

I passed RHCE today, I will be writting some tips and notes.

Here is my score report.

SECTION I: TROUBLESHOOTING AND SYSTEM MAINTENANCE
RHCE requirements: completion of compulsory items (50 points)
overall section score of 80 or higher
RHCT requirements: completion of compulsory items (50 points)

Compulsory Section I score: 50.0
Non-compulsory Section I score: 50.0
Overall Section I score: 100

SECTION II: INSTALLATION AND CONFIGURATION
RHCE requirements: score of 70 or higher on RHCT components (100 points)
score of 70 or higher on RHCE components (100 points)

RHCT requirement: score of 70 or higher on RHCT components (100 points)

RHCT components score: 92.6
RHCE components score: 86.7

RHCE Certification: PASS