Monday, February 15, 2010

File system benchmarking.

A simple file system benchmarking can use dd e.g. creating 100M file on /mnt/iscsi: "dd if=/dev/zero of=/mnt/iscsi/file bs=1024k count=100"
But dd can only measure sequential write/read and as being single thread, it is not powerful enough to overload fast disks.

My experience in file system benchmarking
1. Avoiding file cache effect.
2. Check disk %util from iostat -x command to make sure the load is hevy enough to saturate the disk
3. Run the same test many times to get average value
4. Run with different testing tools if values from one tool are doubtful
There are many File system benchmarking avaiable e.g. bonnie , iozone , fileio, tiobench ...

Bonnie
#simple to use, But doesn't support many options e.g block size, testing type, multiple threads. #Sample: Testing by creating 64M file on dir /mnt/iscsi
[root@centos ~]# bonnie -s 64 -d /mnt/iscsi
Bonnie: Warning: You have 251MB RAM, but you test with only 64MB datasize!
Bonnie: This might yield unrealistically good results,
Bonnie: for reading and seeking and writing.
Bonnie 1.4: File '/mnt/iscsi/Bonnie.1968', size: 67108864, volumes: 1
Writing with putc()... done: 61897 kB/s 48.1 %CPU
Rewriting... done: 569898 kB/s 47.0 %CPU
Writing intelligently... done: 288998 kB/s 31.3 %CPU
Reading with getc()... done: 55108 kB/s 38.1 %CPU
Reading intelligently... done: 1524163 kB/s 97.7 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
centos 1* 64 61897 48.1288998 31.3 569898 47.0 55108 38.11524163 97.7 110628.6 74.7

iozone

#feature rich: options for block size, test type, multiple threads and export output to Excel file format for graphing.
#Sample Test. Run a test by writing 64M file to /mnt/iscsi ; record size is 4k;
#test types are 0=write/rewrite, 1=read/re-read, 2=random-read/write



[root@centos ~]#cd /mnt/iscsi; iozone -s 64M -r 4k -i 0 -i 1 -i 2
Iozone: Performance Test of File I/O
Version $Revision: 3.315 $
Compiled for 32 bit mode.
Build: linux

Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
Al Slater, Scott Rhine, Mike Wisner, Ken Goss
Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

Run began: Mon Feb 15 12:14:05 2010

File size set to 65536 KB
Record Size 4 KB
Command line used: iozone -s 64M -r 4k -i 0 -i 1 -i 2
Output is in Kbytes/sec
Time Resolution = 0.000002 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random bkwd record stride
KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
65536 4 894127 2272316 39813119 53847589 9302484 639088

iozone test complete.

#Output result to Excel file


iozone -Rb perf.xls -s 64M -r 4k -i 0 -i 1 -i 2

Friday, February 12, 2010

Improve OpenSolaris COMSTAR iSCSI Server performance.

I setup OpenSolaris iSCSI and Linux Open iSCSI in Virtualbox to compare the performace. I can get 70~80% of local FS throughput on Linux iSCSI, but it is 10~20% on OpenSolaris iSCSI.
Some tuning can boost OpenSolaris iSCSI performance to be on par with Linux iSCSI

#Environment
Server: OpenSolaris 2009.06 VS Openfiler 2.3 on Virutalbox 3.1
Client: Centos 5.2
Regardless of the iSCSI Server type, the network has to be optimized by tuning network parameters
#OpenSolaris iSCSI tunning
1. Use the bock device(/dsk) as backstore, NOT character device (/rdsk), because the block device (/dev/dsk) has buffer.

$iscsitadm create target -b /dev/zvol/dsk/data/iscsi filera.dbvol

2. Turn on fast-write-ack

$iscsitadm modify admin --fast-write-ack enable

reactivate the zpool for existing target.

$svcadm disable iscsi/target iscsitgt
$zpool export poolname ;zpool import poolname
$svcadm enable iscsi/target iscsitgt


Use the option with care, from iscsitadm man pages:
Enables or disables fast-write acknowledgment. You should enable this option only if a system is connected to the power grid through a UPS.
Otherwise, data corruption could occur if power is lost and some writes that were acknowledged have not been completely flushed to the backing store.

Monday, February 8, 2010

Setup OpenSolaris ISCSI server for Solaris and Linux Client

Setup ISCSI server in Opensolaris 2009.06 and configure Opensolaris/Centos 5.2 client to connect to ISCSI server

Simple ISCSI setup without access-list

####ISCSI Server
##enable iscsi service
svcadm enable iscsi/target iscsitgt

## Create zpool on whole disk ( XXp0 is whole disk convention for i86 arch, SPARC arch use XXs2)
zpool create data c8t1d0p0
##create volume on the zpool
zfs create -V 1G data/iscsi

##Create ISCSI target filera.dbvol, the name is arbitrary, it doesn't need to be volume name

[Server]$iscsitadm create target -b /dev/zvol/dsk/data/iscsi filera.dbvol

[Server]$iscsitadm list target -v
Target: filera.dbvol
iSCSI Name: iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol
Connections: 0
ACL list:
TPGT list:
LUN information:
LUN: 0
GUID: 0
VID: SUN
PID: SOLARIS
Type: disk
Size: 1.0G
Backing store: /dev/zvol/dsk/data/iscsi
Status: online

Command "zfs set shareiscsi=on data/iscsi " can also create target, but the target name wil be volume name

## if ISCSI server has multiple NICS, the target will be exposed on all NICS, following is to limit the target to the desired NIC
[Server]$iscsitadm create tpgt 1
[Server]$iscsitadm modify tpgt -i 172.16.1.12 1
[Server]$iscsitadm modify target -p 1 filera.dbvol

####Client: Solaris
Client-Solaris]$svcadm enable iscsi_initiator
[Client-Solaris]$iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:946808d8ea81.4b6a1021
##Change the long name to a short friendly name
[Client-Solaris]$iscsiadm modify initiator-node -N iqn.1986-03.com.sun:01:opensolaris01
#add the server address to discover
[Client-Solaris]$iscsiadm modify discovery -t enable
[Client-Solaris]$iscsiadm add discovery-address 172.16.1.12
[Client-Solaris]$iscsiadm list target -v
#check new disk discovered
[Client-Solaris]$echo format

####Client: Linux
##assign a friendy name to client
[Client-Linux]$ cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:centos01
[Client-Linux]$service iscsi start
[Client-Linux]$iscsiadm --mode discovery --type sendtargets --portal 172.16.1.12
##login to ISCSI discovered in previous step
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol " --portal "172.16.1.12:3260" --login
##Check the new discovered disk
[Client-Linux]$fidsk -l



ISCSI access list and CHAP authentication


Access list is a must for security reason, it is just like LUN mapping in SAN. If you need additional security, add chap-name and chap-secret, Access list alone works fine without chap initialized


##On Server, Set up CHAP information for Linux client

##Create mapping of initiator ISCSI name to an arbitrary name

[Server]$iscsitadm create initiator --iqn iqn.1994-05.com.redhat:centos01   centos01
[Server]$iscsitadm modify initiator --chap-name user_centos01 centos01
[Server]$iscsitadm modify initiator --chap-secret centos01
[Server]$iscsitadm modify target --acl centos1 filera.dbvol

##On Server, Set up CHAP information for Solaris client

[Server]$iscsitadm create initiator --iqn iqn.1986-03.com.sun:01:opensolaris01   opensolaris01
[Server]$iscsitadm modify initiator --chap-name user_opensolaris01 opensolaris01
[Server]$iscsitadm modify initiator --chap-secret opensolaris01
[Server]$iscsitadm modify target --acl opensolaris01 filera.dbvol

##On Linux Client, use the CHAP info to login

[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --op=update --name node.session.auth.authmethod --value=CHAP
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --op=update --name node.session.auth.username --value=user_centos01
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --op=update --name node.session.auth.password --value=Password0Password0
[Client-Linux]$iscsiadm -m node --targetname "iqn.1986-03.com.sun:02:d4b25dbf-dbc7-4f46-b2b5-c34774a32ae0.filera.dbvol" --portal "172.16.1.12:3260" --login

##On Solaris Client, use the CHAP info to login

[Client-Solaris]$ iscsiadm modify initiator-node --authentication CHAP
[Client-Solaris]$ iscsiadm modify initiator-node --CHAP-name user_opensolaris01
[Client-Solaris]$ iscsiadm modify initiator-node --CHAP-secret
Enter secret :
Re - enter secret :