Dtrace Basics

- D Program Structure
probe descriptions
/ predicate /
{
action statements
}
-- Probe Descriptions
provider : subsystem : function : probeName
syscall::*lwp*:entry, syscall::*sock*:entry #support wildcards, one or more fields of the probe description are omitted(means any value)
--- list available provider moduel function name
$dtrace -l more
ID PROVIDER MODULE FUNCTION NAME
1 dtrace BEGIN
2 dtrace END
3 dtrace ERROR
4 nfsmapid209 nfsmapid check_domain daemon-domain
5 nfsmapid209 nfsmapid resolv_query_thread thread-domain
6 syscall nosys entry
-- Predicates
Predicates are expressions enclosed in slashes / / that are evaluated at probe firing time to determine whether the associated actions should be executed.
D language doesn't has control-flow constructs such as if-statements and loops. it use Predicates
-- Actions
Probe actions are described by a list of statements separated by semicolons (;) and enclosed in braces { }.If no additional action need, an empty set of braces with no statements inside
-- example
$dtrace -n syscall::read:entry #

-n means to match probename from command line, -m=match module name
$dtrace -s counter.d #- s read input from script
$vi counter.d
dtrace:::BEGIN
{
i = 10;
}
profile:::tick-1sec
/i > 0/
{
trace(i--);
}
profile:::tick-1sec
/i == 0/
{
trace("blastoff!");
exit(0);
}

dtrace/profile are providers
tick-5sec #tick-xsec is the function name of provider profile, like sleep in shell
trace (100) #print out a value or string(needs enclosed by " ",trace ("hello")), like echo in shell
printf ("%s","hello") # print out in particular format
/* ... */ #comment lines
if no
dtrace:::END statement, you need to press ctrl+c to see to the result.

--Use built-in variables for predicates
syscall::read:entry,
syscall::write:entry
/pid == 12345/
{
}


execname: Name of the current process's executable file
pid:Process ID of the current process
tid: Thread ID of the current thread
...

- Aggregations
DTrace stores the results of aggregating functions in objects called aggregations. The aggregation results are indexed using a tuple of expressions similar to those used for associative arrays. In D, the syntax for an aggregation is
@name[ keys ] = aggfunc ( args );
Aggregations is used for result data,the entire data set need not be stored
Arrgegations are printed out by default no need to print statement
-- Example
# Syscall count by process,
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

-- DTrace Aggregating Functions
count: The number of times called.
sum: The total value of the specified expressions.
.....

- Structs
If you have programmed in the Java programming language, think of aDstruct as a class, but one with data members only and no methods.
struct callinfo {
uint64_t ts; /* timestamp of last syscall entry */
uint64_t elapsed; /* total elapsed time in nanoseconds */
uint64_t calls; /* number of calls made */
size_t maxbytes; /* maximum byte count argument */
};
You can use the operator -> to access struct members through a pointer. callinfo->ts
- Further reading..
Solaris Dynamic Tracing Guide - Official Dtrace guide
DTrace Tools - collection of useful scripts

Difference between Red Hat Linux and SUSE Linux

Red hat Linux and SUSE linux don't have too much difference, the universal management tool: YaST makes it easier for Red hat Linux users to grasp SUSE Linux: The following is important difference between Red Hat Linux and SUSE Linux in sysadmin perspective (based on SUSE Linux Enterprise Server 10)


- Management tool
YaST is the central system management tool for SUSE Linux.
YaST is collection of modules,e.g "yast users " will go to user management module directly, yast -l to list all modules.

-- launch YaST:
yast: Start in text mode
yast2:Start in text mode/gui mode depending on the terminal capability


-Installation automation
Red Hat Linux: kickstart SUSE linux: AutoYast

- Boot
-- boot files:
/etc/init.d/boot #master boot control file called by init process
/etc/init.d/boot calls various scripts in /etc/init.d/boot.d/* to load modules/mount FS/activate network etc.
Files in /etc/init.d/boot.d/Sxxboot.* are sym link to /etc/init.d/boot.*, e.g. (/etc/init.d/boot.d/S08boot.localfs -> ../boot.localfs).
They are executed in ascending order.
Finally, /etc/init.d/boot call /etc/init.d/boot.local (like /etc/rc.local in RedHat Linux for user defined startup command )

-- Service control
Both can use chkconfig tool to control service, in addition. SuSE linux has rcSVCNAME command linked to the start script to control service directly . e.g
#file /usr/sbin/rcsshd
/usr/sbin/rcsshd: symbolic link to `/etc/init.d/sshd'

- User
New user added to users group by default, RedHat Linux added to new private group
For bash, local profile use ~/.profile, RedHat Linux use ~/.bash_profile

- Network
/etc/sysconfig/network/* #all network related config files
/etc/sysconfig/network-scripts/ifcfg-eth-id-MA-CA-DD-RE-SS #interface config files
/etc/HOSTNAME #set hostname,the file name is upper case.
/etc/sysconfig/network/routes #static routes

- Security
Red Hat Linux: SElinux
SuSE Linux: AppArmor
Both are based on Linux Security Modules (LSM) to overcome the limitation of traditional Discretionary Access Control (DAC), but the configuration details are totally different. The default configuration may cause weird issue, AppArmor can be turned with script /etc/init.d/boot.apparmor

Detailed document for Migrating from RedHat to SUSE Linux Enterprise Server 10

Netapp Notes

#== Basics
- Introduction
Data ONTAP is the name of Netapp's Platform OS, it is based on BSD.
Netapp appliance is based on i86 PC hardware(mostly AMD Opteron nowadays).
NetApp appliance is unified storage supports file-based protocols such as NFS, CIFS, FTP, TFTP, and HTTP, block-based protocols such as FC and iSCSI.
NetApp V-Series can attach and manage third party storage systems.
NVRAM: log transitions, it can replay the log in the event of unplanned shutdown.
RAM: system memory, data read/write cache
Flash card(not all models): system initial boot media

FC Controller port 0a/0b/0c/0d ....
Jumper to setup Disk shelf ID 1 - 7
Disk numbering from right to left 16 ......
FC:
WWNN - world wide name
is a World Wide Name assigned to a port in a Fibre Channel fabric, it performs a function equivalent to the MAC address in Ethernet protocol
WWPN - world wilde port name
is valid for the same WWNN to be seen on many different ports (different addresses)

ISCSI:
Initiator= client, target = server
Software initiator with standard NIC
TCP offload Engine TOE with soft initiator – offload computing from CPU
ISCSI HBA -Hardware initiator and provide diskless boot
No need to use ISCSI HBA, if diskless boot feature is not needed and CPU resource is plenty

Netapp use software ISCSI if no ISCSI HBA present.

# setup iscsi on linux
[root@linux /]# iscsi-iname
iqn.1987-05.com.cisco:01.8494d06dc35d
The ID generated is random, write it to:/etc/initiatorname.iscsi to be persistent

- help
help # list all cmds
cifs help shares #display help for sub command
man cmd
priv set advanced # to use advanced command e.g ls
- account managment
useradmin group list
useradmin user add admin -g Administrators
- Access netapp
http://IP/na_admin # web gui URL.
ssh IP # ssh (or rsh, telnet in) for CLI access, enable ssh with secureadmin cmd
rsh IP #excute cmd remotely from admin server, the admin server should be added to /etc/hosts.equiv
get root mount of /vol/vol0/ in a unix machine to edit config files
- Read/write file
rdfile /etc/exports
wrfile -a /etc/hosts 1.1.1.1 filer1 #append to file
wrfile /etc/hosts.equiv #Rewrite file, type in lines, then ctr+C
- Error messages
rdfile /etc/messages
- backup config file
config dump config.bak
it is saved to /etc/configs/config.bak
- server setting
options command control server setting
options ftpd.enable on #enable ftp server for example
options nfs.export.auto-update off #turn off auto export, otherwise the new volume will be exported automatically
all options are saved to /etc/registry
- system stats
stats show
sysstat -su 1
- copy entire volume
vol copy
ndmpd on
ndmpcopy -f /vol/vol0 /vol/vol0_trad
- Cluster
Netapp cluster doesn't do I/O load balancing, it is just for fail-over purpose.You need to allocate disk in each node for different services


#= =Boot
Ctrl+C to go to boot menu.
there is option to reset password
type in "22/7" to show secret boot menu

#= =Storage

Qtree, and/or subdirectories, export-able
 |
Volume (TradVol, FlexVol), export-able, snapshot configured at this level.
 |
agregate (OnTap 7.0 and up)
 |
plex (relevant mostly in mirror conf)
 |
raid group
 |
disk

disk zero spare # zero all spare disk so they can be added quickly to a volume
Data ONTAP supports 100 aggregates (including traditional volumes) on a single storage system.
Data ONTAP supports 500 volumes per head(FAS2020 and FAS200 series, the limit is 200 FlexVol volumes.), So in cluster enviroment, the combined volumes number in both nodes should not exceed the limit for the sake of failover.
Netapp wafl(Write Anywhere File layout) block size = 4 KB.
Netapp support Raid0, Raid4, RaidDP (double-parity), Raid 1 (via snapmirror)

- Traditional Volume
It is tightly coupled swith its containing aggregate. No other volumes can get their storage from this containing aggregate. It can'b be shrinked. can be expanded by adding more disks

- FlexVol volumes
A FlexVol volume is a volume that is loosely coupled to its containing aggregate. A FlexVol volume
can share its containing aggregate with other FlexVol volumes,Thus, a single aggregate can be the
shared source of all the storage used by all the FlexVol volumes contained by that aggregate.

- flex clone
FlexClone volumes always exist in the same aggregate as their parent volumes
You cannot delete the base Snapshot copy in a parent volume while a FlexClone volume using that
Snapshot copy exists. The base Snapshot copy is the Snapshot copy that was used to create the
FlexClone volume, and is marked busy, vclone in the parent volume.

- snapshot
255 snapshot per volume
A Snapshot copy is a frozen, read-only image of a traditional volume, a FlexVol volume, or an aggregate
that captures the state of the file system at a point in time. It doesn't consume space initially, snapshot grows only as data changes.
- Aggregate
Max space: 16TB Max Number: 100
1 big aggregate runs faster than mulitiple aggregates created on same number of physcial disks
16 disks setup is the sweetspot for space usage utilization and performance
- Lun
Lun is created on top of volume, it can't exceeds the volume size. it is used to for FC/ISCSI mount

#= = command
sysconfig -r # show raid group and spare disks
aggr status -s #only show spare disks
sysconfig -V #show aggregate name and the numer of disk owned
sysconfig -d # show physical disks
storage show disk # show physical disks

- Traditional volume
vol create travol1 3 #create a traditional volume with 3 disks, disk are selected automattically
vol create trad02 -d 0a.24 0a.25 0a.27 #create a traditional volume by specifying disk names

- Flex volume
aggr create aggr2 4 #create aggregate first
vol create flexvol1 aggr2 20M #create volume on the aggregate
vol offline trad02
vol destroy trad02 #delete volume, need to be brought offline first
vol options vol1 nosnap on #turrn off automatic shceduled snapshot, not snapshot ablity
aggr show_space #show aggregate and volume space
aggr options aggr1 raidtype raid_dp #change raidtype between raid4 and raid_dp from raid0 to raid4/raid_dp
df -h #show volume space

- Snapshot
snap create vol0 mysnap0
snap delete vol0 mysnap0
snap list vol0 #list snapshot
snap delta vol0 #show size of changed data
/vol/vol0/.snapshot/mysnap0 #access snapshot data

#= = Network
- show ip/change ip
ifconfig
- add route
route add net 100.100.100.0 192.168.211.2 1
- permanet add
wrfile -a /etc/rc route add net 100.100.100.0 192.168.211.2 1
- vi /etc/rc
routed on # turn on RIP routing
- Link aggregation
vif
- Package tracing
pktt start ns0
pktt dump ns0 #a xx.trc file will be saved to /
pktt stop ns0

#= =NFS
- turn off nfs auto export for new volume
options nfs.export.auto-update off
- Show exports
exportfs -v
- show detailed export options
exportfs -q /vol/vol0
/vol/vol0 -sec=sys,(ruleid=0),rw,anon=0,nosuid
- Permanent export and add entry to /etc/exports
exportfs -p sec=sys,rw,nosuid /vol/vol1
- Temp export and don't add entry to /etc/exports
exportfs -io sec=sys,rw,nosuid /vol/vol1
- permanent unexport, remove from /etc/exports
exportfs -z path
- Temp export
exportfs -u path
- Re-read /etc/exports and re-export
export -r
- Control access
exportfs -io sec=sys,rw=10.10.10.1 /vol/vol1
or
exportfs enable nosave 10.10.10.1 /vol/vol1

#==CIFS
- Stop/start service
cifs terminate / cifs restart
- Initial setup
cifs setup /* select (3) Windows Workgroup authentication using the filer's local user accounts */
- Determine if both nfs client/cifs client access system
options wafl.default_security_style unix ntfs mixed
- display shares
cifs shares
- add share
cifs shares -add HOME /vol/vol0/home
- add permission
cifs access -delete home everyone
cifs access HOME Administrators "Full Control"