Saturday, 31 March 2012

Really quick REGEX guide

Really quick REGEX guide 

Regular expressions… a bunch of brackets and special characters that nobody understands right? Well not exactly true. Although people tend to interpret them as newly discovered ancient language, Unix/Linux shells, text editors, scripting tools and some development environments understand them perfectly. And if you, as a “puny human“, figure them out, the work that usually takes hours can be reduced to minutes or less. So without any further delay here are the basics you can start with.


BRACKETS []: Used to give a list of possibilities.

EXAMPLE: grep [nN]ame /etc/services - This will find either n or N followed by “ame”. If there is ^ character inside of brackets it means that every parameter included after that character will not be included in search, for example: grep [^Nnt]ame /etc/services – it will not include N, n and t letters in search in front of “ame”. Try it, it will make more sense


PERIODS …: Used to match characters.

EXAMPLE: grep n..e /etc/services – This will match letters n and e with any two characters between them.


ASTERIX *: Used to find zero or more occurrences of a character.

EXAMPLE: grep [nN]e*d /etc/services – This will match Ned, need, nd, neeed.


QUOTES ” “: Used to match one or more blank spaces.

EXAMPLE: grep “a lot” /etc/services – This will match zero or more blank spaces which will account for the common misspelling of “a lot” as “alot”.


SPECIAL CHARACTER ^: Used to match the word when it is located at the beginning of the line.

EXAMPLE: grep ^the /etc/services – This will match any line that starts with the.


SPECIAL CHARACTER $: Used to match the word when it’s located at the end of the line.

EXAMPLE: grep the$ /etc/services – This will match any line that ends with the.


There are a bit more regular expressions, but upper ones are most commonly used. At least by me Thanks for reading.

Sunday, 18 March 2012

All about Linux swap space

Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.

Swapping is necessary for two important reasons. First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately. Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

However, swapping does have a downside. Compared to memory, disks are very slow. Memory speeds can be measured in nanoseconds, while disks are measured in milliseconds, so accessing the disk can be tens of thousands times slower than accessing physical memory. The more swapping that occurs, the slower your system will be. Sometimes excessive swapping or thrashing occurs where a page is swapped out and then very soon swapped in and then swapped out again and so on. In such situations the system is struggling to find free memory and keep applications running at the same time. In this case only adding more RAM will help.

Linux has two forms of swap space: the swap partition and the swap file. The swap partition is in independent section of the hard disk used solely for swapping; no other files can reside there. The swap file is a special file in the filesystem that resides amongst your system and data files.
To see what swap space you have, use the command swapon -s. The output will look something like this:

Filename Type        Size       Used Priority
/dev/sda5 partition  859436 0        -1

Each line lists a separate swap space being used by the system. Here, the 'Type' field indicates that this swap space is a partition rather than a file, and from 'Filename' we see that it is on the disk sda5. The 'Size' is listed in kilobytes, and the 'Used' field tells us how many kilobytes of swap space has been used (in this case none). 'Priority' tells Linux which swap space to use first. One great thing about the Linux swapping subsystem is that if you mount two (or more) swap spaces (preferably on two different devices) with the same priority, Linux will interleave its swapping activity between them, which can greatly increase swapping performance.

To add an extra swap partition to your system, you first need to prepare it. Step one is to ensure that the partition is marked as a swap partition and step two is to make the swap filesystem. To check that the partition is marked for swap, run as root:

 fdisk -l /dev/hdb

Replace /dev/hdb with the device of the hard disk on your system with the swap partition on it. You should see output that looks like this:

Device Boot Start   End    Blocks Id    System

/dev/hdb1     2328 2434 859446 82 Linux swap / Solaris

If the partition isn't marked as swap you will need to alter it by running fdisk and using the 't' menu option. Be careful when working with partitions -- you don't want to delete important partitions by mistake or change the id of your system partition to swap by mistake. All data on a swap partition will be lost, so double-check every change you make. Also note that Solaris uses the same ID as Linux swap space for its partitions, so be careful not to kill your Solaris partitions by mistake.

Once a partition is marked as swap, you need to prepare it using the mkswap (make swap) command as root:

mkswap /dev/hdb1

If you see no errors, your swap space is ready to use. To activate it immediately, type:

swapon /dev/hdb1

You can verify that it is being used by running swapon -s. To mount the swap space automatically at boot time, you must add an entry to the /etc/fstab file, which contains a list of filesystems and swap spaces that need to be mounted at boot up. The format of each line is:

Since swap space is a special type of filesystem, many of these parameters aren't applicable. For swap space, add:

 /dev/hdb1 none swap sw 0 0

where /dev/hdb1 is the swap partition. It doesn't have a specific mount point, hence none. It is of type swap with options of sw, and the last two parameters aren't used so they are entered as 0.

To check that your swap space is being automatically mounted without having to reboot, you can run the swapoff -a command (which turns off all swap spaces) and then swapon -a (which mounts all swap spaces listed in the /etc/fstab file) and then check it with swapon -s. 

Swap file

As well as the swap partition, Linux also supports a swap file that you can create, prepare, and mount in a fashion similar to that of a swap partition. The advantage of swap files is that you don't need to find an empty partition or repartition a disk to add additional swap space.

To create a swap file, use the dd command to create an empty file. To create a 1GB file, type:

 dd if=/dev/zero of=/swapfile bs=1024 count=1048576

/swapfile is the name of the swap file, and the count of 1048576 is the size in kilobytes (i.e. 1GB).

Prepare the swap file using mkswap just as you would a partition, but this time use the name of the swap file:

mkswap /swapfile

And similarly, mount it using the swapon command: swapon /swapfile.

The /etc/fstab entry for a swap file would look like this:


 /swapfile none swap sw 0 0

How big should my swap space be? It is possible to run a Linux system without a swap space, and the system will run well if you have a large amount of memory -- but if you run out of physical memory then the system will crash, as it has nothing else it can do, so it is advisable to have a swap space, especially since disk space is relatively cheap.

The key question is how much? Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.


The Linux 2.6 kernel added a new kernel parameter called swappiness to let administrators tweak the way Linux swaps. It is a number from 0 to 100. In essence, higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle. Kernel maintainer Andrew Morton has said that he runs his desktop machines with a swappiness of 100, stating that "My point is that decreasing the tendency of the kernel to swap stuff out is wrong. You really don't want hundreds of megabytes of BloatyApp's untouched memory floating about in the machine. Get it out on the disk, use the memory for something useful."

One downside to Morton's idea is that if memory is swapped out too quickly then application response time drops, because when the application's window is clicked the system has to swap the application back into memory, which will make it feel slow.

The default value for swappiness is 60. You can alter it temporarily (until you next reboot) by typing as root:

echo 50 > /proc/sys/vm/swappiness

If you want to alter it permanently then you need to change the vm.swappiness parameter in the /etc/sysctl.conf file.


Conclusion

Managing swap space is an essential aspect of system administration. With good planning and proper use swapping can provide many benefits. Don't be afraid to experiment, and always monitor your system to ensure you are getting the results you need.

Wednesday, 7 March 2012

Monitor Your System Performance


Monitor Your System Performance

Xxxxxxxxxxxxxxx Monitor Your System Performance xxxxxxxxxxxxxxxxxxxxX

[root@server ~]# uptime
13:14:02 up 1:30, 2 users, load average: 0.04, 0.26, 0.27


[root@server ~]# ps -eo %u | sort | uniq -c | sort -rn
56 root

22 nagios

8 apache

1 xfs

[root@server ~]# sar -c ## Process/sec
Linux 2.6.18-164.el5 (server.scratch.com) 08/24/2011

11:50:01 AM proc/s

12:00:01 PM 6.87

12:10:01 PM 6.90

12:20:01 PM 7.10

12:30:01 PM 6.77

12:40:01 PM 6.99

12:50:01 PM 6.85

01:00:01 PM 6.83

01:10:01 PM 6.92

Average: 6.90


[root@server ~]# sar -b ## Data r/wr ps
Linux 2.6.18-164.el5 (server.scratch.com) 08/24/2011

11:50:01 AM tps rtps wtps bread/s bwrtn/s

12:00:01 PM 24.81 0.18 24.63 6.37 376.94

12:10:01 PM 26.14 0.33 25.81 10.68 388.21

12:20:01 PM 57.68 28.05 29.63 620.21 466.02

12:30:01 PM 24.16 0.13 24.03 1.92 369.00

12:40:01 PM 24.55 0.10 24.44 1.70 374.73

12:50:01 PM 25.07 0.14 24.93 4.31 380.10

01:00:01 PM 24.95 0.58 24.37 6.20 377.88

01:10:01 PM 24.57 0.01 24.56 0.04 376.16

Average: 28.98 3.68 25.30 81.17 388.59


[root@server ~]# iostat -k
Linux 2.6.18-164.el5 (server.scratch.com) 08/24/2011


avg-cpu: %user %nice %system %iowait %steal %idle

0.40 0.00 3.84 0.44 0.00 95.32


Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda 6.60 32.24 65.92 177987 363873

sda1 0.10 0.21 0.00 1163 5

sda2 6.50 32.01 65.92 176684 363868

dm-0 20.19 31.54 65.91 174109 363844

dm-1 0.02 0.08 0.00 440 0

dm-2 0.06 0.25 0.00 1397 20

dm-3 0.03 0.11 0.00 613 4

# cat /proc/stat
cpu 1864 0 16482 416939 2375 34 527 0

cpu0 1864 0 16482 416939 2375 34 527 0

intr 2416023 2368439 12 0 2 2 0 5 0 1 0 0 0 116 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29989 0 0 0 0 0 0 0 37 0 0 0 0 0 0 0 17399 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ctxt 258727

btime 1314166387

processes 32539

procs_running 1

procs_blocked 0

# cat /proc/meminfo

MemTotal: 218872 kB

MemFree: 36680 kB

Buffers: 40416 kB

Cached: 79080 kB

SwapCached: 0 kB

Active: 116728 kB

Inactive: 47320 kB

HighTotal: 0 kB

HighFree: 0 kB

LowTotal: 218872 kB

LowFree: 36680 kB

SwapTotal: 614384 kB

SwapFree: 614384 kB

Dirty: 460 kBWriteback: 0 kB

AnonPages: 44556 kB

Mapped: 15936 kB

Slab: 11628 kB

PageTables: 2320 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

CommitLimit: 723820 kB

Committed_AS: 376900 kB

VmallocTotal: 802808 kB

VmallocUsed: 3564 kB

VmallocChunk: 798832 kB

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

Hugepagesize: 4096 kB

[root@server ~]# cat /proc/cpuinfo
processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 23

model name : Pentium(R) Dual-Core CPU E5300 @ 2.60GHz

stepping : 10

cpu MHz : 2592.870

cache size : 2048 KB

fdiv_bug : no

hlt_bug : no

f00f_bug : no

coma_bug : no

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss constant_tsc up pni

bogomips : 5185.74

[root@server ~]# cat /proc/buddyinfo
Node 0, zone DMA 37 33 31 29 19 12 9 2 0 0 0

Node 0, zone Normal 2217 861 467 317 71 17 6 1 1 0 0

[root@server ~]#


[root@server ~]# cat /proc/execdomains
0-0 Linux [kernel]


[root@server ~]# cat /proc/filesystems
nodev sysfs

nodev rootfs

nodev bdev

nodev proc

nodev cpuset

nodev binfmt_misc

nodev debugfs

nodev securityfs

nodev sockfs

nodev usbfs

nodev pipefs

nodev anon_inodefs

nodev futexfs

nodev tmpfs

nodev inotifyfs

nodev eventpollfs

nodev devpts

ext2

nodev ramfs

nodev hugetlbfs

iso9660

nodev mqueue

ext3

[root@server ~]# cat /proc/loadavg
0.33 0.36 0.29 1/92 2916


[root@server ~]# cat /proc/locks
1: POSIX ADVISORY READ 2689 fd:00:320039 4 4

2: POSIX ADVISORY WRITE 2732 fd:00:931460 0 EOF

3: POSIX ADVISORY READ 2689 fd:00:320021 4 4

[root@server ~]# cat /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode

0: 00000000:0185 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7642 1 c8adbb40 3000 0 0 2 -1

1: 0100007F:0CEA 00000000:0000 0A 00000000:00000000 00:00000000 00000000 27 0 7887 1 c8adb240 3000 0 0 2 -1

2: 00000000:008B 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 8138 1 c8ada4c0 3000 0 0 2 -1

3: 00000000:006F 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7482 1 c9720040 3000 0 0 2 -1

4: 00000000:0015 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7778 1 c8adb6c0 3000 0 0 2 -1

[root@server ~]# cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode

9: 0A08A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 8108 2 c4db8d00

9: 0908A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 8106 2 c4db32c0

9: 0808A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 8104 2 c4db36c0

9: 0708A8C0:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 8102 2 c4db3ac0

9: 00000000:0089 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 8099 2 c8a04080

[root@server ~]# cat /proc/net/protocols
protocol size sockets memory press maxhdr slab module cl co di ac io in de sh ss gs se re sp bi br ha uh gp em

RAWv6 648 -1 -1 NI 0 yes ipv6 y y y n y y y n y y y y n y y y y n n

UDPv6 620 -1 0 NI 0 yes ipv6 y y y n y n y n y y y y n n y y y y n

TCPv6 1248 19 1 no 224 yes ipv6 y y y y y y y y y y y y n n y y y y y

PACKET 444 -1 -1 NI 0 no kernel n n n n n n n n n n n n n n n n n n n

UNIX 416 -1 -1 NI 0 yes kernel n n n n n n n n n n n n n n n n n n n

RAW 500 -1 -1 NI 0 yes kernel y y y n y y n n y y y y n y y y y n n

UDP 508 -1 0 NI 0 yes kernel y y y n y n y n y y y y y n y y y y n

TCP 1136 19 1 no 224 yes kernel y y y y y y y y y y y y n n y y y y y

NETLINK 404 -1 -1 NI 0 no kernel n n n n n n n n n n n n n n n n n n n

[root@server ~]# cat /proc/partitions
major minor #blocks name

8 0 31457280 sda

8 1 104391 sda1

8 2 20860402 sda2

253 0 7168000 dm-0

253 1 512000 dm-1

253 2 3891200 dm-2

253 3 2048000 dm-3

[root@server ~]# cat /proc/schedstat
version 12

timestamp 2436227

cpu0 4336 4336 7 4343 7528 307075 58464 139144 139144 192048 1771799 248611

domain0 00000001 661081 661081 0 0 0 0 0 661081 360 360 0 0 0 0 0 360 58464 58464 0 0 0 0 0 58464 0 0 0 0 0 0 0 0 0 0 0 0

[root@server ~]# cat /proc/stat
cpu 2085 0 18967 484888 2403 42 607 0

cpu0 2085 0 18967 484888 2403 42 607 0

intr 2808623 2749553 12 0 2 2 0 5 0 1 0 0 0 116 0 0 21 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 33013 0 0 0 0 0 0 0 37 0 0 0 0 0 0 0 25861 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ctxt 302029

btime 1314166387

processes 37438

procs_running 1

procs_blocked 0

[root@server ~]# cat /proc/swaps
Filename Type Size Used Priority

/dev/mapper/MYVOL-SWAPer partition 511992 0 -1

/extraswap file 102392 0 -2

[root@server ~]# cat /proc/scsi/scsi
Attached devices:

Host: scsi0 Channel: 00 Id: 00 Lun: 00

Vendor: VMware, Model: VMware Virtual S Rev: 1.0

Type: Direct-Access ANSI SCSI revision: 02

[root@server ~]# cat /proc/tty/drivers
/dev/tty /dev/tty 5 0 system:/dev/tty

/dev/console /dev/console 5 1 system:console

/dev/ptmx /dev/ptmx 5 2 system

/dev/vc/0 /dev/vc/0 4 0 system:vtmaster

serial /dev/ttyS 4 64-95 serial

pty_slave /dev/pts 136 0-1048575 pty:slave

pty_master /dev/ptm 128 0-1048575 pty:master

unknown /dev/tty 4 1-63 console

[root@server ~]# cat /proc/uptime
5253.31 4967.57

[root@server ~]# cat /proc/version
Linux version 2.6.18-164.el5 (mockbuild@builder16.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 03:33:56 EDT 2009

Other that are usefulls :
[root@server ~]# cat /proc/interrupts

[root@server ~]# cat /proc/iomem

[root@server ~]# cat /proc/vmstat

[root@server ~]# vmstat -m


## Check NC status
[root@server ~]# mii-tool eth0
Twitter Bird Gadget