Extending SWAP

Swap extension in linux can be done in two ways.

a) By creating a file

b) By creating a disk

We will see the steps for both the methods.

a) By creating a disk

a.1 Extend SWAP with the existing LVM

To extend an LVM2 swap logical volume (assuming /dev/VolGroup00/LogVol01 is the volume you want to extend):

(i) Disable swap for the associated logical volume:

# swapoff -v /dev/VolGroup00/LogVol01

(ii) Resize the LVM2 logical volume by 256 MB:

# lvresize /dev/VolGroup00/LogVol01 -L +256M

(iii) Format the new swap space:

# mkswap /dev/VolGroup00/LogVol01

(iv) Enable the extended logical volume:

# swapon -va

(v) Test that the logical volume has been extended properly:

# cat /proc/swaps # free

a.2  Extend SWAP by Creating a new LVM

To add a swap volume group (assuming /dev/VolGroup00/LogVol02 is the swap volume you want to add):

(i) Create the LVM2 logical volume of size 256 MB:

# lvm lvcreate VolGroup00 -n LogVol02 -L 256M

(ii) Format the new swap space:

# mkswap /dev/VolGroup00/LogVol02

(iii) Add the following entry to the /etc/fstab file:

/dev/VolGroup00/LogVol02   swap     swap    defaults     0 0

# swapon -va

Test that the logical volume has been extended properly. Above command will enable the extended logical volume:

# cat /proc/swaps # free

b) By creating a Swap File

Determine the size of the new swap file in megabytes and multiply by 1024 to determine the number of blocks. For example, the block size of a 64 MB swap file is 65536.

(i) dd if=/dev/zero of=/swapfile bs=1024 count=65536

Setup the swap file with the command:

(ii) mkswap /swapfile

To enable the swap file immediately but not automatically at boot time:

(iii) swapon /swapfile

To enable it at boot time, edit /etc/fstab to include the following entry:

/swapfile          swap            swap    defaults        0 0

AWK : One Line Commands

In this post we will see one line actions with awk. Moreover all the commands are explained how it works. In case of any queries please comment.

Note : For all the tasks in this post below file1 contents will be used as reference.

Reference File [test@server1 ~]# cat file1

99 1

30 2

40 3

50 4

60 5

70 6

80 7

43 8

42 9

10 10

1)  Calculate Average, Sum, Subtraction and Maximum with AWK 

Case 1) Sum the column one and coloumn two value and print the final value in coloumn 1.

[test@server1 ~]# cat file1 | awk '{sum=$1+$2; print sum}'

100

32

43

54

65

76

87

51

51

20

How it Works: In the above command we are declaring a variable named sum and adding the values of Field 1 & 2. Post that we are printing it in the every line.

Case 2) Find the sum of all values of a particular field

[test@server1 ~]# awk '{sum=sum+$1} END {print sum}' file1

525

How it Works: In the above command awk will create a variable called sum and it's intial value will be 0. Then awk will start it line by line processing, so it will take the first field of column 1 then add it with sum variable which is 0. So after adding the value of sum will be come 99 (0 +99). Now awk processes the 2nd line and its value becomes 99+30 then it continues and the value in final line will be the sum of all fields. 

what is the need of End Block. End Block will be executed in the Last and it wont loop over files. Now at the last the value of the sum Will be 545. If you print without END block as awk will process line by line it will print the value of sum on every line if END block is not used. 

[test@server1 ~]# cat file1 | awk '{sum=sum+$1} {print sum}' file1

99

129

169

219

279

349

429

472

514

525

Case 3) Find the average of all values of a field

[test@server1 ~]# awk '{sum=sum+$1} END {print sum/NR}' file1

How it Works: To find the average of the fields just divide the sum of value by Total number of records. Total number of records can be calculated by "NR" variable. So the command to calculate average looks like this

Case 4) Find the Maximum of all fields

[test@server1 ~]# awk 'BEGIN {max = 0} {if ($1>max) max=$1} END {print max}' file1

1000

[test@server1 ~]# cat file1

01 1

10 2

21 3

45 3

99 1

30 2

40 3

50 4

60 5

70 6

80 7

43 8

42 9

11 10

1000 10

23 1

How it works: Above command will find the maximum value in the respective field. Let us see how it works and what is the reason for declaring max=0 value in BEGIN block.

Initially the value of max(a variable declared by us) will be declared with value 0 then it compares with the field value of the first line (as AWK works line by line). In our case the value is 01 which is greater so value of max will be changed to 01. Then AWK moves to next line and the compares the value once if it gets the maximum value in the file the value of max won’t be changed. In our case it is thousand.

Reason behind declaring in BEGIN block is, it is executed only once but if we declare it in the ACTION block for every line the value of max will get change to 0 and the value in last line will be printed as maximum.

awk  -F"," '{ sum = $1 + $2; print $2, sum }' <filename>

awk -F":" '{print $3","$4-1}' <filename>

awk -F":" '{print $3","$4+1}' <filename>

2) Print all fields except one

(i) cat passwd | awk -F: '{$6=""; print $0}'

Instead of $6 put your field name. But it won't print FS between the fields

(ii) awk -F: '{$6=""; print $0}' OFS=":" passwd

To Print the output with FS we need to use OFS

3)  Print all fields except the last field

cat file1 | awk -F"/" '{NF--; print}' OFS="/"

4) Print nth Field Previous to last field

Prints 2nd Field from Last Field

df -h /opt/ | grep -v Filesystem | tail -n 1 | awk '{print $(NF-1)}'

Prints 3rd Field from Last Field

df -h /opt/ | grep -v Filesystem | tail -n 1 | awk '{print $(NF-3)}'

5) Print last And its Previous Field

cat /etc/passwd | awk -F":" '{print $NF-1, $NF}'

6) To Search and do Requested action

awk '/,20130723/ {print $0}'  O3SoVxmlAppSrv1_00173.txt

awk -F "," ' $3 ~ /^8006/  {print $0}'

7) system function

find /mnt/promptbase/prompts/amoeba/NightRadio_Voda_All -name SubSuccess | awk -F "/" '{print system("mkdir -p /tmp/"$7"")}'

ifconfig | grep Bcast | awk -F " " {'print $2'} | awk -F ":" {'system ("date")'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 > 20) system ("perl /lukman/blackmail.pl")'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 < 5) system ("date")'}

8) Getline function

df -h | awk -F " " '{ if (NF==1) {getline;print $4} else { print $5 }}' | awk -F"%" {'print $1'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {'print $1'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $6'}

df -Ph | grep -v Use |awk -F" " '{print $5}' | awk -F"%" '{"date" | getline d close("date"); if ($1 > 6) print d}'

9) Multiple field seperator

cat output.txt | awk -F"x" '{print $4}' | awk -F"@" '{print $1}' | sort | uniq

cat output.txt | awk -F"x|@" '{print $6","$10}'

cat output.txt | awk -F"x|@" '{print $10}' | sort | uniq | awk -F"-" '{print $1}'

cat output.txt | awk -F"x|@" '{print $10}' | sort | uniq | awk -F"-" '{print $1}' | awk '{for(i=8;i<=16;i+=3)$0=substr($0,1,i)":"substr($0,i+1);print}'

10) To Split files based on a particular field

We can use awk to split files based on a particular field. (i.e) If we have a file with some contents  and we need to segregate the files based on some field. Then we can use below command.

# awk -F\| '{print>$1}' file1

11) if Function

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 > 20) system ("perl /lukman/blackmail.pl")'}

awk -F "/" '{if (NF>3) {print $4} else {print $3}}' to.txt

df -h | grep /dev/cciss/c0d0p2 | awk -F "%" '{print $1}'|awk -F " " '{ if ( $1 > 20) printf ("%d\n",$5)}'

who | awk '!/root/{ cmd="/sbin/pkill -KILL -u " $1; system(cmd)}' OR

### warning must be run as root or via sudo ###

### Safe version :) ###

who | awk '$1 !~ /root/{ cmd="/sbin/pkill -KILL -u " $1; system(cmd)}'

Multipathing Overview

What is Device Mapper

Device Mapper is a kernel driver that provides a framework for volume management. It provides a generic way of creating mapped devices, which may be used as logical volumes. It provides the foundation for a number of higher-level technologies. In addition to LVM, Device-Mapper multipath and the dmraid command use the Device Mapper. The user interface is the dmsetup command. It forms the foundation of LVM2, software RAIDs and dm-crypt disk encryption, and offers additional features such as file system snapshots

What is Multipathing

A path is a connection between a server and the underlying storage. The path can be severed due to many reasons like faulty HBA, faulty cable etc. To avoid such single point of failures, multipathing exists. Multipathing ensures that the system uses multiple physical paths to provide redundancy and increased throughput. There are many vendor specific multipathing implementations like EMC’s powerpath and Symantecs VxDMP.

What is Device Mapper multipath

Device Mapper Multipathing (or DM-multipathing) is a Linux native multipath tool, which allows you to configure multiple I/O paths between server nodes and storage arrays into a single device. These I/O paths are physical SAN connections that can include separate cables, switches, and controllers. Multipathing aggregates the I/O paths, creating a new device that consists of the aggregated paths. Regardless of the vendor hardware in use, device mapper creates a block device under /dev/mapper/ for each LUN attached to the system.

Device Mapper components

The important components of Device Mapper multipathing are :

a) dm-multipath - kernel module responsible for making routing decisions under normal/failure conditions

b) multipath Command used for viewing/listing multipath devices and for initial configuration

c) multipathd - daemon that moitors path, marks failed paths, reactivates restored paths, adds/removes device files as needed.

d) Kpartx command used to create device mapper entries for partitions on multipathed LUN. It is invoked automatically when multipath command is used.

How to verify if DMMP is installed and configured

1. Check whether device-mapper is installed.

# rpm -qa |grep device-mapper

device-mapper-1.02.39-1.el5

device-mapper-multipath-0.4.7-34.el5

device-mapper-1.02.39-1.el5

device-mapper-event-1.02.39-1.el5

2. Check that the following device mapper modules are loaded.

# lsmod |grep dm_multipath

dm_multipath           56921  2 dm_round_robin

scsi_dh                42177  2 scsi_dh_rdac,dm_multipath

dm_mod                101649  11 dm_mirror,dm_multipath,dm_raid45,dm_log

3. If above conditions are met, check whether the file /etc/multipath.conf is configured. Make sure the lines in bold are commented out in order to enable device mapper.

# This is a basic configuration file with some examples, for device mapper multipath

......

# Blacklist all devices by default. Remove this to enable multipathing

# on the default devices.

#blacklist {

#        devnode "*"

#}

......

4. Check whether multipathd is running.

# /etc/init.d/multipathd status

 "multipathd (pid  11405) is running..."

5. If yes, check any devices listed using the command below.

# multipath -v2 or #multipath -ll

mpath15 (3600a0b8000473abc0000bafc52fac127) dm-14 SUN,STK6580_6780

[size=10G][features=0][hwhandler=0][rw]

_ round-robin 0 [prio=1][enabled]

 _ 8:0:0:2  sds 65:32 [active][ready]

_ round-robin 0 [prio=0][enabled]

 _ 9:0:0:2  sdu 65:64 [active][faulty]

mpath13 (3600a0b8000473abc0000bb74530aa7da) dm-12 SUN,STK6580_6780

[size=931G][features=0][hwhandler=0][rw]

_ round-robin 0 [prio=1][enabled]

 _ 9:0:0:0  sdp 8:240 [active][ready]

_ round-robin 0 [prio=0][enabled]

 _ 8:0:0:0  sdo 8:224 [active][faulty]

If all the above steps succeed, the system is configured for DMMP.

Multipathing Configuration

Before starting to configure the multipathing, make sure the device-mapper-multipath package is installed. If not installed, install it using yum :

# yum -y install device-mapper-multipath

The device mapper multipathing uses the configuration file /etc/multipath.conf for the configuration. If you make any changes to this file the multipath command must be run in order to reconfigure the multipathed devices. The easiest way to create this file is to use the mpathconf utility. If there is an existing configuration file mpathconf will edit it, if no such file exists it will copy /usr/share/doc/device-mapper-multipath-*/multipath.conf.

#mpathconf --enable --with_multipathd y --with_chkconfig y

Verifying Configuration

The multipath command can be used to verify the multipath configuration. To list the information about multipathed devices :

#multipath -ll

mpath0 (3600a0b8000473abc0000bafc52fac127) dm-14 SUN,STK6580_6780

[size=10G][features=0][hwhandler=0][rw]

_ round-robin 0 [prio=1][enabled]

 _ 8:0:0:2  sds 65:32 [active][ready]

_ round-robin 0 [prio=0][enabled]

 _ 9:0:0:2  sdu 65:64 [active][faulty]

The output shows a multipathed LUN, mpath0. The number following it is the LUN’s WWID. The status active/ready indicates that the path is ready for I/O. If the path is showing faulty/failed then it needs to be repaired before using it for I/O. After the configuration is completed, we can start the multipathd persistently :

# /etc/init.d/multipathd start

# chkconfig multipathd on

User Friendly Device Names

In order to troubleshoot efficiently, device-mapper can be configured to have human readable, user friendly device names under /dev/mapper instead of using the WWIDs. The user friendly names like /dev/mapper/mpath0 can be created by enabling the user_friendly_names option in /etc/multipath.conf file :

defaults {

    user_friendly_names yes

}

You can also control the name for a particular LUN by using the alias option :

multipaths {

    multipath {

            wwid     3600a0b8000473abc0000bafc52fac127 

            alias    mdisk001

              }

}

Removing Multipath

After removing the all the paths for a multipathed device, run the below command to remove the multipath device completely.

# multipath -f [device]

To flush all the multipathed device after stopping the multipahtd daemon :

# multipath -F

Difference between Block and Character device

When a program reads or writes data from a file, the requests go to a kernel driver. If the file is a regular file, the data is handled by a filesystem driver and it is typically stored in zones on a disk or other storage media, and the data that is read from a file is what was previously written in that place. There are other file types for which different things happen.

When data is read or written to a device file, the request is handled by the driver for that device. Each device file has an associated number which identifies the driver to use. What the device does with the data is its own business.

Block devices (also called block special files) usually behave a lot like ordinary files: they are an array of bytes, and the value that is read at a given location is the value that was last written there. Data from block device can be cached in memory and read back from cache; writes can be buffered. Block devices are normally seekable (i.e. there is a notion of position inside the file which the application can change). The name “block device” comes from the fact that the corresponding hardware typically reads and writes a whole block at a time (e.g. a sector on a hard disk).

Character devices (also called character special files) behave like pipes, serial ports, etc. Writing or reading to them is an immediate action. What the driver does with the data is its own business. Writing a byte to a character device might cause it to be displayed on screen, output on a serial port, converted into a sound, ... Reading a byte from a device might cause the serial port to wait for input, might return a random byte (/dev/urandom), ... The name “character device” comes from the fact that each character is handled individually.

NSLOOKUP, DIG and HOST Command

a) nslookup

This is deprecated by Bind developers. But still used to resolve Hostname into IP address.

b) host  

Host is also used to convert names into IP addresses and vice versa. Without arguments it will give the list of options.

Sample output of host command : ansim0.example.local has address 192.168.1.10

c) dig

Dig and host commands basically output the same info. 

dig +short www.yahoo.com

host www.yahoo.com

dig www.yahoo.com

host -d www.yahoo.com

basically prints the same output. The dig command includes some timing stats and the actual query that will be performed.

nslookup -debug www.yahoo.com

Although the output is in a different format, the information that is printed out is basically the same as dig.

If you want the full enchilada...

nslookup -d2 www.yahoo.com

...or

dig -d2 www.yahoo.com