AWK : One Line Commands

In this post we will see one line actions with awk. Moreover all the commands are explained how it works. In case of any queries please comment.

Note : For all the tasks in this post below file1 contents will be used as reference.

Reference File [test@server1 ~]# cat file1

99 1

30 2

40 3

50 4

60 5

70 6

80 7

43 8

42 9

10 10

1)  Calculate Average, Sum, Subtraction and Maximum with AWK 

Case 1) Sum the column one and coloumn two value and print the final value in coloumn 1.

[test@server1 ~]# cat file1 | awk '{sum=$1+$2; print sum}'

100

32

43

54

65

76

87

51

51

20

How it Works: In the above command we are declaring a variable named sum and adding the values of Field 1 & 2. Post that we are printing it in the every line.

Case 2) Find the sum of all values of a particular field

[test@server1 ~]# awk '{sum=sum+$1} END {print sum}' file1

525

How it Works: In the above command awk will create a variable called sum and it's intial value will be 0. Then awk will start it line by line processing, so it will take the first field of column 1 then add it with sum variable which is 0. So after adding the value of sum will be come 99 (0 +99). Now awk processes the 2nd line and its value becomes 99+30 then it continues and the value in final line will be the sum of all fields. 

what is the need of End Block. End Block will be executed in the Last and it wont loop over files. Now at the last the value of the sum Will be 545. If you print without END block as awk will process line by line it will print the value of sum on every line if END block is not used. 

[test@server1 ~]# cat file1 | awk '{sum=sum+$1} {print sum}' file1

99

129

169

219

279

349

429

472

514

525

Case 3) Find the average of all values of a field

[test@server1 ~]# awk '{sum=sum+$1} END {print sum/NR}' file1

How it Works: To find the average of the fields just divide the sum of value by Total number of records. Total number of records can be calculated by "NR" variable. So the command to calculate average looks like this

Case 4) Find the Maximum of all fields

[test@server1 ~]# awk 'BEGIN {max = 0} {if ($1>max) max=$1} END {print max}' file1

1000

[test@server1 ~]# cat file1

01 1

10 2

21 3

45 3

99 1

30 2

40 3

50 4

60 5

70 6

80 7

43 8

42 9

11 10

1000 10

23 1

How it works: Above command will find the maximum value in the respective field. Let us see how it works and what is the reason for declaring max=0 value in BEGIN block.

Initially the value of max(a variable declared by us) will be declared with value 0 then it compares with the field value of the first line (as AWK works line by line). In our case the value is 01 which is greater so value of max will be changed to 01. Then AWK moves to next line and the compares the value once if it gets the maximum value in the file the value of max won’t be changed. In our case it is thousand.

Reason behind declaring in BEGIN block is, it is executed only once but if we declare it in the ACTION block for every line the value of max will get change to 0 and the value in last line will be printed as maximum.

awk  -F"," '{ sum = $1 + $2; print $2, sum }' <filename>

awk -F":" '{print $3","$4-1}' <filename>

awk -F":" '{print $3","$4+1}' <filename>

2) Print all fields except one

(i) cat passwd | awk -F: '{$6=""; print $0}'

Instead of $6 put your field name. But it won't print FS between the fields

(ii) awk -F: '{$6=""; print $0}' OFS=":" passwd

To Print the output with FS we need to use OFS

3)  Print all fields except the last field

cat file1 | awk -F"/" '{NF--; print}' OFS="/"

4) Print nth Field Previous to last field

Prints 2nd Field from Last Field

df -h /opt/ | grep -v Filesystem | tail -n 1 | awk '{print $(NF-1)}'

Prints 3rd Field from Last Field

df -h /opt/ | grep -v Filesystem | tail -n 1 | awk '{print $(NF-3)}'

5) Print last And its Previous Field

cat /etc/passwd | awk -F":" '{print $NF-1, $NF}'

6) To Search and do Requested action

awk '/,20130723/ {print $0}'  O3SoVxmlAppSrv1_00173.txt

awk -F "," ' $3 ~ /^8006/  {print $0}'

7) system function

find /mnt/promptbase/prompts/amoeba/NightRadio_Voda_All -name SubSuccess | awk -F "/" '{print system("mkdir -p /tmp/"$7"")}'

ifconfig | grep Bcast | awk -F " " {'print $2'} | awk -F ":" {'system ("date")'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 > 20) system ("perl /lukman/blackmail.pl")'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 < 5) system ("date")'}

8) Getline function

df -h | awk -F " " '{ if (NF==1) {getline;print $4} else { print $5 }}' | awk -F"%" {'print $1'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {'print $1'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $6'}

df -Ph | grep -v Use |awk -F" " '{print $5}' | awk -F"%" '{"date" | getline d close("date"); if ($1 > 6) print d}'

9) Multiple field seperator

cat output.txt | awk -F"x" '{print $4}' | awk -F"@" '{print $1}' | sort | uniq

cat output.txt | awk -F"x|@" '{print $6","$10}'

cat output.txt | awk -F"x|@" '{print $10}' | sort | uniq | awk -F"-" '{print $1}'

cat output.txt | awk -F"x|@" '{print $10}' | sort | uniq | awk -F"-" '{print $1}' | awk '{for(i=8;i<=16;i+=3)$0=substr($0,1,i)":"substr($0,i+1);print}'

10) To Split files based on a particular field

We can use awk to split files based on a particular field. (i.e) If we have a file with some contents  and we need to segregate the files based on some field. Then we can use below command.

# awk -F\| '{print>$1}' file1

11) if Function

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 > 20) system ("perl /lukman/blackmail.pl")'}

awk -F "/" '{if (NF>3) {print $4} else {print $3}}' to.txt

df -h | grep /dev/cciss/c0d0p2 | awk -F "%" '{print $1}'|awk -F " " '{ if ( $1 > 20) printf ("%d\n",$5)}'

who | awk '!/root/{ cmd="/sbin/pkill -KILL -u " $1; system(cmd)}' OR

### warning must be run as root or via sudo ###

### Safe version :) ###

who | awk '$1 !~ /root/{ cmd="/sbin/pkill -KILL -u " $1; system(cmd)}'