Removing Duplicates Using AWK and How it Works.

<script data-ad-client="ca-pub-7841181112240136" async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>

As a normal method in unix we need to use sort and then uniq to remove the duplicates from the file. Because without sort, it wont give correct unique values.

Instead of that we can use awk command to remove the duplicates in the file. For this we need to use awk associate array.

Short notes about AWK associative array

Unlike regular arrays in AWK associative arrays the indexes need not to be continuous set of number; you can use either string or number as an array index. Also, there is no need to declare the size of an array in advance – arrays can expand/shrink at runtime.

Its syntax is : array_name[index] = value

Example

Cat tes 

File Contents

Cat test

1

2

3

1

1

3

3a

3a

5

Command to Use: cat tes | awk '!seen[$0]++'  

How it Works

Seen[$0] - Uses the current line as the key to the array a. In our case 1 so array index will become 1 2 3 3a and 5. As we can't have same array index.

Note : Seen is an arbitrary word, it can be any words like a, b or any strings.

if seen[1] is never reference before then a[1] evalutes to empty string as awk will crate empty if it was not initialized before. IN this zero is false. If we negate then we will get true result. if it is non-zero (true) then we will get false result.

uses the current line $0 as key to the array a, taking the value stored there. If this particular key was never referenced before, a[$0] evaluates to the empty string.

!seen[$0]

The ! negates the value from before. If it was empty or zero (false), we now have a true result. If it was non-zero (true), we have a false result. If the whole expression evaluated to true, meaning that a[$0] was not set to begin with, the whole line is printed as the default action.

Also, regardless of the old value, the post-increment operator adds one to a[$0], so the next the same value in the array is accessed, it will be positive and the whole condition will fail.

Below is the actual way how awk with expression works

    awk 'expression' file Is actually a short hand of: awk 'expression {print $0}' file

Whenever the a test with no associated action is true, the default action is triggered. The default action is the equivalent of { print } or { print $0 }, which prints the current record, which for all accounts and purposes in this example is the current unmodified line of input.

AWK match and substr function

MATCH Function

match(string, regexp [, array])

Search string for the longest, leftmost substring matched by the regular expression and return the character position (index) at which that substring begins (one, if it starts at the beginning of string). If no match is found, return zero.

The regexp argument may be either a regexp constant (/…/) or a string constant ("…"). 

Note: The order of the first two arguments is the opposite of most other string functions that work with regular expressions, such as sub() and gsub(). It might help to remember that for match(), the order is the same as for the ‘~’ operator: ‘string ~ regexp’.

The match() function sets the predefined variable RSTART to the index. It also sets the predefined variable RLENGTH to the length in characters of the matched substring. If no match is found, RSTART is set to zero, and RLENGTH to -1.

SUBSTR Function

substr(string, start [, length ])

Return a length-character-long substring of string, starting at character number start. The first character of a string is character number one.49 For example, substr("washington", 5, 3) returns "ing".

If length is not present, substr() returns the whole suffix of string that begins at character number start. For example, substr("washington", 5) returns "ington". The whole suffix is also returned if length is greater than the number of characters remaining in the string, counting from character start.

example:

AB: 20190131  13 J-1|19:30:00.000000000 18:06:00.000000000 123466  50 @TEST . "" 1234 - I . ".." "" "" "TEST TEXT 1" "TEXT 2: SAMPLE TEXT I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find f.==Required file.csv.gz FIELD*SERVER-TIME*05:29:51.981378000" "" NoTime

       
Command : awk 'match($0,/==.*?.csv.gz/){print $3","substr($0, RSTART+2, RLENGTH-2)}'  Sample file


Output  : 13,Required file.csv.gz       
 

AWK : In Built Variables and Functions

NEXT and GETLINE 

The next statement forces awk to immediately stop processing the current record and go on to the next record. This means that no further rules are executed for the current record, and the rest of the current rule’s action isn’t executed.

Contrast this with the effect of the getline function. That also causes awk to read the next record immediately, but it does not alter the flow of control in any way (i.e., the rest of the current action executes with a new input record).

NR and FNR

NR - stores the total number of input records read so far, regardless of how many files have been read. The value of NR starts at 1 and always increases until the program terminates. 

FNR - stores the number of records read from the current file being processed. The value of FNR starts at 1, increases until the end of the current file is reached, then is set again to 1 as soon as the first line of the next file is read, and so on. 

AWK Two File Processing

When Processing multiple files awk reads each file sequentially, one after another, in the order they are specified on the command line. 

$ awk 'NR == FNR { # some actions; next} # other condition {# other actions}' file1.txt file2.txt

How it Works

So, the condition NR == FNR is only true while awk is reading the first file. Thus, in the program above, the actions indicated by # some actions are executed when awk is reading the first file; the actions indicated by # other actions are executed when awk is reading the second file, if the condition in # other condition is met. 

The next at the end of the first action block is needed to prevent the condition in # other condition from being evaluated, and the actions in # other actions from being executed, while awk is reading the first file.

Probably, it all becomes much clearer with some examples. There are really many problems that involve two files that can be solved using this technique. Let's look at this:

# prints lines that are both in file1.txt and file2.txt (intersection)

$ awk 'NR == FNR{a[$0];next} $0 in a' file1.txt file2.txt

Here we see another typical idiom: a[$0] alone has the only purpose of creating the array element indexed by $0, even if we don't assign any value to it. During the pass over the first file, all the lines seen are remembered as indexes of the array a. The pass over the second file just needs to check whether each line being read exists as an index in the array a (that's what the condition $0 in a does). If the condition is true, the line being read from file2.txt is printed (as we already know). In a very similar way, we can easily write the code to print the lines that appear in only one of the two files:

# prints lines that are only in file1.txt and not in file2.txt

$ awk 'NR == FNR{a[$0];next} !($0 in a)' file2.txt file1.txt

Note the order of the arguments. file2.txt is given first. To print lines that are only in file2.txt and not in file1.txt, just reverse the order of the arguments.

AWK : One Line Commands

In this post we will see one line actions with awk. Moreover all the commands are explained how it works. In case of any queries please comment.

Note : For all the tasks in this post below file1 contents will be used as reference.

Reference File [test@server1 ~]# cat file1

99 1

30 2

40 3

50 4

60 5

70 6

80 7

43 8

42 9

10 10

1)  Calculate Average, Sum, Subtraction and Maximum with AWK 

Case 1) Sum the column one and coloumn two value and print the final value in coloumn 1.

[test@server1 ~]# cat file1 | awk '{sum=$1+$2; print sum}'

100

32

43

54

65

76

87

51

51

20

How it Works: In the above command we are declaring a variable named sum and adding the values of Field 1 & 2. Post that we are printing it in the every line.

Case 2) Find the sum of all values of a particular field

[test@server1 ~]# awk '{sum=sum+$1} END {print sum}' file1

525

How it Works: In the above command awk will create a variable called sum and it's intial value will be 0. Then awk will start it line by line processing, so it will take the first field of column 1 then add it with sum variable which is 0. So after adding the value of sum will be come 99 (0 +99). Now awk processes the 2nd line and its value becomes 99+30 then it continues and the value in final line will be the sum of all fields. 

what is the need of End Block. End Block will be executed in the Last and it wont loop over files. Now at the last the value of the sum Will be 545. If you print without END block as awk will process line by line it will print the value of sum on every line if END block is not used. 

[test@server1 ~]# cat file1 | awk '{sum=sum+$1} {print sum}' file1

99

129

169

219

279

349

429

472

514

525

Case 3) Find the average of all values of a field

[test@server1 ~]# awk '{sum=sum+$1} END {print sum/NR}' file1

How it Works: To find the average of the fields just divide the sum of value by Total number of records. Total number of records can be calculated by "NR" variable. So the command to calculate average looks like this

Case 4) Find the Maximum of all fields

[test@server1 ~]# awk 'BEGIN {max = 0} {if ($1>max) max=$1} END {print max}' file1

1000

[test@server1 ~]# cat file1

01 1

10 2

21 3

45 3

99 1

30 2

40 3

50 4

60 5

70 6

80 7

43 8

42 9

11 10

1000 10

23 1

How it works: Above command will find the maximum value in the respective field. Let us see how it works and what is the reason for declaring max=0 value in BEGIN block.

Initially the value of max(a variable declared by us) will be declared with value 0 then it compares with the field value of the first line (as AWK works line by line). In our case the value is 01 which is greater so value of max will be changed to 01. Then AWK moves to next line and the compares the value once if it gets the maximum value in the file the value of max won’t be changed. In our case it is thousand.

Reason behind declaring in BEGIN block is, it is executed only once but if we declare it in the ACTION block for every line the value of max will get change to 0 and the value in last line will be printed as maximum.

awk  -F"," '{ sum = $1 + $2; print $2, sum }' <filename>

awk -F":" '{print $3","$4-1}' <filename>

awk -F":" '{print $3","$4+1}' <filename>

2) Print all fields except one

(i) cat passwd | awk -F: '{$6=""; print $0}'

Instead of $6 put your field name. But it won't print FS between the fields

(ii) awk -F: '{$6=""; print $0}' OFS=":" passwd

To Print the output with FS we need to use OFS

3)  Print all fields except the last field

cat file1 | awk -F"/" '{NF--; print}' OFS="/"

4) Print nth Field Previous to last field

Prints 2nd Field from Last Field

df -h /opt/ | grep -v Filesystem | tail -n 1 | awk '{print $(NF-1)}'

Prints 3rd Field from Last Field

df -h /opt/ | grep -v Filesystem | tail -n 1 | awk '{print $(NF-3)}'

5) Print last And its Previous Field

cat /etc/passwd | awk -F":" '{print $NF-1, $NF}'

6) To Search and do Requested action

awk '/,20130723/ {print $0}'  O3SoVxmlAppSrv1_00173.txt

awk -F "," ' $3 ~ /^8006/  {print $0}'

7) system function

find /mnt/promptbase/prompts/amoeba/NightRadio_Voda_All -name SubSuccess | awk -F "/" '{print system("mkdir -p /tmp/"$7"")}'

ifconfig | grep Bcast | awk -F " " {'print $2'} | awk -F ":" {'system ("date")'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 > 20) system ("perl /lukman/blackmail.pl")'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 < 5) system ("date")'}

8) Getline function

df -h | awk -F " " '{ if (NF==1) {getline;print $4} else { print $5 }}' | awk -F"%" {'print $1'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {'print $1'}

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $6'}

df -Ph | grep -v Use |awk -F" " '{print $5}' | awk -F"%" '{"date" | getline d close("date"); if ($1 > 6) print d}'

9) Multiple field seperator

cat output.txt | awk -F"x" '{print $4}' | awk -F"@" '{print $1}' | sort | uniq

cat output.txt | awk -F"x|@" '{print $6","$10}'

cat output.txt | awk -F"x|@" '{print $10}' | sort | uniq | awk -F"-" '{print $1}'

cat output.txt | awk -F"x|@" '{print $10}' | sort | uniq | awk -F"-" '{print $1}' | awk '{for(i=8;i<=16;i+=3)$0=substr($0,1,i)":"substr($0,i+1);print}'

10) To Split files based on a particular field

We can use awk to split files based on a particular field. (i.e) If we have a file with some contents  and we need to segregate the files based on some field. Then we can use below command.

# awk -F\| '{print>$1}' file1

11) if Function

df -h | grep /dev/cciss/c0d0p2 | awk -F " " {'print $5'} | awk -F "%" {' if ($1 > 20) system ("perl /lukman/blackmail.pl")'}

awk -F "/" '{if (NF>3) {print $4} else {print $3}}' to.txt

df -h | grep /dev/cciss/c0d0p2 | awk -F "%" '{print $1}'|awk -F " " '{ if ( $1 > 20) printf ("%d\n",$5)}'

who | awk '!/root/{ cmd="/sbin/pkill -KILL -u " $1; system(cmd)}' OR

### warning must be run as root or via sudo ###

### Safe version :) ###

who | awk '$1 !~ /root/{ cmd="/sbin/pkill -KILL -u " $1; system(cmd)}'

AWK Syntax and Examples

1) What is AWK

awk is command used for processing files. With the help of awk we can print a particular field of a file or command output. Syntax of awk command is,

Syntax:  awk 'BEGIN {awk-commands} {Action} END {awk-commands}'

To understand more about awk will see some examples.

2) Print function

As mentioned above, with the help of awk we can print a particular field of a file or command.

eg 2. To print the first field of a file. (considering field separator as space )

# cat /etc/services | awk -F" " '{print $1}' 

This will print the first field of the file named services. Where -F" " means field separator is space. We can mention other values also to field separator like F"," (Coma as the field separator). 

Note : If you don't mention any field separator awk will consider space as default field separator. 

3) Inbuilt  Variable's

AWK has some inbuilt variables. Here is some of the list.

FILENAME - It represents the current file name.

FS

NR

FNR

OFS

NF - Number of fields.

When awk reads from the multiple input file, awk NR variable will give the total number of records relative to all the input file. Awk FNR will give you number of records for each input file.

3) sub and gsub function [ for Search and replace ]

In awk we have syntax called sub and gsub function to search for a partial string and perform the action. Below is the detailed explanation of gsub and gsub with examples.

gsub stands for global substitution. It replaces every occurrence of sub with regex. The third parameter is optional. If it is omitted, then $0 is used.

3.a) sub(regexp, replacement [, target])

The 'sub' function alters the value of TARGET.  It searches this value, which should be a string, for the leftmost substring matched by the regular expression, REGEXP, extending this match as far as possible.  Then the entire string      is changed by replacing the matched text with REPLACEMENT. The modified string becomes the new value of TARGET. This function is peculiar because TARGET is not simply used to compute a value, and not just any expression will do: it      must be a variable, field or array reference, so that `sub' can store a modified value there.  If this argument is omitted, then the default is to use and alter `$0'.

eg 3.a):

#echo "water, water, everywhere" | awk '{sub(/at/,"ith")}1'

output = "wither, water, everywhere" - 

Sub will only replace the leftmost occurrence of the regex with replacement(in this case `at'   with `ith') . 'sub' function returns the number of substitutions made (either one or zero).

   

3.b) : Now let's see what happens if the special character `&' appears in REPLACEMENT, it stands for the precise substring that matches the REGEXP. Below is the example for & string.

eg 3.b):

# echo "tommy,tom,water,tomboy" | awk '{ sub(/tom/, "& and his wife"); print }'

tom and his wifemy,tom,water,tomboy

Awk append the regexp instead of replacing it if we use & symbol. and this changes only the first occurance.

Here is another example:

          awk 'BEGIN {

                  str = "daabaaa"

                  sub(/a*/, "c&c", str)

                  print str

          }'

prints `dcaacbaaa'.  This show how `&' can represent a non-constant string, and also illustrates the leftmost rule.

3.c) Turning off Special character's 

Special character's can be turned off by putting a backslash before it in the string. As usual, to insert one backslash in the string, you must write two backslashes. Therefore, write '\\&' in string to include a literal `&' in the replacement.

eg 3.c): Here is how to replace the first `|' on each line with an `&':

          awk '{ sub(/\|/, "\\&"); print }'

Note : as mentioned above, the third argument to `sub' must be an value.  Some versions of `awk' allow the third argument to be an expression which is not an value.  In such a case, `sub' would still search for the pattern and return 0 or 1. 

4) Awk gsub function

List of all examples for gsub

[user@test ~]$ echo "water, water, everywhere" | awk '{gsub(/at/,"&ith");print}'

watither, watither, everywhere

[user@test ~]$ echo "water, water, everywhere" | awk '{gsub(/at/,"ith");print}'

wither, wither, everywhere

[user@test ~]$ echo "water, water, everywhere" | awk '{gsub(/at/,"bd\\&ith");print}'

wbd&ither, wbd&ither, everywhere

[user@test ~]$ echo "water, water, everywhere" | awk '{sub(/at/,"ith");print}'

wither, water, everywhere

[user@test ~]$ echo "water, water, everywhere" | awk '{gsub(/at/,"bd&ith");print}'

wbdatither, wbdatither, everywhere

4.a) Print the count of matched regex then use print before gsub function

eg 4.a):  To search in a file and print the count with line number. 

[user@test ~]$ echo "water, wattter, everywhere" | awk  -F, '{print gsub(/at/,"")}'

2

in the above example "at" occurs twice , so the count is printed as 2.

eg 4.a.2):

[user@test awkregex]$ grep -i NFS passwd_M

rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin

nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

[user@test awkregex]$ grep -i NFS passwd_M | awk -F:  '{print NR "\t" gsub(/rpc/,"")}'

1       1

2       0

in the above example NR is line number and \t represent print in tab space. rpc occurs one time and none in 1st and 2nd line respectively and the same count has been printed.

4.b) to search and print the count on particular field

[user@test awkrgex]$ grep -i NFS passwd_M

rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin

nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

to search in field 1 only we are creating a variable name col and using it in regex syntax and in similar way for column 2

[user@test awkrgex]$ grep -i NFS passwd_M |  awk -F: -v col=1 '{print NR "\t" gsub(/rpc/,"",$col)}'

1       1

2       0

[user@test awkrgex]$ grep -i NFS passwd_M |  awk -F: -v col=2 '{print NR "\t" gsub(/rpc/,"",$col)}'

1       0

2       0

5) Search and print  the lines which matches the recommended count

With the help of awk also we can do search and print lines like sed. Here fldcount is the sample file name. and we will print the lines which has 2nd field length 15 or 16 digit,and 1st field length 12 or 15 digit.

[user@test awkregex]$ cat fldcount

123710337783,351898014413150,123028040249634

123710337785,352934028758390,123028040109275

000123710337785,352934028758390,123028040109275

3710337785,352934028758390,123028040109275

[user@test awkregex]$ cat fldcount | awk -F, '{ if (((length($2) == 15 ) || length($2) == 16) && (length($1) == 12 && length($3) == 15)) print }'

123710337783,351898014413150,123028040249634

123710337785,352934028758390,123028040109275

6) awk printf function example

Printf is similar to #cat passwd

#cat passwd

root:x:0:0:root:/root:/bin/bash

bin:x:1:1:bin:/bin:/sbin/nologin

daemon:x:2:2:daemon:/sbin:/sbin/nologin

adm:x:3:4:adm:/adm:/sbin/nologin

lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin

# awk -F":" '{printf("username=%s,userdid= %d\n", $1, $3)}' passwd | head -n 5

username=root,userdid= 0

username= bin,userdid= 1

username=daemon,userdid= 2

username=adm,userdid= 3

username=lp,userdid= 4

7) next and getline statement

The next statement forces awk to immediately stop processing the current record and go on to the next record. This means that no further rules are executed for the current record, and the rest of the current rule’s action isn’t executed.

Contrast this with the effect of the getline function (see Getline). That also causes awk to read the next record immediately, but it does not alter the flow of control in any way (i.e., the rest of the current action executes with a new input record).

Is there any way to cat a file that has something like:

field1,field2,field number 3,field4,field5

field1,field2,field3,field4,field5

(Some fields have spaces, some fields do not)

I want to print using awk $1 and $3, but only lines that do not have a space in field 3.

I don't know if there is any way to make awk print something only if it starts with , and ends with ,

Lines don't have the space in Field 3

awk -F, '{n=split($3,a," ");if(n==1){print $1,$3}}' filename