About BASH getopts

getopts is the bash version of another system tool, getopt. Notice that the bash command has an s at the end, to differentiate it from the system command.

Advantages

• It can handle things like -xvf filename in the expected Unix way, automatically.
• It makes sure options are parsed like any standard command (lowest common denominator), avoiding surprises.
• Disadvantages
• It cannot handle options with optional arguments à la GNU.
• Options are coded in at least 2, probably 3 places -- in the call to getopts, in the case statement that processes them, and in the help/usage message that documents them.
• Note that getopts is neither able to parse GNU-style long options (--myoption) nor XF86-style long options (-myoption). So, when you want to parse command line arguments in a

Syntax

getopts optstring optname [ arg ]

Usage

- if a character is followed by a colon, the option is expected to have an argument, which should be separated from it by whitespace. The colon (‘:’) and question mark (‘?’) may not be used as option characters

Note

There are two reserved characters which cannot be used as options: the colon (":") and the question mark ("?").

How it Works

getopts processes the positional parameters of the parent command. In bash, this is stored in the shell variable "$@". So, if you run this command:

mycmd -a argument1 -b argument2

During the time that mycmd is running, the variable $@ contains the string "-a argument1 -b argument2". You can use getopts to parse this string for options and arguments.

Every time you run getopts, it looks for one of the options defined in optstring. If it finds one, it places the option letter in a variable named optname. If the option does not match those defined in optstring, getopts sets variable optname to a question mark ("?").

If the option is expecting an argument, getopts gets that argument, and places it in $OPTARG. If an expected argument is not found, the variable optname is set to a colon (":"). Getopts then increments the positional index, $OPTIND, that indicates the next option to be processed.

The special option of two dashes ("--") is interpreted by getopts as the end of options.

getopts is designed to run multiple times in your script, in a loop, for example. It processes one option per loop iteration. When there are no more options to be processed, getopts returns false, which automatically terminates a while loop. For this reason, getopts and while are frequently used together.

Specifying the optstring

optstring is a string which defines what options and arguments getopts look for. For instance, in this call to getopts:
getopts "apZ" optname
The options expected by getopts are -a, -p, and -Z, with no arguments. These options can be combined in any order as -aZ, -pa, -Zap, etc.

Let's say that you'd like the -a and -Z options to take arguments. You can specify this by putting a colon (":") after that option in optstring. For example:
getopts "a:pZ:" optname

Now you can specify arguments to the -a and -Z options such as -a argument1 -pZ argument2. The -p option cannot take arguments, because there is no colon after the p in optstring.

Error checking

By default, getopts will report a verbose error if it finds an unknown option or a misplaced argument. It also sets the value of optname to a question mark ("?"). It does not assign a value to $OPTARG.

If the option is valid but an expected argument is not found, optname is set to "?", $OPTARG is unset, and a verbose error message is printed.
Silent error checking

However, if you put a colon at the beginning of the optstring, getopts runs in "silent error checking mode." It will not report any verbose errors about options or arguments, and you need to perform error checking in your script.

In silent mode, if an option is unexpected, getopts sets optname to "?" and $OPTARG to the unknown option character.

If the option is OK but an expected argument is not found, optname is set to a colon (":") and $OPTARG is set to the unknown option character.
Specifying custom arguments
You will usually want getopts to process the arguments in $@, but in some cases, you may want to manually provide arguments for getopts to parse. If so, you can specify these args as the final argument of the getopts command.

Examples

Here is a bash script using getopts. The script prints a greeting, with an optional name, a variable number of times. It takes two possible options: -n NAME and -t TIMES.

removes n strings from the positional parameters list. Thus shift $((OPTIND-1)) removes all the options that have been parsed by getopts from the parameters list, and so after that point, $1 will refer to the first non-option argument passed to the script.
Update

As mikeserv mentions in the comment, shift $((OPTIND-1)) can be unsafe. To prevent unwanted word-splitting etc, all parameter expansions should be double-quoted. So the safe form for the command is

Note that $OPTIND is a shell variable managed by the getopts builtin command of the shell. When the shell starts, $OPTIND equals 1 and is incremented every time a getopts call discoveres options or option arguments.

he exact use of shift $(($OPTIND-1)) is to skip the options parsed and standardized by getopts (i.e. -uvh will become -u -v -h), so by doing so, you will have your first "non-option" argument in the $0 position of the argument array.

shift "$((OPTIND-1))"
getopts is used by shell scripts to parse positional parameters. optstring contains the option characters to be recognized; if a character is followed by a colon, the option is expected to have an argument, which should be separated from it by whitespace. The colon (‘:’) and question mark (‘?’) may not be used as option characters. Each time it is invoked, getopts places the next option in the shell variable name, initializing name if it does not exist, and the index of the next argument to be processed into the variable OPTIND. OPTIND is initialized to 1 each time the shell or a shell script is invoked. When an option requires an argument, getopts places that argument into the variable OPTARG. The shell does not reset OPTIND automatically; it must be manually reset between multiple calls to getopts within the same shell invocation if a new set of parameters is to be used.

When the end of options is encountered, getopts exits with a return value greater than zero. OPTIND is set to the index of the first non-option argument, and name is set to ‘?’.
shift n

removes n strings from the positional parameters list. Thus shift $((OPTIND-1)) removes all the options that have been parsed by getopts from the parameters list, and so after that point, $1 will refer to the first non-option argument passed to the script.

As mikeserv mentions in the comment, shift $((OPTIND-1)) can be unsafe. To prevent unwanted word-splitting etc, all parameter expansions should be double-quoted. So the safe form for the command is
shift "$((OPTIND-1))"

getopts normally parses the positional parameters, but if more arguments are given in args, getopts parses those instead.
OPTIND is initialized to 1 each time the shell or a shell script is invoked. When an option requires an argument, getopts places that argument into the variable OPTARG
While the getopt system tool can vary from system to system, bash getopts is defined by the POSIX standard. So if you write a script using getopts, you can be sure that it runs on any system running bash in POSIX mode (e.g., set -o posix).
getopts parses short options, which are a single dash ("-") and a letter or digit. Examples of short options are -2, -d, and -D. It can also parse short options in combination, for instance -2dD.
However, getopts cannot parse options with long names. If you want options like --verbose or --help, use getopt instead.

Difference with getopt
Never use getopt(1). getopt cannot handle empty arguments strings, or arguments with embedded whitespace. Please forget that it ever existed.

Positional And Optional Argument

Positional And Optional Argument

positional and optional arguments. The positional arguments must occur at the location specified (ex: $1, $2) while the optional arguments can occur at any position denoted by their command line flag

Command in script

awk '{print >> ( $3 ".txt")}' filename

Input file

sample_text1 text3 20190426

sample_text2 text4 20190426

text1 abc 20190425

text2 def 20190425

generated output's (20190426.txt and 20190425.txt)

20190426.txt

sample_text1 text3 20190426

sample_text2 text4 20190426

20190425.txt

text1 abc 20190425

text2 def 20190425

You can change print to print $1 – steeldriver Apr 27 '19 at 1:41

i want to create output file based on third fields but in output can we have only $1 – upkar Apr 27 '19 at 1:42    

Got the answer now. As the 3rd field holds the essential part of the target file name, the latter is contructed from it and then used for the redirection of the print statement, which by default prints $0, the entire input line

List of DNS Reocrds and Its explanations

Differences between the A, CNAME, ALIAS and URL records

A, CNAME, ALIAS and URL records are all possible solutions to point a host name (name hereafter) to your website. 

However, they have some small differences that affect how the client will reach your site.

Before going further into the details, it’s important to know that A and CNAME records are standard DNS records, whilst ALIAS and URL records are custom DNS records. Both are translated internally into A records to ensure compatibility with the DNS protocol.

Here’s the main differences:

The A record maps a name to one or more IP addresses, when the IP are known and stable.

The CNAME record maps a name to another name. It should only be used when there are no other records on that name.

The ALIAS record maps a name to another name, but in turns it can coexist with other records on that name. The URL record redirects the name to the target name using the HTTP 301 status code.

Some important rules to keep in mind:

The A, CNAME, ALIAS records causes a name to resolve to an IP. Vice-versa, the URLrecord redirects the name to a destination. The URL record is simple and effective way to apply a redirect for a name to another name, for example to redirect www.example.com to example.com.

The A name must resolve to an IP, the CNAME and ALIAS record must point to a name.

Which one to use

Understanding the difference between the A name and the CNAME records will help you to decide.

The general rule is:

use an A record if you manage what IP addresses are assigned to a particular machine or if the IP are fixed (this is the most common case)

use a CNAME record if you want to alias a name to another name, and you don’t need other records (such as MX records for emails) for the same name

use an ALIAS record if you are trying to alias the root domain (apex zone) or if you need other records for the same name

use the URL record if you want the name to redirect (change address) instead of resolving to a destination.

You should never use a CNAME record for your root domain name (i.e. example.com).

The A and CNAME records are the two common ways to map a host name (name hereafter) to one or more IP address. Before going ahead, it’s important that you really understand the differences between these two records. I’ll keep it simple.

The A record points a name to a specific IP. For example, if you want the name blog.dnsimple.com to point to the server 185.31.17.133 you will configure:

blog.dnsimple.com.     A        185.31.17.133

The CNAME record points a name to another name, instead of an IP. The CNAME source represents an alias for the target name and inherits its entire resolution chain.

Let’s take our blog as example:

blog.dnsimple.com.      CNAME   aetrion.github.io.

aetrion.github.io.      CNAME   github.map.fastly.net.

github.map.fastly.net.  A       185.31.17.133

We use GitHub Pages and we set blog.dnsimple.com as a CNAME of aetrion.github.io, which in turns is itself a CNAME of github.map.fastly.net, which is an A record pointing to 185.31.17.133. In short terms, this means that blog.dnsimple.com resolves to 185.31.17.133.

To summarize, an A record points a name to an IP. CNAME record can point a name to another CNAME or an A record.

The chief difference between a CNAME record and an ALIAS record is not in the result—both point to another DNS record—but in how they resolve the target DNS record when queried.  As a result of this difference, one is safe to use at the zone apex (e.g., naked domain, such as example.com) and the other is not.

Let’s start with the CNAME record type.  It simply points a DNS name, like www.example.com, at another DNS name, like lb.example.net.  This introduces a performance penalty, since at least one additional DNS lookup must be performed to resolve the target (lb.example.net).  In the case of neither record ever having been queried before by your recursive resolver, it’s even more expensive time-wise, as the full DNS hierarchy must be traversed for both records:

You as the DNS client (or stub resolver) query your recursive resolver for www.example.com.

Your recursive resolver queries the root name server for www.example.com

The root name server refers your recursive resolver to the .com Top-Level Domain (TLD) authoritative server.

Your recursive resolver queries the .com TLD authoritative server for www.example.com.

The .com TLD authoritative server refers your recursive server to the authoritative servers for example.com.

Your recursive resolver queries the authoritative servers for www.example.com, and receives lb.example.net as the answer.

Your recursive resolver caches the answer, and returns it to you.

You now issue a second query to your recursive resolver for lb.example.net.

Your recursive resolver queries the root name server for lb.example.net.

The root name server refers your recursive resolver to the .net Top-Level Domain (TLD) authoritative server.

Your recursive resolver queries the .net TLD authoritative server for lb.example.net.

The .net TLD authoritative server refers your recursive server to the authoritative servers for example.net.

Your recursive resolver queries the authoritative servers for lb.example.net, and receives an IP address as the answer.

Your recursive resolver caches the answer, and returns it to you.

Each of these steps consumes at least several milliseconds, often more, depending on network conditions.  This can add up to a considerable amount of time that you spend waiting for the final, actionable answer of an IP address.

In the case of an ALIAS record, all of the same actions are taken as with the CNAME, except the authoritative server for example.com performs steps six through thirteen for you, and returns the final answer of an IP address.

This offers two advantages and one significant drawback:

Advantages

Faster final answer resolution speed. In most cases, the authoritative servers for example.com are more powerful and have faster Internet connectivity than your own computer and connection.  They can therefore traverse the DNS hierarchy and retrieve the final answer much faster than you can.

Answer looks like an A record. Since an ALIAS record returns the final answer consisting of one or more IP addresses, it can be used anywhere an A record can be used—including the zone apex. This makes it more flexible than a CNAME, which cannot be used at the zone apex.

Disadvantages

Geotargeting information is lost. Since it is the authoritative server for example.com that is issuing the queries for lb.example.net, then any intelligent routing functionality on the lb.example.net record will act upon the location of the authoritative server, not on your location. The EDNS0 edns-client-subnet option does not apply here. This means that you may be potentially mis-routed: for example, if you are in New York, and the authoritative server for example.com is in California, then lb.example.com will believe you to be in California and will return an answer that is distinctly sub-optimal for you in New York.

One important thing to note is that NS1 collapses CNAME records provided they all fall within the NS1 system, i.e., NS1’s nameservers are authoritative for both the CNAME and the target record. Collapsing simply means that the NS1 nameserver will return the full chain of records, from CNAME to final answer, in a single response.  This eliminates all the additional lookup steps, and allows you to use CNAME records, even in a nested configuration, without any performance penalty.

And even better, NS1 supports a unique record type called a Linked Record. This is basically a symbolic link within DNS that acts as an ALIAS record might, except with sub-microsecond resolution speed.  To use a Linked Record, simply create the target record as you usually would (it can be of any type) and then create a second record to point to it, and select the Linked Record option.  Note that Linked Records can cross domain (zone) boundaries and even account boundaries within NS1, and offer a powerful way to organize and optimize your DNS record structure.

Removing Duplicates Using AWK and How it Works.

<script data-ad-client="ca-pub-7841181112240136" async src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>

As a normal method in unix we need to use sort and then uniq to remove the duplicates from the file. Because without sort, it wont give correct unique values.

Instead of that we can use awk command to remove the duplicates in the file. For this we need to use awk associate array.

Short notes about AWK associative array

Unlike regular arrays in AWK associative arrays the indexes need not to be continuous set of number; you can use either string or number as an array index. Also, there is no need to declare the size of an array in advance – arrays can expand/shrink at runtime.

Its syntax is : array_name[index] = value

Example

Cat tes 

File Contents

Cat test

1

2

3

1

1

3

3a

3a

5

Command to Use: cat tes | awk '!seen[$0]++'  

How it Works

Seen[$0] - Uses the current line as the key to the array a. In our case 1 so array index will become 1 2 3 3a and 5. As we can't have same array index.

Note : Seen is an arbitrary word, it can be any words like a, b or any strings.

if seen[1] is never reference before then a[1] evalutes to empty string as awk will crate empty if it was not initialized before. IN this zero is false. If we negate then we will get true result. if it is non-zero (true) then we will get false result.

uses the current line $0 as key to the array a, taking the value stored there. If this particular key was never referenced before, a[$0] evaluates to the empty string.

!seen[$0]

The ! negates the value from before. If it was empty or zero (false), we now have a true result. If it was non-zero (true), we have a false result. If the whole expression evaluated to true, meaning that a[$0] was not set to begin with, the whole line is printed as the default action.

Also, regardless of the old value, the post-increment operator adds one to a[$0], so the next the same value in the array is accessed, it will be positive and the whole condition will fail.

Below is the actual way how awk with expression works

    awk 'expression' file Is actually a short hand of: awk 'expression {print $0}' file

Whenever the a test with no associated action is true, the default action is triggered. The default action is the equivalent of { print } or { print $0 }, which prints the current record, which for all accounts and purposes in this example is the current unmodified line of input.