If Command Options - Complete List

Here is the complete list of if command options 

a) For file related operations

-a FILE : True if FILE exists.

-b FILE : True if FILE exists and is a block-special file.

-c FILE : True if FILE exists and is a character-special file.

-d FILE : True if FILE exists and is a directory.

-e FILE : True if FILE exists.

-f FILE : True if FILE exists and is a regular file.

-g FILE : True if FILE exists and its SGID bit is set.

-h FILE : True if FILE exists and is a symbolic link.

-k FILE : True if FILE exists and its sticky bit is set.

-p FILE : True if FILE exists and is a named pipe (FIFO).

-r FILE : True if FILE exists and is readable.

-s FILE : True if FILE exists and has a size greater than zero.

-t FD : True if file descriptor FD is open and refers to a terminal.

-u FILE : True if FILE exists and its SUID (set user ID) bit is set.

-w FILE : True if FILE exists and is writable.

-x FILE : True if FILE exists and is executable.

-O FILE : True if FILE exists and is owned by the effective user ID.

-G FILE : True if FILE exists and is owned by the effective group ID.

-L FILE : True if FILE exists and is a symbolic link.

-N FILE : True if FILE exists and has been modified since it was last read.

-S FILE : True if FILE exists and is a socket.

b) For String Comparison

-z STRING : True of the length if "STRING" is zero.

-n STRING : or  STRING : True if the length of "STRING" is non-zero.

STRING1 == STRING2 : True if the strings are equal. 

STRING1 != STRING2 : True if the strings are not equal.

STRING1 < STRING2 : True if "STRING1" sorts before "STRING2" lexicographically in the current locale.

STRING1 > STRING2 : True if "STRING1" sorts after "STRING2" lexicographically in the current locale.

c) For Integer Comparison

Hope below examples clarify how to do integer comparison

if [ "$a" -eq "$b" ]

if [ "$a" -ne "$b" ]

if [ "$a" -gt "$b" ]

if [ "$a" -ge "$b" ]

if [ "$a" -lt "$b" ]

if [ "$a" -le "$b" ]

d) For Logical Operations

EXPR1 -a EXPR2 : True if both EXPR1 and EXPR2 are true.

EXPR1 -o EXPR2 : True if either EXPR1 or EXPR2 is true.

What is Proxy, Types of proxy and Uses of Proxy

What is a Proxy Server?

A Proxy server is an intermediary machine, between a client and the actual server, which is used to filter or cache requests made by the client.

Types of Proxy

1) Caching Proxy
2) Transparent Proxy
3) Reverse Proxy

Normal (Regular/Caching) Proxy:

A regular caching proxy server is a server which listens on a separate port (e.g. 3128) and the clients (browsers) are configured to send requests for connectivity to that port. So the proxy server receives the request, fetches the content and stores a copy for future use. So next time when another client requests for the same webpage the proxy server just replies to the request with the content in its cache thus improving the overall request-reply speed.

Transparent Proxy:

A transparent proxy server is also a caching server but the server is configured in such a way that it eliminates the client side (browser side) configuration. Typically the proxy server resides on the gateway and intercepts the WWW requests (port 80, 443 etc.) from the clients and fetches the content for the first time and subsequently replies from its local cache. The name Transparent is due to the fact that the client doesn't know that there is a proxy server which mediates their requests. Transparent proxy servers are mostly used in big corporate organizations where the client side configuration is not easy (due to the number of clients). This type of server is also used in ISP's to reduce the load on the bandwidth usage.

Reverse Proxy:

A reverse proxy is totally different in its usage because it is used for the benefit of the web server rather than its clients. Basically a reverse proxy is on the web server end which will cache all the static answers from the web server and reply to the clients from its cache to reduce the load on the web server. This type of setup is also known as Web Server Acceleration

Configuration file -  /etc/squid/squid.conf 

You need to add three lines to the squid.conf file in the /etc/squid/ directory before activating Squid. First editing is about hostname locate visible_hostname tag near about line no 2835

Go in the end of this tag near about line no and add the hostname which you have checked in pervious command

eg : visible hostname

By default squid works on port no 3128 but can change this. Port tag is located near line no 73

ACL Syntax - acl <acl name> <acl module> <acl argument>

We will create three access list.

First to block host with ip address 192.168.1.7 from accessing internet.

Second to block a particular site.

Third to allow our lab network for accessing internet.

Go in the end of access control tag near about line 2410 and create access list as show here

access list tag in squid.conf

Final editing is to implement whatever access list you have configured in access list tag go to http access tag near line no 2482

http access tag

In the end of this tag near line no 2529 apply the configured access list

http access

Be very careful about the order of access list alway put http_access deny all line in the end of all line. Whatever access list defined below the http_access deny all line will never be checked.

You have made necessary changed in squid.conf now save it and return to command prompt.

We have created a access list web_deny to filter the web traffic. We have set http_access deny web_deny tag in squid.conf. Now you can add the url of those websites in this file which you want block.

acl Safe_ports port 777 II multiling http

acl CONNECT method CONNECT

acl deny_host src 192.168.1.7

acl allow_network src 192.168.1.0/24

acl web_deny dstdomain "/etc/squid/web_deny:

it And finally deny all other access to this proxy

http_access allow localhost

http_access deny deny_host

http_access deny web_deny

http_access allow allow_network

http_access deny all

it TAG: http_access2

SAMPLE ACL's

acl allowfacebooktime time MTWHF 12:15-13:45

# Facebook ACL

acl facebookdotcom  dstdomain .facebook.com

# Only allow Facebook as described by allowfacebooktime

http_access allow facebookdotcom allowfacebooktime

# Else block facebook

http_access deny facebookdotcom

ACL TYPES

src: source (client) IP addresses

dst: destination (server) IP addresses

myip: the local IP address of a client's connection

arp: Ethernet (MAC) address matching

srcdomain: source (client) domain name

dstdomain: destination (server) domain name

srcdom_regex: source (client) regular expression pattern matching

dstdom_regex: destination (server) regular expression pattern matching

src_as: source (client) Autonomous System number

dst_as: destination (server) Autonomous System number

time: time of day, and day of week

url_regex: URL regular expression pattern matching

urlpath_regex: URL-path regular expression pattern matching, leaves out the protocol and hostname

port: destination (server) port number

myport: local port number that client connected to

myportname: name tag assigned to the squid listening port that client connected to

proto: transfer protocol (http, ftp, etc)

method: HTTP request method (get, post, etc)

http_status: HTTP response status (200 302 404 etc.)

browser: regular expression pattern matching on the request user-agent header

maxconn: a limit on the maximum number of connections from a single client IP address

max_user_ip: a limit on the maximum number of IP addresses one user can login from

req_mime_type: regular expression pattern matching on the request content-type header

req_header: regular expression pattern matching on a request header content

rep_mime_type: regular expression pattern matching on the reply (downloaded content) content-type header. This is only usable in the http_reply_access directive, not http_access.

rep_header: regular expression pattern matching on a reply header content. This is only usable in the http_reply_access directive, not http_access.

external: lookup via external acl helper defined by external_acl_type

user_cert: match against attributes in a user SSL certificate

ca_cert: match against attributes a users issuing CA SSL certificate

ext_user: match on user= field returned by external acl helper defined by external_acl_type

ext_user_regex: regular expression pattern matching on user= field returned by external acl helper defined by external_acl_type

Combining ACLs (AND/OR)

Often you need to combine ACLs. Let’s say you want to allow access to google.com only for the back office. This combines two ACLS with an AND. This would look like this:

http_access allow accesses_to_google.com accesses_from_back_office

If you wanted to use an OR and say either accesses from the back office or accesses to google.com are allowed then the line would look like this:

http_access allow accesses_to_google.com

http_access allow accesses_from_back_office

To summarize: AND means putting the conditions in one line. OR means using seperate lines.

Custom error pages (deny_info)

By default when you deny access the user gets the error page that is stored in the ERR_ACCESS_DENIED file. But luckily you can define your own custom error pages and display them when you deny certain accesses. A simple example:

acl google dstdomain google.com

deny_info error-google google

http_access deny google

Put an error page into the directory where the HTML files are stored (look for error_directory in your squid.conf) and name it error-google. If the user tries to access www.google.com the access is denied and your error page is shown.

Careful when you combine ACLs on a http_access line. Example:

acl google dstdomain google.com

acl admin src 10.0.5.16

deny_info google error-google

http_access deny admin google

This will deny access only for the user from the IP address 10.0.5.16 when www.google.com is accessed. As you can see I have combined the ACLs admin and google. In such a combination the last ACL in the line is taken into account for lookups of deny_info. So it’s important that you define a deny_info for the google ACL.

How to enable Proxy in Linux

Type the following command to set proxy server:

$ export http_proxy=http://proxy-server.example.com:3128/

If the proxy server requires a username and password then add these to the URL. For example, to include the username foo and the password bar:

$ export http_proxy=http://USERNAME :PASSWORD@proxy server:port/ 

Port Numbers and It protocol's

1) FTP – 20 (Data), 21 (Command) TCP
2) TFTP - UDP

3) SSH - 22

4) DNS – 53 ( Zone Transfer in TCP, DNS Queries in UDP) 

A client computer will always send a DNS Query using UDP Protocol over Port 53. If a client computer does not get response from a DNS Server, it must re-transmit the DNS Query using the TCP after 3-5 seconds of interval.

5) DHCP – 67, 68 (Client)  - UDP

6) NTP – 123

7) POP3 - 110

8) POP3S - 993

9) IMAP - 143

10) IMAPS – 995

11) NFS – 2049 (Default is TCP, UDP can also be used)

12) SMB – 137, 138, 139
13) SQUID - 3128 

How the Domain Name System (DNS) works

1. Client enters ‘www.example.com’ internet address. Client computer needs the IP address translation of ‘example.com’ and first checks its own DNS cache for this information. If this is the first time using this website or the cache has been cleared it cannot find the IP address here.

2. The client computer (or “query”?) is then redirected to the Internet Service Provider’s (ISP’s) DNS Server. The ISP’s DNS server checks its own cache but it will not be there if the site has not been accessed before.

3. The ISP’s DNS server redirects the query to the Root DNS Server. Every DNS server has a file that contains a list of all of the root DNS servers. Totally there are 13 root DNS servers.

4. The root DNS server maintains information about where a top-level (like .com, .in) DNS server is located and returns this information to the ISP’s DNS Server.

5. The ISP’s DNS server redirects the query to a top-level (.com) DNS server.

6. The top-level (.com) DNS server knows the IP address of the DNS server(Authoritative DNS server) for the example.com domain and returns that information to the ISP’s DNS server.

7. The ISP’s DNS server redirects the query to the actual authoritative DNS server for the example.com domain.

8. The DNS server for www.example.com returns the IP address of the host of www.example.com to the ISP’s DNS server.

9. Lastly, the ISP’s DNS server sends the IP address to the client computer so the client can access www.example.com

When your browser connects to a website say example.com, the browser first queries your local DNS server to get the IP address of example.com.

If the local DNS server doesn't have the A record of example.com, it will query one of the root DNS servers.

The root DNS server will say: I don't have the A record but I know the top-level domain DNS server which is responsible for .com domains.

Then your local DNS server query the top-level domain DNS server which is responsible for .com domains. The TLD DNS server will respond: I don't know either but I know which DNS 
server is authoritative for example.com.

So your local DNS server queries the authoritative DNS server. Because the actual DNS record is stored on that authoritative DNS server, so it will give your local DNS server an answer.

First you have to understand how DNS system works. DNS system can be divided into three tiers. They are:

a) root DNS servers

b) top-level domain DNS servers

c) authoritative DNS servers
d) Local DNS server (Which Will be IP of ISP) whose IP address is specified on your operating system.

When your browser connects to a website say example.com, the browser first queries your local DNS server to get the IP address of example.com.

If the local DNS server doesn't have the A record of example.com, it will query one of the root DNS servers.

The root DNS server will say: I don't have the A record but I know the top-level domain DNS server which is responsible for .com domains.

Then your local DNS server query the top-level domain DNS server which is responsible for .com domains. The TLD DNS server will respond: I don't know either but I know which DNS 
server is authoritative for example.com.

So your local DNS server queries the authoritative DNS server. Because the actual DNS record is stored on that authoritative DNS server, so it will give your local DNS server an answer.

Then this query result is cached on your local DNS server but it can be outdated. When the TTL time has expired, your local DNS server will update the query result from the authoritative DNS server. Whenever you query a DNS record on your local DNS server, it returns a non-authoritative (unofficial) answer. If you want an authoritative answer, you must explicitly specify the authoritative DNS server when you use nslookup or other utilities. I think a local DNS server should be called caching DNS server.

HOW DHCP Works

What is DHCP?

DHCP stands for Dynamic Host Configuration Protocol.

As the name suggests, DHCP is used to control the network configuration of a host through a remote server. DHCP functionality comes installed as a default feature in most of the contemporary operating systems. DHCP is an excellent alternative to the time-consuming manual configuration of network settings on a host or a network device.

DHCP works on a client-server model. Being a protocol, it has it’s own set of messages that are exchanged between client and server. Here is the header information of DHCP

How DHCP Works?

Before learning the process through which DHCP achieves it’s goal, we first have to understand the different messages that are used in the process.

1. DHCPDISCOVER

It is a DHCP message that marks the beginning of a DHCP interaction between client and server. This message is sent by a client (host or device connected to a network) that is connected to a local subnet. It’s a broadcast message that uses 255.255.255.255 as destination IP address while the source IP address is 0.0.0.0

2. DHCPOFFER

It is DHCP message that is sent in response to DHCPDISCOVER by a DHCP server to DHCP client. This message contains the network configuration settings for the client that sent the DHCPDISCOVER message.

3. DHCPREQUEST

This DHCP message is sent in response to DHCPOFFER indicating that the client has accepted the network configuration sent in DHCPOFFER message from the server.

4. DHCPACK

This message is sent by the DHCP server in response to DHCPREQUEST recieved from the client. This message marks the end of the process that started with DHCPDISCOVER. The DHCPACK message is nothing but an acknowledgement by the DHCP server that authorizes the DHCP client to start using the network configuration it received from the DHCP server earlier.

5. DHCPNAK

This message is the exact opposite to DHCPACK described above. This message is sent by the DHCP server when it is not able to satisfy the DHCPREQUEST message from the client.

6. DHCPDECLINE

This message is sent from the DHCP client to the server in case the client finds that the IP address assigned by DHCP server is already in use.

7. DHCPINFORM

This message is sent from the DHCP client in case the IP address is statically configured on the client and only other network settings or configurations are desired to be dynamically acquired from DHCP server.

8. DHCPRELEASE

This message is sent by the DHCP client in case it wants to terminate the lease of network address it has be provided by DHCP server.

Here are the steps :

The client, Host A, sends a DHCPDISCOVER broadcast message to locate a DHCP server. A relay agent forwards the packets between the DHCP client and server. A DHCP server offers configuration parameters (such as an IP address, a MAC address, a domain name, and a lease for the IP address) to the client in a DHCPOFFER unicast message.

Step 1: When the client computer (or device) boots up or is connected to a network, a DHCPDISCOVER message is sent from the client to the server. As there is no network configuration information on the client so the message is sent with 0.0.0.0 as source address and 255.255.255.255 as destination address. If the DHCP server is on local subnet then it directly receives the message or in case it is on different subnet then a relay agent connected on client’s  subnet is used to pass on the request to DHCP server. The transport protocol used for this message is UDP and the port number used is 67. The client enters the initializing stage during this step.

Step 2: When the DHCP server receives the DHCPDISCOVER request message then it replies with a DHCPOFFER message. As already explained, this message contains all the network configuration settings required by the client. For example, the yaddr field of the message will contain the IP address to be assigned to client. Similarly the the subnet mask and gateway information is filled in the options field. Also, the server fills in the client MAC address in the chaddr field. This message is sent as a broadcast (255.255.255.255) message for the client to receive it directly or if DHCP server is in different subnet then this message is sent to the relay agent that takes care of whether the message is to be passed as unicast or broadcast. In this case also, UDP protocol is used at the transport layer with destination port as 68. The client enters selecting stage during this step

Step 3: The client forms a DHCPREQUEST message in reply to DHCPOFFER message and sends it to the server indicating it wants to accept the network configuration sent in the DHCPOFFER message. If there were multiple DHCP servers that received DHCPDISCOVER then client could receive multiple DHCPOFFER messages. But, the client replies to only one of the messages by populating the server identification field with the IP address of a particular DHCP server. All the messages from other DHCP servers are implicitly declined. The DHCPREQUEST message will still contain the source address as 0.0.0.0 as the client is still not allowed to use the IP address passed to it through DHCPOFFER message. The client enters requesting stage during this step.

Step 4: Once the server receives DHCPREQUEST from the client, it sends the DHCPACK message indicating that now the client is allowed to use the IP address assigned to it. The client enters the bound state during this step.

The Concept of Lease

With all the necessary information on how DHCP works, one should also know that the IP  address assigned by DHCP server to DHCP client is on a lease. After the lease expires the DHCP server is free to assign the same IP address to any other host or device requesting for the same. For example, keeping lease time 8-10 hours is helpful in case of PC’s that are shut down at the end of the day.  So, lease has to be renewed from time to time. The DHCP client tries to renew the lease after half of the lease time has expired. This is done by the exchange of DHCPREQUEST and DHCPACK messages. While doing all this, the client enters the renewing stage.

What is DHCP Relay

DHCP relay agent is any host that forwards DHCP packets between clients and servers. Relay agents are used to forward requests and replies between clients and servers when they are not on the same physical subnet. 

What is the use of SYSLINUX

In particular, 'Syslinux' is a collection of boot loaders that includes 'SYSLINUX', 'ISOLINUX', 'EXTLINUX' and 'PXELINUX'

syslinux has pretty much taken a more minimalist approach, whereas GRUB was much more extensive. GRUB also supports different filesystems through the use of a secondary loader. If memory serves, syslinux supports a few formats and doesn't require a secondary boot loader.

If you want to install a Linux distribution on a local drive, use GRUB. If you want to create a bootable media, use Syslinux. This is not a general recipe, as you can use Syslinux on local installs.

Syslinux isn’t a single bootloader, rather it is a collection of lightweight bootloaders. SYSLINUX is a boot loader for the Linux operating system which runs on an MS-DOS/Windows FAT filesystem. It is intended to simplify first-time installation of Linux, and for creation of rescue and other special purpose boot disks.

GRUB was initially developed by Erich Boleyn as part of work on booting the operating system GNU/Hurd, developed by the Free Software Foundation. In 1999, Gordon Matzigkeit and Yoshinori K. Okuji made GRUB an official software package of the GNU Project and opened the development process to the public

What is DD command and what is the use of DD command

dd “data duplicator” -  Will be used for copying and converting data.

Below are the some of the tasks

1) Backup and restore the entire hard disk or partition.

2) Wipe/delete content of a disk so that it will be empty for some one to use it.

3) Backing Up and Restoring MBR (Master Boot Record)

4) Convert lower case to upper case and vice versa

5) It can also be used by Linux kernel make files to make boot images.

SYNTAX

dd if=<source file name> of=<target file name> [Options]

1) Backup and restore the entire hard disk or partition.

Clone one hard disk to another hard disk. This is useful when we are building many machines with same configuration. We no need to install OS on all the machines. Just install OS and required software on machine then clone with below example.

dd if=/dev/sda of=/dev/sdb

EXAMPLE-2:

To create a disk image. 

# dd if=/dev/sda of=/tmp/sdadisk.img

The above creates the image of a harddisk /dev/hda. Backing up a disk to an image will be faster than copying the exact data. Also, disk image make the restoration much more easier.

EXAMPLE-3:

To create a compressed disk image.

# dd if=/dev/sda | gzip >/tmp/sdadisk.img.gz

EXAMPLE-4:

To restore hard disk image.

# dd if=hdadisk.img of=/dev/hdb

The image file hdadisk.img file, is the image of a /dev/hda, so the above command will restore the image of /dev/hda to /dev/hdb.

2) Deleting the Data from Disk

a) Wipe/delete content of a disk so that it will be empty for some one to use it.

# dd if=/dev/zero of=/dev/sdb

b) What to hide your ass by deleting your personal data. Many people think if we do rm -rf /<your data> will do the needful. But we can recover those deletion by using disk recovery tools like Photorec or some forensic tools. But if you want some not to recover your data you have to write random data on your partition where you data resides.

dd if=/dev/random of=/dev/sdb

3. Backing up and restoring MBR

As you are aware MBR makes up the first 512 bytes of the disk, in which 466 bytes about boot loader info. The additional space will be used to store the partition table for that drive.  If MBR gets corrupted, we will not be able to boot into Linux.

a. Backing up MBR

Because the MBR makes up the first 512 bytes of the disk, we just need to copy that block size

# dd if=/dev/sda of=/tmp/sdambr.img bs=512 count=1

With the count=1 and bs=512, only 512 bytes will be copied which corr

b. Backing up the boot data of MBR excluding the partition table

The MBR 512 bytes data is located at the first sector of the hard disk. It consists of 446 bytes bootstrap, 64 bytes partition table and 2 bytes signature. It means that we can exclude the partition table and bytes signature while backing up the MBR with conserving only a block size equal to the bootstrap size.

# dd if=/dev/sda of=/tmp/sdambr2.img bs=446 count=1

c. Restoring MBR from MBR image

You can restore your MBR as shown on the previous commands with

# dd if=/tmp/sdambr.img of=/dev/sda

4. Converting case of a file

dd command can be also used for an amazing thing. It can convert all text (alphabets) in a file to upper or lower case and vice versa. For the example below, we will have a file for the tests.

# cat file10

test dd convert

a. Converting a file to uppercase

Because our text file example is on lowercase, we will convert it to uppercase

# dd if=~/file10 of=~/file20 conv=ucase

The command will create the new file indicated. See that now conv option takes ucase value. Let's check the result

# cat file20

TEST DD CONVERT

b. Converting a file to lowercase

Now we will do the reverse operation which will convert to lowercase

# dd if=~/file20 of=~/file30 conv=lcase

See that we use lcase of conv option to convert from upper case to lower case.

# cat file30

test dd convert

Note: dd command does not convert the file names, only its content

2. Creating virtual filesystem and backup images of CD or DVDs as iso files

You can need to create a virtual filesystem on Linux for some reasons as creating a virtual machine on your Linux host. You can also need to create a backup iso image of a CD or DVD

a. Creating a virtual filesystem

A virtual filesystem is a filesystem that exists in a file, which in turn exists on a physical disk. You can need it to create for example an additional swap or loop device or a virtual machine. We need /dev/zero which is a file used to create a file with no data but with required size (a file with all zero’s). In other words, this will create a data file with all zeros in the file which will give the size to a file.

# dd if=/dev/zero of=/file bs=1024K count=500

500+0 records in

500+0 records out

524288000 bytes (524 MB) copied, 1.21755 s, 431 MB/s

The option count refers to the number of input blocks to be copied. Combined with block size value, it indicates the total size to copy. For example bs=1024k and count=500 give a size=1024K*500 =524288000 bytes =524MB

Now let's check the size of our file

# ls -lh /file

-rw-r--r-- 1 root root 500M May 17 18:57 /file

You can see that we have our virtual filesystem created with the size indicated. You can now use it to create loop device or a virtual disk or anything else.

b. Modify the first 512 bytes of a file with null data

If during the operation you indicate an existing output file, you will lose its data. For some reasons, you can need to replace a block size of the output file.

#dd if=/dev/zero of=file1 bs=512 count=1 conv=notrunc

The notrunc option refers to do not truncate the file, only replace the first 512 bytes, if it exists. Otherwise, you will get a 512 byte file

c. Creating a backup iso image of CD or DVD

You may wonder why not just copy the contents of your CD to a directory. How would you handle the boot sector of a CD? You can’t find that as a file on the device because it’s just the first sector. Because dd copies sector by sector, on the other hand, it will copy that information as well.

# dd if=/dev/cdrom of=/mycd.iso

You need to know that you have to use the -o loop option, which allows you to mount a file like any normal device. So, to mount /mycd.iso on the /mnt/cd directory, do as below