AWK Associative array

arrayname[string]=value

Syntax:

for (var in arrayname)

actions

Code to identify shell used by how many users

Sample Code

BEGIN {

        FS=":"

}

{user_shells[$7] +=1}

END{

        for (shell in user_shells)

                print shell "\t\t" user_shells[shell]

}

How it Works

We are creating an array name called user_shells for which we are mentioning index item is  present on field 7. If the value of field 7 repeats then the respective index value is increased by 1. 

to access the array we will be using for loop. for loop of awk will work slight differently compare to bash for loop. 

In case of awk for loop, variable mentioned will be used as a index of the array. So printing the declared variable in awk will print the array index and the second part will print the array values for the respective index.

rename Command

About rename

The rename command is used to change the name of multiple files.This command is slightly more advanced than mv because it requires the knowledge of, or at least a basic familiarity with, regular expressions.

We have two versions of rename package, a simpler version and a perl based version. perl based version has more advanced features.

If you have the simpler rename installed, you will see something like this instead.

       
$ rename --version 

rename from util-linux 2.33.1      
 

Perl based rename has below type of version.

       
$ rename --version

perl-rename 1.9

In this post, we are going to see the example for simpler version.

Syntax : rename 'old pattern' 'new pattern' input files

example : If you want to replace txt to text, below is the syntax

       
 rename 'txt' 'text' *.txt
       

What is .Vimrc file

What is .vimrc

A file that contains initialization commands is called a "vimrc" file

Types of .vimrc

• User vimrc 
• System vimrc

Location of vimrc

• User vimrc in $HOME (The user vimrc file often does not exist until created by the user. If you cannot find $HOME/.vimrc (or $HOME/_vimrc on Windows) then you can, and probably should, just create it.)

• The system vimrc is normally under /etc in linux. The system vimrc should normally be left unmodified and it is not a good place you keep your personal settings. If you modify this file your changes may be overwritten if you ever upgrade vim. Also, changes here will affect other users on a multi-user system. In most cases, settings in the user vimrc will override settings in the system vimrc.

TIPS

1) :scriptnames list all the .vim files that Vim loaded for you, including your .vimrc file.

2) :e $MYVIMRC open & edit the current .vimrc that you are using, then use Ctrl + G to view the path in status bar.

To create your vimrc, start up vim and do one of the following:

3) :e $HOME/.vimrc  " on Unix, Mac or OS/2

Examples of vimrc contents

# Show line numbers

set number

# Show file stats

set ruler

AWK match and substr function

MATCH Function

match(string, regexp [, array])

Search string for the longest, leftmost substring matched by the regular expression and return the character position (index) at which that substring begins (one, if it starts at the beginning of string). If no match is found, return zero.

The regexp argument may be either a regexp constant (/…/) or a string constant ("…"). 

Note: The order of the first two arguments is the opposite of most other string functions that work with regular expressions, such as sub() and gsub(). It might help to remember that for match(), the order is the same as for the ‘~’ operator: ‘string ~ regexp’.

The match() function sets the predefined variable RSTART to the index. It also sets the predefined variable RLENGTH to the length in characters of the matched substring. If no match is found, RSTART is set to zero, and RLENGTH to -1.

SUBSTR Function

substr(string, start [, length ])

Return a length-character-long substring of string, starting at character number start. The first character of a string is character number one.49 For example, substr("washington", 5, 3) returns "ing".

If length is not present, substr() returns the whole suffix of string that begins at character number start. For example, substr("washington", 5) returns "ington". The whole suffix is also returned if length is greater than the number of characters remaining in the string, counting from character start.

example:

AB: 20190131  13 J-1|19:30:00.000000000 18:06:00.000000000 123466  50 @TEST . "" 1234 - I . ".." "" "" "TEST TEXT 1" "TEXT 2: SAMPLE TEXT I must explain to you how all this mistaken idea of denouncing pleasure and praising pain was born and I will give you a complete account of the system, and expound the actual teachings of the great explorer of the truth, the master-builder of human happiness. No one rejects, dislikes, or avoids pleasure itself, because it is pleasure, but because those who do not know how to pursue pleasure rationally encounter consequences that are extremely painful. Nor again is there anyone who loves or pursues or desires to obtain pain of itself, because it is pain, but because occasionally circumstances occur in which toil and pain can procure him some great pleasure. To take a trivial example, which of us ever undertakes laborious physical exercise, except to obtain some advantage from it? But who has any right to find f.==Required file.csv.gz FIELD*SERVER-TIME*05:29:51.981378000" "" NoTime

       
Command : awk 'match($0,/==.*?.csv.gz/){print $3","substr($0, RSTART+2, RLENGTH-2)}'  Sample file


Output  : 13,Required file.csv.gz       
 

What is Interpreter and Complier

We generally write a computer program using a high-level language. A high-level language is one which is understandable by us humans. It contains words and phrases from the English (or other) language. But a computer does not understand high-level language. It only understands program written in 0's and 1's in binary, called the machine code. A program written in high-level language is called a source code. We need to convert the source code into machine code and this is accomplished by compilers and interpreters. Hence, a compiler or an interpreter is a program that converts program written in high-level language into machine code understood by the computer.

The difference between an interpreter and a compiler is given below:

Interpreter	Compiler
Translates program one statement at a time.	Scans the entire program and translates it as a whole into machine code.
It takes less amount of time to analyze the source code but the overall execution time is slower.	It takes large amount of time to analyze the source code but the overall execution time is comparatively faster.
No intermediate object code is generated, hence are memory efficient.	Generates intermediate object code which further requires linking, hence requires more memory.
Continues translating the program until the first error is met, in which case it stops. Hence debugging is easy.	It generates the error message only after scanning the whole program. Hence debugging is comparatively hard.
Programming language like Python, Ruby use interpreters.	Programming language like C, C++ use compilers.

 

How to set Quota in Linux

What is Quota

Disk space can be restricted by implementing disk quotas which alert a system administrator before a user consumes too much disk space or a partition becomes full.

Disk quotas can be configured for individual users as well as user groups. This makes it possible to manage the space allocated for user-specific files (such as email) separately from the space allocated to the projects a user works on (assuming the projects are given their own groups).

In addition, quotas can be set not just to control the number of disk blocks consumed but to control the number of inodes (data structures that contain information about files in UNIX file systems). Because inodes are used to contain file-related information, this allows control over the number of files that can be created.

Prerequisite : The quota RPM must be installed to implement disk quotas.

How to Enable

To implement disk quotas, use the following steps:

Enable quotas per file system by modifying the /etc/fstab file.

Remount the file system(s).

Create the quota database files and generate the disk usage table.

Assign quota policies.

Each of these steps is discussed in detail in the following sections.

Enabling Quotas

As root, using a text editor, edit the /etc/fstab file.

Add the usrquota and/or grpquota options to the file systems that require quotas:

Example 16.2. Add quotas

/dev/VolGroup00/LogVol02 /home     ext3    defaults,usrquota,grpquota  1 2 

/dev/VolGroup00/LogVol01 swap      swap    defaults        0 0 . . .

In this example, the /home file system has both user and group quotas enabled.

Note

The following examples assume that a separate /home partition was created during the installation of Red Hat Enterprise Linux. The root (/) partition can be used for setting quota policies in the /etc/fstab file.

Remounting the File Systems

After adding the usrquota and/or grpquota options, remount each file system whose fstab entry has been modified. If the file system is not in use the following commands:

umount /mount-point

For example, umount /work.

mount /file-system /mount-point

For example, mount /dev/vdb1 /work.

If the file system is currently in use, the easiest method for remounting the file system is to reboot the system.

Creating the Quota Database Files

After each quota-enabled file system is remounted run the quotacheck command.

The quotacheck command examines quota-enabled file systems and builds a table of the current disk usage per file system. The table is then used to update the operating system's copy of disk usage. In addition, the file system's disk quota files are updated.

To create the quota files (aquota.user and aquota.group) on the file system, use the -c option of the quotacheck command.

Create quota files

For example, if user and group quotas are enabled for the /home file system, create the files in the /home directory:

# quotacheck -cug /home

The -c option specifies that the quota files should be created for each file system with quotas enabled, the -u option specifies to check for user quotas, and the -g option specifies to check for group quotas.

If neither the -u or -g options are specified, only the user quota file is created. If only -g is specified, only the group quota file is created.

After the files are created, run the following command to generate the table of current disk usage per file system with quotas enabled:

# quotacheck -avug

The options used are as follows:

a

Check all quota-enabled, locally-mounted file systems

v

Display verbose status information as the quota check proceeds

u

Check user disk quota information

g

Check group disk quota information

After quotacheck has finished running, the quota files corresponding to the enabled quotas (user and/or group) are populated with data for each quota-enabled locally-mounted file system such as /home.

Assigning Quotas per User

The last step is assigning the disk quotas with the edquota command.

To configure the quota for a user, as root in a shell prompt, execute the command:

# edquota username

Perform this step for each user who needs a quota. For example, if a quota is enabled in /etc/fstab for the /home partition (/dev/VolGroup00/LogVol02 in the example below) and the command edquota testuser is executed, the following is shown in the editor configured as the default for the system:

Disk quotas for user testuser (uid 501):   

Filesystem                blocks     soft     hard    inodes   soft   hard   

/dev/VolGroup00/LogVol02  440436        0        0     37418      0      0

Note : The text editor defined by the EDITOR environment variable is used by edquota. To change the editor, set the EDITOR environment variable in your ~/.bash_profile file to the full path of the editor of your choice.

The first column is the name of the file system that has a quota enabled for it. The second column shows how many blocks the user is currently using. The next two columns are used to set soft and hard block limits for the user on the file system. The inodes column shows how many inodes the user is currently using. The last two columns are used to set the soft and hard inode limits for the user on the file system.

The hard block limit is the absolute maximum amount of disk space that a user or group can use. Once this limit is reached, no further disk space can be used.

The soft block limit defines the maximum amount of disk space that can be used. However, unlike the hard limit, the soft limit can be exceeded for a certain amount of time. That time is known as the grace period. The grace period can be expressed in seconds, minutes, hours, days, weeks, or months.

If any of the values are set to 0, that limit is not set. In the text editor, change the desired limits.

Change desired limits

For example:

Disk quotas for user testuser (uid 501):   

Filesystem                blocks     soft     hard   inodes   soft   hard   

/dev/VolGroup00/LogVol02  440436   500000   550000    37418      0      0

To verify that the quota for the user has been set, use the command:

# quota username

Disk quotas for user username (uid 501): 

   Filesystem  blocks   quota   limit   grace   files   quota   limit   grace

     /dev/sdb    1000*   1000    1000               0       0       0

Assigning Quotas per Group

Quotas can also be assigned on a per-group basis. For example, to set a group quota for the devel group (the group must exist prior to setting the group quota), use the command:

# edquota -g devel

This command displays the existing quota for the group in the text editor:

Disk quotas for group devel (gid 505):   

Filesystem                blocks    soft     hard    inodes    soft    hard   

/dev/VolGroup00/LogVol02  440400       0        0     37418       0       0

Modify the limits, then save the file.

To verify that the group quota has been set, use the command:

# quota -g devel

Setting the Grace Period for Soft Limits

If a given quota has soft limits, you can edit the grace period (i.e. the amount of time a soft limit can be exceeded) with the following command:

# edquota -t

This command works on quotas for inodes or blocks, for either users or groups.

Important

While other edquota commands operate on quotas for a particular user or group, the -t option operates on every file system with quotas enabled.

What is atime and noatime in fstab

Linux has a special mount option for file systems called noatime that can be added to each line that addresses one file system in the /etc/fstab file. If a file system has been mounted with this option, reading accesses to the file system will no longer result in an update to the atime information associated with the file like we have explained above. The importance of the noatime setting is that it eliminates the need by the system to make writes to the file system for files which are simply being read. Since writes can be somewhat expensive, this can result in measurable performance gains. Note that the write time information to a file will continue to be updated anytime the file is written to. In our example below, we will set the noatime option to our /chroot file system.

Edit the fstab file vi /etc/fstab and add in the line that refer to /chrootfile system the noatime option after the defaults option as show below:

            /dev/sda7          /chroot          ext2          defaults,noatime          1  2

           

You need not reboot your system for the change to take effect, just make the Linux system aware about the modification you have made to the /etc/fstab file. This can be accomplished with the following commands:

           [root@deep] /#mount -oremount /chroot/

           

Then test your results with the flowing command:

           [root@deep]# cat /proc/mounts

           

 /dev/root / ext2 rw 0 0 /proc /proc proc rw 0 0 /dev/sda1 /boot ext2 rw 0 0 /dev/sda8 /cache ext2 rw 0 0 /dev/sda7 /chroot ext2 rw,noatime 0 0 /dev/sda6 /home ext2 rw 0 0 /dev/sda11 /tmp ext2 rw 0 0 /dev/sda5 /usr ext2 rw 0 0 /dev/sda9 /var ext2 rw 0 0 none /dev/pts devpts rw 0 0

If you see something like: /dev/sda7 /chroot ext2 rw,noatime 0 0

SED Options

We know sed command is used for file processing.

To Match Digit

sed -r 's/([^0-9]*([0-9]*)){2}.*/\2/' file

This extracts the second number:


sed -r 's/([^0-9]*([0-9]*)){1}.*/\2/' file

sed 's/[[:digit:]]\+\.//g'

$ echo "This is an example: 65 apples" | sed -r  's/^[^0-9]*([0-9]+).*/\1/'

65

59

Two problems:

sed does not support \d. Use [0-9] or [[:digit:]].


+ must be backslashed to get the special meaning: \+.

sed -r 's/.*_([0-9]*)\..*/\1/g'