AWK : In Built Variables and Functions

NEXT and GETLINE 

The next statement forces awk to immediately stop processing the current record and go on to the next record. This means that no further rules are executed for the current record, and the rest of the current rule’s action isn’t executed.

Contrast this with the effect of the getline function. That also causes awk to read the next record immediately, but it does not alter the flow of control in any way (i.e., the rest of the current action executes with a new input record).

NR and FNR

NR - stores the total number of input records read so far, regardless of how many files have been read. The value of NR starts at 1 and always increases until the program terminates. 

FNR - stores the number of records read from the current file being processed. The value of FNR starts at 1, increases until the end of the current file is reached, then is set again to 1 as soon as the first line of the next file is read, and so on. 

AWK Two File Processing

When Processing multiple files awk reads each file sequentially, one after another, in the order they are specified on the command line. 

$ awk 'NR == FNR { # some actions; next} # other condition {# other actions}' file1.txt file2.txt

How it Works

So, the condition NR == FNR is only true while awk is reading the first file. Thus, in the program above, the actions indicated by # some actions are executed when awk is reading the first file; the actions indicated by # other actions are executed when awk is reading the second file, if the condition in # other condition is met. 

The next at the end of the first action block is needed to prevent the condition in # other condition from being evaluated, and the actions in # other actions from being executed, while awk is reading the first file.

Probably, it all becomes much clearer with some examples. There are really many problems that involve two files that can be solved using this technique. Let's look at this:

# prints lines that are both in file1.txt and file2.txt (intersection)

$ awk 'NR == FNR{a[$0];next} $0 in a' file1.txt file2.txt

Here we see another typical idiom: a[$0] alone has the only purpose of creating the array element indexed by $0, even if we don't assign any value to it. During the pass over the first file, all the lines seen are remembered as indexes of the array a. The pass over the second file just needs to check whether each line being read exists as an index in the array a (that's what the condition $0 in a does). If the condition is true, the line being read from file2.txt is printed (as we already know). In a very similar way, we can easily write the code to print the lines that appear in only one of the two files:

# prints lines that are only in file1.txt and not in file2.txt

$ awk 'NR == FNR{a[$0];next} !($0 in a)' file2.txt file1.txt

Note the order of the arguments. file2.txt is given first. To print lines that are only in file2.txt and not in file1.txt, just reverse the order of the arguments.

regex Patterns

Lookbehind Regex 

Normally we can grep will search for a word and prints the entire line which has the match word. 

Using -O option we can print only the matching word.

If we want to search a word and print the word next to it. then we need to use Look behind in regex.

Syntax for Look Behind is enter the match word between the parenthesed followed by ? and <. Add = if you want to do positive lookbehind or add ! if you want to do negative lookbehind. Something like -v in grep option.

(?<=matchword)\w*

       

Examples 







1) Lookbehind







[test@localhost ]# echo "Input : File Java.lanag.xyz..File 




copied completed : one.txt" | grep -oP '(?<=File copied 




completed : 




)\w*.*' 







Output : one.txt







2 ) Lookahead






[test@localhost ~]# echo "Input : File 



:Java.lanag.xyz..File copied completed : one.txt" | grep 



-oP '\w*(?= : \w*.txt)'



completed





[root@localhost log]#







3) Both





Input Text: applebananabananaapple 





(i)  banana(?=banana) - finds the 1st test ("banana" which 



has "banana" after it) 





(ii) banana(?!banana) - finds the 2nd test ("banana" which 



does not have "banana" after it)





(iii) (?<=apple)banana- finds the 1st test ("banana" which 



has "apple" before it)







(iv) (?<!apple)banana- finds the 2nd test ("banana" which 



does not have"apple" before it)       

What is Dump and Types of Dump

What is Dump and Types of Dump

In computing, a dump, core dump , memory dump, or system dump consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally. Core dumps are often used to assist in diagnosing and debugging errors in computer programs

core dump created by netdump, diskdump, xendump, or kdump. In this post we will see more about Kernel dump (Kdump). 

What is Kernel Dump (Kdump)

Kdump is a kernel crash dumping mechanism that allows you to save the contents of the system's memory for later analysis. It relies on kexec tools package. 

This second kernel resides in a reserved part of the system memory that is inaccessible to the first kernel. The second kernel then captures the contents of the crashed kernel's memory (a crash dump) and saves it

a) How to enable Kdump

To enable Kdump we need to add below keyword-value in the boot.conf at kernel line. While enabling crash kernel we need to reserve the memory for crash kernel. You can  set it to either auto or specific value. It is recommended to use minimum of 128M for a machine with 2G memory or higher.

root (hd0,0)

kernel /vmlinuz-2.6.32-419.el6.x86_64 ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

initrd /initramfs-2.6.32-419.el6.x86_64.img

b) Configure Dump Location

Once the kernel crashes, the core dump can be captured to local filesystem or remote filesystem(NFS) based on the settings defined in /etc/kdump.conf

This file is automatically created when the kexec-tools package is installed.

path /var/crash

core_collector makedumpfile -c --message-level 1 -d 31

In the file:

To write the dump to a raw device, you can uncomment “raw /dev/sda5” and change it to point to correct dump location.

For NFS, you can uncomment “#net my.server.com:/export/tmp” and point to the current NFS server location.

c) Configure Core Collector

The next step is to configure the core collector in Kdump configuration file. It is important to compress the data captured and filter all the unnecessary information from the captured core file.

To enable the core collector, uncomment the following line that starts with core_collector.

# core_collector makedumpfile -c --message-level 1 -d 31

makedumpfile specified in the core_collector actually makes a small DUMPFILE by compressing the data. makedumpfile provides two DUMPFILE formats (the ELF format and the kdump-compressed format). By default, makedumpfile makes a DUMPFILE in the kdump-compressed format.

The kdump-compressed format can be read only with the crash utility, and it can be smaller than the ELF format because of the compression support.

The ELF format is readable with GDB and the crash utility.

-c is to compresses dump data by each page

-d is the number of pages that are unnecessary and can be ignored

If you uncomment the line #default shell then the shell is invoked if the kdump fails to collect the core. Then the administrator can manually take the core dump using makedumpfile commands.

d) Restart kdump Services

Once kdump is configured, restart the kdump services,

If you have any issues in starting the services, then kdump module or crashkernel parameter has not been setup properly. So, verify /proc/cmdline and make sure it reflects to include the crashkernel value.

How to check core dump

Crash utility is used to analyze the core file captured by kdump.

It can also be used to analyze the core files created by other dump utilities like netdump, diskdump, xendump.

You need to ensure the “kernel-debuginfo” package is present and it is at the same level as the kernel.

Launch the crash tool as shown below. Once after entering this you will get a cash prompt, where you can execute crash commands:

# crash /var/crash/127.0.0.1-2014-03-26-12\:24\:39/vmcore /usr/lib/debug/lib/modules/`uname –r`/vmlinux

crash>

To view the Process when System Crashed

Execute ps command at the crash prompt, which will display all the running process when the system crashed.

crash> ps

   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM

      1      0   0  ffff88013e7db500  IN   0.0   19356   1544  init

      2      0   0  ffff88013e7daaa0  IN   0.0       0      0  [kthreadd]

      3      2   0  ffff88013e7da040  IN   0.0       0      0  [migration/0]

      4      2   0  ffff88013e7e9540  IN   0.0       0      0  [ksoftirqd/0]

      7      2   0  ffff88013dc19500  IN   0.0       0      0  [events/0]

Crash is the utility allows you to interactively analyze a running Linux system as well.

How to manually Trigger the Core Dump

You can manually trigger the core dump using the following commands:

echo 1 > /proc/sys/kernel/sysrq

echo c > /proc/sysrq-trigger

The server will reboot itself and the crash dump will be generated.

What is Primary, Extended and Logical Volumes

What is the need of Extended and Logical Partition

The original partitioning scheme for PC hard disks allowed only four partitions. This quickly turned out to be too little in real life, partly because some people want more than four operating systems (Linux, MS-DOS, OS/2, Minix, FreeBSD, NetBSD, or Windows/NT, to name a few), but primarily because sometimes it is a good idea to have several partitions for one operating system. For example, swap space is usually best put in its own partition for Linux instead of in the main Linux partition for reasons of speed.

To overcome this design problem, extended partitions were invented. This trick allows partitioning a primary partition into sub-partitions. The primary partition thus subdivided is the extended partition; the sub-partitions are logical partitions. They behave like primary partitions, but are created differently. There is no speed difference between them. By using an extended partition you can now have up to 15 partitions per disk.

The extended partitions is a way to get around the fact you can only have four primary partitions on a drive. You can put lots of logical partitions inside it

You will never see hda4 mounted, just hda5 and hda6, in this case. 

Note: 

Linux numbers primary partitions 1-4, logical partitions start at 5 and up, even if there are less than 4 primary partitions.

On an IDE drive you can have up to 63 partitions, 3 primary and 60 logical ( contained in one extended partition )

What is partition table

A partition table is a 64-byte data structure that provides basic information for a computer's operating system about the division of the hard disk drive (HDD).

Partition table is part of the master boot record (MBR), which is a small program that is executed when a computer boots (i.e., starts up) in order to find the operating system and load it into memory. 

A partition is a division of a HDD into logically independent sections. Primary partitions are the first four partitions on a HDD.

The partition table begins at the hexadecimal (i.e., base 16) position 0x1BE in the boot sector. It contains four entries, each of which is 16 bytes in length, one for each partition.

The partition table entry for each partition consists of six items: the active flag, with 0x00 for off and 0x80 for on (one byte); the starting head, cylinder and sector (three bytes); the filesystem descriptor (one byte); the ending head, cylinder and sector (three bytes); the starting sector relative to the disk beginning (four bytes); and the number of sectors in the partition (four bytes).

Where MBR and Partition table are stored

The MBR, and thus the partition table, is stored in the boot sector, which is the first physical sector of a HDD. A sector is a segment of a track on a magnetic disk (i.e., a floppy disk or a platter in a HDD).

A track is any of the concentric circles on the magnetic media on a disk or platter over which one magnetic head (i.e., a device used for reading and writing data on the disk) passes while the head is stationary but the disk is spinning. A platter is a thin, high-precision aluminum or glass disk that is coated on both sides with a high-sensitivity magnetic material and which is used by a HDD to store data.

The MBR reads the partition table to determine which partition is the active partition. The active partition is the partition that contains the operating system that a computer attempts to load into memory by default when it is booted or rebooted.