Pages

Tuesday, April 4, 2017

Array

List of Topics

1) What is array

2) Options with Array

1) What is Array

2) Syntax of Array

Name [index] = value

Where,

Name - Is any name for an array

Index could be any number or expression that must evaluate to a number greater than or equal to zero.You can declare an explicit array using declare -a arrayname.

If the elements has the white space character, enclose it with in a quotes.

3) Syntax of Array

declare -a Unix=('Debian' 'Red hat' 'Suse' 'Fedora');

declare -a arrayn=(element1 element2 element3)

echo ${arrayn[1]}



In bash, array is created automatically when a variable is used in the format like,

There are different ways to print the whole elements of the array. If the index number is @ or *, all members of an array are referenced. You can traverse through the array elements and print it, using looping statements in bash.


4) Other options with the array

a) To find length of the Array

echo ${#arrayname[@]} 
echo ${#arrayname[Index]} 


[user@Server ~]$ cat a.sh
#! /bin/bash
## Start of Script
Unix[0]='Debian'
Unix[1]='Red hat'
Unix[2]='Ubuntu'
Unix[3]='Suse Linux'



echo ${#Unix}  #Number of characters in the first element of the array.i.e Debian
echo ${#Unix[3]}
echo ${#Unix[@]}

##End of Script
[user@Server ~]$ sh a.sh
6
10

4

14. Deleting an Entire Array

unset is used to delete an entire array.

$cat arraymanip.sh

#!/bin/bash
Unix=('Debian' 'Red hat' 'Ubuntu' 'Suse' 'Fedora' 'UTS' 'OpenLinux');
Shell=('bash' 'csh' 'jsh' 'rsh' 'ksh' 'rc' 'tcsh');
UnixShell=("${Unix[@]}" "${Shell[@]}")
unset UnixShell
echo ${#UnixShell[@]}


$ ./arraymanip.sh
0

regex[1]="line1\|here1"
regex[2]="line2\|here2"
regex[3]="line3\|here3"
regex[4]="line4\|here4"

for i in "${!regex[@]}"
do
        grep -h -r --color=always "${regex["$i"]}" file.txt >> "pattern$i.txt"

done

#!/bin/bash

array=(un1Z1269E9909 rb1Z129098515)

echo ${array[0]:0:2}
echo ${array[1]:0:2}
three=${(array[3]:0:2):=word}
echo $three
var="$1"
var=${var:=word}
echo "$var"

FTP Configuration files and Port numbers

List of Topics

1) What is FTP
2) List of configuration files

3) Types of FTP
4) Sample Commands

1) What is FTP

FTP Linux Booting involves following five steps. BIOS, BOOT LOADER, KERNEL, INIT and RUNLEVEL. Below diagram is the exact procedure how Linux OS is booting. In the following sections we will see about  the below processes the in detail


2) Port Numbers

20 - Used for data channel
21 - Used for command channel

3) List of Configuration files

/etc/vsftpd/vsftpd.conf

/etc/vsftpd.ftpusers - blacklist file, users having entry cant login

/etc/vsftpd.user_list - This file can be configured to either deny access to the users listed or allow the listed users access, depending on whether the userlist_deny directive is set to YES (default) or NO in /etc/vsftpd/vsftpd.conf.

<br> # If userlist_deny=NO, only allow users in this file

# If userlist_deny=YES (default), never allow users in this file, and

/var/ftp/ - The directory containing files served by vsftpd. It also contains the /var/ftp/pub/ directory for anonymous users. Both directories are world-readable, 

ASCII is the default transmission mode of FTP

Changes in ftpusers and user_list doesnt require ftp daemon restart

4) Types of FTP

Active and Passive

Active - Client establishes command channel and Server establishes data channel
Passive - Client establishes both command and data channel.

Active - Client connects from a random unprivileged port number to command port (21) of server . Server then connects to N+1 port number from Data port (20). 

Passive - Client opens two unpriviledge port numbers  n and n+1. rom a random unprivileged port number to command port (21) of server. Client uses PASV command for the server to respond port number. Then again client initiates dat from N+1 port number to Server M port number

5) Sample Commands

cd - 
lcd - 
!dir 
!<command>
prompts
ASCII
Binary
get/mget
put/mput
delete
mls
mdelete

Saturday, April 1, 2017

All About Xargs

List of Topics

1) What is Xargs

2) Options with example's

1) What is Xargs

xargs command by default  expects the input from stdin, and executes /bin/echo command over the input.This manual page documents the GNU version of xargs.  xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial-arguments followed by items read from standard input.  Blank lines on the standard input are ignored.

2) Options with Examples

(i) -0     Input items are terminated by a null character instead of by whitespace, and the quotes and backslash are not special (every character is taken literally).  Disables the end of file string, which  is treated like any other argument.  Useful when input items might contain white space, quote marks, or backslashes.  The GNU find -print0 option produces input suitable for this mode.

(i) -a <file> - This will take file as an input.

(ii) -p - Will prompt the user before executing ever command 

(ii) -n <Integer> - n will split the number of inputs per line

(iv) -i - Should be used while using curley braces to replcae string

find . -name 'Type*' | xargs -i mv {} test

ls *.txt | xargs -p -n 1 -i mv {} xargs_test

xargs as called above does not need the {}. It appends the STDIN to the end of the command without a placeholder. The use of {} usually means you want the STDIN somewhere in the middle of the cmd to execute, as in:

$ ls -1 | xargs -i mv {} /path/to/someplace/.

The -I requires a defined placeholder. The -i option will assume {} is the placeholder. This was where I found any assumption of {} 


eg 2: cat 10linefl | awk -F":" '{print $1}' | xargs -n 1 id

Sample file


[bhr_moham607@l0202 ~]$ cat tlinefile

root:x:0:0:root:/root:/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin

Eg : 


[bhr_moham607@l0202 ~]$ xargs -a tlinefile

root:x:0:0:root:/root:/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin


Eg : for this example will take a file which is having 10 lines. 

#
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin

# xargs -n 2 -a 10lf
root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin  ucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin

Thursday, March 23, 2017

What is a Filesystem and Hierarchy of Linux File system

List of Topics

1) What is File system

2) What is Block
3) About Linux File system Hierachy
4) What is Boot block
5) What is Super block
6) What is Inode
7) What is Dentry

1) What is File system

In General a HDD is partitioned into sections. Partition is divided into number of data blocks with a fixed size which are used to store content of file. . Each section of a hard disk contains a single type of file system. Each type of file system has its own set of rules for controlling the allocation of disk, space to files and for associating data about each file(metadata)

2) What is Block

A block is a uniformly sized unit of data storage for a file system. Block size will be an important consideration when setting up a system that is designed for maximum performance. In case if file is of greater size than of block then it is stored in multiple blocks

Block size is selected at the time of formatting, i.e., preparing the hard disk drive (HDD) or other media for creation of a filesystem. If the mke2fs (i.e., make ext2 filesystem) command is used to create the filesystem, 

dumpe2fs command with the device name as an argument. Thus, for example, to find the block size for the second partition of the first HDD, the following can be used:

#/sbin/dumpe2fs /dev/hda2 | grep 'Block size'

#blockdev --getbsz /dev/sda1
4096

3) About Linux File system Hierarchy

Linux File System is mainly divided  into following 5 blocks
a) Boot block
b) Super block
c) Data Block
d) iNode
e) Dentry

4) What is Boot block

Boot Block is located in the first few sectors of a the root file system. The boot block contains the initial bootstrap program used to load the operating system. 

5) What is Superblock

Superblock is the metadata of file system and has information about the file system type, size, status, and information about structures. Superblock is very critical for the file system so it is stored in multiple places for each file system. Superblock is a very "high level" metadata structure for the file system. 

Will see what does metadata means. Metadata is a piece of information about the data. For example, if you own a Bike, then you will be having some set of information about the Bike like bike number,Manufacturer name, Model, insurance information, and so on which is not part of the Bike itself. All of that information is collectively referred to as the metadata. Similarly in Linux file systems metadata exists which holds information about file system.

Super Block: it contains info about
1. type of filesystem (ext2, ext3...)
2. the block size
3. pointers to a list of free blocks
4. the inode number of the root directory5
5. magic number

For example, if the superblock of a /opt gets corrupt then the file system cannot be mounted by the operating system. Normally in this case, you need to run fsck which will automatically select an alternate, backup copy of the superblock and attempt to recover the file system. The backup copies themselves are stored in block groups spread through the file system with the first stored at a 1 block offset from the start of the partition. 

In case if you want to do a manual recovery then you may need to know information about superblock backups. Following  command will give the location of super blocks.

#dumpe2fs /dev/<Filesystem> | grep -i superblock 


Output:
dumpe2fs 1.43-WIP (20-Jun-2013)
  Primary superblock at 0, Group descriptors at 1-1

  Backup superblock at 163840, Group descriptors at 163841-163841
  Backup superblock at 32768, Group descriptors at 32769-32769

  Backup superblock at 98304, Group descriptors at 98305-98305

Suppose if dumpe2fs command outputs the Backup superblock at 32768, Group descriptors at 32769-32769. Then to repair it we may need to the command like below.

#/sbin/fsck -b 163840  /dev/<partition name>

6) What is Inode

An inode is an entry in inode table, containing information ( the metadata ) about a regular file and directory. 

1. file ownership indication
2. file type (e.g., regular, directory, special device, pipes, etc.)
3. file access permissions. May have setuid (sticky) bit set.
4. time of last access, and modification
5. number of links (aliases) to the file
6. pointers to the data blocks for the file
7. size of the file in bytes (for regular files), major and minor  numbers for special devices
8. File types ( executable, block special etc )
9. Permissions ( read, write etc )
10. UID ( Owner )
11. GID ( Group )

An inode exists in, or on, a file system and represents metadata about a file. For clarity, all objects in a Linux or UNIX system are files; actual files, directories, devices, and so on. Please note that, among the metadata contained in an inode, there is no file name as humans think of it, this will be important 

Note that the mapping from dentries to inodes given by d_inode is in general a many-to-one mapping; a single file may be pointed to by multiple paths in the same filesystem (called "hard links"), in which case it will not be deleted as long as any such path exists.

Files and directories may also be opened by processes, of course, and a struct file is used to represent this.  The struct file contains a pointer to the dentry.  The underlying file will also not be deleted as long as there are processes holding the file open, even though that file may no longer be accessible by any path in the filesystem.

The namespace that a process sees, however, is normally made up of more than just one filesystem; instead it is patched together from multiple filesystems that are mounted on top of each other.  The structure of mountpoints is represented by a tree of vfsmount structures, one for each mountpoint.

7) What is Dentry 

A dentry is the glue that holds inodes and files together by relating inode numbers to file names. Dentries also play a role in directory caching which, ideally, keeps the most frequently used files on-hand for faster access. File system traversal is another aspect of the dentry as it maintains a relationship between directories and their files.

A filesystem is represented in memory using dentries and inodes.  Inodes are the objects that represent the underlying files (and also directories).  A dentry is an object with a string name (d_name), a pointer to an inode (d_inode), and a pointer to the parent dentry (d_parent).

           /

            |
            test
            |   \
            file1  file2

is represented by four inodes: one each for foo, bar, and file2, and the root; and three dentries: one linking bar to test, one linking file2 to foo, and one linking foo to the root.  The first of those dentries, for example, has a name of "fille1", a d_inode pointing to the underlying file file1, and a d_parent pointing to the dentry for test (which in turn would have a d_parent pointing to the dentry for the root).  The root dentry has a d_parent that points to itself.

Monday, March 20, 2017

About Logger Command

List of Topics

1) What is Logger

2) Examples of Logger with options
3) Where it can be used

1) What is Logger

logger - a shell command interface to the syslog system log module. In simple logger command can be used to add a message to /var/log/messages(by default)  and configurable to other files also. Below are some of the options of logger with example

Logger will exit with 0 on success, and greater than 0 in case of any errors.

2) Examples of Logger with options

Without any option logger will print the message in "messages" file like below.

# logger "hi test"
Output entry in messages : Mar 21 05:32:08 lapp02 bhr_moham607: hi test

a) - i  = Print the process id of the “logger” process.
Eg) for Option -i
# logger -i "test: print pid"

Output entry in messages : Mar 21 06:00:17 lapp02 bhr_moham607[2199]: test: print pid
In the above output 2199 is the PID of the logger.

b) -t tag   = Add the tag will printing the message in the logger.
Eg) for Option -t


# logger -t "TAG test" -i "test message"

Output entry in messages : Mar 21 05:55:48 lapp02 TAG test[1957]: test message




3) Where it can be used

Logger will be mainly useful while automating the task in background and also also while working with shell script.