What is a Filesystem and Hierarchy of Linux File system

List of Topics

1) What is File system
2) What is Block
3) About Linux File system Hierachy
4) What is Boot block
5) What is Super block
6) What is Inode
7) What is Dentry

1) What is File system

In General a HDD is partitioned into sections. Partition is divided into number of data blocks with a fixed size which are used to store content of file. . Each section of a hard disk contains a single type of file system. Each type of file system has its own set of rules for controlling the allocation of disk, space to files and for associating data about each file(metadata)

2) What is Block

A block is a uniformly sized unit of data storage for a file system. Block size will be an important consideration when setting up a system that is designed for maximum performance. In case if file is of greater size than of block then it is stored in multiple blocks

Block size is selected at the time of formatting, i.e., preparing the hard disk drive (HDD) or other media for creation of a filesystem. If the mke2fs (i.e., make ext2 filesystem) command is used to create the filesystem, 

dumpe2fs command with the device name as an argument. Thus, for example, to find the block size for the second partition of the first HDD, the following can be used:

#/sbin/dumpe2fs /dev/hda2 | grep 'Block size'

#blockdev --getbsz /dev/sda1

4096

3) About Linux File system Hierarchy

Linux File System is mainly divided  into following 5 blocks

a) Boot block

b) Super block
c) Data Block

d) iNode
e) Dentry

4) What is Boot block

Boot Block is located in the first few sectors of a the root file system. The boot block contains the initial bootstrap program used to load the operating system. 

5) What is Superblock

Superblock is the metadata of file system and has information about the file system type, size, status, and information about structures. Superblock is very critical for the file system so it is stored in multiple places for each file system. Superblock is a very "high level" metadata structure for the file system. 

Will see what does metadata means. Metadata is a piece of information about the data. For example, if you own a Bike, then you will be having some set of information about the Bike like bike number,Manufacturer name, Model, insurance information, and so on which is not part of the Bike itself. All of that information is collectively referred to as the metadata. Similarly in Linux file systems metadata exists which holds information about file system.

Super Block: it contains info about

1. type of filesystem (ext2, ext3...)

2. the block size

3. pointers to a list of free blocks

4. the inode number of the root directory5
5. magic number

For example, if the superblock of a /opt gets corrupt then the file system cannot be mounted by the operating system. Normally in this case, you need to run fsck which will automatically select an alternate, backup copy of the superblock and attempt to recover the file system. The backup copies themselves are stored in block groups spread through the file system with the first stored at a 1 block offset from the start of the partition. 

In case if you want to do a manual recovery then you may need to know information about superblock backups. Following  command will give the location of super blocks.

#dumpe2fs /dev/<Filesystem> | grep -i superblock 


Output:

dumpe2fs 1.43-WIP (20-Jun-2013)

  Primary superblock at 0, Group descriptors at 1-1

  Backup superblock at 163840, Group descriptors at 163841-163841

  Backup superblock at 32768, Group descriptors at 32769-32769

  Backup superblock at 98304, Group descriptors at 98305-98305

Suppose if dumpe2fs command outputs the Backup superblock at 32768, Group descriptors at 32769-32769. Then to repair it we may need to the command like below.

#/sbin/fsck -b 163840  /dev/<partition name>

6) What is Inode

An inode is an entry in inode table, containing information ( the metadata ) about a regular file and directory. 

1. file ownership indication

2. file type (e.g., regular, directory, special device, pipes, etc.)

3. file access permissions. May have setuid (sticky) bit set.

4. time of last access, and modification

5. number of links (aliases) to the file

6. pointers to the data blocks for the file

7. size of the file in bytes (for regular files), major and minor  numbers for special devices

8. File types ( executable, block special etc )

9. Permissions ( read, write etc )

10. UID ( Owner )

11. GID ( Group )

An inode exists in, or on, a file system and represents metadata about a file. For clarity, all objects in a Linux or UNIX system are files; actual files, directories, devices, and so on. Please note that, among the metadata contained in an inode, there is no file name as humans think of it, this will be important 

Note that the mapping from dentries to inodes given by d_inode is in general a many-to-one mapping; a single file may be pointed to by multiple paths in the same filesystem (called "hard links"), in which case it will not be deleted as long as any such path exists.

Files and directories may also be opened by processes, of course, and a struct file is used to represent this.  The struct file contains a pointer to the dentry.  The underlying file will also not be deleted as long as there are processes holding the file open, even though that file may no longer be accessible by any path in the filesystem.

The namespace that a process sees, however, is normally made up of more than just one filesystem; instead it is patched together from multiple filesystems that are mounted on top of each other.  The structure of mountpoints is represented by a tree of vfsmount structures, one for each mountpoint.

7) What is Dentry 

A dentry is the glue that holds inodes and files together by relating inode numbers to file names. Dentries also play a role in directory caching which, ideally, keeps the most frequently used files on-hand for faster access. File system traversal is another aspect of the dentry as it maintains a relationship between directories and their files.

A filesystem is represented in memory using dentries and inodes.  Inodes are the objects that represent the underlying files (and also directories).  A dentry is an object with a string name (d_name), a pointer to an inode (d_inode), and a pointer to the parent dentry (d_parent).

           /

            |

            test

            |   \

            file1  file2

is represented by four inodes: one each for foo, bar, and file2, and the root; and three dentries: one linking bar to test, one linking file2 to foo, and one linking foo to the root.  The first of those dentries, for example, has a name of "fille1", a d_inode pointing to the underlying file file1, and a d_parent pointing to the dentry for test (which in turn would have a d_parent pointing to the dentry for the root).  The root dentry has a d_parent that points to itself.

About Logger Command

List of Topics

1) What is Logger
2) Examples of Logger with options
3) Where it can be used

1) What is Logger

logger - a shell command interface to the syslog system log module. In simple logger command can be used to add a message to /var/log/messages(by default)  and configurable to other files also. Below are some of the options of logger with example

Logger will exit with 0 on success, and greater than 0 in case of any errors.

2) Examples of Logger with options

Without any option logger will print the message in "messages" file like below.

# logger "hi test"

Output entry in messages : Mar 21 05:32:08 lapp02 bhr_moham607: hi test

a) - i  = Print the process id of the “logger” process.

Eg) for Option -i

# logger -i "test: print pid"

Output entry in messages : Mar 21 06:00:17 lapp02 bhr_moham607[2199]: test: print pid

In the above output 2199 is the PID of the logger.

b) -t tag   = Add the tag will printing the message in the logger.

Eg) for Option -t

# logger -t "TAG test" -i "test message"

Output entry in messages : Mar 21 05:55:48 lapp02 TAG test[1957]: test message

3) Where it can be used

Logger will be mainly useful while automating the task in background and also also while working with shell script.

What is DAS, SAN, NAS

List of Topics

1) What is DAS
2) What is SAN
3) What is NAS

1) What is DAS
DAS is  known as Directly Attached Storage, which means physically a storage medium will be attached to a server eg: an USB drive, a firewire... Unfortunately this method does not have the ablity to scale.So this was not a much succesful method for storage.

2) What is SAN
SAN is known as Storage Area Network. To configure a SAN we need to attach a special device to server known as HOST BUS ADAPTER (HBA). From this host bus adapter a cable will connected to a gateway known as storage gateway. This network dedicated for storage and backup purpose is known as Storage area network. 

SAN allows multiple servers to connect to SAN network  to access centralized storage. Advantage of SAN is Maintenance and administration can be done from a  single location. 

LUN - A Logical unit which is storage attached to a server. SAN is bit expensive. and requires a special administration.

3) What is NAS
If we don't want to create a separate network for Storage we can use existing network for storage which does'nt need HBA and will communicate with NIC and existing LAN. But NAS traffic has to compete with existing LAN traffic.

All About SSH Keys & It's files

List of Topics

1) Types of Authentication of SSH
2) Files for Host Based Authentication
3) Files for Key Based Authentication

4) Last method of authentication
5) What is known_hosts file

1) Types of Authentication in SSH

SSH has two types of Authentication mode, below are the types of authentication with its small description. 

Host based authentication is where the host (or server) is authenticated so users can just authenticate without the need of a key.

Key base authentication is where the user generates public and private key, where the public key need to copied to the remote server(i.e wherever the user has to login). By using this method we can login without password's.

2) Files for Host Based Authentication

Host-based authentication works as follows: If the machine the user logs in from is listed in /etc/hosts.equiv or /etc/ssh/shosts.equiv on the remote machine, and the user names are the same on both sides, or if the files ~/.rhosts or ~/.shosts exist in the userâs home directory on the remote machine and contain a line containing the name of the client machine and the name of the user on that machine, the user is considered for login.   Additionally, the server must be able to verify the clientâs host key (see the description of /etc/ssh/ssh_known_hosts and ~/.ssh/known_hosts, below) for login to be permitted.  This authentication method closes security holes due to IP spoofing, DNS spoofing, and  routing spoofing.  [Note to the administrator: /etc/hosts.equiv, ~/.rhosts, and the login/rsh protocol in general, are inherently insecure and should be disabled if security is desired.]

3) Files for Key Based Authentication

Key based authentication uses the following files  
~/.ssh/authorized_keys lists the public keys that are permitted for logging in.  When the user logs in, the ssh program tells the server which key pair it would like to use for thentication.

id_dsa, id_rsa, id_ecdsa -  if you use dsa protocol then these files will be created id_dsa.pub is for public key and without extension is for private key

The user creates his/her key pair by running ssh-keygen(1).  This stores the private key in ~/.ssh/identity (protocol 1), ~/.ssh/id_dsa (protocol 2 DSA), ~/.ssh/id_ecdsa (protocol 2 ECDSA), or ~/.ssh/id_rsa (protocol 2 RSA) and stores the public key in ~/.ssh/identity.pub (protocol 1), ~/.ssh/id_dsa.pub (protocol 2 DSA), ~/.ssh/id_ecdsa.pub (protocol 2 ECDSA), or ~/.ssh/id_rsa.pub (protocol 2 RSA) in the user’s home directory.  The user should then copy the public key to ~/.ssh/authorized_keys in his/her home directory on the remote machine.  The authorized_keys file corresponds to the conventional ~/.rhosts file, and has one key per line, though the lines can be very long.  After this, the user can log in without giving the password.

3) What are id_dsa, id_rsa, id_ecdsa files

The user creates his/her key pair by running ssh-keygen(1).  This stores the private key in ~/.ssh/identity (protocol 1), ~/.ssh/id_dsa (protocol 2 DSA), ~/.ssh/id_ecdsa (protocol 2 ECDSA), or ~/.ssh/id_rsa (protocol 2 RSA) and stores the public key in ~/.ssh/identity.pub (protocol 1), ~/.ssh/id_dsa.pub (protocol 2 DSA), ~/.ssh/id_ecdsa.pub (protocol 2 ECDSA), or ~/.ssh/id_rsa.pub (protocol 2 RSA) in the user’s home directory.  The user should then copy the public key to ~/.ssh/authorized_keys in his/her home directory on the remote machine.  The authorized_keys file corresponds to the conventional ~/.rhosts file, and has one key per line, though the lines can be very long.  After this, the user can log in without giving the password.

4) Last method of authentication

Finally, if other authentication methods fail, ssh prompts the user for a password.  The password is sent to the remote host for checking; however, since all communications are encrypted, the password cannot be seen by someone listening on the network. ssh automatically maintains and checks a database containing identification for all hosts it has ever been used with.  

5) What is known_hosts file

Host keys are stored in ~/.ssh/known_hosts in the user’s home directory.  Additionally, the file /etc/ssh/ssh_known_hosts is automatically checked for known hosts.  Any new hosts are automatically added to the user’s file.  If a host’s identification ever changes, ssh warns about this and disables password authentication to prevent server spoofing or man-in-the-middle attacks, which could otherwise be used to circumvent the encryption.  The StrictHostKeyChecking option can be used to control logins to machines whose host key is not known or has changed.

6) Generating ssh keys

ssh key without pasphrase

# ssh-keygen -t rsa -N "" -f explore

To mention the private key file name while logging in

#ssh -i <PRIVATE_KEY_FILE>   <USER_NAME>@<HOSTNAME>

The StrictHostKeyChecking option can be used to control logins to machines whose host key is not known or has changed.

What is Throughput and Latency

List of Topics

1) What is Throughput

2) What is Latency

3) Example for Throughput and Latency

1) What is Throughput

In Simple terms, Throughput is the maximum rate of production or the maximum rate at which something can be processed. Data transfer rates for disk drives and networks are measured in terms of throughput. Typically, throughput is measured in kbps, Mbps and Gbps.

Network throughput is the rate of successful message delivery over a communication channel

2) What is Latency 

Latency, therefore, is wasted time. For example, in accessing data on a disk, latency is defined as the time it takes to position the proper sector under the read/write head. In networking, the amount of time it takes a packet to travel from source to destination. Together, latency and bandwidth define the speed and capacity of a network

3) Example for Throughput and Latency

Eg 1), To Understand in laymen way. Let us take speed as the parameter of throughput. There is Source and destination and the car is allowed to run at a speed of 80 KM/h. So the throughput at this point is 80 Km/H. Normally car will reach the destination in 1 hour but in case of any delay due to traffic that delay is named as Latency. 

Eg 2), Throughput is basically how long it takes for a company to get a product from the beginning to the end. The objective of throughput is to minimize the amount of time needed to get a product, or good, through the entire production process so that a company can ultimately increase their profitability. The faster a company can move a product, the more they can produce per day, and the more money they will make

What is SELINUX

Security Enhanced Linux is a Mandatory Access Control Security Mechanics implemented in the Kernel. Selinux follows the model of Least privilege more closely.

List of Topics

1) What is Selinux

2) What is the difference between MAC and DAC

3) List of Configuration Files

4) Modes of Operation 

5) Policy  

6) SELinux Context

7) How to see SELinux Context
8) List of Commands and its Explanation

1) What is SELinux

As noted, SELinux follows the model of least-privilege; by default everything is denied and then a policy is written that gives each element of the system only the access required to function. This description best describes the strict policy

2) What is the difference between MAC and DAC

Discretionary Access Controls (DAC) - Is a basic access control policies to files. These are set at the discretion of the owner of the objects. Example, user and group ownership or file and directory permissions.

Mandatory Access Controls (MAC) are system-controlled access control policies where the system dictates and controls the level of access to an object, even a user created one. 

3) List of Configuration files

(i)  /etc/selinux/config - Default configuration file of SELinux. Moded of the SELinux need to be configured for permanent change.
(ii)  /etc/sestatus.conf - sestatus -v, will refer this file and displays the context

(iii) /etc/selinux/semanage.conf

(iv) /etc/selinux/restorecond.conf

(v)  /etc/security/sepermit.conf 

4) Modes of Operation

SELinux Operates in following three modes
(a) Enforcing
(b) Permissive

(c) Disabled

a) Enforcing - This is Default Mode of SELinux, which will enable the SELinux. 

b) Permissive - In this mode, SELinux is enabled but it will not enforce the security policy, It will only warn and log the actions. Permissive mode is useful to troubleshoot SELinux issues
c) Disabled - SELinux is turned off.

Note:- While changing the SELinux mode from disabled to Enforce/Permissive reboot is recommended as the filesystem need to be relabelled. Relabelling will take some time and it depends upon the size of filesystem.

5) Policy

SELinux allows different policies to be written that are interchangeable. The default policy in RHEL is the targeted policy which "targets" and restricts selected system processes. In RHEL 4 only 15 defined targets existed (including httpd, named, dhcpd, mysqld). Later, in CentOS 5 this number had risen to over 200 targets.

The default policy in RHEL is targeted policy which targets and restricts selected system process.

By convention all confined(restricted) executable have a label type that ends with exec_t

6) Selinux Context 

Selinux context are named based on below three

(a) User - Is the First attribute in the Security context
(b) Roles based access control - Is the Second attribute in the Security context
(c) Type Enforcement - Is the Third attribute in the Security context. Also Known as domain type.

(d) MLS/MCS (Multi Level Security/Multi Category Security) - Multi Category Security and Multi Level Security are mutually exclusive, This is hidden field and it is in the fourth attribute of the Security context.

To check the selinux context execute ls -Z <filename>

Fields - User:Role:Type:MLS (Hidden Field)

7) How to see the SELinux Context

ls -Z filename  - To check the context of a file

id -Z                - To check the context of a user

ps -eZ             - To check the context of processes

8) What is Boolean

SELinux has a set of built-in switches named Booleans or conditional policies that you can use to turn specific SELinux features on or off.

Entering the "getsebool -a | grep http" command lists the 23 Booleans related to the http daemon, which are a subset of the 234 Booleans currently defined in the selinux-policy-2.4.6-203.el5 policy. 

getsebool allow_console_login

getsebool allow_console_login allow_cvs_read_shadow allow_daemons_dump_core

These 23 Booleans allow you to customize SELinux policy for the http daemon during runtime without modifying, compiling, or loading a new policy. You can customize the level of http security by setting the relevant Boolean values or toggling between on and off values.

Acceptable values to enable a Boolean are 1, true, and on. Acceptable values to disable a Boolean are 0, false, and off

Viewing affected policy rules

To view the policy rules that are enabled (or disabled) when a boolean is set (or unset), use sesearch:

root #sesearch -b user_dmesg -AC

Found 4 semantic av rules:

ET allow user_t kernel_t : system syslog_read ; [ user_dmesg ]

ET allow user_t user_t : capability2 syslog ; [ user_dmesg ]

ET allow staff_t kernel_t : system syslog_read ; [ user_dmesg ]

ET allow staff_t staff_t : capability2 syslog ; [ user_dmesg ]

In the above example, user_dmesg is currently enabled. As a result, all four rules are shown as enabled as well (E). The second character (T) tells us that the rule becomes active if the boolean is enabled (T stands for True).

9) List of Commands and its Explanation

a) setstatus - Will display the status of SELinux 

    Options:

    -b Displays all Booleans and their statuses

    -v Provides verbose output

b) getenforce - prints the current mode of SELinux

c) setenforce – command to change the SELinux mode on the fly, but changes do not persist through a system reboot. For permanent change do the change in  /etc/selinux/config. Also with setenforce command we can't change the mode to disabled. Either permissive or enabled only possible and even if the SELinux is disabled we can't use setenforce to enable it

d) getsebool - Returns the Boolean value of a service option

e) setsebool - Sets the Boolean value of a service option, -P Makes the changes persistent

    

f) chcon - Changes the context of a file, directory, or service

   Options:

   -f  To Suppresses error messages

   -u To change user context

   -r  To change role context

   -t  To change domain context

   -R To do the changes recursively

   -v Provides verbose output

g) restorecon - Resets the context of an object

    Options:

    -i Ignores files that don’t exist

    -p Shows progress

    -v Shows changes as they happen

    -F Resets context

h) semanage -To review the status of current users, run the semanage login -l command

Use of Touch Command

List of Topics

1) About Touch Command and its uses

2) Available time stamp's

3) How to Check time stamp of a file

4) Command line options to modify time stamp

1) About Touch Command and its uses

Everyone is aware that “touch” is the command to create an empty file but apart from that touch command is mainly used to change the time stamp's of the file.

2) Available Time stamp's

Every file/folder in linux has below three time stamps

a) Last access time

b) Last Modification Time

c) Last Change time

Whenever we create a file above mentioned time stamp's will be updated automatically.

3) How to check time stamp of a file

We can change the above time stamp's with the help of touch command. 

Before modifying will first see how to check these time stamps. To check these time stamps we need to use "stat" command. An Sample output of stat command.

[XXXX@server2 ~]# stat test_file

  File: `test_file'

  Size: 0               Blocks: 0          IO Block: 4096   regular empty file

Device: fd00h/64768d    Inode: 791598      Links: 1

Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2017-03-07 02:19:25.612999258 +0530

Modify: 2017-03-07 02:19:25.612999258 +0530

Change: 2017-03-07 02:19:25.612999258 +0530

4) Command line options to modify time stamp

Below are the command line options to be used with touch command to change the access and modification time

Note: It is not possible to change ctime using touch command

(i)  -a  -> To Change access time alone. To Change the access time of the file just execute touch -a <filename>. By default it will take the current time and update the access time

(ii)  -m -> Similar to -a option just use -m to change the file modification time.

(iii) -r   -> This will copy the timestamp from another file eg: touch <file1> -r <file2>

(iv) -t & -d -> this options are used to explicitly change the time to our value.

      Syntax -> touch -t  [[Century]YY]MMDDhhmm[.ss]

                      touch -d  "YYYY-MM-DD hh:mm:ss" <filename>