UNIX

Yum for CentOSians

Lets say someone wants to install a package, or update a package or just wanted to check if there is any update of a package. The command "yum" would be pretty helpful for them. The basic syntax for yum would be like below:

yum install 
Used to install the latest version of a package or group of packages.

yum update 
Used to update the specified packages to the latest available version.

yum update stuck due to broken package: (Lets say due to firefox and java packages)

                                  vi /etc/yum.conf --> In main section add below line:
                                  exclude=firefox* java*

Then run the update again like below:
                                  yum update --skip-broken

yum check-update
This command allows you to determine whether any updates are available for your installed packages. yum returns a list of all package updates from all repositories if any are available.

yum localinstall 
Used when using yum to install a package located locally in the machine

yum provides 
Used to determine which packages provide a specific file or feature.

yum search 
This command is used to find any packages containing the specified keyword in the description, summary, packager and package name fields of RPMs in all repositories.

To search for a specific package by name, use the list function. To search for the package tsclient, use the command: (provide root's password when prompts)

su -c 'yum list tsclient'

What about uninstalling a package? Yes yum will again help you in this regard.

yum remove 
Used to remove specified packages, along with any other packages dependent on the packages being removed.

By default, yum is configured through /etc/yum.conf.

cat /etc/yum.conf

[main]
cachedir=/var/cache/yum
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
metadata_expire=1800

[myrepo]
name=RHEL 5 $releasever - $basearch
baseurl=http://local/path/to/yum/repository/
enabled=1
A typical /etc/yum.conf file is made up of two types of sections: a [main] section, and a repository section. There can only be one [main] section, but you can specify multiple repositories in a single /etc/yum.conf.

Extract rpm/jar/tar/iso in Windows 7

Windows 7-Zip from All Programs Menu

How about extracting .rpm/iso/tar/jar files on windows 7 machine?

Once of the best utilities i have come across in recent years in 7-Zip. I was particularly interested in jar, tar, iso and rpm files extraction. Other than these four  extensions, 7-Zip supports a number of other compression and non-compression archive formats (both for packing and unpacking) including 7z, ZIP, GZip, bzip2, xz and WIM. The utility also supports unpacking APM, ARJ, CHM, cpio, DEB, FLV, JAR, LHA/LZH, LZMA, MSLZ, onepkg, RAR, RPM, smzip, SWF, XAR and Z archives and CramFS, DMG, FAT, HFS, ISO, MBR, NTFS, SquashFS, UDF and VHD disk images.

7-Zip can open some MSI files, allowing access to the meta-files within along with the main contents. Some Microsoft CAB (LZX compression) and NSIS (LZMA) installer formats can be opened. Similarly, some Microsoft executable programs (.EXEs) which are self-extracting archives or otherwise contain archived content (e.g., some setup files) may be opened as archives.

The utility can be downloaded from here:
http://www.7-zip.org/download.html

fuser and pfiles: lsof equivalent in Solaris

Lets talk about lsof. It displays information about files which are opened by running(active) processes. An open file may be a regular file like text file or any script, directory, NFS file, block special file, character special file, shared library, regular pipe, named pipe, symbolic link, socket, anything. In Unix, we consider everything is a file, so you can think of how incredibly smart utility lsof is. Generally it is supplied with Linux.

But the question is, what is its Solaris equivalent? 

After some hit and misses i have managed to run an equivalent command. fuser. The fuser syntax that worked for me is :

sudo fuser -vm / 2>&1 | awk '$3 ~ /f|F/' | less

Another one is pfiles. It is not necessarily supplied with the default package.

pfiles /proc/*

Run fsck on a filesystem to repair it..

fsck -y -o b=32 /filesystem

This will repair the corrupted file system to its best and restore the data in /filesystem/lost+found directory. Data in this directory will have "#inode name" nomenclature. From there the data can be copied back to the filesystem. Ofcourse the techie who is repairing the FS must have prior knowledge about the directory structure in filesystem otherwise he would not be able to decide where to copy which file from lost+found directory. 

Note: This is system administrator's work, so please do not run mount/umount/fsck sort of commands with minimal knowledge. The fsck command can have other parameters as well. So depending on the need of the work, one can decide the exact fsck command. Above fsck command is most generic one i came across and applied while **working on real time issues**. Also please keep in mind that the filesystem has to be unmounted first (umount /filesystem) then run fsck and then mount (mount /filesystem /mountpoint) the filesystem back if properly repaired. Then only you will be able to see the files for restore in /lost+found directory.

Mount ISO image in Linux

Getting a lot of emails regarding mounting ISOs in linux. Please use the following steps in order to mount the image to FS.

Procedure to mount ISO images in Linux.

1) Login as a root user:
$ su -

2) Create a mount point:
# mkdir -p /mnt/disk

3) Mount iso file called mountthisdisk.iso:
# mount -o loop mountthisdisk.iso /mnt/disk

4) Verify it:
# cd /mnt/disk
# ls -l

In case you have not noticed "loop" in the mount command, let me tell you that A loop device is a pseudo-device that makes a file accessible as a block device. Loop devices are often used for CD ISO images and floppy disc images. Mounting a file containing a filesystem via such a loop mount makes the files within that filesystem accessible. They appear in the mount point directory using above commands.

Command to covert decimal to binary in shell:

echo "ibase=10;obase=2;128" | bc

Find files which are older than 15 minutes in Linux and Solaris..

The below example is for moving (mv) files, same can be changes easily for rm, ls, cp etc. One has to make change in xargs paramaeter.

In LINUX:
=========

find /path_to_directory/ -name "*.rpt" -type f -mmin +15 | xargs -I {} mv {} /target_directory_path/


In Solaris:
===========

#! /usr/bin/sh

#### Create a reference file minute_test
touch /path_to_directory/minute_test

#### Make minute_test 15 minutes old
sleep 900

#### Move all .rpt files to some other directory which are older than reference file minute_test
find /path_to_directory/ -name "*.rpt" -type f -newer minute_test | xargs -I {} mv {} /target_directory_path/

#### Remove reference file
rm -f /path_to_directory/minute_test

xargs with cp and xargs with mv command

Find with xargs and mv and cp, On public demand :)
=======================================
find . -type f  | xargs -I {} mv {} /target_directory_path/
find . -type f  | xargs -I {} cp -p {} /target_directory_path/

Print lines of file 1 which are not present in file 2. Also second field of file 2 may be different from file 1.

#!/usr/bin/perl -w
use strict;
use English qw(-no_match_vars);
die "Manadatory Parameter missing!Exiting..." if @ARGV < 2;
my ($file1, $file2) = @ARGV;
open my $file_handle1, "<", $file1 || die "Error opening $file1 - $OS_ERROR";
open my $file_handle2, "<", $file2 || die "Error opening $file2 - $OS_ERROR";
while (<$file_handle1>) {
    my $file1_line = $_;
    my $file1_id   = ($file1_line =~ /^(.+?)\;/) [0];
    my $match_found;
    while (<$file_handle2>) {
        my $file2_line = $_;
        my $file2_id   = ($file2_line =~ /^(.+?)\;/) [0];
        if ($file1_id == $file2_id) {
            $match_found = 1;
            last;
        }
    }
    seek $file_handle2, 0, 0;
    print $file1_line if ! $match_found;

}

Count the number of occurances of a pattern in a file using awk

awk '{ for (i=1;i<=NF;i++) if ( $i == "word" ) count++ } END{print count}' filename

This will print the count of the occurance of  a pattern "word" in a file.

How to compare the values of a column in awk in a same file and consecutive lines..

If one would like to compare the values of 2nd column of consecutive lines of same file in such a way so that if the difference between first value and second value is more than 100 it should print complete line else ignore line.

Input File:

=======

ABC 2500

ABCD 123

XYZ 122

WXYZ 2565

Desired Output:

==========

ABC 2500 (i.e. difference between 2500 and 123 is greater than 100 here)

XYZ 122 (i.e. difference between 122 and 2565 is greater than 100 here)

Command:

=======

awk 'NR % 2 != 0 {a=$1; b=$2} NR % 2 == 0 {if (b - $2 > 100 || $2 - b > 100){print a,b}}' inputfile

Count the number of files in each directory recursively

find ./ -type d -exec sh -c "echo -n {} ' ' ; ls -l {} | wc -l" \;

Use awk in following cases

Joining two files parallely using awk:
========================
awk 'NR==FNR{a[NR]=$0; next} {print a[FNR], $0}' file1.txt file2.txt

Changing extension of all .txt to .sh:
========================
for i in *.txt;do mv "$i" "${i%.txt}".sh;done

Print 3 lines before and 5 lines after pattern match abcd in a file a.txt

In Linux:

awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=3 a=5 s="abcd" a.txt

In SOLARIS:

nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=3 a=5 s="abcd" a.txt

ftp in UNIX

ftp -n 10.200.120.12 << EOF
quote user username
quote pass password
cd /path/to/directory/
prompt off
put dump.txt
exit
EOF

Adding 30 days to current date using PERL inside SHELL

#! /usr/bin/ksh

dt1=`perl -l -e '$date=(localtime(time + (30 * 24 * 60 *60))); print $date;'`

dt=`echo $dt1|awk '{if($3<10) print $1" "$2" 0"$3" "$4" "$5;else print $0}'`

echo $dt|awk '{print $5"-"$2"-"$3"23:59:59"}'|sed -e 's/Jan/01/g'\;'s/Feb/02/g'\;'s/Mar/03/g'\;'s/Apr/04/g'\;'s/May/05/g'\;'s/Jun/06/g'\;'s/Jul/07/g'\;'s/Aug/08/g'\;'s/Sep/09/g'\;'s/Oct/10/g'\;'s/Nov/11/g'\;'s/Dec/12/g'

Print 3 lines before pattern match using sed and awk ..

To reverse a file using awk and print next three lines after pattern match (Including pattern):

awk '{ a[i++] = $0 } END { for (j=i-1; j>=0;) print a[j--] }' abcd.txt |sed -n '/PRDS/{p;n;p;n;p;n;p;}' >abcd_new.txt

(This can be used when you want to print 3 lines before pattern match, including pattern. Logic: reverse a file and print 3 lines after pattern match.)

How to reverse a file using awk:

awk '{ a[i++] = $0 } END { for (j=i-1; j>=0;) print a[j--] }' abcd.txt

How to reverse a file using sed:

sed -n '1!G;h;$p' abcd.txt

Print one line before pattern match using sed in UNIX.

sed -n -e '/prds/{x;p;x;p;}' -e h test.txt

Adding one month (30-31 days) in UNIX SOLARIS

#! /usr/bin/ksh

sort a delimited file on the basis of a column

sort -t":" -k3 filename.txt

(The above will sort a file filename.txt which is ":" delimited on the basis of third column.)

Print every 6th line of a file ..

The below awk command will print every 6th line of a file test.txt starting from line number one.

awk '{if(NR%6==1) print $0}' test.txt

(Note: Print every nth line using awk.)

Sed Options

use sed when ....

# insert a blank line above every line which matches "regex"
 sed '/regex/{x;p;x;}' OR  sed '/regex/{x;G;}'

# insert a blank line below every line which matches "regex"
 sed '/regex/G'

# insert a blank line above and below every line which matches "regex"
 sed '/regex/{x;p;x;G;}'

# remove the header of a file

sed '/1,1d' filename

Basic UNIX Questions

All questions are for command line use in UNIX environments. 

Q: How do you display your running kernel version? (Solaris, AIX, Linux)
A: Linux # uname –r , Solaris # showrev

Q: Which command do you use to display a table of running processes? (Solaris, AIX, Linux)
A: Linux # ps –ef and top , Solaris # prstat 

Q: Which file do you modify to configure a domain name resolver? (Solaris, AIX, Linux)
A: Linux # /etc/resolv.conf , Solaris # /etc/resolv.conf

Q: Which file contains a list of locally defined hostnames and corresponding IP addresses? (Solaris, AIX, Linux)
A: Linux # /etc/hosts , Solaris # /etc/hosts and linked file /etc/inet/hosts 

Q: How do you display a routing table? (Solaris, AIX, Linux)
A: Linux # ip route show or #netstat –nr or #route –n and Solaris # netstat –nr and #route -n

Q: Which command would you use to view partitions and their sizes on Solaris?
A: # df -kh

Q: Which OpenBoot command would you use to print/view OpenBoot environment variables on a SUN server?
A: #printenv

Q: What does "ypwhich" command do? (Solaris, AIX, Linux)
A: # Will display NIS server to which client is connected to and which NIS Server is master for particular map specified with this command

Q: which command would you use to create an OS user on Solaris and Linux?
A: Linux # useradd and Solaris #useradd

Q: Which file contains passwords for local users on Solaris, Linux and on AIX?
A: Linux #/etc/shadow and Solaris # /etc/shadow

Q: Which command would you use to list partitions on Linux?
A: Linux # mount –l or # df -kh

Q: Which command/commands would you use to manage installed packages on RedHat Linux?
A: Linux # rpm 

Q: What is the default port for SSH server? 
A: 22

Q: Which command/commands would you use to manage installed packages on Solaris?
A: #pkginfo #pkgrm # pkgadd #pkgchk

Q: What command would you use to install an OS patch on Solaris?
A: #showrev –p and #patchadd -p

Q: Which Veritas command would you use to display a list of Veritas volumes?
A: # vxprint

Q: Which Veritas command would you use to display a list of disks on a system?
A: # vxdx list

Q: What is the main system configuration utility in AIX?
A: 

Q: Which file has a list of filesystems to be mounted at boot time on Solaris, Linux and AIX?
A: Linux # /etc/fstab and Solaris #/etc/vfstab

how to get patterns of file 1 "a" in file 2 "b"

#! /usr/bin/sh
 









while read line





do





grep $line b >>final



done<a

how to get yesterday date in unix

YEST=`TZ="GMT+24" date +'%m/%d/%Y'`; echo $YEST

Print next 7 lines of the file sample2 after pattern match from file sample1:

for i in `cat sample1`;do sed -n "/$i/{p;n;p;n;p;n;p;n;p;n;p;n;p;n;G;p;}" sample2 ; done

UNIX interview questions and best practices.....

Q1.Write a shell script in Linux to shift all characters in a file forward by five characters. (Thus "a" becomes "f'").
Solution: cat test tr '[a-z A-Z]' '[f-za-e F-ZA-E]'

Q2.I have 2 files:-
file1 and file2

file1
SEED
RPTT
TST8

file2
SEED:db:Y
RPTT:db:Y
SED8:db:N
SEED:db:N
TST8:db2:Y
TRN8:db3:N
CNV8:db4:Y

I have to change third field of file2 to "y" for every entry in file1 matches first filed of file 2
and rest to N

I want file 2 like :-
as file 1 has (SEED,RPTT,TST8) so in file2 I want to do third field "y" for those entries(which is first field in file2).
In rest I want to convert to "N"
SEED:db:Y
RPTT:db:Y
SED8:db:N
SEED:db:Y
TST8:db2:Y
TRN8:db3:N
CNV8:db4:N

Solution: while IFS=: read a b _ ; do echo -n "$a:$b:" ; ([[ $(grep -w $a file1.txt) ]] && echo Y) echo N ; done <>

Q3. How u convert string "hi prds how are you?" to "Hi prds How Are You?" 
Solution: echo "hi prds how are you"sed -e 's/ [a-z]/\U&/g'\;'s/^[a-z]/\U&/g'

OR

echo "hi prds how are you" sed 's/\<[a-z]*\>/\u&/g'

Q4. write a shell script that counts a number of unique word contained in the file and print them in alphabetical order line by line?
Solution: $ rm -f res res1 ; while read line ; do cat uniqtest grep -wc $line >>res1 ; echo "$line :-> Count=" >>res ;done final_temp ; cat final_tempsort -u >final ; rm -f res res1 final_temp1
(Check file "final" for output)

Q5. How to rename all the files in a folder having specific extension? Example: I have some files with extension (.txt) in a folder name 'Test'. I have to rename all the .txt files in a test and its subdirectories to .my extension. 
Solution: for file in `ls` ; do NEW=`echo $file sed 's/.txt/.my/g'` ; mv $file $NEW ; done

Q6. write a shell script to identify the given string is palindrome or not? 
Solution: 
#! /usr/bin/sh
len=0
i=1
tag=0
echo -n "Enter a String: "
read str
len=`echo $str wc -c`
len=`expr $len - 1`
halfLen=`expr $len / 2`

while [ $i -le $halfLen ]

do
c1=`echo $strcut -c$i`
c2=`echo $strcut -c$len`
if [ $c1 != $c2 ] ; then
i=$halfLen
tag=1
fi
i=`expr $i + 1`
len=`expr $len - 1`
done


if [ $tag -eq 0 ]

then
echo "String is Palindrome"
else
echo "String is not Palindrome"

fi

Q7. one file is a.txt which has 4 fields/columns, where as each field has separated by space and that field contains some data like below

ram usa 105 25
rag uk 115 26
pat ind 234 23
sah 425 24
tat usa 344 28
rat brz 536 29

now i want only 2nd field data, that should be like this

usa
uk
ind

usa
brz

(i.e. It should print blank line for the second field if it doesnt have country defined)
Solution: awk 'NF < 4 {$2=""} { print $2 }' a.txt

Q8. How to change all .my files to .txt in current directory??
Solution: for i in `ls *.my` ; do NEW=`echo $ised -e 's/.my/.txt/g'` ; mv $i $NEW ; done

Q9. How to change all .extn files to .my in entire directory recursively??
Solution:
find ./ -type f grep ".extn$" >files.txt;for i in `cat files.txt`;do NEW=`echo $ised 's/.extn/.my/g'`;mv "${i}" "${NEW}";done;rm -f files.txt

Q10: What is umask??
Answer: umask is used to set default permission level in entire unix system. It can be set in .kshrc/.bashrc/.cshrc or init (So that to load at the time of login into shell)

Q11. What is inode??
Answer : inode represents any file/directory residing in your unix system with a numner known as inode number. It is unique for every file/dir. It is basically a reference that kernel use for any work related to that entity. "ls -i filename" will give you the inode number of that particu;ar file.

Q12. Explain general commands like ps,df, export, env, ufsdump, tar, cron, system.
Answer: EASY ONE

Q13. Explain the make utility in unix?? what are the parameters of a makefile and how to use them. What are macros in makefile.
Answer: Unix make utility is used to build a software package. It requires a makefile where all the bunch of the scripts/codes/programs are declared/defined. The main parameters of the make file are variable declaration, Macro declaration, Phony Target declaration and Clean-up block. Macros in a makefile represents the order in which the scripts would get executed.

Q14. What are the benefits of makefile over shell script??
Answer: Using makefile one can run the different scripts or programs or codes in one go. This can also be done by making a wrapper shell script which can execute the different scripts in one go. But then it would not be termed as a package because when we say a package it means that we have one installer (.exe) which can execute the bunch more efficiently and also this .exe will be in encrypted format. So in order to maintain efficiency and security makefile is given precedence over wrapper shell scripts.

Q15. What are hard links in UNIX??
Answer: A hard link is a reference to a file or directory that appears just like a file or directory, not a link. Hard links only work within a filesystem. In other words, don't use hard links between mounted filesystems. A hard link is only a reference to the original file, not a copy of the file. If the original file is deleted, the information will be lost.
To create a hard link of the file /export/home/fred/stuff to /var/tmp/thing, use:

ln /export/home/fred/stuff /var/tmp/thing

The syntax for creating a hard link of a directory is the same. To create a hard link of /var/www/html to /var/www/webroot, use:

ln /var/www/html /var/www/webroot

Q16. What are the soft/symbolic links in unix ??
Answer: A symbolic link is a pointer to another file or directory. It can be used just like the original file or directory. A symbolic link appears in a long listing (ls -l) with a reference to the original file/directory. A symbolic link, as opposed to a hard link, is required when linking from one filesystem to another and can be used within a filesystem as well.
To create a symbolic link, the syntax of the command is similar to a copy or move command: existing file first, destination file second. For example, to link the directory /export/space/common/archive to /archive for easy access, use:

ln -s /export/space/common/archive /archive

To link the runtime control script /etc/init.d/httpd to /etc/rc2.d/S77httpd, use:

cd /etc/rc2.d
ln -s ../init.d/httpd S77httpd

Q17. What is a command to kill the last background job??
Answer: kill $!

Q18. Bring a job to foreground by specifying its job number after the percent sign.
Answer: fg %jobnumber
(Note: jobnumber can be obtained by using jobs -l command)

Q19. Explain top command.
Answer: top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes. It can sort the tasks by CPU usage, memory usage and runtime. can be better configured than the standard top from the procps suite. Most features can either be selected by an interactive command or by specifying the feature in the personal or system-wide configuration file. 

Q20. Write a script which give you those records which are having CPU utilization more than 80%.
Answer: df -k awk 'NR==1{print $1"\t"$5"\t"$6}sub("%","",$5){if($5 >= 80) print $1"\t"$5"%\t"$NF}'
Q21. How to pass arguments in an awk script??Answer: Using, cat a.txtawk -v awk_var="$shell_var" '{print $1 "\t" awk_var}'
e.g. #! /usr/bin/sh
var=$*
cat ussd.txt >offpeak_input.txt
cat offpeaknawk -v var1="$var" '{print "\"1-"$1"\"""""\""$2"\"""""1006"""""$3"""0""""-1""""-1""""\""var1" .Account balance is Rs..Bonus minutes.\""}'>>offpeak_input.txt

Daily Purpose sed commands

Replace Text : sed -e "s/Old/New/g"

Remove Text : sed -e "s/Text//g"

Remove Text From Front : sed -e "s/^Text//g"

Remove Text From End : 
sed -e "s/Text\$//g"
sed -e 's/Text$//g'

Insert Text in Front : sed -e "s/^/Text/g"

Append Text to End :
sed -e "s/\$/Text/g"
sed -e 's/$/Text/g'

Truncate :
sed -e "s/Text.*//"
cut -c5-8

Downshift and Upshift :
tr '[A-Z]' '[a-z]'
tr '[a-z]' '[A-Z]'

Change Tabs to Spaces : sed -e "s/<tab>/ /g"

Change Multiple Spaces to a Single Space :
sed -e "s/  */ /g"

Change Whitespaces to a Single Space :
sed -e "s/[<tab><space>][<tab><space>]*/<space>/g"

Delete Leading Whitespaces:
sed -e "s/^[<tab><space>]*//g"

Delete Trailing Whitespaces:
sed -e 's/[<tab><space>]*$//g'

Delete Lines:
sed -e "/Text/d" 
grep -v "Text"

Delete Empty Lines:
sed -e '/^*$/d'
sed -e '/^[<tab><space>]*$/d'

Delete First Line:
sed -e '1d'
sed -e '1,7d'

Delete Last Line : sed -e '$d'

Print First Line :
sed -n '1p'
sed -e '2,$d'
sed -e '1q'

Print Last Line :
sed -n '$p'

Delete Comments :
sed -e '/^#/d'
sed -e 's/#.*//'

Delete Text Between Keywords :
sed -e '/Keyword1/,/Keyword2/d'
sed -e '/^Keyword1$/,/^Keyword2$/d'

Extract Text Between Keywords : 
sed -e '/^Keyword1$/,/^Keyword2$/!d'

Print Odd Lines : cat -n file |  sed -e '/.....[02468]/d' -e 's/^.......//'

Print Even Lines : cat -n file |  sed -e '/.....[13579]/d' -e 's/^.......//'

Read a File Backwards : cat -n file | sort -nr | sed 's/^.......//'

Convert < > into LTGT : cat $* | sed -e 's/</\&lt;/g' -e 's/>/\&gt;/g' 

awk oneliners explained

At below link the famous awk one liners by Eric Pement is explained.
http://www.catonmat.net/blog/awk-one-liners-explained-part-one/

SED One liners explained .....

Below is the link, helpful in understanding most of the sed onliners :

http://www.catonmat.net/blog/sed-one-liners-explained-part-one/

Useful sed one liners

FILE SPACING:
# double space a file
sed G
# double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text.
sed '/^$/d;G'
# triple space a file
sed 'G;G'
# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'
# insert a blank line above every line which matches "regex"
sed '/regex/{x;p;x;}'
# insert a blank line below every line which matches "regex"
sed '/regex/G'
# insert a blank line above and below every line which matches "regex"
sed '/regex/{x;p;x;G;}'
NUMBERING:
# number each line of a file (simple left alignment). Using a tab (see # note on '\t' at end of file) instead of space will preserve margins.
sed = filename sed 'N;s/\n/\t/'
# number each line of a file (number on left, right-aligned)
sed = filename sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
# number each line of file, but only print numbers if line is not blank
sed '/./=' filename sed '/./N; s/\n/ /'
# count lines (emulates "wc -l")
sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # gsed 3.02.80, but top script is easier

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$//" # method 1
sed -n p # method 2
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format Can only be done with UnxUtils sed, version 4.0.7 or higher. Cannot be done with other DOS versions of sed. Use "tr" instead.
sed "s/\r//" infile >outfile # UnxUtils
sed v4.0.7 or higher tr -d \r outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line # aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file
# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file
# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'
# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'
# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space
# center all text in the middle of 79-column width. In method 1, spaces at the beginning of the line are significant, and trailing # spaces are appended at the end of the line. In method 2, spaces at # the beginning of the line are discarded in centering the line, and no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # method 2

# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
sed 's/\(.*\)foo/\1bar/' # replace only the last case
# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'
# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds gsed 's/scarlet\ruby\puce/red/g' # GNU sed only
# reverse order of lines (emulates "tac") bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2

# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'
# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'
# if a line begins with an equal sign, append it to the previous line # and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds
# add commas to numbers with decimal points and minus signs (GNU sed)
gsed ':a;s/\(^\[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g;ta'
# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.) gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds

SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
sed 10q
# print first line of file (emulates "head -1")
sed q
# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'
# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'
# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2
# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2
# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax
# print the line immediately before a regexp, but not the line # containing the regexp
sed -n '/regexp/{g;1!p;};h'
# print the line immediately after a regexp, but not the line # containing the regexp
sed -n '/regexp/{n;p;}'
# print 1 line of context before and after regexp, with line number indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'
# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'
# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\BBB\CCC/!d' # GNU sed only
# print paragraph if it contains AAA (blank lines separate paragraphs) HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\BBB\CCC/b;d' # GNU sed only
# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'
# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax
# print section of file from regular expression to end of file
sed -n '/regexp/,$p'
# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2
# print line number 52 sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files
# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds
# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:
# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'
# delete duplicate, consecutive lines from a file (emulates "uniq"). First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'
# delete duplicate, nonconsecutive lines from a file. Beware not to overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
# delete all lines except duplicate lines (emulates "uniq -d").
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
# delete the first 10 lines of a file
sed '1,10d'
# delete the last line of a file
sed '$d'
# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'
# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2
# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds
# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2
# delete all CONSECUTIVE blank lines from file except the first; also deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF
# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'
# delete all leading blank lines at top of file
sed '/./,$!d'
# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02*
# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:
# remove nroff overstrikes (char, backspace) from man pages. The 'echo' command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed v1.5
# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line
# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line
# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'
# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
# parse out the address proper. Pulls out the e-mail address by itself from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//' # add a leading angle bracket and space to each line (quote a message) sed 's/^/> /'
# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'
# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/zipup.bat dir /b *.txt sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/">>zipup.bat

TYPICAL USE: Sed takes one or more editing commands and applies all ofthem, in sequence, to each line of input. After all the commands havebeen applied to the first input line, that line is output and a secondinput line is taken for processing, and the cycle repeats. Thepreceding examples assume that input comes from the standard inputdevice (i.e, the console, normally this will be piped input). One ormore filenames can be appended to the command line if the input doesnot come from stdin. Output is sent to stdout (the screen). Thus:
cat filename sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk

On line selection or deletion in which you only need to output linesfrom the first part of the file, a "quit" command (q) in the scriptwill drastically reduce processing time for large files. Thus:
sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster

Useful awk oneliners

FILE SPACING:
# double space a file
awk '1;{print ""}' awk 'BEGIN{ORS="\n\n"};1'

# double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text. NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are # often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}' 
# triple space a file
awk '1;{print "\n"}'
NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment). Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*

# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files* 
# number each line of a file (number on left, right-aligned) # Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}' 
# number each line of file, but only print numbers if line is not blank Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};{print}' awk '{print (NF? ++a " :" :"") $0}' 
# count lines (emulates "wc -l")
awk 'END{print NR}'

# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}' # print every line after replacing each field with its absolute value awk '{for (i=1; i<=NF; i++) if ($i < i =" -$i;" i="1;" i =" ($i">
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file

# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file 
# print the largest first field and the line that contains it Intended for finding the longest string in field 1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'

# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '

# print the last field of each line
awk '{ print $NF }'

# print the last field of the last line
awk '{ field = $NF }; END{ print field }'

# print every line with more than 4 fields
awk 'NF > 4'

# print every line where the value of the last field is > 4
awk '$NF > 4'

TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"");print}' # assumes EACH line ends with Ctrl-M

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r");print}

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format # Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile

# Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line # aligns all text flush left
awk '{sub(/^[ \t]+/, ""); print}'

# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "");print}'

# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+[ \t]+$/,"");print}' awk '{$1=$1;print}' # also removes extra space

# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ");print}'

# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*

# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*

# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar");print}' # replaces only 1st instance
gawk '{$0=gensub(/foo/,"bar",4);print}' # replaces only 4th instance
awk '{gsub(/foo/,"bar");print}' # replaces ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")};{print}'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz" awk '!/baz/{gsub(/foo/, "bar")};{print}'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarletrubypuce/, "red"); print}'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{ print $1 "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '! a[$0]++' # most concise script
awk '!($0 in a) {a[$0];print}' # most efficient script
# concatenate every 5 lines of input, using a comma separator between fields
awk 'ORS=%NR%5?",":"\n"' file

SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR <>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print the line immediately before a regex, but not the line # containing the regex
awk '/regex/{print x};{x=$0}' awk '/regex/{print (x=="" ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order)
awk '/AAA/; /BBB/; /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < nr="="8,NR="="12'" nr="="52'" nr="="52">
CREDITS AND THANKS:http://student.northpark.edu/pemente/awk/awk1line.txt

Taking count of all files , where file names were passed as parameters from another file

Taking a count of all files listed in a diirectory
=================================

#! /bin/ksh
cd /home/prds for i in `cat file1.txt`
do
echo "$i" | cat $i | wc -l
done

Removing a particular prefix from all files listed by ls

Removing a particular prefix from all file names listed by ls -lrt e.g. dbo.prds.sql-->prds.sql ......
------------------------------------------------------------------------

#! /bin/ksh
for i in `ls *.sql` 
do 
NEW=`echo $i | sed -e 's/^dbo.//g'` 
mv ${i} ${NEW} 
done

use of sed inside shell script for removing a particular pattern

Use of sed in removing a particular pattern from a file name
=============================================

#! /bin/ksh
M=`echo prds_rpt | sed 's/_rpt//'`
echo "$M"
R="$M"_sync1
echo "$R"

Comparing Partial strings and use of config file inside script

This shell script compares the partial strings and also says how do you use another file as config file inside the script.
========================================================

#! /bin/ksh

. `pwd`/config.txt

export RPT_DB_NAME="$1"
string='_rpt'
echo $RPT_DB_NAME | grep $string >> /dev/null
if [ $? = 0 ]
then
if [[ -e check.txt ]]
then rm check.txt
echo "prds"
echo "prds1"
echo "File Check.txt is removed"
fi
echo "first loop"
echo "first loop"
echo "first loop"
else
echo "No Match Found"
echo "second loop"
echo "second loop"
echo "second loop"
fi
#print prds from the file config.txt
echo "$PRDS"

Invisible Password

#!/bin/ksh

/bin/stty -echo echonl
read up?password:
/bin/stty echo

# you should get user input password in this up varialbe now
# print it out to verify
echo password is "$up"

Comparing two columns

#! /bin/ksh

########################
### Created on : 10-sep-2008 ###
### Created By : prds ###
#########################

############## How to use -- USAGE ###############

usage="$0 File1 File2"

############ Setting up the environment variables #####################

datetime=`date '+%Y%m%d'`
export FILE1="$1"
export FILE2="$2"

########## If either of the mandatory parameters is missing raise error #############

if [[ "$FILE1" = "" ]] || [[ "$FILE2" = "" ]] then
print "Invalid usage, Usage is : "
print "$usage"
exit 1
fi
############################################################

echo "\n"
echo "#### Contents of "$1" which are not present in "$2" ####"
for i in $(cat "$1")
do

cat "$2" | grep "^"$i"$">check.txt

if [ -s check.txt ]
then
cat check.txt>>result.txt
else
echo "Not found --> "$i" : in the file "$2".."
fi

done

echo "\n"
echo "#### Contents of "$2" which are not present in "$1" ####"
for j in $(cat "$2")
do

cat "$1" | grep "^"$j"$">checkf.txt

if [ -s checkf.txt ]
then
cat checkf.txt>>result.txt
else
echo "Not found --> "$j" : in the file "$1".."
fi

done

echo "\n"
echo "Cleaning up temporary files, if there is any...."
rm result.txt
rm check.txt
rm checkf.txt
echo "Temp Section cleaned."

echo "\n"

Replace semicolon with @ in a new line

#!/bin/ksh
for i in `ls *.sql`
do
cat $i | sed '/^$/d' | sed '/;/G' | sed 's/;//g'| sed 's/^$/@/g' >> prds.txt
done