Difference between revisions of "Bash"

Latest revision as of 18:36, 30 April 2010

Here are some of my BASH tricks

Remove non-ascii chars from file

 cat compile.py |tr -cd '\11\12\40-\176' >compile2.py

Find files containing MMU in the name, do a line count, and them add them up. The first part finds files, then redirects the output of the line counts to a temp file

[tom@garak gaisler]$ for i in $(find|grep mmu); do wc $i -l; done>~/leon3_mmu_linecount

The output file:
655 ./leon3/mmutlb.vhd
351 ./leon3/mmuconfig.vhd
258 ./leon3/mmutw.vhd
176 ./leon3/mmulru.vhd 
1559 ./leon3/mmu_dcache.vhd
202 ./leon3/libmmu.vhd
245 ./leon3/mmuiface.vhd
609 ./leon3/mmu.vhd
148 ./leon3/mmu_cache.vhd
189 ./leon3/mmutlbcam.vhd
380 ./leon3/mmu_acache.vhd
111 ./leon3/mmulrue.vhd
662 ./leon3/mmu_icache.vhd

The second part cuts the first field of the file, which is an integer containing the line count, and add all of them up

[tom@garak gaisler]$ PASS=0;for i in $(cat ~/leon3_mmu_linecount|cut -d" " -f1); do PASS=$(($PASS+$i)); done; echo $PASS lines total
5545 lines total

--Done--

Remove all synopsys dirs in path, so we can use a different version (need a bit of work, since it doesn't redirect to stdout for some reason)

echo $PATH|sed s/\:/\\n/g|grep -v synopsys|tr '\n' ':'

Find Hacks

Find using wildcards, and multiple names

find -name "*.v" -o -name "*.h"   
This finds all *.v and *.h files. Note that wildcards are enclosed in quotes; or the shell will interpret it!
The -o option is necessary when specifying multiple instances of an option.

Find files, then grep them.

find "*.h" -exec grep "FE_BITS" -Hn {} \; 
This greps every file, and prints the filename and line number of each instance

Sometimes, you want to do something for all files of a particular class. For example, grep *.cpp and *.h files, and see if they contain a specific string or variable. The previous method is to use the find -exec option. This has inherit weaknesses. Using a pipe is very difficult if not impossible with the exec option ie -exec grep "a"|grep "b" will not work, nor will redirects.

What you really want to do in that case, is similar to a bash for loop.

 $for i in `find ~/Download/ -name "*.pdf"`; do echo pdftk \"${i}\" dump_data output ; done

This for loop will work correctly, but only if the filename DON'T contain SPACES. Here's a quick digression of why. The 'for i in' bash construct parses tokens by both the newline, and the space. So when find returns this file:

 /home/tom/Download/lxf/LinuxFormat 2009.pdf

The i variable will become the value "/home/tom/Download/lxf/LinuxFormat" then in the next iteration, it will progress onto "2009.pdf". Clearly not the behaviour we want.

Bash has the read utility, so that is how we will fix the for loop. First, we need to use a while loop, second, it must use the read utility.

 find ~/Download/ -name "*.pdf"|while read filename; do echo " FILE: $filename"|grep -i linux; done

As you can see in this cumbersome example, we use the while read construct. This will set the filename variable to that of the output of the find utility per line, not per space and per line. Going through all this trouble is quite worth it though, since sometimes you need to pipe and redirect to get what you want. The find -exec option does not allow that.

Another Solution: Create a bash script, and have the find -exec option invoke the script with the filename. Of-course, I am focusing on one-liners here,so my preffered method is the one-liner.

Example: Search for all Makefiles, replace -O2 option with -O0 option

 find -name Makefile|while read aline; do  sed -i $aline -e s/"\-O2"/"\O0"/g; done

For more information on bash scripts with filename containing spaces: http://www.cyberciti.biz/tips/handling-filenames-with-spaces-in-bash.html

AWK Hacks

--Currently, awk cannot handle files in-place. So redirect to a new_file, then mv the new_file to the old_file.

Print out n'th field, where deliminator is 1 or more spaces, replace $1 with the number field you want e.g. $3 is third field.

cat orig_file|awk '{print $1}'> new_file

Print out only unique lines, first instance only. The uniq command does not always work correctly!

cat orig_file|awk '!x[$0]++' > new_file

BASH Inline Tricks

bash is very powerful. Knowing how to use the 'one-liners' will save significant amounts of time.

Inline Loop

This is a simple for loop, parsing in every line of ls and echoing it.

 for i in `ls`; do echo $i; done

for each `define in fe.h, take second field, and make a $display statement with. Notice the semicolons

 for i in `cat fe.h|grep ^'\`define'|awk '{print $2}' `; do echo "\$display (\"" $i = \%d\"\, $i \)\;; done>fe_print.v

for every line that contains "4'", replace the preceding const with const OP4Type.

 for i in `grep "4'" sparcv8.v -n|awk '{print $1}'|sed 's/://g'`; do sed  ${i}s/"const"/"const OP4Type"/g sparcv8.v -i; done
 

The preceding example is special because it goes line for line, it can be tweaked to match first n occurences.It may be complex, but it's more versatile than the find | sed example

Kill all processes containing 'main.rb' The killall command does something similar, with the "-r, --regex - Interpret process name pattern as an extended regular expression." option

 for i in `ps -aux|grep main.rb|awk '{print $2}' `; do kill -9 $i; done

The second field is the pid (process id)

SED Tricks

Sed is an extremely powerfull Stream EDitor. Its popular use is search replace, and infile editing. You can replace all instances of DEBUG to DEBUG_PRINT for *.h *.cpp files in a tree.

Replace newline with a
and a newline (great for formatting text for wiki)

  cat test|tr  '\n' "BR\n"|sed s/"B"/"\<BR\>\\n"/g

For more SED tricks, visit http://sed.sourceforge.net/sed1line.txt