grep with color output

grep is capable of color-highlighting the matched string in its output. But, by default, that option is turned off.

$ grep abc a_file.txt
abcdef

There are 3 color options available to you:

• --color=auto
  
• --color=always
  
• --color=never
  

With color=always, it colors the matched string.

$ grep --color=always  abc  a_file.txt
abcdef

Quite often, you want to page through the output:

$ grep --color=always  abc  a_file.txt |less 
ESC[01;31mabcESC[00mdef
(END)

The problem is that less does not understand those control characters, by default. You need to use the -R parameter.

$ grep --color=always  abc  a_file.txt |less -R
abcdef

Alternatively, use more.

$ grep --color=always  abc  a_file.txt | more 
abcdef

Another problematic scenario is when you want to save the grep output to a file. The output file will contain those control characters.

$ grep --color=always  abc  a_file.txt > myoutput.txt
$ less myoutput.txt
ESC[01;31mabcESC[00mdef
myoutput.txt (END)

With color=auto, it displays color in the output unless the output is piped to a command, or redirected to a file.

Lastly, you can specify the color parameter in a grep-specific environment variable. Then, you don't have to enter it in the command line.

$ export GREP_OPTIONS='--color=always' 

So, go ahead, and add color to your Linux world.

P.S. Additional grep articles from this blog:

• Use the OR operator in grep to search for words and phrases.
  
• Trick grep not to report itself in a search.
  

The find / xargs Linux command-pipe

My task at hand is to delete all backup files ("*~") located anywhere in my home directory hierarchy.

$ find /home/peter -name *~ |xargs rm

This works reasonably well, but some targeted backup files are actually not deleted, namely:

1. backup files in a sub-directory that is symbolically linked.
   
2. backup files that have spaces in their file name or path name.
   

In addition, the xargs command handles zero argument poorly. If no match is found by the find command, xargs is not smart enough to terminate right away, but will still try to execute the target command (rm) in some way.

$ find /home/peter -name no-such-thing* |xargs rm
rm: missing operand


Symbolic Links

find does not follow symbolic links, by default. To make it follow symbolic links, add the -L parameter.

$ find -L /home/peter -name *~ |xargs rm

Names with spaces

xargs splits up its input arguments at spaces (and newlines). If a file name (or path name) has spaces in it, e.g., "can not do this.pdf", xargs will misinterpret it and thinks there are multiple files.

The solution is to invoke xargs with the -0 (zero) parameter. xargs will separate filenames by NUL instead of whitespace. Also, the find command needs the -print0 parameter: this puts a NUL between filenames instead of a newline.

Note that the -print0 parameter must be put last in find.
$ find /home/peter -name *~ -print0 |xargs -0 rm


Zero argument
Use the -r flag with xargs. If stdin is empty, xargs will not run the command, and exit.

Putting it altogether, the command I will use for my task is:

$ find -L /home/peter -name *~ -print0 |xargs -0 -r rm

Crunch numbers

You can crunch numbers on the command line using the command "bc". Instead of firing up a GUI-based calculator application, you can do a quick compute right on the command line.

Running bc will put you in interactive mode by default. You type in statements, and it returns the answer.
$ bc
5 * 3
15
$

The same calculation can be done non-interactively.
$ echo '5 * 3' | bc
15


By default, division returns truncated whole numbers.
$ bc
10 / 3
3

You can define the number of decimal places you want in the answer. This is the "scale".
$ bc
scale=2
10 / 3
3.33

Alternatively, you can invoke the bc command with -l. This preloads the math library and the default scale is set to 20.
$ bc -l
10 / 3
3.33333333333333333333

You can convert a number from one base (ibase) to another base (obase). The default ibase and obase are base 10.

To convert the number 255 in base 10 to base 2,
$echo 'obase=2; 255' | bc
11111111

In reverse,
$ echo 'ibase=2; 11111111' | bc
255

Shred it!

Have a top-secret file that you want to delete? A file that you don't want to recover by you or anyone else?

Use the shred command:
#shred

By default, the shred overwrites the target file 25 times. But, it does not delete the file per se.

I usually want to overwrite AND delete it.

#shred -uvz -n 50 topsecret.txt

The -u flag will actually remove the file after overwriting it.
The -n specifies the number of overwriting passes: 50 in this example.
The -z flag will add a pass at the end to zero out the data.

Add in a -v (verbose) flag to see the shredding in action.

Disclaimer: Please read the man page of the shred command. Shred is not effective against certain types of file systems, journaling file systems being the notable example. Shred is NOT effective for ext3, (the default file system for most modern Linux distributions), if your ext3 partition is mounted in data=journal mode. In the journal mode, file data in addition to just metadata are stored in the journal. According to the man page, shred works as usual in both the data=ordered (default) and data=writeback modes.

If you are not sure about the data mode for your ext3 partitions, display /etc/fstab and look for data= for your ext3 partitions.
$more /etc/fstab

Dictionary Lookup via Command Line

#curl dict://dict.org/d:YourWord
To help you remember, d: stands for DEFINE.

For example, to look up the word "bash", (all output in this article are greatly abbreviated for clarity)
# curl dict://dict.org/d:bash
151 "Bash" gcide "The Collaborative International Dictionary of English v.0.48"
Bash \Bash\, n.
1. a forceful blow, especially one that does damage to its
target.
[PJC]
2. a elaborate or lively social gathering or party.
[PJC]


The dict.org web site supports 77 dictionaries. To list them:
#curl dict://dict.org/show:db

So, if you want to know the computer meaning of "bash", specify foldoc "The Free On-line Dictionary of Computing" like this:
#curl dict://dict.org/d:bash:foldoc
151 "bash" foldoc "The Free On-line Dictionary of Computing (27 SEP 03)"
bash
Bourne Again SHell. {GNU}'s {command interpreter} for {Unix}.
Bash is a {Posix}-compatible {shell} with full {Bourne shell}
syntax, and some {C shell} commands built in. The Bourne
Again Shell supports {Emacs}-style command-line editing, job
control, functions, and on-line help. Written by Brian Fox of
{UCSB}.

To display results from all libraries, do this:
#curl dict://dict.org/d:bash:*


Fuzzy matching is also possible. Replace the "d" or Define command with "m" which stands for MATCH.
#curl dict://dict.org/m:bash

With the match command, you can also specify the strategy such as exact, prefix, suffix, or even soundex.

#curl dict://dict.org/m:bash::prefix
gcide "Bash"
gcide "Bashed"
gcide "Bashful"
gcide "bashful Billy"


To list all supported strategies:
#curl dict://dict.org/show:strat

Recalling command history

The bash shell has a command history. You can repeat a command which you executed before by specifying the command number, e.g., !398 to run the command numbered 398 in the history.

Alternatively, you can specify a partial search string like this: !rm

Either way, the corresponding command is executed right away. Kind of dangerous, you say??

To be on the safe way, you want to see the exact command before the command is executed.

Just add :p to the end, like
!398:p
!rm:p

It will just print out the command that matched the command number (or the search string), but not execute it.

If it is really the command you want, just do !! to run the last command.

Better safe than sorry.

You can always have the Last Word In ..

!$ is a handy shorthand for the last word in the previous command.

At first, it may not seem that useful.

But wait, ...

cd somedir
cp some_file_a /some/long/dir/path/some_file_b

Now you want to edit some_file_b

Just type
vi !$

That translates to vi /some/long/dir/path/some_file_b

It saves quite a bit of typing. Not bad.

More context with less

less is a text file viewer. By default, it will print out one screenful of file contents at a time, and wait for you to tell it what to do next. Scroll forward? Backward? Search for a string?

Sometimes, when I scroll through an unfamiliar file, I don't know where I am in the file. Am I near the beginning, or the end? Am I in the middle?

To get some context, you can specify some run-time options for the less command.

#less -mN somefile.txt

-m will display the percentage complete at the bottom of the screen.
-N will display line numbers.

Instead of typing out the command-line options every time, you can specify the options in an environment variable.

#export LESS="-mN"

Finally, put the above command in your .bashrc file.

Command Line editing: emacs or vi shortcuts

You can edit your command line using emacs style shortcuts. For example, to go to the beginning of the command line, enter Control-a via keyboard input; to go to the end of line, Control-e. If you are a emacs user like me, you feel right at home.


One time, I logged in to someone else's Linux machine, and arrived at the bash shell. To my surprise, a Control-a did not take me to the start of line. Instead, I saw:

$ ./somecommand ^A^A


It did not seem to understand my emacs-styled keyboard shortcuts.

It turns out that for the bash shell, you can use other styles of command line edits besides the emacs style. Some users prefer the vi/vim style of command line edits.

To switch, set the relevant shell option. In the above example, to switch to the emacs style, type:

$ set -o emacs

If you want to switch from the default emacs to the vi style, type:


$ set -o vi


For the vi style of command line edits, you must first press the Esc key to enter the command mode. Then, you can enter the command, e.g., the ^ key to go to the beginning of the line, or the $ key to the end of the line. To return to regular typing mode, type the letter i for insert, or a for append.

My #1 beloved tool: emacs

It is fitting that my first post is about the tool I love the most: GNU emacs. It is the text editor, plus a lot more.

Normally, when you just run emacs, it will load your initialization file (.emacs). But, what if you just want to do something quick, and don't want to wait the extra couple of seconds for emacs to load all these other stuff in your init file?

Just run emacs with the -q option. You will have a bare-bone emacs.