Use of --parents flag in mkdir and cp

Occasionally, you want to create a directory structure several levels deep.
For example, /home/peter/status/2007/november.

Your first attempt may be something like this:

$ mkdir /home/peter/status/2007/november
mkdir: cannot create directory `/home/peter/status/2007/november': 
No such file or directory

The problem is that the intermediate directories (status and 2007) do not exist.

The following will work, but it is quite clumsy.

$ cd /home/peter; mkdir status
$ cd status; mkdir 2007
$ cd 2007; mkdir november

A much shorter way is simply:

$ mkdir --parents /home/peter/status/2007/november

With the --parents option, mkdir will actually create the intermediate parent directories if needed.

-p is the short equivalent of --parent for the mkdir command.

When the time comes to create the december directory, you can issue this:

$ mkdir -p /home/peter/status/2007/december

Although you have specified the -p (--parents) option, and the parent
directory structure already exists, it will quietly create the december
directory below it. This is just what we expect.


To check your results, do a ls -R (recursively list the directory)

$ cd /home/peter; ls -R status
status:
2007

status/2007:
december  november

status/2007/december:

status/2007/november:

Let's create a file in /home/peter/status/2007/november.

 $ touch /home/peter/status/2007/november/nov12.txt

The above will create an empty file named nov12.txt.

Next, we will copy the nov12.txt file like this:

$ cd /home/peter
$ cp --parents status/2007/november/nov12.txt /home/peter/tmp

The --parents flag will cause the full path to be copied to tmp ("status/2007/november/nov12.txt")

$ cd /home/peter/tmp/
$ ls -R status
status:
2007

status/2007:
november

status/2007/november:
nov12.txt

How is cp --parents different from cp -r (recursive copy)?

$ cd /home/peter
$ cp -r status /home/peter/tmp

The recursive copy will have copied everything under status (including the contents of the december directory).

ls -R /home/peter/tmp/status
/home/peter/tmp/status:
2007

/home/peter/tmp/status/2007:
december  november

/home/peter/tmp/status/2007/december:

/home/peter/tmp/status/2007/november:
nov12.txt

sudo hacks: making cd and redirection work

Suppose you want to run a command that requires root privileges. Conventional wisdom says don't login as root. Instead, login as a non-root user, and then sudo.

$ sudo 'cd /root/restricted'
Password:
sudo: cd /root/restricted: command not found
$

cd Not Found ?? That is RIGHT !

cd is a shell built-in command. It cannot be run in a child process. The child process simply cannot change the working directory of its parent shell process.

Redirection also does not work with sudo for the same reason (redirection being a shell "thing")

$ sudo 'ls /root/restricted >/root/out.txt'
sudo: ls /root/restricted >/root/out.txt: command not found
$

The workaround in both cases is to execute the command in a subshell.

$ sudo sh -c 'cd /root/restricted'

$ sudo sh -c 'ls /root/restricted >/root/out.txt'

How to Look Up a Process

You can look up a process in many ways. Often, you have the name of a daemon, command, or a program, and you want to know its process ID. For example, you may want to find the process ID of firefox.

• pgrep
  

  
  $ pgrep firefox
  4559

  
  
  You do not need to specify the complete name.
  

  
  $ pgrep fox
  4559

  
  
  Because of the partial matching, it is prudent to have pgrep report both the process ID and the matching name:
  
  

  $ pgrep -l fox
  4559 firefox-bin

  
  
  
• ps
  

  $ ps aux |grep fox 
  peter     4559  4.0  8.7 211680 84668 ?        Sl   08:58   6:20 /usr/lib/iceweasel/firefox-bin -a firefox
  root     10611  0.0  0.0   2848   696 pts/0    R+   11:36   0:00 grep fox

  
  
  Alternatively,
  

  $ ps -ef |grep fox
  peter     4559     1  4 08:58 ?        00:06:26 /usr/lib/iceweasel/firefox-bin -a firefox
  root     10645  8518  0 11:38 pts/0    00:00:00 grep fox

  
  
  The second output line for each of the 2 commands refers to the grep itself, and should be ignored.
  

The pgrep and the ps approaches can be adjusted to look up a process using other criteria besides the name.

For example, to look up all processes being executed by an user ID (0 = root):

• 
  

  $ pgrep -lu 0
  pgrep -lu 0
  1 init
  2 migration/0
  3 ksoftirqd/0
  4 events/0
  5 khelper
  6 kthread
  9 kblockd/0
  ....

  
  
• 
  

  $ ps -ef |grep root 
  root         1     0  0 08:56 ?        00:00:01 init [2]
  root         2     1  0 08:56 ?        00:00:00 [migration/0]
  root         3     1  0 08:56 ?        00:00:00 [ksoftirqd/0]
  root         4     1  0 08:56 ?        00:00:00 [events/0]
  root         5     1  0 08:56 ?        00:00:00 [khelper]
  root         6     1  0 08:56 ?        00:00:00 [kthread]
  root         9     6  0 08:56 ?        00:00:00 [kblockd/0]
  ...

  
  The grep command will match root along anywhere on the output line. This is sufficient for most casual command line lookup of processes. Of course, you can write to your heart's content a pattern-matching awk/perl script to only output lines that have root in the user name field.
  

For details, please read the pgrep and ps man pages.

How to Scroll the Command Window

How many times have you been annoyed by the command output spanning over the length of the window? If the window is in the X-Window environment, then you could use the scroll bars to scroll back. What if this is your console window, and it does not have scroll bars?

Sure, if only you had remembered, you could have piped the command to the less command. For example,

ps -ef |less

It turns out that you can scroll the bash command window.

To scroll up, hit Shift-Up.

To scroll down, hit Shift-Down.

Simple? Yes. And Effective !!

Get into the right groups

In Linux, belonging to the right group gives you permission to use restricted-access resources.

When you create a new user, by default,

$ useradd -m george
$ passwd george
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

Of course, you could have done the above using the wizard-styled adduser command.

Now, see what default groups george is in:

$ groups george
george : george

Poor george: he is a loner: being a group of oneself.

george may one day try to use some peripheral devices like the CDROM drive, USB drive, or floppy drive, and discovers that he simply cannot.

What do you do?

First, make sure that for pre-configured drives like the CDROM and floppy drives, the user option is specified. The default option is nouser which restricts access to root only.

cat /etc/fstab
...
/dev/hdb        /media/cdrom0   udf,iso9660 user,noauto     0       0
/dev/fd0        /media/floppy0  auto        rw,user,noauto  0       0
...

If the drive is configured correctly, then determine if george is in the right groups. The easiest way is to reference some user whom you know can access the target devices.

$ groups peter
peter : peter dialout cdrom floppy audio video plugdev netdev powerdev

Most group names are self-explanatory. For USB thumb drives, make sure that george is in plugdev.

To add george to the right groups:

usermod -aG  cdroom,floppy,plugdev  george

usermod -aG appends the new groups to george's existing groups.

$ groups george
george : george cdrom floppy plugdev

Remember this: if you are not in the right group, you don't get invited to the right parties ....

Click here for an updated post about group membership.

Tail multiple files

In an earlier post, I discussed 2 commands for continuously displaying the contents of a file: tail -f and less. I mentioned that with less, you can scroll back, perform searches, and, in general, use all the interactive commands that come with less.

There is (at least) one scenario where tail -f is more appropriate than less:
when you need to tail more than 1 file.

$ tail -f /var/log/messages    /var/log/mail.log
==> /var/log/messages <==
Nov  6 20:17:54 tiger -- MARK --
Nov  6 20:38:04 tiger -- MARK --

==> /var/log/mail.log <==
Nov  5 18:04:51 tiger sendmail[3090]: gethostbyaddr(192.168.0.102) failed: 3
Nov  5 18:04:52 tiger sendmail[3192]: gethostbyaddr(192.168.0.102) failed: 3
Nov  5 18:05:07 tiger sm-mta[3733]: starting daemon (8.13.8): SMTP+queueing@00:1 0:00
Nov  5 18:05:07 tiger sm-mta[3733]: restarting /usr/sbin/sendmail-mta due to sig nal

==> /var/log/messages <==
Nov  6 20:58:05 tiger -- MARK --

The contents of each file are interspersed throughout the output, making it somewhat hard to read.

Alternatively, you can use another tool named multitail to tail multiple files. Multitail divides the console into multiple windows: one for each file.

multitail  /var/log/messages  /var/log/mail.log

To scroll back, type b. Then, select the file to scroll. The selected file will be displayed in a new window. Now, you can use the arrow keys as well as the PageUp/PageDown keys to scroll. To exit the scroll window, type x.

I find multitail a bit clumsy to use. For simple log file viewing I tend to stick with tail -f.

Note that multitail may not be installed by default in your Linux distribution.

Keeping Command History across Multiple Sessions

The bash shell maintains a history of the commands you entered. You can re-execute a command by recalling it from the history, without having to re-type it.

The command history is stored in a file, specified by an environment variable.
$ echo $HISTFILE
/home/peter/.bash_history

The maximum number of commands that it will save depends on the value of this environment variable:
$ echo $HISTSIZE
500

Life is simple if we operate on a single shell session at any given time. If you have 2 simultaneous sessions, you may be surprised that the history does not have the commands you expect.

The default behavior is that the command history is saved ONLY on session exit. Moreover, the existing contents in the history are overwritten by the new contents.

Assuming you already have a session running, opening a second session will not have the latest commands from the first session. This is because the first shell hasn’t exited yet. Ultimately, the contents of the history file depends on which session exits last, hence overwriting the commands from the session that finishes earlier.

This can become quite confusing.

The remedy is simple. Change the history behavior as follows:

1. Append commands to the history file, rather than overwrite it.
   

   $ shopt -s histappend

   
   
2. Save each command right after it has been executed, not at the end of the session.
   

   $ PROMPT_COMMAND='history -a'

   
   

Insert the 2 statements into ~/.bashrc
shopt -s histappend
PROMPT_COMMAND='history -a'

Note: If the PROMPT_COMMAND is already initialized, then you probably want to concatenate the history -a part to what is already there.

To determine if PROMPT_COMMAND has a value:

echo $PROMPT_COMMAND

To concatenate:

PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

Log watching using tail or less

Log files typically grow in size, with the latest contents appended to the end of the log. I often need to watch a log file in live action, for error detection.

The command tail -f will display the last 10 lines of a file, and then continuously wait for new lines, and display them as they appear.

$ tail -f /var/log/messages

If you want to see more than ten lines at the outset, specify the new number (say 50 lines) like this:

$ tail -50   -f /var/log/messages

The tail command is fast and simple. But if you want more than just following a file (e.g., scrolling and searching), then less may be the command for you.

$ less /var/log/messages

Press Shift-F. This will take you to the end of the file, and continuously display new contents. In other words, it behaves just like tail -f.

To start less in the tail mode (thanks to Seth Milliken for this tip), execute:

$ less +F /var/log/messages

To scroll backwards, you must first exit the follow mode by pressing Control-c. Then, you can scroll back by pressing b. In fact, all the less commands are available to you once you are in the regular less mode. You can start a search by typing / followed by the string you want to search for.

Happy Log Watching !!!

P.S.
Related articles on tailing files:
Tail multiple files
Two additional ways to tail a log file