Two additional ways to tail a log file

I want to revisit a topic: how to tail a log file. My earlier posts discussed the use of tail and less commands to tail a log file, and multitail if you need to tail multiple files at once.

This followup article discussed two other ways to tail a log file: the use of the most command, and to tail within emacs.

The most command bills itself as a replacement for less. Like its predecessors less and more, most is a file pager program.

To install most on my Debian Etch system:

$ apt-get update && apt-get install most

To page the log file say /var/log/messages:

$ most /var/log/messages

This opens the log file, and displays the first page.

To tail the file, put most into Tail Mode by pressing the F key (capital F).

You should see the following line in the status area at the bottom of the tail window.

Most Tail Mode--  MOST keys are still active.

If a new line is appended to the log file, most will automatically reposition the file to the end and display the new line.

Note that you can scroll the file or do a search when you are in Tail Mode. For example, you can press the PageUp/PageDown, UpArrow/DownArrow keys to scroll the file. Also, you can press / to search forward or ? to search backward. However, when you press a key in Tail Mode, you break out of Tail mode in the sense that newly appended lines are not automatically displayed in the window. Yet, you can hit F again to re-enter Tail Mode, or simply use the DownArrow scroll key to scroll past the end of file to fetch the new lines.

To quit most, press the q key.

For my fellow emacs users out there, another way to tail a log file is to do it right within emacs.

You need the tail-file elisp function which is in the emacs-goodies-el package. Debian users can install this package like this:

$ apt-get update && apt-get install emacs-goodies.el

To tail a file in emacs: start emacs, hit M-x (Alt and x keys together), and type tail-file. Then, enter the filename to tail. The net result is that this will spawn an external tail -f process.

Note that you can't read the entire file under emacs using tail-file, only the tail initially and the newly appended lines portion afterwards.

How to disable SSH host key checking

Remote login using the SSH protocol is a frequent activity in today's internet world. With the SSH protocol, the onus is on the SSH client to verify the identity of the host to which it is connecting. The host identify is established by its SSH host key. Typically, the host key is auto-created during initial SSH installation setup.

By default, the SSH client verifies the host key against a local file containing known, trustworthy machines. This provides protection against possible Man-In-The-Middle attacks. However, there are situations in which you want to bypass this verification step. This article explains how to disable host key checking using OpenSSH, a popular Free and Open-Source implementation of SSH.

When you login to a remote host for the first time, the remote host's host key is most likely unknown to the SSH client. The default behavior is to ask the user to confirm the fingerprint of the host key.

$ ssh peter@192.168.0.100
The authenticity of host '192.168.0.100 (192.168.0.100)' can't be established.
RSA key fingerprint is 3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Are you sure you want to continue connecting (yes/no)? 

If your answer is yes, the SSH client continues login, and stores the host key locally in the file ~/.ssh/known_hosts. You only need to validate the host key the first time around: in subsequent logins, you will not be prompted to confirm it again.

Yet, from time to time, when you try to remote login to the same host from the same origin, you may be refused with the following warning message:

$ ssh peter@192.168.0.100
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Please contact your system administrator.
Add correct host key in /home/peter/.ssh/known_hosts to get rid of this message.
Offending key in /home/peter/.ssh/known_hosts:3
RSA host key for 192.168.0.100 has changed and you have requested strict checking.
Host key verification failed.$

There are multiple possible reasons why the remote host key changed. A Man-in-the-Middle attack is only one possible reason. Other possible reasons include:

• OpenSSH was re-installed on the remote host but, for whatever reason, the original host key was not restored.
  
• The remote host was replaced legitimately by another machine.

If you are sure that this is harmless, you can use either 1 of 2 methods below to trick openSSH to let you login. But be warned that you have become vulnerable to man-in-the-middle attacks.

The first method is to remove the remote host from the ~/.ssh/known_hosts file. Note that the warning message already tells you the line number in the known_hosts file that corresponds to the target remote host. The offending line in the above example is line 3("Offending key in /home/peter/.ssh/known_hosts:3")

You can use the following one liner to remove that one line (line 3) from the file.

$ sed -i 3d ~/.ssh/known_hosts

Note that with the above method, you will be prompted to confirm the host key fingerprint when you run ssh to login.

The second method uses two openSSH parameters:

• StrictHostKeyCheckin, and


• UserKnownHostsFile.

This method tricks SSH by configuring it to use an empty known_hosts file, and NOT to ask you to confirm the remote host identity key.

$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no peter@192.168.0.100
Warning: Permanently added '192.168.0.100' (RSA) to the list of known hosts.
peter@192.168.0.100's password:

The UserKnownHostsFile parameter specifies the database file to use for storing the user host keys (default is ~/.ssh/known_hosts).

The /dev/null file is a special system device file that discards anything and everything written to it, and when used as the input file, returns End Of File immediately.

By configuring the null device file as the host key database, SSH is fooled into thinking that the SSH client has never connected to any SSH server before, and so will never run into a mismatched host key.

The parameter StrictHostKeyChecking specifies if SSH will automatically add new host keys to the host key database file. By setting it to no, the host key is automatically added, without user confirmation, for all first-time connection. Because of the null key database file, all connection is viewed as the first-time for any SSH server host. Therefore, the host key is automatically added to the host key database with no user confirmation. Writing the key to the /dev/null file discards the key and reports success.

Please refer to this excellent article about host keys and key checking.

By specifying the above 2 SSH options on the command line, you can bypass host key checking for that particular SSH login. If you want to bypass host key checking on a permanent basis, you need to specify those same options in the SSH configuration file.

You can edit the global SSH configuration file (/etc/ssh/ssh_config) if you want to make the changes permanent for all users.

If you want to target a particular user, modify the user-specific SSH configuration file (~/.ssh/config). The instructions below apply to both files.

Suppose you want to bypass key checking for a particular subnet (192.168.0.0/24).

Add the following lines to the beginning of the SSH configuration file.

Host 192.168.0.*
   StrictHostKeyChecking no
   UserKnownHostsFile=/dev/null

Note that the configuration file should have a line like Host * followed by one or more parameter-value pairs. Host *means that it will match any host. Essentially, the parameters following Host * are the general defaults. Because the first matched value for each SSH parameter is used, you want to add the host-specific or subnet-specific parameters to the beginning of the file.

As a final word of caution, unless you know what you are doing, it is probably best to bypass key checking on a case by case basis, rather than making blanket permanent changes to the SSH configuration files.

If you make it this far in the article, you may find the following more recent ssh articles interesting:

Allow root ssh login with public key authentication
How to auto fill in ssh client parameters
One-liner to shutdown remote host
X11 Forwarding over SSH

Upgrade individual packages for Debian-based systems

If you are using Debian-based distributions (Debian, Ubuntu, etc), you are probably familiar with the apt-get update followed by the apt-get upgrade routine. That is what I regularly use to upgrade ALL packages that have an update available.

But what if you only want to upgrade certain individual packages?

apt-get upgrade will upgrade ALL or nothing. So, that is out of the question.

What you need is apt-get install. apt-get install serves dual purposes: install and upgrade. It installs a package if it is not already installed. If it is already installed, apt-get install will upgrade the package to the latest version.

You should still run apt-get update before apt-get install. You can specify one or more package names as arguments to the apt-get install command.

For example, to upgrade these 2 packages: libfreetype6, libtiff4:

$ sudo apt-get update
$ sudo apt-get install libfreetype6 libtiff4

How to get the process start date and time

How can we determine when a running process was started?

The venerable ps command deserves first consideration.

Most Linux command-line users are familiar with either the standard UNIX notation or the BSD notation when it comes to specifying ps options.

If ps -ef is what you use, that is the UNIX notation.

$ ps -ef
UID     PID  PPID C STIME TTY      TIME CMD
root      1    0  0 Sep20 ?    00:00:03 init [3]
  
peter  1218    1  0 Sep20 ?    00:21:35 /usr/lib/iceweasel/firefox-bin -a firefox
  
peter  4901    1  1 16:34 ?    00:01:12 /usr/bin/emacs-snapshot-gtk

The STIME column displays the start time or date. From the above, we can tell that process 4901 (emacs) began execution at 16:34 (4:34 pm). But on what day, today?

From the ps man page: 'Only the year will be displayed if the process was not started the same year ps was invoked, or "mmmdd" if it was not started the same day, or "HH:MM" otherwise.'

So, emacs was started at 16:34 TODAY.

What is the start time for the other process, the firefox process with pid 1218?

The STIME for process 1218 reads Sep20 (which was yesterday). But what time yesterday?

The default ps -ef only tells you the start date but NOT the time if a process was NOT started on the same day.

If the BSD notation is more familiar to you, you will find that ps aux yields similar results.

$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY STAT START   TIME COMMAND
root         1  0.0  0.1   1576   540 ?   Ss   Sep20   0:03 init[3]
  
peter     1218  0.5  8.9 201252 45456 ?   Sl   Sep20  21:35 /usr/lib/iceweasel/firefox-bin -a firefox

The Start column in the above output only reveals the start date if the process is older than the current day.

There are (at least) 2 ways to determine the exact start time if the process was started before the current day.

Solution 1
Specify elapsed time in the ps output format.

$ ps -eo pid,cmd,etime
  PID CMD                             ELAPSED
 1218 /usr/lib/iceweasel/firefox-bin -  2-16:04:45

The above ps command specifies 3 fields to be included in the output: the process pid, the command, and the elapsed time, respectively.

etime is the elapsed time since the process was started, in the form dd-hh:mm:ss. dd is the number of days; hh, the number of hours; mm, the number of minutes; ss, the number of seconds.

The firefox command started execution 2 days, 16 hours, 4 minutes and 45 seconds ago. To find the exact time, you need to do some simple math.

If you prefer the BSD notation, issue this command:

$ ps axo pid,cmd,etime
  PID CMD                             ELAPSED
  
 1218 /usr/lib/iceweasel/firefox-bin -  2-16:04:57 

Solution 2
Get the process pid and read off the timestamp in the corresponding subdirectory in /proc.

First, get the process pid using the ps command (ps -ef or ps aux)

Then, use the ls command to display the creation timestamp of the directory.

$ ls -ld /proc/1218
dr-xr-xr-x 5 peter peter 0 Sep 20 16:14 /proc/1218

You can tell from the timestamp that the process 1218 began executing on Sept 20, 16:14.

If you can think of another clever way to get the start time and date, please let us know.

How to find and delete all hard links to a file

Deleting a file is deceptively simple. You can simply use the rm command like this.

$ rm file1

However, if the file has one or more hard links to it, life gets more interesting. You need to seek and destroy all hard links to the file.

A hard link is essentially another name for a file. Hard links to file1 can be created as follows:

$ ln file1 file2
$ ln file1 tmp/file3

file2 and file3 become another name of file1.

How do I know if some file has a hard link to it?

Do a long listing of the file like this.

$ ls -l file1
-rw-r--r-- 3 peter peter 0 2008-09-01 16:15 file1

Note that the hard link count has the value 3 (this is the number to the left of the file owner). It means that the physical file has 3 names: file1 and two others.

If you only rm file1, the other 2 names still exist, and users can still access the physical file using these 2 names.

To find out all hard links to file1, use this command.

$ find /home -xdev -samefile file1
/home/peter/file1
/home/peter/tmp/file3
/home/peter/file2

Note that we specified /home as the search starting point. You should replace /home with the mount point for the filesystem in which your file1 was created. (This is likely to be either /home or / for files in your home directory.)

The reason for using the filesystem's mount point is that hard links are restricted to reside in the same filesystem as the physical file. This means that file2 and file3 must be in the same filesystem as file1. Therefore, to find ALL hard links to a file, you must start search at the mount point of the filesystem.

The -xdev option instructs find NOT to traverse directories in other filesystems.

To find and delete all hard links to a file:

$ find /home -xdev -samefile file1 | xargs rm

An alternative (and slightly more complicated) method is by using inodes. An inode is a data structure assigned to a physical file that contains all its meta-information except the file's name(s).

All hard links to a physical file share the same inode, and have the same inode id number. Therefore, if you know the inode number of a file, you can find all hard links to the file by searching for files with the same inode number.

To display a file's inode number, use ls -i:

$ ls -li file1
2655341 -rw-r--r-- 3 peter peter 0 2008-09-02 19:09 file1

The first column in the above listing shows the file's inode number.

Now that you know the inode number of file1, you can use the find command to search for all files that have the same inode number:

$ find /home -xdev -inum 2655341
/home/peter/file1
/home/peter/tmp/file3
/home/peter/file2

To delete all hard links to the physical file:

$ find /home -xdev -inum 2655341 | xargs rm