Saturday, October 18, 2008

How to disable SSH host key checking

Remote login using the SSH protocol is a frequent activity in today's internet world. With the SSH protocol, the onus is on the SSH client to verify the identity of the host to which it is connecting. The host identify is established by its SSH host key. Typically, the host key is auto-created during initial SSH installation setup.

By default, the SSH client verifies the host key against a local file containing known, trustworthy machines. This provides protection against possible Man-In-The-Middle attacks. However, there are situations in which you want to bypass this verification step. This article explains how to disable host key checking using OpenSSH, a popular Free and Open-Source implementation of SSH.

When you login to a remote host for the first time, the remote host's host key is most likely unknown to the SSH client. The default behavior is to ask the user to confirm the fingerprint of the host key.
$ ssh peter@192.168.0.100
The authenticity of host '192.168.0.100 (192.168.0.100)' can't be established.
RSA key fingerprint is 3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Are you sure you want to continue connecting (yes/no)?

If your answer is yes, the SSH client continues login, and stores the host key locally in the file ~/.ssh/known_hosts. You only need to validate the host key the first time around: in subsequent logins, you will not be prompted to confirm it again.

Yet, from time to time, when you try to remote login to the same host from the same origin, you may be refused with the following warning message:
$ ssh peter@192.168.0.100
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Please contact your system administrator.
Add correct host key in /home/peter/.ssh/known_hosts to get rid of this message.
Offending key in /home/peter/.ssh/known_hosts:3
RSA host key for 192.168.0.100 has changed and you have requested strict checking.
Host key verification failed.$

There are multiple possible reasons why the remote host key changed. A Man-in-the-Middle attack is only one possible reason. Other possible reasons include:
  • OpenSSH was re-installed on the remote host but, for whatever reason, the original host key was not restored.
  • The remote host was replaced legitimately by another machine.

If you are sure that this is harmless, you can use either 1 of 2 methods below to trick openSSH to let you login. But be warned that you have become vulnerable to man-in-the-middle attacks.

The first method is to remove the remote host from the ~/.ssh/known_hosts file. Note that the warning message already tells you the line number in the known_hosts file that corresponds to the target remote host. The offending line in the above example is line 3("Offending key in /home/peter/.ssh/known_hosts:3")

You can use the following one liner to remove that one line (line 3) from the file.
$ sed -i 3d ~/.ssh/known_hosts

Note that with the above method, you will be prompted to confirm the host key fingerprint when you run ssh to login.

The second method uses two openSSH parameters:
  • StrictHostKeyCheckin, and

  • UserKnownHostsFile.

This method tricks SSH by configuring it to use an empty known_hosts file, and NOT to ask you to confirm the remote host identity key.
$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no peter@192.168.0.100
Warning: Permanently added '192.168.0.100' (RSA) to the list of known hosts.
peter@192.168.0.100's password:

The UserKnownHostsFile parameter specifies the database file to use for storing the user host keys (default is ~/.ssh/known_hosts).

The /dev/null file is a special system device file that discards anything and everything written to it, and when used as the input file, returns End Of File immediately.

By configuring the null device file as the host key database, SSH is fooled into thinking that the SSH client has never connected to any SSH server before, and so will never run into a mismatched host key.

The parameter StrictHostKeyChecking specifies if SSH will automatically add new host keys to the host key database file. By setting it to no, the host key is automatically added, without user confirmation, for all first-time connection. Because of the null key database file, all connection is viewed as the first-time for any SSH server host. Therefore, the host key is automatically added to the host key database with no user confirmation. Writing the key to the /dev/null file discards the key and reports success.

Please refer to this excellent article about host keys and key checking.

By specifying the above 2 SSH options on the command line, you can bypass host key checking for that particular SSH login. If you want to bypass host key checking on a permanent basis, you need to specify those same options in the SSH configuration file.

You can edit the global SSH configuration file (/etc/ssh/ssh_config) if you want to make the changes permanent for all users.

If you want to target a particular user, modify the user-specific SSH configuration file (~/.ssh/config). The instructions below apply to both files.

Suppose you want to bypass key checking for a particular subnet (192.168.0.0/24).

Add the following lines to the beginning of the SSH configuration file.
Host 192.168.0.*
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null

Note that the configuration file should have a line like Host * followed by one or more parameter-value pairs. Host *means that it will match any host. Essentially, the parameters following Host * are the general defaults. Because the first matched value for each SSH parameter is used, you want to add the host-specific or subnet-specific parameters to the beginning of the file.

As a final word of caution, unless you know what you are doing, it is probably best to bypass key checking on a case by case basis, rather than making blanket permanent changes to the SSH configuration files.

If you make it this far in the article, you may find the following more recent ssh articles interesting:

Allow root ssh login with public key authentication
How to auto fill in ssh client parameters
One-liner to shutdown remote host
X11 Forwarding over SSH

Saturday, September 27, 2008

Upgrade individual packages for Debian-based systems

If you are using Debian-based distributions (Debian, Ubuntu, etc), you are probably familiar with the apt-get update followed by the apt-get upgrade routine. That is what I regularly use to upgrade ALL packages that have an update available.

But what if you only want to upgrade certain individual packages?

apt-get upgrade will upgrade ALL or nothing. So, that is out of the question.

What you need is apt-get install. apt-get install serves dual purposes: install and upgrade. It installs a package if it is not already installed. If it is already installed, apt-get install will upgrade the package to the latest version.

You should still run apt-get update before apt-get install. You can specify one or more package names as arguments to the apt-get install command.

For example, to upgrade these 2 packages: libfreetype6, libtiff4:
$ sudo apt-get update
$ sudo apt-get install libfreetype6 libtiff4

Sunday, September 21, 2008

How to get the process start date and time

How can we determine when a running process was started?

The venerable ps command deserves first consideration.

Most Linux command-line users are familiar with either the standard UNIX notation or the BSD notation when it comes to specifying ps options.

If ps -ef is what you use, that is the UNIX notation.

$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Sep20 ? 00:00:03 init [3]

peter 1218 1 0 Sep20 ? 00:21:35 /usr/lib/iceweasel/firefox-bin -a firefox

peter 4901 1 1 16:34 ? 00:01:12 /usr/bin/emacs-snapshot-gtk


The STIME column displays the start time or date. From the above, we can tell that process 4901 (emacs) began execution at 16:34 (4:34 pm). But on what day, today?

From the ps man page: 'Only the year will be displayed if the process was not started the same year ps was invoked, or "mmmdd" if it was not started the same day, or "HH:MM" otherwise.'

So, emacs was started at 16:34 TODAY.

What is the start time for the other process, the firefox process with pid 1218?

The STIME for process 1218 reads Sep20 (which was yesterday). But what time yesterday?

The default ps -ef only tells you the start date but NOT the time if a process was NOT started on the same day.

If the BSD notation is more familiar to you, you will find that ps aux yields similar results.

$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 1576 540 ? Ss Sep20 0:03 init[3]

peter 1218 0.5 8.9 201252 45456 ? Sl Sep20 21:35 /usr/lib/iceweasel/firefox-bin -a firefox


The Start column in the above output only reveals the start date if the process is older than the current day.

There are (at least) 2 ways to determine the exact start time if the process was started before the current day.

Solution 1
Specify elapsed time in the ps output format.

$ ps -eo pid,cmd,etime
PID CMD ELAPSED
1218 /usr/lib/iceweasel/firefox-bin - 2-16:04:45


The above ps command specifies 3 fields to be included in the output: the process pid, the command, and the elapsed time, respectively.

etime is the elapsed time since the process was started, in the form dd-hh:mm:ss. dd is the number of days; hh, the number of hours; mm, the number of minutes; ss, the number of seconds.

The firefox command started execution 2 days, 16 hours, 4 minutes and 45 seconds ago. To find the exact time, you need to do some simple math.

If you prefer the BSD notation, issue this command:
$ ps axo pid,cmd,etime
PID CMD ELAPSED

1218 /usr/lib/iceweasel/firefox-bin - 2-16:04:57


Solution 2
Get the process pid and read off the timestamp in the corresponding subdirectory in /proc.

First, get the process pid using the ps command (ps -ef or ps aux)

Then, use the ls command to display the creation timestamp of the directory.
$ ls -ld /proc/1218
dr-xr-xr-x 5 peter peter 0 Sep 20 16:14 /proc/1218


You can tell from the timestamp that the process 1218 began executing on Sept 20, 16:14.

If you can think of another clever way to get the start time and date, please let us know.

Tuesday, September 2, 2008

How to find and delete all hard links to a file

Deleting a file is deceptively simple. You can simply use the rm command like this.
$ rm file1


However, if the file has one or more hard links to it, life gets more interesting. You need to seek and destroy all hard links to the file.

A hard link is essentially another name for a file. Hard links to file1 can be created as follows:
$ ln file1 file2
$ ln file1 tmp/file3


file2 and file3 become another name of file1.

How do I know if some file has a hard link to it?

Do a long listing of the file like this.
$ ls -l file1
-rw-r--r-- 3 peter peter 0 2008-09-01 16:15 file1


Note that the hard link count has the value 3 (this is the number to the left of the file owner). It means that the physical file has 3 names: file1 and two others.

If you only rm file1, the other 2 names still exist, and users can still access the physical file using these 2 names.

To find out all hard links to file1, use this command.
$ find /home -xdev -samefile file1
/home/peter/file1
/home/peter/tmp/file3
/home/peter/file2


Note that we specified /home as the search starting point. You should replace /home with the mount point for the filesystem in which your file1 was created. (This is likely to be either /home or / for files in your home directory.)

The reason for using the filesystem's mount point is that hard links are restricted to reside in the same filesystem as the physical file. This means that file2 and file3 must be in the same filesystem as file1. Therefore, to find ALL hard links to a file, you must start search at the mount point of the filesystem.

The -xdev option instructs find NOT to traverse directories in other filesystems.

To find and delete all hard links to a file:
$ find /home -xdev -samefile file1 | xargs rm


An alternative (and slightly more complicated) method is by using inodes. An inode is a data structure assigned to a physical file that contains all its meta-information except the file's name(s).

All hard links to a physical file share the same inode, and have the same inode id number. Therefore, if you know the inode number of a file, you can find all hard links to the file by searching for files with the same inode number.

To display a file's inode number, use ls -i:

$ ls -li file1
2655341 -rw-r--r-- 3 peter peter 0 2008-09-02 19:09 file1


The first column in the above listing shows the file's inode number.

Now that you know the inode number of file1, you can use the find command to search for all files that have the same inode number:

$ find /home -xdev -inum 2655341
/home/peter/file1
/home/peter/tmp/file3
/home/peter/file2


To delete all hard links to the physical file:
$ find /home -xdev -inum 2655341 | xargs rm 

Sunday, August 17, 2008

How I repaired a corrupted grub menu.lst config file

Imagine the shock when I discovered my Debian Etch machine would not boot after I ran a "routine" apt-get upgrade. The upgrade involved quite a number of packages, including the kernel image.

GRUB, the default boot loader, came to an abrupt stop with the error message
Error 15: File Not found.

The culprit was the file / [01;31mvmlinuz- [00m2.6.18-6-k7.

The weird-looking file name suggested strongly that the GRUB config file, /boot/grub/menu.lst, got corrupted by the upgrade.

At this point, I had the following options:
  • Use a rescue CD/DVD like Knoppix to boot into the system, correct the menu.lst file, and reboot.

  • While in GRUB, repair the corrupted pre-set GRUB commands, boot up the system, then correct the menu.lst file, and reboot.

I chose the second option. Below was my experience, followed by some suggestions based on the hard lessons I learned.

I rebooted the system. At the GRUB menu, I selected the corrupted OS entry, and typed e to edit this entry's associated pre-set boot commands. (Note that the OS entries and their associated commands are taken from the menu.lst file.)

The boot commands associated with the OS entry were:
root  (hd0,0)
kernel / [01;31mvmlinuz- [00m2.6.18-6-k7 root=/dev/mapper/tiger-root ro
savedefault

That did not look right because of the funny looking characters in the kernel command and the fact that the boot commands from before were root, kernel, and initrd, but NOT savedefault.

ROOT

The root command specifies and mounts GRUB's root drive and partition where the boot directory is located (/boot). This is usually (hd0,0) which means the first partition of the first hard disk. Note, GRUB's notation for numbering drives and partitions starts from 0, not 1.

If you are not sure what to set the root drive, enter the GRUB batch command mode by pressing b, and enter the following find command.
grub> find /grub/stage1
(hd0,0)

The find command searches for the file named grub/stage1 and displays the root drive and partition which contains the file. Note that if the drive does not have a partition designated for /boot, you need to prepend /boot to the command argument (find /boot/grub/stage1)

So far so good: I did not have to modify the root drive.

KERNEL

The kernel command specifies and loads the Linux kernel image. The default image file name was corrupted. To correct, I selected the kernel command, and pressed e to edit the line.

What should the file name be? Different Linux distributions name the kernel image file differently. Don't fret if you can't remember its name. The GRUB command line offers file name completion. So just enter kernel / and then hit tab.
grub> kernel /
Possible files are: System.map-2.6.18-6-k7 config-2.6.18-5-k7 config-2.6.18-6-k7 initrd.img-2.6.18-5-k7 vmlinuz-2.6.18-5-k7 grub initrd.img-2.6.18-6-k7 System.map-2.6.18-5-k7 vmlinuz-2.6.18-6-k7

For Debian, Ubuntu, Fedora, and Mandriva, the kernel image file is named vmlinuz followed by the kernel release number and the machine architecture (e.g., vmlinuz-2.6.18-6-k7). From the options returned by the file name completion feature, choose the kernel image with the latest release number.

The rest of the kernel parameters looked OK, and required no change.
grub> kernel /vmlinuz-2.6.18-6-k7 root=/dev/mapper/tiger-root ro

INITRD

Next, I had to replace the savedefault command with the initrd command.

initrd specifies the ramdisk image file. The RAM disk is used for loading modules required to access the root filesystem.

I first selected the savedefault command and pressed e to edit the line. Again, you could use the Tab key to help you complete the filename for the RAM disk file.
grub> initrd   /initrd.img-2.6.18-6-k7 


Boot & Edit menu.lst

After I made the above changes, I went back to the GRUB main menu, and pressed b to boot.

This time, the machine booted up successfully, and everything worked just fine for me.

I was not done however. Unless I changed the source of the problem (the corrupted menu.lst), the machine would come up with the same boot error in the next reboot. So, as root, I opened the file /boot/grub/menu.lst and edited the commands.

Before I could modify the corrupted commands, I needed to first locate the corresponding OS entry in the file. Each OS entry occupies a separate section in the file, beginning with its own title line. So, I scrolled down the file until I reached the target title line.
title Debian GNU/Linux, kernel  [01;31m- [00m2.6.18-6-k7
root (hd0,0)
kernel / [01;31mvmlinuz- [00m2.6.18-6-k7 root=/dev/mapper/tiger-root ro
savedefault


After correcting the commands, save the file, and reboot the machine.

Lessons Learned

The GRUB config file (menu.lst) can get corrupted, and when it does, it spells real trouble.

The commands to boot the OS are not something anyone tends to remember. So, it makes sense to have a print out of the menu.lst file or a backup copy.

What I do is backup the menu.lst file in the same directory as menu.lst (say, call it menu.lst.bak).

The advantage of saving it in the same directory (as opposed to somewhere over the network) is that if the menu.lst file ever gets corrupted again, you can still display the backup copy at the grub command prompt. During the GRUB boot up process, you can display the backup file by simply entering the following at the GRUB command prompt.
 grub> cat /grub/menu.lst.bak 


The above cat command displays the backup menu.lst file. Armed with the knowledge of the correct commands to use, you can then edit the commands as shown in this article.