Create File of a Given Size ... with random contents

A while back, I wrote about how to create a zero-filled file of any arbitrary size. This is part 2 where I share how to create a file of random contents (not just zeroes).

Recently, I ran into a situation where a zero-filled file is insufficient. I needed to create a log file of size 2 MB in order to be zipped up and copied to another server.

To create the 2MB file (with all zeroes), I run the dd command:
$ dd if=/dev/zero of=a.log bs=1M count=2

I quickly realized that the test result would be invalid because zipping a all-zero file dramatically reduced its size.

$ gzip a.log
$ ls -hl a.log*
-rw-r--r-- 1 peter peter 2.1K 2008-06-14 14:36 a.log.gz
$

I decided to create a 2 MB file of random contents instead. This is how.

$ dd if=/dev/urandom of=a.log bs=1M count=2
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 1.00043 seconds, 2.1 MB/s
$ gzip a.log
$ ls -hl a.log*
-rw-r--r-- 1 peter peter 2.1M 2008-06-14 14:43 a.log.gz
$

To take a look at the random contents of a.log, use the hexdump command:

$ hexdump a.log |head
0000000 c909 2da7 4a77 22fc 88b6 b394 be42 b0c1
0000010 1531 f9d5 4b3d 390d e670 da2c e7e9 b681
0000020 0518 2b5d 5a66 ef76 c297 7f73 2d0b 453e
0000030 ba47 c268 26f9 79b5 1816 82ac 2e76 0ff2
0000040 c1e8 e14f 898f 2507 9c29 83b7 226c 0d65
0000050 f3f6 6eb4 62d9 410b b566 c522 ffca fbac
0000060 81f6 d91c dd34 18cd f873 8073 fa02 20c1
0000070 06bb 7e32 dc2e 13b2 a345 aadd 8700 fa9e
0000080 e28e 1b58 c25f 4619 c8bc 8110 6306 a2fc
0000090 9766 d98f 648e cec7 d654 2eaa 1f6f 839f

Smart case-insensitive, incremental search using vim

My previous article describes my top annoyance with the vim text editor, namely, its syntax highlighting. In this article, I will tell on a close second annoyance, and what to do about it.

vim, by default, searches case sensitively. If you search for apple, you will find exactly that, but not Apple or APPLE.

In most situations, I want my searches to be case insensitive. To make search case sensitive, set the corresponding vim option by typing :set ignorecase (and press the return key).

ignorecase has a shorter alias called ic. You can type :set ic and it will have the same effect.

Now searching for apple will give you Apple, APPLE as well as apple.

But, what about the situations where you DO want case-sensitive searching?

You can always disable the ignorecase search, by typing the following and hit return:
:set noignorecase

Flipping between ignorecase and ignorecase can be tiresome for even the most patient. Luckily, vim has the smartcase option that you can use TOGETHER with ignorecase.

Type the following:
:set ignorecase (and hit return)
:set smartcase (and hit return)

With both ignorecase and smartcase turned on, a search is case-insensitive if you enter the search string in ALL lower case. For example, searching for apple will find Apple and APPLE.

However, if your search string has one or more characters in upper case, it will assume that you want a case-sensitive search. So, searching for Apple will only give you Apple but not apple or APPLE. It turns out to be quite satisfactory for most people (including yours truly).

While we are on the topic of vim search options, there is a third option that you should know:
:set incsearch (and hit return)

incsearch stands for incremental search. It means that you will see what vim matches as you type in each letter of your search string (without having to hit return before search is even attempted).

For example, you type / to initiate search, and right after you type the letter a, vim will highlight the a in apple. As you type the next letter p, vim will highlight ap in the word apple.

You can often find what you are looking for before you finish typing in the entire search string. It is also helpful if you are not quite sure of what you are searching for, and depending on the instant feedback as you type, you can make corrections to the search string by backspacing.

If you want to enable those options permanently, insert the following lines into your ~/.vimrc file.
set ignorecase
set smartcase
set incsearch

Happy searching!

How to find a file and cd to its dirname using command substitution

Many times I know the name of a file on my Linux machine, say unknown.txt, but I don't know the directory the file is in.

If I want to change directory to the directory folding the file, I used to do a 2-step process:

$ find / -name unknownfile.txt 2>/dev/null
/home/peter/status/2007/november/unknownfile.txt

Note: if you know more about where that file may be, you can always make the starting point of the find more specific (say "/home/peter" instead of just "/").

Then, I manually enter the cd command with the path discovered from the last step.

$ cd /home/peter/status/2007/november
$

I got tired of re-entering the directory name in this 2 step process. So, I set out to see if I can automate it further.

Here we go.

There may be more than 1 file of that file name, and if so, let's take the first one.

$ find / -name unknownfile.txt  2>/dev/null | head -n 1
/home/peter/status/2007/november/unknownfile.txt

We need to extract just the directory path, and leave out the filename (so that we can cd to the directory).

The dirname command should be able to do that, but alas, dirname cannot take its input from STDIN. dirname expects the file name as a command line parameter. The following will fail.

$ find / -name unknownfile.txt  2>/dev/null | head -n 1  | dirname
dirname: missing operand
Try `dirname --help' for more information.

This is where command substitution comes in.

The final command is:

$ cd $(dirname $(find / -name unknownfile.txt  2>/dev/null | head -n 1))
$ pwd
/home/peter/status/2007/november
$

Let's break it down. Using command substitution, the output of one command becomes the input parameter to another command.

The first command substitution we use is:

dirname $(find / -name unknownfile.txt  2>/dev/null | head -n 1)

In this case, the output of "find / -name unknownfile.txt 2>/dev/null | head -n 1" becomes the input to dirname.

Then, we again use command substitution to make available the output of dirname as the input to cd.

cd $(dirname $(find / -name unknownfile.txt  2>/dev/null | head -n 1))

Show progress during dd copy

(2016-01-09 Update)
I owe RapidElectronic for his excellent comment regarding the new dd command. Beginning with version 8.24, you can specify the parameter, status=progress, to the dd command. By using this new parameter, you no longer need to send an explicit USR1 signal to the dd process to request an update of the disk copy statistics; it will automatically print periodic updates in the standard output.


$ sudo df if=/dev/sda of=/dev/sdb status=progress


Note that sending the USR1 signal will continue to work for the new dd.

(Original article)

dd is a popular, generic command-line tool for copying files from 1 location to another. It is often used to copy entire disk images.

Like many Linux command line tools, it operates silently unless something unexpected happens. Its lack of visual progress feedback is a nice feature for scripting. However, it can leave you wondering about its progress if you are interactively dd-copying a large disk.

To illustrate, you run the following (valid, but perhaps not very useful) dd copy:

$ dd if=/dev/random of=/dev/null bs=1K count=100 

It will run for a few minutes as it copies (and immediately discards) 100 blocks of randomly generated data, each of size 1 KB.

To get a progress report while dd is running, you need to open another virtual terminal, and then send a special USR1 signal to the dd process.

First, find out the process id of the dd process by running the following in the new virtual terminal.

$ pgrep -l '^dd$'
8789 dd
$

To send the USR1 signal to the dd prcoess:

$ kill -USR1  8789
$

Note that as soon as the USR1 signal is detected, dd will print out the current statistics to its STDERR.

$ dd if=/dev/random of=/dev/null bs=1K count=100
0+14 records in
0+14 records out
204 bytes (204 B) copied, 24.92 seconds, 0.0 kB/s

After reporting the status, dd will resume copying. You can repeat the above kill command any time you want to see the interim statistics. Alternatively, you can use the watch command to execute kill at a set interval.

$ watch -n 10 kill -USR1 8789

P.S.

Other articles from this blog on the dd command:
Create files of a given size

How to number each line in a text file on Linux

Some Linux commands support options that will number the input lines as a side effect, e.g., grep, and cat. The nl command is on the other hand dedicated to the task of numbering lines in a text file. If you want maximum flexibility, sed or perl is your best bet.

Suppose you want to number the lines in the input.txt file which contains:

123

456

789


abc

def

ghi

Below are some ways to number input.txt:

• cat -n
  

  $ cat -n input.txt
       1  123
       2
       3  456
       4
       5  789
       6
       7
       8  abc
       9
      10  def
      11
      12  ghi

  
  
  
• grep -n
  

  $ grep -n '^' input.txt
  1:123
  2:
  3:456
  4:
  5:789
  6:
  7:
  8:abc
  9:
  10:def
  11:
  12:ghi

  
  
  
  
• nl
  

nl inserts the line number at the beginning of each non-empty line. By default, the line number is 6 characters wide, right justified with leading spaces. A tab is inserted by default after the line number.

$ nl input.txt
     1  123

     2  456

     3  789


     4  abc

     5  def

     6  ghi

If your file is small, 6 is perhaps too wide for the line number field. To adjust the width of the line number, use the -w option. To make nl number all lines including blank ones, add -ba option.

$ nl -ba -w 3 input.txt
  1     123
  2
  3     456
  4
  5     789
  6
  7
  8     abc
  9
 10     def
 11
 12     ghi

If you don't want a tab after the line number, you can replace the tab with null (no separator between line number and rest of line), or any string you want. Use -s '' for no separator or -s ' ' for a space.

$ nl -ba -w 3  -s ' ' input.txt
  1 123
  2
  3 456
  4
  5 789
  6
  7
  8 abc
  9
 10 def
 11
 12 ghi

If you prefer left justifying the line numbers, set the field width to 1 (or use -n ln option).

$ nl -ba -w 1  input.txt
1       123
2
3       456
4
5       789
6
7
8       abc
9
10      def
11
12      ghi

nl is flexible enough to only number lines that match a regular expression. This is done by specifying the option -b p followed by a regular expression on the command line.

Number only those lines that start with either the character 1 or 4.

$ nl -b 'p^[14]' -w 3 -s ' ' input.txt
  1 123

  2 456

    789


    abc

    def

    ghi

Note that we specify -s ' ' to use a single space as the separator between line number and body text. This is used to preserve text alignment (the default tab will cause output to look messy).

Number only those lines that contain the "words" 12 or ef.

$ nl -b 'p12\|ef' -w 3 -s ' ' input.txt
  1 123

    456

    789


    abc

  2 def

    ghi