Thursday, June 5, 2008

Show progress during dd copy

(2016-01-09 Update)
I owe RapidElectronic for his excellent comment regarding the new dd command. Beginning with version 8.24, you can specify the parameter, status=progress, to the dd command. By using this new parameter, you no longer need to send an explicit USR1 signal to the dd process to request an update of the disk copy statistics; it will automatically print periodic updates in the standard output.


$ sudo df if=/dev/sda of=/dev/sdb status=progress

Note that sending the USR1 signal will continue to work for the new dd.

(Original article)

dd is a popular, generic command-line tool for copying files from 1 location to another. It is often used to copy entire disk images.

Like many Linux command line tools, it operates silently unless something unexpected happens. Its lack of visual progress feedback is a nice feature for scripting. However, it can leave you wondering about its progress if you are interactively dd-copying a large disk.

To illustrate, you run the following (valid, but perhaps not very useful) dd copy:

$ dd if=/dev/random of=/dev/null bs=1K count=100 

It will run for a few minutes as it copies (and immediately discards) 100 blocks of randomly generated data, each of size 1 KB.

To get a progress report while dd is running, you need to open another virtual terminal, and then send a special USR1 signal to the dd process.

First, find out the process id of the dd process by running the following in the new virtual terminal.

$ pgrep -l '^dd$'
8789 dd
$

To send the USR1 signal to the dd prcoess:

$ kill -USR1  8789
$

Note that as soon as the USR1 signal is detected, dd will print out the current statistics to its STDERR.

$ dd if=/dev/random of=/dev/null bs=1K count=100
0+14 records in
0+14 records out
204 bytes (204 B) copied, 24.92 seconds, 0.0 kB/s

After reporting the status, dd will resume copying. You can repeat the above kill command any time you want to see the interim statistics. Alternatively, you can use the watch command to execute kill at a set interval.

$ watch -n 10 kill -USR1 8789

P.S.

Other articles from this blog on the dd command:
Create files of a given size

71 comments:

GiNeR said...

Thank U!

Anonymous said...

thanks too. i was trying SIGINFO but that isnt implemented on linux. a wiki article said SIGPWR is SIGINFO - after 2 hrs of cloning my new HDD i managed to kill dd because of a power problem. Gutted. SIGPWR wiki article is a troll :( nice watch idea - i was going to script it :D

Cullen said...

Thanks. This was very helpful!

sergerus said...

Use "pv" command from "pv" package.

Anonymous said...

Try for example:
pv -ptre FILE | dd of=FILE bs=1M
Gives a nice output with a progressbar, good for writing cf cards from images.

beambot said...

Just use dcfldd instead. It is an "enhanced version of dd for forensics and security" and is readily available in most *nix package mangers!

Past said...

you can use killall -s USR1 dd

totedati said...

can be ever better, using only tr as an extracommand:

dd if=/dev/random of=/dev/null bs=1K count=1000

then:

pgrep -l '^dd$' | tr -d [:alpha:]
12320

watch -n 1 kill -USR1 $(pgrep -l '^dd$' | tr -d [:alpha:])

and output will be:

dd if=/dev/random of=/dev/null bs=1K count=1000
0+85 records in
0+85 records out
709 bytes (709 B) copied, 93.5436 s, 0.0 kB/s
0+113 records in
0+113 records out
933 bytes (933 B) copied, 119.764 s, 0.0 kB/s
0+113 records in
0+113 records out
.
.

and so on ...

Anonymous said...
This comment has been removed by a blog administrator.
hERB said...

Handy tip, thanks!

/hERB

Anonymous said...

Thanks alot

Anonymous said...

If you're the only one on the system, just do:

pkill -USR1 dd

Saves yourself a step!

kc7zzv said...

By the way, using SIGUSR1 on MacOS will kill dd. You need to use SIGINFO instead.

Unknown said...

Careful, I did this on Ubuntu 8.10 and it just killed the process >:(

DJ Gruby said...

Thank you! Very useful piece of advice! :)

Anonymous said...

Thanks a lot. Nice tip! Just helped me to see the status if my prolong running 'dd'!

Dipak

Anonymous said...

1. Nice hack :)
2. Ubuntu doesn't really count as Linux :)

Daniel said...

Thanks. This was exactly what I was looking for.

Ubuntu definitely counts as Linux.

Tobias said...

You are a genius, and I thank you so very much. I am glad that I don't have to kill dd and start it over again with a --verbose flag!

Tim said...

Neat!

kriss said...

thanks, very helpfull - I've used:

while sleep 10; do sudo kill -USR1 [PID]; done

Anonymous said...

Good stuff, very helpful while low-level copying a 750Gb drive. Thanks!!

Dextor said...

thanks!!! was very helpfull...

Anonymous said...

You made my day. Thanks.

Emine said...

very helpful, thank you

I did the following:

$dd if=/dev/sda of=/dev/sdb conv=sync

and on another terminal:

$watch -n 480 kill -USR1 4292

The final message was:

488397168+0 records in
488397168+0 records out
250059350016 bytes (250 GB) copied, 53809 s, 4.6 MB/s


53809 s = 14 h 56 mn 49 s



Before copying I got :

$ fdisk -l

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x35e920cb

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 6092 48829567+ 83 Linux
/dev/sda3 6093 13387 58597087+ 7 HPFS/NTFS
/dev/sda4 13388 30401 136664955 f W95 Ext'd (LBA)
/dev/sda5 13388 21036 61440561 7 HPFS/NTFS


Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe8cd189d

Device Boot Start End Blocks Id System
/dev/sdb1 1 60801 488384001 c W95 FAT32 (LBA)



After copying I got :

$ fdisk -l

Disk /dev/sda: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x35e920cb

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 6092 48829567+ 83 Linux
/dev/sda3 6093 13387 58597087+ 7 HPFS/NTFS
/dev/sda4 13388 30401 136664955 f W95 Ext'd (LBA)
/dev/sda5 13388 21036 61440561 7 HPFS/NTFS

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x35e920cb

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 13 104391 83 Linux
/dev/sdb2 14 6092 48829567+ 83 Linux
/dev/sdb3 6093 13387 58597087+ 7 HPFS/NTFS
/dev/sdb4 13388 30401 136664955 f W95 Ext'd (LBA)
/dev/sdb5 13388 21036 61440561 7 HPFS/NTFS

Anonymous said...

Thanks, i knew about the USR1 trick, but learnt about the watch command from your blog. It's really useful.

Anonymous said...

Thx! :)

Knapsu said...

Exactly what I needed. Thank you!

Anonymous said...

Thanks. but you need to explain what STDERR means. You don't explain what STDERR is

JVMHost said...

STDERR is error message destination, a console/monitor by default. Thanks for the article.

Anonymous said...

Typing ^T (CTRL-T) in the console where dd is running should achieve the same result as the USR1 signal.

Regards

Juanj said...

Just thank you for this tip.

Szymon said...

Thumb up! Thank you mate!

Anonymous said...

Really.. nice job! This is what I wanted.

Thank You.


http://webhoststore.tk

Anonymous said...

Thanks for the help!

Anonymous said...

Excellent, thank you Master!

Anonymous said...

Warning: using the USR1 signal on Mac OSX kills dd!

Anonymous said...

Thank you So Much! I was in pain because I didn't knew if the command was working correctly or not.... i am zeroing my hard drive.Thank you again for your advice.

Timothy Tripp said...

Warning: using the USR1 signal on Mac OSX kills dd! Use this on OSX:

killall -SIGINFO dd

or (if dd was started as sudo)

sudo killall -SIGINFO dd

Anonymous said...

dd if=/dev/zero of=/dev/null & pid=$! ; watch -n 3 kill -USR1 $pid

:c)

DailyRazor said...

Thanks. This was very helpful!

@iambaskar said...

I just did

sudo kill -USR1

on Mac, It killed the process!! Damn! it was running more than 1 hour :( Lost it!

ok said...

I just used it after booting my machine from System Rescue CD and using the dd command to clone my disk to another disk of the same size. Worked like a charm. Thank you for the tip!

Anonymous said...

Thank you!!! Had many months that I was seeking for this tip!!! tks :))

Unknown said...

@iambaskar
yeah, I just did that too, and it killed the process w/o restarting. Also on a Mac.

Unknown said...

@Joel Riggs and Mac OS friends: check the man page of your dd implementation. In Linux, "dd" is part of the GNU coreutils package. The man page in a recent Linux system says:

Sending a USR1 signal to a running 'dd' process makes it print I/O sta‐
tistics to standard error and then resume copying.

$ dd if=/dev/zero of=/dev/null& pid=$!
$ kill -USR1 $pid; sleep 1; kill $pid


Other "dd" imprementation may not handle an USR1 signal in which case the process will terminate.

Everything is in the manuals (TM).

Anonymous said...

Cool stuff. Thank you.

Gustavo Diaz said...

dd if=/dev/sda1 bs=1M |pv -brtp -s 500M| gzip -1 > /home/boot-backup.gz

Bing Ren said...

Cool, and thank you!

Anonymous said...

Thanks for the info!

I'm in the process of dd a 300GB disk. With help of
$ pgrep -l '^dd$'
8789 dd
$
and

$ kill -USR1 8789

I find I have 10 hours still to wait to complete dd from the 5 hours so far.

I am hindered by the fact I am booting from USB HDD and copying files to another HDD from laptop HDD.
@ 5.9MB/s

As a result I can now kick back and get on with something else rather than staring at what appeared to be a Terminal window doing nothing :)

Anonymous said...

I forgot to add.
I had to use sudo kill -USR1 8789

Anonymous said...

Try out **pv** or pipeviewer :D

xdpirate said...

You could just do

watch -n 1 sudo kill -USR1 `pidof dd`

Anonymous said...

Dooooood. Thats awesome. THANKS!

Anonymous said...

Excellent info - really useful - I'm trying low level format a flash drive, and this really helps to tell me how far its got.

Anonymous said...

err, can't you simply press Ctrl+T in the terminal running dd?
That works on FreeBSD and other BSDs.

flitjes said...

you could also use:
watch -n 10 kill -USR1 `pidof dd`

furriephillips said...

If your device has no activity lights, you've no idea that
anything's even happening with "dd" :(

The progress info that "dcfldd" shows is just what you need.

I've now got this in my .zshrc: -

## Conditional Aliases
## dd -> dcfldd
if type dcfldd 2>&1 >/dev/null ; then
alias dd='dcfldd'
fi

Thanks!

Anonymous said...

On OpenSUSE this seems to work only partially. When I was reading from a HDD to file it worked like a charm but when I was writing it back to another HDD it killed the process. Very strange.

Anonymous said...

thank u!

Anonymous said...

Thank you.
Clearest description and works.

I had "bash" errors so had to add "sudo" to the front.

Anonymous said...

Very useful. Thank you for this post!

Anonymous said...

Thank you very much friend, that really works :-)

Miguel Ortiz said...

Thank you man, this is very usefull!!! :) -USR1 and watch will become my new friends.

Hope you check my project, it has information about technology and virtual education in Argentina. hugs!!

Anonymous said...

I don't know if you still maintain this page and if you know that this is page #1 when your searching "dd progress".

So maybe you could enhance your post that your post is more useful than the comments below. ;)

I would recommend this e.g. command:

while true; do sudo kill -USR1 $(pgrep '^dd$'); sleep 5m; done

where sudo is just needed if you e.g. dd-ing from /dev/sda and the loop just if you want it to repeat by itself. Sure it's also possible with watch... but at least

kill -USR1 $(pgrep '^dd$')

is a really useful combination. This maybe will have some issues when using multiple dd's, but whatever. :) Guess that's easy to fix then on the fly.

Anonymous said...

Just press CTRL+t to see the interim statistics

Rapid Electronic said...

New version of coreutils (8.24) adding a status progress to dd tool:

Usage on Xubuntu 15.10:

Open terminal shell and type these commands:

wget ftp://ftp.gnu.org/pub/gnu/coreutils/coreutils-8.24.tar.xz
tar -xf coreutils-8.24.tar.xz
cd coreutils-8.24
./configure && make -j $(nproc)

Run dd as root:

sudo ./dd if=/dev/sdc of=/dev/sda conv=noerror status=progress

You will see: Bytes, Seconds and Velocity (Bytes/seconds)

To check versions of dd:

Native:

dd --version

New (cd coreutils-8.24/src):

./dd --version

Peter Leung said...

Thanks, Rapid Electronic. I've updated this article with the new information you provided.

Anonymous said...

Thanks, I don't use linux much often and everytime I do I wonder how to check dd status :-)
Anyway, you should mention that you're talking about GNU version of dd, which is pure linux thing. BSD and other unices (including OS X, which is based on Darwin, which is Apple derivate of FreeBSD) use another version of dd and kill and everything. So the INFO kill signal and Ctrl+T keypress are lethal for linux/GNU version of dd!


Little flame about Ubuntu:
Ubuntu definitely is linux, but.. It's linux for (dumb) masses. Folks who know linux rarely install it on their machines. Ordinary linux guy uses Debian, Slackware, RedHat or OpenSUSe. Some maniacs use Arch. I'd like to watch Ubuntu user installing Arch linux or set up devices on any other distro :-)

Anonymous said...

Shouldn't it be $ sudo dd if=/dev/sda of=/dev/sdb status=progress instead of $ sudo df if=/dev/sda of=/dev/sdb status=progress ?

Anonymous said...

You can add watch to the kill command to repeat the command.

For exemple every 60 seconds :

watch -n 60 kill -USR1 2334