▼

Thursday, June 5, 2008

Show progress during dd copy

(2016-01-09 Update)
I owe RapidElectronic for his excellent comment regarding the new dd command. Beginning with version 8.24, you can specify the parameter, status=progress, to the dd command. By using this new parameter, you no longer need to send an explicit USR1 signal to the dd process to request an update of the disk copy statistics; it will automatically print periodic updates in the standard output.


$ sudo df if=/dev/sda of=/dev/sdb status=progress

Note that sending the USR1 signal will continue to work for the new dd.

(Original article)

dd is a popular, generic command-line tool for copying files from 1 location to another. It is often used to copy entire disk images.

Like many Linux command line tools, it operates silently unless something unexpected happens. Its lack of visual progress feedback is a nice feature for scripting. However, it can leave you wondering about its progress if you are interactively dd-copying a large disk.

To illustrate, you run the following (valid, but perhaps not very useful) dd copy:

$ dd if=/dev/random of=/dev/null bs=1K count=100

It will run for a few minutes as it copies (and immediately discards) 100 blocks of randomly generated data, each of size 1 KB.

To get a progress report while dd is running, you need to open another virtual terminal, and then send a special USR1 signal to the dd process.

First, find out the process id of the dd process by running the following in the new virtual terminal.

$ pgrep -l '^dd$'
8789 dd
$

To send the USR1 signal to the dd prcoess:

$ kill -USR1  8789
$

Note that as soon as the USR1 signal is detected, dd will print out the current statistics to its STDERR.

$ dd if=/dev/random of=/dev/null bs=1K count=100
0+14 records in
0+14 records out
204 bytes (204 B) copied, 24.92 seconds, 0.0 kB/s

After reporting the status, dd will resume copying. You can repeat the above kill command any time you want to see the interim statistics. Alternatively, you can use the watch command to execute kill at a set interval.

$ watch -n 10 kill -USR1 8789

P.S.

Other articles from this blog on the dd command:
Create files of a given size

71 comments:

  1. GiNeRAugust 21, 2008 at 10:03 AM

    Thank U!

    ReplyDelete
  2. AnonymousAugust 27, 2008 at 10:35 AM

    thanks too. i was trying SIGINFO but that isnt implemented on linux. a wiki article said SIGPWR is SIGINFO - after 2 hrs of cloning my new HDD i managed to kill dd because of a power problem. Gutted. SIGPWR wiki article is a troll :( nice watch idea - i was going to script it :D

    ReplyDelete
  3. CullenOctober 12, 2008 at 4:27 PM

    Thanks. This was very helpful!

    ReplyDelete
  4. sergerusNovember 11, 2008 at 5:58 AM

    Use "pv" command from "pv" package.

    ReplyDelete
  5. AnonymousFebruary 11, 2009 at 11:38 AM

    Try for example:
    pv -ptre FILE | dd of=FILE bs=1M
    Gives a nice output with a progressbar, good for writing cf cards from images.

    ReplyDelete
  6. beambotApril 15, 2009 at 3:10 PM

    Just use dcfldd instead. It is an "enhanced version of dd for forensics and security" and is readily available in most *nix package mangers!

    ReplyDelete
  7. PastMay 28, 2009 at 2:15 AM

    you can use killall -s USR1 dd

    ReplyDelete
  8. totedatiJune 6, 2009 at 5:45 PM

    can be ever better, using only tr as an extracommand:

    dd if=/dev/random of=/dev/null bs=1K count=1000

    then:

    pgrep -l '^dd$' | tr -d [:alpha:]
    12320

    watch -n 1 kill -USR1 $(pgrep -l '^dd$' | tr -d [:alpha:])

    and output will be:

    dd if=/dev/random of=/dev/null bs=1K count=1000
    0+85 records in
    0+85 records out
    709 bytes (709 B) copied, 93.5436 s, 0.0 kB/s
    0+113 records in
    0+113 records out
    933 bytes (933 B) copied, 119.764 s, 0.0 kB/s
    0+113 records in
    0+113 records out
    .
    .

    and so on ...

    ReplyDelete
  9. AnonymousJuly 6, 2009 at 8:52 PM

    This comment has been removed by a blog administrator.

    ReplyDelete
  10. hERBJuly 12, 2009 at 7:09 PM

    Handy tip, thanks!

    /hERB

    ReplyDelete
  11. AnonymousAugust 3, 2009 at 8:55 AM

    Thanks alot

    ReplyDelete
  12. AnonymousAugust 23, 2009 at 6:59 PM

    If you're the only one on the system, just do:

    pkill -USR1 dd

    Saves yourself a step!

    ReplyDelete
  13. kc7zzvOctober 15, 2009 at 9:49 AM

    By the way, using SIGUSR1 on MacOS will kill dd. You need to use SIGINFO instead.

    ReplyDelete
  14. UnknownNovember 26, 2009 at 10:25 AM

    Careful, I did this on Ubuntu 8.10 and it just killed the process >:(

    ReplyDelete
  15. DJ GrubyJanuary 1, 2010 at 10:12 AM

    Thank you! Very useful piece of advice! :)

    ReplyDelete
  16. AnonymousFebruary 17, 2010 at 7:25 AM

    Thanks a lot. Nice tip! Just helped me to see the status if my prolong running 'dd'!

    Dipak

    ReplyDelete
  17. AnonymousMarch 1, 2010 at 1:33 AM

    1. Nice hack :)
    2. Ubuntu doesn't really count as Linux :)

    ReplyDelete
  18. DanielMarch 29, 2010 at 1:26 PM

    Thanks. This was exactly what I was looking for.

    Ubuntu definitely counts as Linux.

    ReplyDelete
  19. TobiasApril 13, 2010 at 3:35 PM

    You are a genius, and I thank you so very much. I am glad that I don't have to kill dd and start it over again with a --verbose flag!

    ReplyDelete
  20. TimMay 21, 2010 at 10:18 PM

    Neat!

    ReplyDelete
  21. krissJune 27, 2010 at 1:51 PM

    thanks, very helpfull - I've used:

    while sleep 10; do sudo kill -USR1 [PID]; done

    ReplyDelete
  22. AnonymousJuly 7, 2010 at 10:15 PM

    Good stuff, very helpful while low-level copying a 750Gb drive. Thanks!!

    ReplyDelete
  23. DextorJuly 30, 2010 at 4:20 PM

    thanks!!! was very helpfull...

    ReplyDelete
  24. AnonymousSeptember 10, 2010 at 11:42 AM

    You made my day. Thanks.

    ReplyDelete
  25. EmineFebruary 13, 2011 at 3:29 AM

    very helpful, thank you

    I did the following:

    $dd if=/dev/sda of=/dev/sdb conv=sync

    and on another terminal:

    $watch -n 480 kill -USR1 4292

    The final message was:

    488397168+0 records in
    488397168+0 records out
    250059350016 bytes (250 GB) copied, 53809 s, 4.6 MB/s


    53809 s = 14 h 56 mn 49 s



    Before copying I got :

    $ fdisk -l

    Disk /dev/sda: 250.1 GB, 250059350016 bytes
    255 heads, 63 sectors/track, 30401 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x35e920cb

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 13 104391 83 Linux
    /dev/sda2 14 6092 48829567+ 83 Linux
    /dev/sda3 6093 13387 58597087+ 7 HPFS/NTFS
    /dev/sda4 13388 30401 136664955 f W95 Ext'd (LBA)
    /dev/sda5 13388 21036 61440561 7 HPFS/NTFS


    Disk /dev/sdb: 500.1 GB, 500107862016 bytes
    255 heads, 63 sectors/track, 60801 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0xe8cd189d

    Device Boot Start End Blocks Id System
    /dev/sdb1 1 60801 488384001 c W95 FAT32 (LBA)



    After copying I got :

    $ fdisk -l

    Disk /dev/sda: 250.1 GB, 250059350016 bytes
    255 heads, 63 sectors/track, 30401 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x35e920cb

    Device Boot Start End Blocks Id System
    /dev/sda1 * 1 13 104391 83 Linux
    /dev/sda2 14 6092 48829567+ 83 Linux
    /dev/sda3 6093 13387 58597087+ 7 HPFS/NTFS
    /dev/sda4 13388 30401 136664955 f W95 Ext'd (LBA)
    /dev/sda5 13388 21036 61440561 7 HPFS/NTFS

    Disk /dev/sdb: 500.1 GB, 500107862016 bytes
    255 heads, 63 sectors/track, 60801 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x35e920cb

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 1 13 104391 83 Linux
    /dev/sdb2 14 6092 48829567+ 83 Linux
    /dev/sdb3 6093 13387 58597087+ 7 HPFS/NTFS
    /dev/sdb4 13388 30401 136664955 f W95 Ext'd (LBA)
    /dev/sdb5 13388 21036 61440561 7 HPFS/NTFS

    ReplyDelete
  26. AnonymousApril 25, 2011 at 7:47 AM

    Thanks, i knew about the USR1 trick, but learnt about the watch command from your blog. It's really useful.

    ReplyDelete
  27. AnonymousMay 17, 2011 at 1:52 PM

    Thx! :)

    ReplyDelete
  28. KnapsuJuly 5, 2011 at 1:33 AM

    Exactly what I needed. Thank you!

    ReplyDelete
  29. AnonymousOctober 10, 2011 at 7:42 PM

    Thanks. but you need to explain what STDERR means. You don't explain what STDERR is

    ReplyDelete
  30. JVMHostDecember 20, 2011 at 3:57 AM

    STDERR is error message destination, a console/monitor by default. Thanks for the article.

    ReplyDelete
  31. AnonymousDecember 30, 2011 at 9:42 AM

    Typing ^T (CTRL-T) in the console where dd is running should achieve the same result as the USR1 signal.

    Regards

    ReplyDelete
  32. JuanjFebruary 21, 2012 at 2:57 PM

    Just thank you for this tip.

    ReplyDelete
  33. SzymonMarch 21, 2012 at 6:53 AM

    Thumb up! Thank you mate!

    ReplyDelete
  34. AnonymousApril 18, 2012 at 3:14 AM

    Really.. nice job! This is what I wanted.

    Thank You.


    http://webhoststore.tk

    ReplyDelete
  35. AnonymousMay 21, 2012 at 6:32 AM

    Thanks for the help!

    ReplyDelete
  36. AnonymousJuly 1, 2012 at 8:53 AM

    Excellent, thank you Master!

    ReplyDelete
  37. AnonymousJuly 26, 2012 at 3:23 PM

    Warning: using the USR1 signal on Mac OSX kills dd!

    ReplyDelete
  38. AnonymousJuly 29, 2012 at 6:47 AM

    Thank you So Much! I was in pain because I didn't knew if the command was working correctly or not.... i am zeroing my hard drive.Thank you again for your advice.

    ReplyDelete
  39. Timothy TrippJuly 30, 2012 at 6:55 AM

    Warning: using the USR1 signal on Mac OSX kills dd! Use this on OSX:

    killall -SIGINFO dd

    or (if dd was started as sudo)

    sudo killall -SIGINFO dd

    ReplyDelete
  40. AnonymousOctober 18, 2012 at 1:12 AM

    dd if=/dev/zero of=/dev/null & pid=$! ; watch -n 3 kill -USR1 $pid

    :c)

    ReplyDelete
  41. DailyRazorNovember 27, 2012 at 6:48 AM

    Thanks. This was very helpful!

    ReplyDelete
  42. @iambaskarFebruary 14, 2013 at 1:14 PM

    I just did

    sudo kill -USR1

    on Mac, It killed the process!! Damn! it was running more than 1 hour :( Lost it!

    ReplyDelete
  43. okMarch 22, 2013 at 7:35 PM

    I just used it after booting my machine from System Rescue CD and using the dd command to clone my disk to another disk of the same size. Worked like a charm. Thank you for the tip!

    ReplyDelete
  44. AnonymousMarch 23, 2013 at 6:00 AM

    Thank you!!! Had many months that I was seeking for this tip!!! tks :))

    ReplyDelete
  45. UnknownMarch 23, 2013 at 2:45 PM

    @iambaskar
    yeah, I just did that too, and it killed the process w/o restarting. Also on a Mac.

    ReplyDelete
  46. UnknownMay 7, 2013 at 8:27 AM

    @Joel Riggs and Mac OS friends: check the man page of your dd implementation. In Linux, "dd" is part of the GNU coreutils package. The man page in a recent Linux system says:

    Sending a USR1 signal to a running 'dd' process makes it print I/O sta‐
    tistics to standard error and then resume copying.

    $ dd if=/dev/zero of=/dev/null& pid=$!
    $ kill -USR1 $pid; sleep 1; kill $pid

    Other "dd" imprementation may not handle an USR1 signal in which case the process will terminate.

    Everything is in the manuals (TM).

    ReplyDelete
  47. AnonymousJune 12, 2013 at 2:24 AM

    Cool stuff. Thank you.

    ReplyDelete
  48. Gustavo DiazJune 17, 2013 at 8:32 AM

    dd if=/dev/sda1 bs=1M |pv -brtp -s 500M| gzip -1 > /home/boot-backup.gz

    ReplyDelete
  49. Bing RenJuly 8, 2013 at 9:30 PM

    Cool, and thank you!

    ReplyDelete
  50. AnonymousOctober 13, 2013 at 11:15 AM

    Thanks for the info!

    I'm in the process of dd a 300GB disk. With help of
    $ pgrep -l '^dd$'
    8789 dd
    $
    and

    $ kill -USR1 8789

    I find I have 10 hours still to wait to complete dd from the 5 hours so far.

    I am hindered by the fact I am booting from USB HDD and copying files to another HDD from laptop HDD.
    @ 5.9MB/s

    As a result I can now kick back and get on with something else rather than staring at what appeared to be a Terminal window doing nothing :)

    ReplyDelete
  51. AnonymousOctober 13, 2013 at 11:22 AM

    I forgot to add.
    I had to use sudo kill -USR1 8789

    ReplyDelete
  52. AnonymousOctober 15, 2013 at 6:38 PM

    Try out **pv** or pipeviewer :D

    ReplyDelete
  53. xdpirateNovember 19, 2013 at 4:09 AM

    You could just do

    watch -n 1 sudo kill -USR1 `pidof dd`

    ReplyDelete
  54. AnonymousNovember 22, 2013 at 10:33 AM

    Dooooood. Thats awesome. THANKS!

    ReplyDelete
  55. AnonymousJanuary 17, 2014 at 3:04 PM

    Excellent info - really useful - I'm trying low level format a flash drive, and this really helps to tell me how far its got.

    ReplyDelete
  56. AnonymousFebruary 9, 2014 at 3:29 PM

    err, can't you simply press Ctrl+T in the terminal running dd?
    That works on FreeBSD and other BSDs.

    ReplyDelete
  57. flitjesFebruary 10, 2014 at 12:55 AM

    you could also use:
    watch -n 10 kill -USR1 `pidof dd`

    ReplyDelete
  58. furriephillipsApril 21, 2014 at 6:13 AM

    If your device has no activity lights, you've no idea that
    anything's even happening with "dd" :(

    The progress info that "dcfldd" shows is just what you need.

    I've now got this in my .zshrc: -

    ## Conditional Aliases
    ## dd -> dcfldd
    if type dcfldd 2>&1 >/dev/null ; then
    alias dd='dcfldd'
    fi

    Thanks!

    ReplyDelete
  59. AnonymousJune 27, 2014 at 3:34 AM

    On OpenSUSE this seems to work only partially. When I was reading from a HDD to file it worked like a charm but when I was writing it back to another HDD it killed the process. Very strange.

    ReplyDelete
  60. AnonymousJuly 17, 2014 at 9:36 AM

    thank u!

    ReplyDelete
  61. AnonymousOctober 23, 2014 at 7:23 AM

    Thank you.
    Clearest description and works.

    I had "bash" errors so had to add "sudo" to the front.

    ReplyDelete
  62. AnonymousDecember 5, 2014 at 12:40 PM

    Very useful. Thank you for this post!

    ReplyDelete
  63. AnonymousDecember 28, 2014 at 9:54 AM

    Thank you very much friend, that really works :-)

    ReplyDelete
  64. Miguel OrtizMay 26, 2015 at 7:23 AM

    Thank you man, this is very usefull!!! :) -USR1 and watch will become my new friends.

    Hope you check my project, it has information about technology and virtual education in Argentina. hugs!!

    ReplyDelete
  65. AnonymousJuly 12, 2015 at 10:09 AM

    I don't know if you still maintain this page and if you know that this is page #1 when your searching "dd progress".

    So maybe you could enhance your post that your post is more useful than the comments below. ;)

    I would recommend this e.g. command:

    while true; do sudo kill -USR1 $(pgrep '^dd$'); sleep 5m; done

    where sudo is just needed if you e.g. dd-ing from /dev/sda and the loop just if you want it to repeat by itself. Sure it's also possible with watch... but at least

    kill -USR1 $(pgrep '^dd$')

    is a really useful combination. This maybe will have some issues when using multiple dd's, but whatever. :) Guess that's easy to fix then on the fly.

    ReplyDelete
  66. AnonymousDecember 12, 2015 at 9:25 AM

    Just press CTRL+t to see the interim statistics

    ReplyDelete
  67. Rapid ElectronicDecember 30, 2015 at 8:47 AM

    New version of coreutils (8.24) adding a status progress to dd tool:

    Usage on Xubuntu 15.10:

    Open terminal shell and type these commands:

    wget ftp://ftp.gnu.org/pub/gnu/coreutils/coreutils-8.24.tar.xz
    tar -xf coreutils-8.24.tar.xz
    cd coreutils-8.24
    ./configure && make -j $(nproc)

    Run dd as root:

    sudo ./dd if=/dev/sdc of=/dev/sda conv=noerror status=progress

    You will see: Bytes, Seconds and Velocity (Bytes/seconds)

    To check versions of dd:

    Native:

    dd --version

    New (cd coreutils-8.24/src):

    ./dd --version

    ReplyDelete
  68. Peter LeungJanuary 9, 2016 at 4:31 PM

    Thanks, Rapid Electronic. I've updated this article with the new information you provided.

    ReplyDelete
  69. AnonymousFebruary 14, 2016 at 3:17 AM

    Thanks, I don't use linux much often and everytime I do I wonder how to check dd status :-)
    Anyway, you should mention that you're talking about GNU version of dd, which is pure linux thing. BSD and other unices (including OS X, which is based on Darwin, which is Apple derivate of FreeBSD) use another version of dd and kill and everything. So the INFO kill signal and Ctrl+T keypress are lethal for linux/GNU version of dd!


    Little flame about Ubuntu:
    Ubuntu definitely is linux, but.. It's linux for (dumb) masses. Folks who know linux rarely install it on their machines. Ordinary linux guy uses Debian, Slackware, RedHat or OpenSUSe. Some maniacs use Arch. I'd like to watch Ubuntu user installing Arch linux or set up devices on any other distro :-)

    ReplyDelete
  70. AnonymousMarch 10, 2017 at 9:13 AM

    Shouldn't it be $ sudo dd if=/dev/sda of=/dev/sdb status=progress instead of $ sudo df if=/dev/sda of=/dev/sdb status=progress ?

    ReplyDelete
  71. AnonymousMay 2, 2017 at 1:25 AM

    You can add watch to the kill command to repeat the command.

    For exemple every 60 seconds :

    watch -n 60 kill -USR1 2334

    ReplyDelete