Show HN: Linux tool to show progress for cp, rm, dd, etc. | Hacker News Reader

Show HN: Linux tool to show progress for cp, rm, dd, etc.

github.com

Slightly related: what many people don't know is that on OS X and BSDs, commands often react to SIGINFO (Ctrl-t), giving progress information. E.g. from the cp man page:

     If cp receives a SIGINFO (see the status argument for stty(1)) signal,
     the current input and output file and the percentage complete will be
     written to the standard output.

dd has a similar feature, but with SIGUSR (kill -USR1), and it works on Linux too.

How do you fire that signal from the console ?

By using

    kill -s <Signal> <Process ID>

  pkill -USR1 dd

You'll need to use ps to get the pid if you've got multiple dd instances running, and you'll of course need to run that as root if your dd instance is running as root.

Personally, I think the experience of typing ctrl+t is better than using pkill. Just 2 keyboard taps and you're done.

Very cool. Is there a reason why Linux tools don't react to SIGINFO, other than just "not implemented"?

As far as I'm aware, Linux tools do react to various signals, just not SIGINFO because it comes from BSD and Linux just doesn't use that particular signal.

You can check dd progress by sending USR1, for example.

I just wish there was a control sequence keystroke like Ctrl+I to get programs to do that.

For a verbose or simple progress `cp` equivalent I tend to use RSync. Both RedHat/CentOS/etc and Debian/Ubuntu all come with RSync as standard. The only caveat is that I find RSync to be slightly slower than coreutils cp; however you do gain all the advantages of RSync if you then need to sync files across the network for example.

> rsync --progress <source> <destination>

> rsync --progress /home/me/music/*.mp3 /mnt/shared/music

It is slightly slover, but one huge benifit is, that you get -P or --partial, which often ends up saving me lots of time.

I also like -avh and --stats for rsync, I use rsync many times a day for transferring files and folders across networks. It's brilliant.

I've found Midnight Commander (https://www.midnight-commander.org/) good if you just need the progress bar. In fact, you get not just one, but TWO - one for the individual file, and one for the transfer as a whole. Perhaps I'm easily pleased, but I find this useful.

(It doesn't do all the rsync fanciness, but if you don't need that and/or you've got some unwieldy subset of files to transfer - a situation the GUI-style approach makes light work of - then it comes in handy.)

So it's a centralized `pv` ?

http://www.catonmat.net/blog/unix-utilities-pipe-viewer/

Yeah .. I think I prefer pv for all my cp progressbar needs .. keeps things light and simple, and is already onboard.

I like pv, but I thought that it didn't work with "cp -r" (let alone mv)… Am I wrong?

It would be nice for cp and mv to just have built-in progress bars (only active in interactive mode, etc).

About 10 years back I tried to do this by defining terminal escape codes for this. I was also implementing a new terminal as a POC. Unfortunately didn't find the time to go beyond it.

Here's the POC if anyone would like to take it further: http://anthias.sourceforge.net/

pv is by no means a standard include. Also the obvious advantage here is being able to monitor progress for processes you didn't pipe to pv initially (e.g., forgot or didn't think it would be a long transfer).

pv is just cat with a progress bar, so it only handles the case where you are copying a single big file.

Indeed. You can use it to show the progress of a recursive directory copy with this trick:

tar c -C /tmp/source . | pv | tar x -C /tmp/dest

you can also pass an approximated size of your source to pv, however the tar output will be slightly bigger.

how would you estimate the size quickly? 'du' can take quite long

You can start pv without an estimated size, run du in the background, and then supply the estimate to pv once it's calculated using the '-R' (remote control) option.

A bit clumsy, but it shouldn't be hard to write a little script to do it.

Cool, didn't know about that pv had a remote control.

Anyway, if cp -R had a progress bar it would behave the same way, i.e. it would have first to recursively stat the source the same way as du does, and only then it could start reporting a completion percentage.

Using /proc is pretty damn clever! I thought it would be some sort of wrapper for strace, but your solution is so much simpler.

Here's a single command which should do the same. It supports cp and dd, but anything else that supports SIGINFO could be added in.

    ps -A | grep -P 'cp|dd' | sed -r 's/^\s*([0-9]+).+$/\1/' | xargs kill -s SIGINFO

This assumes the command supports SIGINFO, but that's a BSD thing. Linux commands don't support that, so this is not a replacement.

And did you really just reinvent pkill?

Off topic, hoe do I get a PS1 like those screenshots?

Looks just like vim's powerline/airline status bars. IIRC it takes a bit of font wrangling to get those chevron colors working right.

https://github.com/Lokaltog/powerline

https://github.com/milkbikis/powerline-shell

dd has progress built in, under Linux it is kill -USR1 <pid> to get a progress report, the signal varies by OS. cp has a -v option, if tahts not enough rsync has a --progress option too that gives within file progress.

`cv` is a coreutils viewer, rsync is not included in coreutils.

cv is not included in coreutils either, so I still have to install something.

And dcfldd (not a coreutils app you have to install it separately) gives you a progress report, as well as using all the usual dd syntax.

To expand the parent comment, in Terminal (OS X), Ctrl-T prints the progress of dd and cp so you needn't look up their PIDs

If anybody is using dd to rescue a failing drive, I can very much recommend using the spectactular ddrescue instead.

I had a 2TB drive turn up five figure error rates (Raw Read Error rate ended up at 90k - when I started, it was already no longer possible to mount it, but it did show up in /dev) and thought I had lost an entire year of family pictures (mea culpa on not having a backup, of course). It took four months, 24/7 of meticulous work, but it ended up giving me a near perfect copy to an identical drive (save for ~20kb of truly dead sectors).

Amongst many other features, it has a logfile where it keeps track of the overall process - so that it can be killed or paused and restarted later.

Very nice! I should package this.

Anyone know a similar tool available to OSX or if its feasible to port or recreate this for MAC?

I'm not too familiar with what file libs or utils can be used to handle this as the author says this relies on a linux specific header file and tool.

See https://github.com/Xfennec/cv/issues/6

Just hit ctrl-t to get status on OS X.

ctrl+t doesn't show percent complete

Yes it does

  aeg@bigwibble ~/ % cp bigfile /Volumes/data_sd1
  load: 1.60  cmd: cp 17686 uninterruptible 0.00u 0.09s
  bigfile -> /Volumes/data_sd1/bigfile   1%

Here's me hitting it 4 times:

    [/tmp]$ cp bigfile bigfile2
    load: 1.78  cmd: gcp 93420 running 0.00u 0.04s
    load: 1.78  cmd: gcp 93420 running 0.00u 0.10s
    load: 1.80  cmd: gcp 93420 uninterruptible 0.00u 0.16s
    load: 1.80  cmd: gcp 93420 uninterruptible 0.00u 0.26s
    [/tmp]$

What is gcp? It sounds like you have some alias setup for cp and that's interfering.

Try: /bin/cp bigfile bigfile2

Entirely feasible, as far as I know; OS X lets you discover processes and access their devices. OS X, being BSD-based, doesn't have a /proc file system; instead, you have to use sysctl() to get at the data.

I really hope this stays on the HN front page for awhile because what a great utility.

I've been using patched copies of coreutils to do the same thing, this is much better.

By the way, if you ever need to quickly spy on rsync you can try this:

       watch lsof -ad3-999 -c rsync

but there is a chance cv will work with rsync too and be more helpful

In the old days I've done this using lsof -p PID and looking at where the files are in relationship to their size. If you have a long running process lsof can be a wonderful tool to figure out how it's doing.

My (joke) version:

https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-prog...

Wow, that's such a clever idea! I'm impressed at the simplicity.

Sweet ! It's even in the AUR repositories !

Very nice idea!

Ehm, etc is a directory, not a program?

(Sorry, really bad joke...)

A completely unrelated question: in the terminal screenshots, what is being used for PS1? I love that display of the CWD and git repo on the command line.

It's called Powerline, and it has bindings and configs for nearly every shell, editor, and everything in between including tmux.

http://powerline.readthedocs.org/en/latest/overview.html

Reply on news.ycombinator.com