GCP - cp with a progress bar | Hacker News Reader

GCP - cp with a progress bar

hecticgeek.com

40 pointsdannyrosen37 comments

Nifty, lightweight copy tool. Completely replace cp with: alias cp="/usr/bin/gcp"

For single files or monitoring other pipe based operations (like network transfers) pv [1] is also a useful tool.

1 - http://www.ivarch.com/programs/pv.shtml

Using pv:

  gcp() {
    cat "$1" | pv | cat > "$2"
  }

useless use of cat

    pv < "$1" > "$2"

Even if the cat at the beginning of a pipe isn't strictly necessary, it still helps readability to have everything flowing from left to right.

I also advocate this "useless" use of cat, but you can write the redirection before the command:

    <"$1" pv >"$2"

The bigger problem is that using cat destroys the file size information.

pv reads content from a file by default, so that can actually be reduced further to:

    pv "$1" > "$2"

Which will actually give better results this way, since it'll read the size of the file and use that to provide an estimate even.

I find pv useful when moving MySQL databases between servers.

mysqldump -u user dbname | pv | mysql -u user -h some_remote server dbname

to copy a tree of files:

  tar c SOURCE_DIR | pv | tar xv -C DEST_DIR

  # on source computer
  tar c SOURCE_DIR | pv > /dev/tcp/192.168.1.2/4567

  # on destination computer computer
  nc -l -p 4567 | pv | tar xv -C DEST_DIR

For anyone curious as I was why the /dev/tcp/... redirection works, the Bash manual contains the following:

    Bash  handles  several filenames specially when they are used in
    redirections, as described in the following table:

      /dev/fd/fd
         If fd is a valid integer, file  descriptor  fd  is
         duplicated.
      /dev/stdin
         File descriptor 0 is duplicated.
      /dev/stdout
         File descriptor 1 is duplicated.
      /dev/stderr
         File descriptor 2 is duplicated.
      /dev/tcp/host/port
         If  host  is a valid hostname or Internet address,
         and port is an  integer  port  number  or  service
         name,  bash  attempts  to open a TCP connection to
         the corresponding socket.
      /dev/udp/host/port
         If host is a valid hostname or  Internet  address,
         and  port  is  an  integer  port number or service
         name, bash attempts to open a  UDP  connection  to
         the corresponding socket.

pv is a great tool and one of the first things I install on new machines. I hate having no idea how long an operation is going to take. For example, loading a SQL dump

pv foo.sql | mysql

The 'g' prefix is a bad idea. On BSD systems that prefix is used to refer to binaries in the GNU toolset.

I'd recommend "pcp," for "progress copy"...I'd use it for the name alone!

Why? rsync does all of this and a whole lot more.

He didn't want all the complication of rsync, he just wanted something simple with a progress bar. Rsync isn't really that simple. Sure, he could of wrote a shell script on top of rsync, but he probably learned a lot more from this.

For local copies, rsync with a progress meter is as simple as

  rsync --progress <source> <destination>

I usually use it like this for local copy:

    rsync -rvh --progress <source> <destination>

Oh-my-zsh has a plugin 'cp' which is basically an alias: alias cpv="rsync -poghb --backup-dir=/tmp/rsync -e /dev/null --progress --"

When I read the installation example, I see that you also need the python-progressbar library, which means it depends on Python. So I would say rsync is simpler.

I was thinking (and didn't convey in writing) more on the use side, not the install side. It just bugs me that everything someone gets some joy from trying has a first comment of "why?".

The "why" generally comes into play when you project it as a solution to an unsolved problem(or at least a problem which can't be easily solved). If you leave it as "because it's fun", I am fine with it. If you present it as "now you can have progress bars with cp", I will ask "why? I can already do that with rsync". That's not because I want to put down your work; that's just thinking out loud.

And therein lies the sickness that is affects HN sometimes. For a bunch of hackers there are some fairly closed off minds. I did think rsync could do this as well, but there are other ways to skin a cat, perhaps not all in the most efficient ways, but what's wrong with exploring doing it this way? As developers and hackers, aren't we supposed to explore and try stuff out? FWIW, I learned some new python odds and sods out of this, even if I don't use this.

> And therein lies the sickness that is affects HN sometimes.

Sickness? Really? Did you come up looking for a question for which you already had an answer, and finding none, just put words in my mouth, and responded to an argument I didn't make.

What part of "just for fun" is fine with me was difficult to comprehend?

I have an honest question then, "What benefit does the why post have?". I can see it if you're trying to warn someone of something, but otherwise it just comes off wrong.

If you are going for a teaching moment, then saying "I wanted the same thing and I chose to solve it with rsync with the following alias / script" is better than telling someone they are just a waste of time.

Make what you will of it. I am more likely to phrase it as "Why? rsync works just fine" than ""I wanted the same thing and I chose to solve it with rsync with the following alias / script". FWIW, as far as being educational goes, your phrasing says the same thing with padding.

Also, "how" matters only when I have "why" answered. If it isn't mentioned on the project page, or I skim too fast to grasp it, "why" would be the first thing on my mind.

This one is also pretty clever (based on catching the write() syscall with strace) : https://chris-lamb.co.uk/posts/can-you-get-cp-to-give-a-prog...

Thanks.

I think someone made a patch which works with the original gnu coreutils

Oh here it is http://beatex.org/web/advancedcopy.html

This seems relevant:

http://www.commandlinefu.com/commands/view/4553/copy-a-file-...

dd does not show a progress bar, unfortunately but it does report its progress.

Shouldn't cp have "-h" (human readable) flag that does exactly that?

UNIX Style, or cat -v Considered Harmful: http://harmful.cat-v.org/cat-v/

That's not an argument. There's no reason why cp shouldn't report progress if you want to. Not one.

SCP works locally too and has a progress bar.

scp just falls back to cp when copying locally, so doesn't show any progress bar.

If you use oh-my-zsh, gcp is binded to "git cherry-pick" (with auto-completion etc.), so I did:

    sudo ln -s /usr/bin/gcp /usr/local/bin/cp_

A good alternative to cp.

Reply on news.ycombinator.com