Hacker News Reader

Comment by ktm5j | Hacker News Reader

I like it! But for me ssh already provides a simple and secure way to move files from one place or another..

  tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)"

Given that ssh it so ubiquitous I think this will always be my go to.

View on HN

I use netcat a lot with my friends, we don't even have to do the SSH dance.

On the receiving end:

    nc -vll 0.0.0.0 12345 | pv | tar xv

On the sending end:

    tar cv files... | pv | nc -N "destination IP" 12345

pv is a nice tool that just reports on the rate and number of bytes moving through the pipe, optional if you don't have it. Throw in gpg --symmetric + gpg --decrypt for good measure if you want to encrypt it on the wire with a password.

That works assuming the client has a public IP address or port forwarding set up, or you're on a local network together.

That throws your files over the wire with no encryption, signing, or integrity verification. It might well be fine for your use, but I don't think I'd ever be comfortable with it.

Did you miss the part of my comment where I explained how to add encryption?

...yep, I only read the command itself and skimmed right over that. My bad.

What about good old scp?

    scp file.txt user@host:/dest/dir/

Or rsync?

rsync -zvh file.txt user@host:/dest/dir/

So many ways to do this :)

rsync has always been my favourite because it makes the most sense to me (and the --help/man page is easy to read).

rsync -n -avh --progress source destination:~/asdf/ for a dry run followed by ctrl-p, ctrl-a, alt-f, alt-d, alt-d to remove the -n flag and then execute that for the real thing.

Occasionally though, I'll also use sftp if I'm just pulling one thing - perhaps even after sshing to the remote machine.

For all of these, SSH keys should be set up (and desktop logins secured) to make life easier.

As for Android, adb push and adb pull -a seems to work better than mtp:// or AirDroid in my experience.

After all these years, I still can't keep straight when I need a trailing slash in rsync, and when I need to not have it.

If you think of it in terms of archives and whether you want to "extract" into the current directory, or a new directory within the current one; that might help.

rsync source destination will plonk the entire source directory and put it inside destination as a neat bundle.

rsync source/ destination will take the contents of source (but not the directory source itself) and plonk it in destination

I found the info page a little dry but it does describe it succintly:

    rsync -av /src/foo /dest
    rsync -av /src/foo/ /dest/foo

For some reason though, my head freaks out when it sees "foo" and "bar", but all they're saying is that it does the same thing.

If in doubt though, just chuck everything into the destination ~/temp/ or ~/asdf/ and sort it out later.

To be honest though, most of the time I just use fish shell's autosuggestions to guide me along.

I usually go straight to tar & ssh over scp since you can do directories where you can't with scp. But yeah, if you just have some files scp works.

You can obviously send directories with scp, just use -r. Or am I missing something?

Recently used it to copy half a terabyte of stuff on my home network. Unsure about the exact specification but it supported the same flags as cp as far as I could tell.

Yes -r works for copying directories. I do it all the time

Well I'll be damned.. not sure what lead me to think this.

scp performs very slowly with directories containing many small files; "tar|ssh tar" does fine with that.

What if you don't have network and login access to the other person's computer?

> tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)"

You should use '&&' instead of ';' on the host side. That way you don't accidentally dump your transfer contents into a wrong directory if the existing host dir doesn't exist.

e.g.: tar cf - stuff | ssh host "(cd /dest/dir && tar xf -)"

Or just skip the shell entirely:

  tar -C /dest/dir -xf -

Yeah that's the right way to do it, but I always forget which flag it is for tar and this is faster than pulling up the manpages ;)

Could you give a full example of how this would work? Receiver and sender..

    tar cf - src/ | ssh $host "tar -C /dest/dir -xf -"

Do you have any idea about a crazy fast way to transfer files between two machines on the internal network?

No encryption necessary No integrity checks No resume support Just plain and as fast it could get..

I need to try the netcat version but I'm hoping someone could show me a concurrent version of it that is mad fast.

You can use nc (netcat). If you have a 10G ethernet you will not saturate it because you will be limited by disk IO (I get 1Gb/s disk reads for a M.2 SSD). I you can read from disk faster, netcat alone will not saturate a 10G link, probably will hit a 3Gb/s limit (depending on your hw). You will need parallel transfers (xargs, a bit of scripting etc). You can also try rsync, maybe it is good enough for you.

From your source machine:

  < /path/to/source ncat remote-host 8001

On your destination machine:

  ncat -l 8001 > /path/to/dest

Though in most situations with files of reasonable size, you're probably going to be better off running through `gzip -c`.

Good point!

This particular tool seems useful for sharing files with people who might not be comfortable with the command-line, or where you don't have an account on the receiver's computer, since it produces a regular http link to download the data.

That makes sense, I hadn't thought about that.

> ssh already provides a simple way

:-D

> tar -cf - ./files.txt ./orDir/ | ssh host "(cd /dest/dir; tar -xf -)"

:-O

reminds me of this gem https://news.ycombinator.com/item?id=9224

2007. Wow. That's a lot in Internet time.

It is a lot in computer time as well. The other gem in that thread IMHO is how they marketed Dropbox as a replacement for USB thumb drives (which it also is, but it just goes to show how niche the concept of cloud was 11 years ago).

Yeah, and that guy is actually still active on HN and is always decent whenever someone calls him out about his ancient Dropbox comment.

If you can trust to have gnu tar on both side, you can reduce typing. I usually do: tar c files.txt orDir | ssh -C host tar x -C /dest/dir

What about just typing rsync -havz --progress source destination?

-h for human-readable numbers -a for archive mode -v and --progress for verbose info -z for compression during transfer

Add a -n for a dry run if required.

https://linux.die.net/man/1/rsync

One of the people who came up with it - Andrew Tridgell - was more or less "responsible" for Linux and BitKeeper parting ways, which in turn ultimately lead to the creation of Git. I think it's a fascinating story. Excellent tools.

The progress / speed display is the main thing that keeps me coming back to rsync even though other tools might manage the same job - not seeing the progress of a copy is what had me searching for that solution in the very first place.

Yo dawg, I heard you like progress, so check out rsync's

--info=progress2

You can use it since rsync version 3.1.0.

Hah I've seen that in the man a few times but always feel too lazy to call it up. Next time I'll give it another go!

>Ha ha, I remember writing a command like that (made it into a script for repeated use, with args) when one of our HP-UX servers in the company where I worked then, had a DAT drive failure. The script allowed us to take backups of the source code and data on one such box from another box (connected on the same network), until the bad drive was replaced a few days later.

Didn't know about nc or rysnc at the time. Good to see those other solutions, and good that there are many ways to do it, with different pros and cons.

Reminds me of how they left recursions out of the 'cp' command on plan9, so the recommended way to copy directories is:

@{cd fromdir && tar cp .} | @{cd todir && tar xT}

I have a lot of friends who don't know what the fuck SSH is. For them, these services are great at sending them files.

Reply on news.ycombinator.com