Hacker News Reader

Comment by gerdesj | Hacker News Reader

It handles as many files as I've ever thrown at it - often in the millions. The great thing about rsync is it is restart-able and the source and destination don't even have to be local.

View on HN

the biggest weakness of rsync is that it's single-threaded. Single-thread-copy with millions of small files is painfully slow, Rsync is as good as it can be but you just need more threads.

I think that's what a sibling is getting into with the "better to tar files and send them over ssh in some cases" thing. And yes, you can hack it in after-the-fact with xargs/etc but it's clunky compared to just having native multithreading like rclone/etc.

If you use GNU Parallel, there's a fairly easy way to parallelize [1].

That being said, I hadn't heard of rclone - thanks for mentioning it, it looks amazing. I'll definitely be trying this out for my use cases...

[1] http://www.gnu.org/software/parallel/man.html#example-parall...

Wow, Parallel also looks awesome. Never seen it before.

It handles millions, but it can be a lot faster to just pipe output from tar through the ssh connection.

Got any benchmarks/write ups on the subject? I did a bit of testing myself a long time ago and basically the answer just ended up being to use rsync because any differences were marginal. That said, I didn't test with millions of files.

I think perhaps this was a bigger issue back in the day when we were using rotating harddisks. In those day doing a seek would be a lot slower than doing a write.

Today seeks are mostly instants, so maybe my experience isn't valid anymore.

Your experience is valid today but for a different reason: if you're comparing millions of tiny files there's a lot of back and forth. If you're streaming a single archive, it only checks if that single file has been modified.

Like everyone here I've no benchmarks but have got burned trying to rsync around too many small files.

I got into the « lot of small files » case, and I ended up generating a file list, split it and feed multiple rsync instances with xargs.

We've put rsync in a HPC scheduler and used it, or some tooling ontop of it really, to copy billions of files for a large-ish HPC compute cluster with many P of data.

That's my experience - millions or billions of files - who cares?

rsync is time served. It just works. I'm sure there are other funky solutions but they are not proven over decades.

we have had problems with millions of rsunk files.

Reply on news.ycombinator.com