Hacker News Reader

I've found when there are many files but overall size being small i.e. many many small files, archiving, copy and unpacking works faster. Perhaps due to enumerating, analysing overall size and then copying vs archiving directly.

Would be happy to learn the real reason.

View on HN

cp is single-threaded and only blocks on one IOP at a time.

With a tarpipe, you can block on two IOPs at a time, and they're decoupled by the pipe buffer.

This primarily makes a difference because the kernel cannot issue the I/O for small files ahead of time, like it does when you sequentially read a large file, so you do actually end up blocking and waiting.

This is the strategy I always use with that use-case. Especially with "assets" folders where thousands of little icons and drawables and the like.

Reply on news.ycombinator.com