Hacker News Reader

Comment by Svetlitski | Hacker News Reader

The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.

On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).

View on HN

This works out assuming that file copying is CPU bound. Traditionally this hasn't been the case but maybe things have changed with widespread fast SSDs and usage of disk crypto.

Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ)

(Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)

> Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput

I would assume that is the case here too. The threads are likely hiding the latency of the IOPS. I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not.

A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)

Ok, so the core count is not the right level of parallelism (except by accident).

> The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.

CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.

Does this software gain any speed by sacrificing any safety? As in, does it switch off fsync in any way? Are the files guaranteed durable by the time the software terminates?

Is it available on homebrew?

No, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1].

[1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...

Reply on news.ycombinator.com