Hacker News Reader

Comment by Svetlitski | Hacker News Reader

Inspired by many of the amazing new-wave Unix tools that have come about over the past several years (fzf, ripgrep, fd, lsd, etc.), I set out to create a more modern and performant version of the classic Unix cp command. Written in Rust due to the excellent ergonomics and performance characteristics of the language, fcp copies large files and directories in a fraction of the time that cp does.

Feel free to ask me any questions about the project!

View on HN

My only gripe with those new unix tools is that many of them don't come with a man page, even though they have essentially the same information elsewhere

So, perhaps, take a look at https://github.com/rust-cli/man or something

IMHO, just convert README.md into the man page using a md2man converter, and be done with it.

This assumes README.md lists all the command-line options like in a manpage, which is not often the case. Also, manpages follow a conventional structure. You can expect the content in a manpage. Most READMEs are free-style. You don't know what is there. md2man doesn't solve the problem.

Ok. Then just fire browser, then google project page, and then read free-style README.md at it original location. It's just a dozen clicks away.

You assume that I have a browser that starts up quickly, or that it's not already loaded with a hundred other tabs on a good day.

Well tldr (or tealdeer) or whatever is a nice example of another modern (semi-)alternative to man. tldr usually just gives a handful of explained example commands rather than an actual manual, but it's often enough information to solve the problem. I bet these some of these tools have pages on that at least.

(I agree that a real manual is necessary too. Maybe you could convert one from the online docs?)

Not a question, but I made a similar tool in c++[1], for Linux only using io_uring, and a blog post explaining its internals [2]. I'll definitely have a look some time soon, I'd be interested to see how performance compares (I gathered from some other comments here that you're using blocking io in threads?)

1: https://github.com/wheybags/wcp

2: https://wheybags.com/blog/wcp.html

Sounds super interesting, I'll definitely check out the blog post. Yes, fcp is indeed using blocking io in threads. In an earlier prototype I was using async (i.e. M on N threading), but was running into issues with exhausting file descriptors. Because of this (and because writing async Rust is quite difficult) I settled on using blocking io in threads. Perhaps in a future version I'll try to implement the io_uring approach for Linux.

Ah, I also tried unlocking the file descriptor 1024 limit, which worked fine most of the time, but didn't actually give me a perf boost in any of my tests. I got rid of it in the end because it caused problems when copying from an smb mount. iirc I exhausted the file descriptor limit on the samba server.

_Why_ is it faster? (particularly on Linux, I see you're using syscalls on mac that mac's cp doesn't, since apple doesn't really maintain their unix userspace)

The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.

On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).

Is it available on homebrew?

No, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1].

[1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...

This works out assuming that file copying is CPU bound. Traditionally this hasn't been the case but maybe things have changed with widespread fast SSDs and usage of disk crypto.

Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ)

(Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)

> Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput

I would assume that is the case here too. The threads are likely hiding the latency of the IOPS. I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not.

A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)

Ok, so the core count is not the right level of parallelism (except by accident).

Does this software gain any speed by sacrificing any safety? As in, does it switch off fsync in any way? Are the files guaranteed durable by the time the software terminates?

> The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.

CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.

Does GNU cp (installed from brew) use those syscalls?

What's the performance like with large numbers of small files? I find that's typically much slower than copying a few big files.

Thanks for this. I'll be testing it on Apple Silicon. Here's the build: https://workupload.com/file/kvwMb2DGSrS

Reply on news.ycombinator.com