Show HN: fcp – A significantly faster alternative to cp(1), written in Rust | Hacker News Reader

Show HN: fcp – A significantly faster alternative to cp(1), written in Rust

github.com

88 pointsSvetlitski86 comments

Inspired by many of the amazing new-wave Unix tools that have come about over the past several years (fzf, ripgrep, fd, lsd, etc.), I set out to create a more modern and performant version of the classic Unix cp command. Written in Rust due to the excellent ergonomics and performance characteristics of the language, fcp copies large files and directories in a fraction of the time that cp does.

Feel free to ask me any questions about the project!

_Why_ is it faster? (particularly on Linux, I see you're using syscalls on mac that mac's cp doesn't, since apple doesn't really maintain their unix userspace)

The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.

On macOS, this is augmented by regular files being copied using the OS's fclonefileat and fcopyfile mechanisms under the hood, which allows even very large files to be copied near-instantly (when compared to copying them block-by-block as classic cp does).

Is it available on homebrew?

No, it is not. I would've submitted a Homebrew formula for it, but that would go directly against Homebrew's guidelines, which frown upon authors submitting formulas for their own tools [1].

[1] https://docs.brew.sh/Acceptable-Formulae#niche-or-self-submi...

This works out assuming that file copying is CPU bound. Traditionally this hasn't been the case but maybe things have changed with widespread fast SSDs and usage of disk crypto.

Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput. (They might end up hitting different disks in RAID, or taking advantage SSD internal parallelism, or whatever tricks spinning disks use to get more throughput with TCQ)

(Re your MacOS remark, Linux has support for copy-on-write file copies as well, see reflink and FICLONE / copy_file_range)

> Traditionally the reason to use threads in small-requests i/o is to get multiple outstanding requests in flight in the hardware io stack which some hardware can use to get you more IOPS throughput

I would assume that is the case here too. The threads are likely hiding the latency of the IOPS. I can't see any CPU load statistics, so it is hard to say, if it is indeed CPU bound or not.

A quick run with time shows user+sys being only slightly higher than real, so with 6 threads for my 6 cpus, it uses a single cpu. (and practically no time in user-space)

Ok, so the core count is not the right level of parallelism (except by accident).

Does this software gain any speed by sacrificing any safety? As in, does it switch off fsync in any way? Are the files guaranteed durable by the time the software terminates?

> The primary reason is that it uses all of your computer's available cores to walk through directories and copy files, effectively speeding things up by a factor directly proportional to how many cores you have.

CPU cores can't read/write files, those are IO operations by the storage device. The CPU could have 64 cores but on a system with a single hard disk that can do just one concurrent operation.

Does GNU cp (installed from brew) use those syscalls?

My only gripe with those new unix tools is that many of them don't come with a man page, even though they have essentially the same information elsewhere

So, perhaps, take a look at https://github.com/rust-cli/man or something

IMHO, just convert README.md into the man page using a md2man converter, and be done with it.

This assumes README.md lists all the command-line options like in a manpage, which is not often the case. Also, manpages follow a conventional structure. You can expect the content in a manpage. Most READMEs are free-style. You don't know what is there. md2man doesn't solve the problem.

Ok. Then just fire browser, then google project page, and then read free-style README.md at it original location. It's just a dozen clicks away.

You assume that I have a browser that starts up quickly, or that it's not already loaded with a hundred other tabs on a good day.

Well tldr (or tealdeer) or whatever is a nice example of another modern (semi-)alternative to man. tldr usually just gives a handful of explained example commands rather than an actual manual, but it's often enough information to solve the problem. I bet these some of these tools have pages on that at least.

(I agree that a real manual is necessary too. Maybe you could convert one from the online docs?)

What's the performance like with large numbers of small files? I find that's typically much slower than copying a few big files.

Thanks for this. I'll be testing it on Apple Silicon. Here's the build: https://workupload.com/file/kvwMb2DGSrS

Not a question, but I made a similar tool in c++[1], for Linux only using io_uring, and a blog post explaining its internals [2]. I'll definitely have a look some time soon, I'd be interested to see how performance compares (I gathered from some other comments here that you're using blocking io in threads?)

1: https://github.com/wheybags/wcp

2: https://wheybags.com/blog/wcp.html

Sounds super interesting, I'll definitely check out the blog post. Yes, fcp is indeed using blocking io in threads. In an earlier prototype I was using async (i.e. M on N threading), but was running into issues with exhausting file descriptors. Because of this (and because writing async Rust is quite difficult) I settled on using blocking io in threads. Perhaps in a future version I'll try to implement the io_uring approach for Linux.

Ah, I also tried unlocking the file descriptor 1024 limit, which worked fine most of the time, but didn't actually give me a perf boost in any of my tests. I got rid of it in the end because it caused problems when copying from an smb mount. iirc I exhausted the file descriptor limit on the samba server.

For SSDs this is probably the right thing (maybe use uring tho?)

For hard drives this is the opposite of what you want. For hard drives, especially when copying large trees, you only want one thread to access the disk, and preferably sort its stat()s and open/read/close() by inode.

Also this tool seems to lack most options that cp(1) has.

> preferably sort its stat()s and open/read/close() by inode

A better approach would be to use information about location of data extents. Luckily there's already a crate that can help with that: https://github.com/the8472/platter-walk

That's better for reading the data, but doesn't work for ordering stat()s because looking up extent information requires reading the inode, but that's what we're trying to make linear-ish at that point.

you can optimize for both. obtain directory entries, sort by inode, do a batch stat (ideally via io_uring) and fiemap and then sort again by extent order.

And if you only need the `d_type` instead of the full stat struct, then you get that for free from getdents(2) on some filesystems, this is what platter_walk does.

options that cp(1) has

OK, this is something that's bugged me for a long time. I've been in the field forever, but working on Windows. So I've seen this kind of thing many times before, particularly when referencing man, and it seems completely cryptic.

The way linux people occasionally put a parenthetic number after a generic command seems to me one of the ways that normal civilians, and even pros like me, are intimidated into staying away from serious linux usage.

So what gives? What the heck is the parenthetic number for?

It refers to the manpage section. Good SO post: https://unix.stackexchange.com/questions/3586/what-do-the-nu...

It’s the man page section. cp(1) is documented in …/man1/…. You can type man 1 cp.

Roughly, 1 is unprivileged shell commands, 2 is syscalls, 3 is library functions, and 8 is administrative commands.

The number denotes what section of the manual the command (or function, etc.) comes from. Aside from that, it also helps clear up potential confusion about what you’re talking about—a good example is stat(1) (the command) and stat(2) (the system call). Without the number, you’d have to rely on context clues to infer which stat was the relevant one. https://unix.stackexchange.com/questions/3586/what-do-the-nu...

I can’t lie, if you spend most of your time in Windows, there’s probably diminished reasons to understand manpages, but for those that do any degree of meaningful work on a Linux or Unix (e.g. macOS) system, not knowing how to navigate man pages separates entire classes of developer professionalism to me.

It’s night and day working with people who know how to be approachable in development, but also realize you need to understand some basic flags for everyday tools and not turn to Google or Stack Overflow for what is directly in your manpages.

Admittedly those who cannot stay approachable concoct all sorts of shell hell for everyone, even those who are experienced.

Excellent question, by the way. I wish we’d get some more candid questions like these on HN when relevant to the thread, like this one. It’s a learning opportunity for all readers.

What's wrong with Stack over man pages? Most man pages don't you give advice or context, just the raw reference, but sometimes the best way to do something is not even the tool you originally looked up (say, scp vs rsync).

Stack combines the reference (which flags) with the expertise (of educated guesses by sometimes wiser users).

I would never trust a man page over a good Stack answer. (edit: as a web dev who uses *nix all day, but has no desire to become a full fledged sysadmin... Just give me the tool I need to get the job done without sacrificing performance or security. I don't need to know every possible gnu tool and flag and distro dependent options...)

You’re certainly better off reading both.

But I have a bigger issue with the fact that developers who use Stack Overflow are more often than not just going to copy and paste and not understand what they’re doing.

And that’s predominantly how the site is used.

Yeah, there are plenty of manpages that are verbose, but frankly a lot of them are easier to parse than reading hacked together Stack Overflow answers that don’t explain how the poster arrived at their answer.

They literally don’t show their work.

one of the things that surprised me about large cluster filesystems (although it's obvious in retrospect) is that the solution to scaling performance is many threads, not one, because the backend can handle many concurrent requests. In fact at one place I worked, the cluster filesystem wouldn't even take you seriously until you hit it with GB/s of requests.

Definitely valid points, as I mentioned in a comment below I'll be updating the README to mention this is geared towards systems with SSDs.

As for lacking most of cp(1)'s options, that's a deliberate choice. My goal with fcp is to make the most common use case of cp(1) (i.e. copy some files/directories with no options) fast, not to replace cp(1) completely.

It looks like this is using multiple threads to copy files. That's fine for NVMe which tends to achieve more throughput with higher queue depths. But it will degrade performance when copying large files on spinning rust.

My experiments confirms your analyze.

On a SATA hard drive using ext4, I copied a directory with a large hierarchy of file and 216MB overall. After 2 warm-ups, I launch this simple benchmark.

    time cp -r moodle X
    13.76s s (0.06s u + 2.44s k) 18% CPU

    time fcp-0.1.0-x86_64-unknown-linux-gnu moodle X
    46.12s s (0.15s u + 2.47s k) 3% CPU

I emptied linux cache before each run with `echo 1 > /proc/sys/vm/drop_caches`, but this does not remove the HDD buffering. I suspect that fcp would be worse in real life conditions (no warm-up).

This does not mean the fcp is bad or not performant. But it some cases, it can be 3x slower.

Since you've already started, can you do a benchmark of a pair of copies started at same time?

One copy operation is the little files and deep hierarchy. The other copy operation is just a big file. Both operations start at same time and use the same tool.

I'm curious what cp and fcp do to the cache as they operate.

What do you mean by "After 2 warm-ups"? I've seen this expression in the context of benchmarks before and it puzzles me.

Does it mean you are doing this operation enough times to trick the kernel to cache everything in RAM? Or you run it 3 times and chose the fastest timings?

I'm not sure about the specific context here, but benchmark warm-ups are often used to make sure that the program's run time is nothing but that, and doesn't include things like setting up an environment or compiling to a cache.

I think there are sometimes arguments to be made that, for the user, that is run time. When using benchmarks to compare programs like this, though, I think isolating the "actual" (i.e. useful) program execution is productive.

First run: fcp binary has to be read from disk, so it's slow(ish) to start.

Second run: fcp binary is cached by the OS in RAM so it starts up faster.

Third run: just to make sure :)

This is just one of the things of a type of cache that can be warmed up.

The irony is that spinning rust is everything but ;) Most platters are made of glass, or maybe aluminum.

I learned this the hard way when dismantling and abusing a dead HDD. Had to vacuum all the tiny shards from the floor.

The rust in this case is the iron oxide coating on the inert platters.

Soooo all my data is stored on magnetized rust?

Yep. See [0] for a demonstration of do-it-yourself audio tape but the same applies for hard drives.

https://youtu.be/g1JlUcFKm5o?t=537

Hah, this is amazing! I wish they taught that in schools. I don't know who they are but they're like a mix between Bill Nye, Carl Sagan, and the Mythbusters. Cool duo.

Great old video. Thanks for sharing. Terrifying goose though.

This was definitely an oversight on my part; the benchmarks I ran were all conducted on systems with SSDs (as noted in the README). Especially considering the testing done in some of the comments below this one, I'll update the README to mention that fcp is geared towards SSDs. As a developer tool though, I still think this is the right choice, as at least anecdotally the vast majority of developers I know have an SSD in their primary machine.

The only thing I'd like "new" in cp is an optional progress indicator, for large files

Btw, copying files is mostly bound by disk speed (or network if share), not so much by CPU, so it doesn't seem to be a valid reason to rewrite it in Rust for cpu performance...

You can get progress for cp by piping through pv from moreutils. What I do though is use rsync -P, which accepts most of cp syntax.

I also use pv when dd'ing stuff. Super useful, no need to send USR1 signals

There is status=progress dd option.

Depends which dd you use and what version.

That's for one file. It won't help for directories, unless you tar on one side and then untar on the other.

“rewrite (the easy parts of) $battle-tested-ancient-coreutil in rust” seems interesting as an exercise or as a way to learn a new language, but i’m not convinced of it’s broader utility.

here, the utility could be proof of concept that can be implemented in original tool

But then again there's ripgrep, which did just that and is much faster than grep and works perfectly.

i use ripgrep all the time, it's great.

i don't think it was ever meant to be "exactly GNU/coreutils grep", and I don't think it ever even tried. The API is totally different, i don't think it's ever been the same, it scratches a similar-but-not-identical itch.

https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...

I would argue that "the easy parts" of cp account for a very significant fraction of its usage (i.e. copy some files/directories without any other options). The idea with fcp is to make that common use case very fast, not to replace cp and all the options it handles.

I definitely can understand the fatigue with "rewrite it in Rust" though, as there is a lot of that.

Something you might be interested in is that people have actually gone through the effort of rewriting the actual coreutils to spec in Rust (https://github.com/uutils/coreutils).

Does this support reflinks on filesystems that support deduplication? (like cp --reflink)

The implementation of `fs::copy` uses `copy_file_range` on Linux, which does allow reflink optimizations.

What filesystem on Linux? Show us some benchmarks on other filesystems.

Good question. I've already shut down the EC2 instance I used for testing, but I'll fire up another one later today and get back on this. Great point though, performance characteristics will almost certainly vary quite a bit between different filesystems.

Genuine question: Is running EC2 instances easier than spawning a local VM?

No, it isn't. My motivation for running on EC2 was that benchmarking on a laptop is somewhat noisy due to thermal constraints [1]. I specifically used a bare-metal EC2 instance in an attempt to have a less noisy environment (since VM performance is less predictable and depends on the other workloads running on the same hardware) with proper cooling.

[1] https://lemire.me/blog/my-sayings/

Don't take my word for it, Prof. Lemire is the expert on performance.

I totally buy your explanation. Consumer devices are extremely noisy. Back when I worked on embedded, the only tests that I could trust were running on the boxes

I lost the link, but I remember reading an article in the last year or so about tricks for benchmarking on a laptop. Various things you could do to avoid the thermal throttling. The catch was that it was only possible to do approximate, comparative benchmarks, but never get a result that showed the "true" unconstrained performance.

The most important thing to do would be disabling turbo boost. On linux that's

    echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo # intel
    echo 0 > /sys/devices/system/cpu/cpufreq/boost # amd

together with disabling noisy background services that's the most significant reduction in noise. There are further steps that can be taken but for CPU-bound microbenchmarks they only provide incremental improvements in my experience.

IO benchmarks are trickier. SSDs have thermal throttling too, there's the OS write cache, the SLC cache performance cliff and other factors.

This discussion is starting to make me think that the industry needs benchmarks that take into account modern hardware behavior better, somehow. The only benchmarking I've done is on deployed software for comparative performance tuning, determining if a given change runs faster or slower as deployed. I've long been a skeptic of benchmarks that try to test across heterogenous systems. My dad, who was a systems programmer, helped teach me how benchmarks could be deceptive.

While that's correct, it's misleading in the big picture.

On full load, any machine hits thermal constraints, so, considering clock changes a problem, a desktop is problematic as well. A desktop will certainly have a higher ceiling, and/or a laptop may have too much of a low one, but it's incorrect to assume that the problem is inherent to laptops.

The solution is therefore, which is canonical in serious benchmarks and it's not mentioned, is to ensure that the machine runs on a fixed clock rate. This is easily accomplished, at least on desktops, by changing the BIOS. I wasn't personally able to do this from the O/S (Linux).

Using a machine without control on the clock (which a cloud one can be, even if bare metal) is subject to generate inaccurate benchmark results.

Depending on the scaling governor you're using, you should be able to adjust behavior from the command line: https://www.kernel.org/doc/html/v4.12/admin-guide/pm/cpufreq...

https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_p...

E.g., if you're using the Intel pstate driver in active mode, you can disable turbo via:

  echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo

(and re-enable by writing "0".)

Thanks! I'll check the reference document to search for something that works. In the past, I had tried (to no avail) with the `userspace` governor, `cpufreq-set`, `cpupower` and also `/sys/*/cpufreq/scaling_max_freq`; hopefully, I'll find other info in the document.

I spun up another instance, and now know that the filesystem used on the Linux benchmarks is xfs.

Why do people think it is acceptable for utility programs like file copies to launch many threads and saturate my CPU? It's usually the case that there are many things being done on the system, and I don't want one task take up all the resources I have. Most classic Unix command line tools only use a single thread, so I know they won't bog down the entire system without any effort on my part (adjusting niceness, cgroups, ulimit, etc).

I just tried this going between my 2 servers with NFS, as compared to cp, rsync, and rclone. This tool wins by >2x. Great work! ~1.5Gbps vs ~3-4Gbps.

There was a tool for Solaris called "mtwrite" that would use LD_PRELOAD to intercept writes and farm them out to threads. This way, it would work with not just cp, but also tar extracts to files, etc.

http://www.maier-komor.de/mtwrite.html

> The massive difference in performance in this case is due to fcp using fclonefileat and fcopyfile under the hood

GNU Coreutils' cp, the thing you use on most GNU/Linux systems, is not the "Classic Unix cp".

It has options for copy-on-write cloning via kernel-specific methods.

COW copying is opt-in probably because it's not a true copy. If either file sustains damage, both are toast.

https://www.unix.com/man-page/mojave/2/fclonefileat/

summary: this is a nice cow/cloning wrapper for use on a single ssd partition!

-- but cloning not copying between volumes or partitions is .. and when you actually want to copy with fcp, it can be very slow as detailed in other comments.

I worked with a large cluster filesystem and it had some interesting properties. In particular, files were effectively append-only and immutable when closed, but the FS supported "snapshot for append" which allowed to to make a snapshot of the file, and append to it. Under the hood it managed all the pointers but when the FS designers found out we used the feature heavily they got worried.

I'm afraid the only way there can be a faster file copying tool is by sacrificing contiguousness and increasing fragmentation. Isn't this the case?

I want fragmentation to be as little as possible so it wouldn't be too hard to manually recover a file if I delete something accidentally or damage the file system.

Unfortunately I find it's not enough to copy large files, you need to verify them as well. I don't trust the hard drive controller or the nvme controller to have not f'ed something up.

You're in the wrong abstraction layer. There are a whole bunch of intervening levels between the layer where you've imagined that you "need to verify them as well" and the layer where the "hard drive controller" that you don't trust lives.

Those layers may conceal a real problem (if there is one) so that you can't see it even though it exists, or they can create problems, which you will then mistakenly blame on the "hard drive controller" even if it was not at fault.

So you've created a situation where you can at best confuse yourself into not understanding what's actually going on, and at worst satisfy yourself nothing is wrong when actually there's a catastrophe awaiting you. The worst of all worlds.

What file systems are supported here?

An even faster O(1) alternative

  ln -s /path/to/source /path/to/dest

Or, even closer, drop the -s. That way deleting the source doesn’t delete the destination.

That said, neither of these are truly the same as an edit to one will impact the other.

He might've gotten confused by newer filesystems like btrfs which let you create links to files instantly which effectively work exactly like a copy. The second file however isn't actually created until a change has occurred, so the linking is only an entry in the catalog to the inodes i believe.

i.e. btrfs snapshots use this feature.

This can be enabled as available with the --reflink=auto flag for cp.

When copying between different filesystems it still makes sense to add the --sparse=auto flag to keep a copy of a sparse file sparse, too.

My alias for cp is:

  cp -i --reflink=auto --sparse=auto

Reply on news.ycombinator.com