Hacker News Reader

Comment by est31 | Hacker News Reader

This is an extremely amazing project because it's actually trying to come up with a replacement that can be used as such. Nobody is going to rewrite scripts to use exa instead of ls for example. The coreutils are also somewhat the "basis" of the shell userspace.

Outside of the test failures and missing features, the only reason one might not want the coreutils is the size.

Right now, cat on my system is 44K large, while the default output size for the cat executable in release mode is 4.4 megabytes in release mode. If you enable a bunch of options in Cargo.toml to reduce the default bloaty settings (lto = true, codegen-units = 1, strip = true, debug = false), you get that down to 876K. Which is still 20 times larger than the native cat. For true, it's a similar story. 40K vs 812K.

View on HN

Rust executables are statically linked by default. This makes them oftentimes substantially larger than their dynamically linked counterparts. For me it is not a deal breaker.

My guess is people will start "solving" this kind of problem through multi-call binaries a la busybox.

uutils already (optionally) builds that way. The GitHub release has a single 8.4M binary.

That doesn't seem so bad, the GNU coreutils Debian package has an Installed-Size of 17.9MB. Though there might be hidden drawbacks I don't know about.

> The GitHub release has a single 8.4M binary.

That's a definitely acceptable size, thanks for pointing that out. Yeah a bunch of symlinks should solve the issue. If the binary is kept cached in RAM at least, there should be no overhead in starting it. I think it's pretty rare to have environments that are so memory constrained that they can't hold onto 8 MB of RAM.

Note that not all targets are that small though. Musl has indeed 8.3 MiB, but in coreutils-0.0.12-x86_64-unknown-linux-gnu.tar.gz, the size of the coreutils executable is 12.6 MiB. On the bright side, on the same target, a local build with the cargo features to reduce bloat enabled gives only 7.1M. All these numbers are acceptable.

Is anything in coreutils setuid? That’s the only possible drawback I have in mind.

Yes, the only reason I say "solved" is that this approach will only help for suites of many small tools that ship together.

Is that a typical ratio for software ported to Rust? Where is all the extra size coming from?

As someone else mentioned, shared libraries. If you built ls and statically linked libc and whatever else it needs, ls would be a lot bigger.

According to cargo bloat, 40.1% of the .text section comes from clap, and 46.7% of the .text section comes from std. clap is quite famous to be extremely bloaty.

To give an answer to your other question: a lot of the additional code size is a one time cost. That means that the relative difference is the largest in these cases, for binaries with little logic of their own. The more complex the program gets, the less the relative difference is. However, even if you don't account for these one time costs, Rust creates larger programs than C, due to heavy use of generics compared to C.

Maybe clap should store text strings in compressed form, and then decompress as needed. Could greatly reduce the bloat.

.text is the section of the binary where the code is.

At least 500K comes from clap, the argument parsing library. That's how much true's size drops if you comment it out.

It's a lovely library. Generated help text with colors and line wrapping based on your terminal width, informative error messages that suggest corrections for typos, and a declarative model that lets you express subtle relationships between options. You don't get that from getopt.

But it's also pretty large. https://github.com/rust-cli/argparse-benchmarks-rs has some numbers.

(Disclaimer: I wrote one of the competitors on that page, lexopt. But I also use clap, depending on the project, and I'm happy with it.)

Out of curiosity, why should each application implement line wrapping for terminals. That seems like a feature the terminal should implement, and from what I've read there is no standard way of getting the terminal size anyway.

The terminal isn't aware that you're, for example, using an indent because the help text of an option should appear indented from the option it's explaining. So the line wrap will go into the next line and ruin the readability.

Ah, that makes sense.

There's the TIOCGWINSZ ioctl which seems pretty standard.

500K to parse options is frankly too much...

In most software people will care more about helpful error messages and readable help texts than about half a megabyte of code size. Coreutils with its tiny frequently called binaries is an outlier where the size adds up, but that's not the norm.

Not sure how you are serious about your reply. PCs are a small market and a solution fitting them hardly represents a valid approach. There is an entire world of small devices that don't vomit terabytes of disk space.

One of the most attractive features of Linux is how it suits a large array of hardware. Replacing C solutions with some bloated bandwagon alternative is a bad idea.

Yes, if you write software for devices with megabytes of disk space then don't use Clap for argument parsing in your software. I'm not really sure how that's a criticism of Clap in general, or the rust coreutils specifically. Rust coreutils built as a single binary (as opposed to one binary per tool) is about 8MB. Anyone who is agonizing over 500kB of RAM or disk space is going to use BusyBox instead of either the GNU coreutils or a rust replacement for them. Why limit the implementation to satisfy the needs of somebody who wouldn't choose you anyways?

This thread inspired me—I found a way to remove 40K: https://github.com/clap-rs/clap/pull/3370

I'm sure it's possible to cut it more. It'll never be tiny, but it can be smaller than it is now.

The added compile times scared me off of pretty much all the bigger options. Like, six seconds? Wooooooof.

I believe what you are seeing there is the power of shared libraries.

It'd also be a great time to establish a better standard for using these programs with a shell or scripting.

For example, if ls had a --shell option that would write out file information as quoted eval'able variables, or even JSON, or anything that was easily and reliably parsable it would remove a huge portion of scripting headaches and errors.

I added some CI task and replayed the history to graph the evolution of the size: https://github.com/uutils/coreutils-tracking#binary-size-evo...

This will help making sure that we aren't regressing (more) ;)

Reply on news.ycombinator.com