Comment by jerf | Hacker News Reader

Comment by jerf | Hacker News Reader

The article does mention this a bit, but I would suggest to people to be a bit more careful about accusing everything of executable-size-bloat. We have the mentioned Unicode tables. Can you cut them? Sure, but now you've got a binary that only works in English and very-English like languages for no reason. We use 64-bit stuff for good reason, because we routinely blow out 32-bit limits, but that's 8 bytes for all sorts of things that used to be 4, pervasively throughout the code. We have localization support. We have some sort of async runtime or memory managing runtime or other thing like that. We have all sorts of things like this. If you sit down and take an inventory of what's going on, the minimum size for some sort of "serious" executable is still going to be quite a lot.

(Could you move some of this stuff out of the executable? Yeah, but that really only addresses the immediately visible executable size and comes with tradeoffs of its own... it doesn't mean the executable is "smaller", it just means it's split into pieces.)

It's a bit of a pipe dream now to expect some sort of "serious" executable (one that actually has all these concerns) to fit into half-a-megabyte anymore, just because Commodore 64 programs used to fit into a hundred kilobytes.

100 KBytes is quite bloaty for a C64 program and wouldn't even fit into RAM in one piece ;)

But anyway... wouldn't the better and "obvious" solution be to move foundational things like the above mentioned UNICODE tables into operating systems (if they're not already provided) instead of shipping them with every application?

After all, what's the point of operating systems if they don't provide commonly needed services (and I wonder if such solutions had even been considered for said Rust libraries).

PS: One situation where binary size matters a lot is WASM running in browsers. It's not just download bandwidth, but also the time it takes to compile the executable. We're now back at WASM binaries stuttering on startup for a few seconds because the JIT needs to "warm up". For a very short time, when WASM was AOT compiled, this wasn't an issue. But it became an issue as people became careless about binary size and just-in-time compilation had to be reintroduced to cope with massively big WASM binaries.

Does this mean you need to ship a new Rust binary for every application you build when the Unicode consortium updates the tables?

If your application needs support for new emojis or some dead language, then yes.

> After all, what's the point of operating systems if they don't provide commonly needed services

My answer is "managing access to shared resources"--the primary role of an OS is to allow multiple programs to run on the same hardware without needing to be aware of each other. I specifically don't think the OS should be a kitchen sink for common utilities / services. I would actually prefer fewer things managed by the OS (microkernel if not unikernel). In my utopia, the line between "OS" and "Hypervisor" would be very thin/blurry.

If we really want to be space-conscious, I think the answer is dynamic linkage, although I think people have forgotten the horror that is dependency hell. Nix conceptually provides an answer for dependency hell, but in my experience there remains quite a chasm between what Nix promises or aspires towards and what it actually delivers. For the time being, I have to think that static binaries are overwhelmingly the most economical solution when you account for time spent mucking with dependencies.

> what's the point of operating systems if they don't provide commonly needed services

To abstract over hardware, including to allow running multiple programs simultaneously on the same hardware.

I'd argue that there's pretty widespread agreement in how we actually deploy modern systems when given the choice, that this is all we want the operating system to do too. There's all these operating systems that try to do more (e.g. approximately every linux distribution ever), and yet modern software primarily runs in some sort of sandbox who doesn't allow any of that to seep into the environment. On servers that sandbox looks like "containers" or less frequently "jails". On clients that sandbox looks like "a browser" or "electron".

Empirically the idea of shipping commonly used dependencies with the hardware abstraction layer, seems to have been the wrong idea. Even if you disagree with that, certainly it's not the case that it's the only point of shipping the hardware abstraction layer (operating system) in the first place.

That's a fairly "mainframe era" definition of an operating system though ;) Let me reformulate: If an "operating system" or any other sort of application runtime (like a web browser) needs to handle UNICODE strings anyway in its UI layer, doesn't it make sense to expose an UNICODE API to the applications because all the code and data exists anyway in the OS.

This is one of my main gripes with web browsers: They have excellent text rendering, but there are no APIs which expose the text rendering capabilities to APIs like WebGL.

Now you're getting into nuanced answers anyways

Does it make sense to expose them. Yes, probably, if there's no cost to doing so (which is approximately what you said when you says "needs to handle unicode strings anyways").

Does it make sense to use the exposed dependency? That depends

- What is the cost of including an extra copy (high in wasm, low in server side containers)

- [Edit: Added] What is the cost of using the system copy (typically low, but non 0 performance impact)

- What set of symbols is more likely to be outdated?

- How many times are you going to run into issues where you are missing the dependency in the runtime?

- How expensive is it going to be to implement a different solution for every different runtime that exposes it in a different way?

- How likely is the runtimes API ever to change in a backwards incompatible way?

Most of the time, for things that are not downloaded just when the user tries to run them (i.e. websites), I think it doesn't make sense to use the exposed dependency, even if it exists.

Agreed. As always, the only correct answer is "it depends".

But even considering those questions is probably much more effort than most application developers put into selecting their dependencies, unfortunately.

There's also the question of scale.

Currently, whether your calculator.exe is 50 KB or 5 MB doesn't matter much, trimming down the size towards 50 KB is a nice exercise and deserves praise, but is probably not even noticeable by users.

50 MBytes for a calculator deserves raised eyebrows, something must be wrong which should be investigated. 500 MB would be unacceptable IMHO. Even if users wouldn't complain, I would be deeply ashamed to produce such a turd of a program ;)

Yes, it depends in a very absolute sense, but for most cases, efficient use of memory is a hugely important factor in a large software system for performance. It’s worth doing an abstraction layer. ICU has a stable API, to the best of my knowledge.

Operating systems like Windows and Mac OS X do ship those tables, but there is no Rust crate to use them directly. It's easier to include them once rather than use a different API per operating system.

Unicode tables are gigantic. If it’s possible to build an abstraction layer, it should be done.

The ICU4X project[꧁] is working on some of those goals, and they're also using Rust as a primary implementation language.

[꧁]: https://github.com/unicode-org/icu4x

Unrelated but wow that’s a nice flourish character

꧁ TEXT ꧂

"100 KBytes is quite bloaty for a C64 program and wouldn't even fit into RAM in one piece ;)"

I was thinking of the numerous games that involving loading from floppy disks quite routinely. There's a ton of C64 programs over what fit in RAM at once. Cutting it down to what fits in RAM itself just seemed a bit harsh since even the C64 back in the day wasn't actually that limited.

Also, possibly you wrote your reply after I edited my comment about the ability to move pieces out, but I've seen people still whine about the "bloat" programs have from the sum of their linked libraries. My point is more that a modern program really does get into the dozens of megabytes fairly quickly on modern systems, for what are generally agreed to be good reasons... until we all start talking about "bloat", after which all sense is thrown out the window.

If you honestly sit down and look at how many numbers it takes to be a modern program, it gets large fast.

You know, I agree that shipping the same damm Unicode table over and over is stupid.

However, if they were part of the OS sooner rather than later, certain OS vendors would have found a way to make them incompatible or distribute them separately and make the user explicitly install them. (if you play in Windows, how many times and how many versions of VC++ redistributable you had to install?)

I remember there were projects where I opted-in into writing my own parser (with Pest or nom) just to avoid depending on regex crate. Which is usually the biggest dependency in my projects.

Reminds me of the articles about text processing in haskell vs C. haskell wasn't as lean but it was more general (full unicode support). There was no mystery in the size difference.

Reply on news.ycombinator.com