Hacker News Reader

> 2) Rust's ecosystem makes it easy to find and integrate solutions not offered by the standard API

This is a double-edged sword. I'm a huge fan of Rust's tooling, but when it's so easy to add dependencies, this inevitably leads to deep dependency trees, slower compile times, and less knowledge about what's actually happening inside most rust code bases. Often times when asking about how to solve a problem in Rust, the answer comes in the form of "use crate X" rather than an explanation of how to solve the problem using the language itself.

I think it's important to say the silent part out loud: "just use crate x and you can look up how the crate works behind the scenes if you are curious".

This statement is a bit confusing: are you saying that there is such a thing as too much code reuse? That it should be necessarily a bit difficult to integrate a third party library in your codebase?

There's always a balance point. For something like JSON parsing, or an HTTP client, for sure I want to use a library. But when code reuse means pulling in a giant library to do one specific thing, or else introducing a runtime like tokio where it otherwise might not be needed, I would probably rather implement a small tailored solution specifically for my use-case.

In one sense, the way that larger crates in Rust are fragmented does address some of that.

This does result in large dependency trees, but it does also mean that if you need that one specific functionality, you may not need to pull in the full library, but only that smaller section.

Ideally, there should be no harm in pulling in a library for just 1% of its functionally. A combination of careful library design and tooling should take care of dependency bloat ("shaking the tree"). In practice of course, libraries don't always consider this aspect, and the tooling never seems quite up to the task.

Yeah I think it’s also a matter of API design. Like a crate which by necessity has to handle the “general case” to some degree will often have more API complexity than a solution which is tailored to one specific use-case. For instance, you might have to concern yourself with configuration parameters which have zero relevance to your use-case.

> Often times when asking about how to solve a problem in Rust, the answer comes in the form of "use crate X" rather than an explanation of how to solve the problem using the language itself.

Do you think this is a bad thing? How frequently would you say it's a better use of a software engineer's time to carefully reimplement a commoditized library that already exists for a speed improvement or to understand it?

I don't really understand arguments that you shouldn't introduce convenience and quality of life features because programmers will lean on them too much. Leaning on them is the point; the implicit thesis is that programmers no longer have to understand what's happening under the hood. It's specialization and division of labor applied to programming.

And if you really care, you can probably just look at the source code or pick up a book to learn how it works?

For almost any real world application, where people's data and businesses are at stake, developer convenience is a distant priority to security. Of course it's great when we can have both, and making secure solutions the most convenient is key, but Rust doesn't quite get this right yet. If there's a security issue in one of those dependencies you are importing, how do you even know that you are impacted? What about when it's a dependency of a dependency? Making it easy for developers to suck in a huge TCB at build time has some serious downsides.

When security is a concern, developers can simply avoid using untrusted crates. But concerns over security are not reason for a package manager to not even exist. Pretty much every major language and operating system has a package manager. While supply chain attacks are real, they are not unique to Rust in any way.

The ecosystem also includes stuff like cargo-audit[0], with which you can easily add dependency auditing to your CI pipelines or whatever. People are definitely thinking about this problem.

[0]: https://github.com/rustsec/cargo-audit

I very much agree with your point, and want to add that in certain contexts, the same goes for performance.

I do think that Rust making it easy doesn't necessarily mean that you have to use it. It's fine to write applications using only the stdlib, and only link to some known libs like openssl or libpq.

In Rust's defense, it is easy to list all dependencies, and they don't automagically upgrade from one checkout to another.

I think it's a subtle problem and it takes more than a few sentences to make the required nuance in the argument clear (or the one line "zingers" found elsewhere in this very thread). But just as one example, see my comment elsewhere in this thread about the use of globwalk in this example. That's bringing in a ton of code when compared to just using walkdir and checking the file extension directly. The crate ecosystem encourages this kind of emergent behavior.

People make the mistake of treating this issue as black and white: either you're for or against code reuse. But the reality is far more nuanced. Often, a dependency will solve a much more general problem than what you actually need, and thus, avoiding the dependency might result in solving a considerably simpler problem than what the dependency does. In exchange, you use less code, which means less to audit/review and less to compile.

Given my position as the author of a few core crates, I actually often find myself advocating against the use of those very crates when the problem could be solved nearly as simply without beinging in the dependency. (I did not author globwalk, but I did author its 'ignore' and 'walkdir' dependencies.)

it would be great to have an https://alternativeto.net/ style recommendation engine integrated into crates.io (or npm) that can narrow down the minimal crate required for a specific use case.

It's an interesting idea, but would take a fair bit of creativity to execute well I think. It's really hard to enumerate this grey area! But I think something that elaborates on this grey area with lots of examples would be great. Sounds like a blog post I should write. :)

> It's an interesting idea, but would take a fair bit of creativity to execute well I think.

yeah for sure, it's a hard problem. but at least a high level listing of similar libraries in the same category with # of recursive deps and total bloat size clearly visible so it was easy to shop around and sort by whatever attribute the developer wants to prioritize.

it would be similar to category-specific filters & comparison tables for features of e.g. SLR cameras, usb3 power supplies on amazon.

The problem is that walkdir isn't really an alternative to globwalk. It's only a good alternative in this specific simple case. globwalk even depends on walkdir. :)

i don't see a problem with listing both, as long as the high-level featureset is sufficiently enumerated in a comparison table for me to quickly discard or add the lib to my to-try list.

There's no problem in listing both. The point is that a high level feature list comparison wouldn't have necessarily helped here.

Having lists of alternative libraries to solve a particular problem is great. It is valuable on its own. But it doesn't really fix the nuance that I'm focusing on here in this example. And my general claim is that this sort of nuance is a fairly common problem that leads to unnecessary dependencies.

What about a less ambitious goal: fuzzy keyword search that displays the dependency graph and cumulative build time of each node in the package graph (activity stats on hover)? Users can assess for themselves whether a simpler library meets their needs, but this would make the relevant information more accessible.

Sure. Also sounds useful. crates.rs (an alternative UI for crates.io) has the beginnings of this by listing the binary size of the crate's dependencies. e.g., https://lib.rs/crates/globwalk vs https://lib.rs/crates/walkdir

I would throw in that it's very hard to design APIs well and it's very hard to not couple things and include the kitchen sink of features.

Let's say someone makes an XML parser (just trying to pick an example). IMO a bad XML library would read files. Instead it should just, at best, take some abstract interface to read text and outside the library the docs should provide examples of how to provide a file reader, a network reader, an in memory reader, etc...

But, I rarely see that. Instead (this is more the npm experience) such library would include as many possible integrations as possible. Parse XML from files, from the network, from fetch, from web sockets, there would be a command line utility to parse XML to JSON, and some watch option that continually parses any new or updated XML file and spawns an task to do something with it, all integrated into this "XML Library". Parts of it will respond to environment variables and it will have prettifiers integrated with ANSI color codes and you'll be able to choose themes, and it will have a progress bar integrated into the command line tool for large files

And the worst part is the noobs all love it. They down load this 746 dependency XML library and then ask for even more things to be integrated.

Maybe someday there will be a language with a package manager/community/guidelines that mostly somehow rejects bad bloated packages. It seems like a nearly impossible task though.

Note: I don't know rust well but I tried up fix a bug in the docs. The docs are built with rust. The doc builder downloaded a ton of packages which to me at least was not a good sign

Personally, I don't see this particular problem as widespread in the Rust crate ecosystem. It's definitely one way in which dependencies can proliferate, but not a terribly common one in my experience. There's a lot of focus on inter-operation using common interfaces.

It's interesting that this line of argument perhaps leads to the conclusion "keep libraries small, to allow more granular selection of dependencies".

But this is the kind of reasoning that leads to the situation in npm of thousands of tiny libraries such as "leftpad" that many systems programmers are so derisive of.

I'm not sure that's the only choice. I think another choice is to help educate folks when a dependency could be removed in favor of a little code. Take regex or aho-corasick for example. aho-corasick is a lot of code and only has one tiny dependency itself (memchr). There's really not much left to break apart. But I wouldn't recommend the use of aho-corasick for every case in which you need to search for multiple strings in one pass. A trivial solution using multiple passes is quite serviceable in a number of cases.

I think that Clojure does something interesting here, but I can't quite put my finger on it. Apparently, it uses very small and abstract libraries and seems to lead to narrowed down dependencies.

Very true.

Yet another possibility that I think has a place in this nuance conversation is adapting existing code with changes. It's interesting that we don't culturally today have an accepted way of doing copying + adaptation, but over most of the history of computing it has been very common. It has the obvious downsides of not easily getting improvements and bug fixes of later versions of the upstream code, and rightly is passed over in most cases, but in some cases it's still a win.

It does (inevitably) really depend on the quality of the library. We really saw this for say, SSL deprecation.

If you made good choices (e.g. maybe you depend on Python's Requests) you had to go out of your way to make problems for yourself, by default the software understands that it should prefer shiny modern cryptography, allow anything that's probably fine, and reject nasty broken stuff. What counts as "shiny modern" or "nasty broken" will vary over time, but it's not likely you, as the application developer, were best placed to decide when you wrote the code, so unless you specifically did so the runtime decides.

In contrast Microsoft shipped not one but several distinct C++ APIs and .NET APIs where you'd need to go back and periodically modify old code (e.g. to specify newer TLS versions) or now it doesn't work because there was no provision for programmers to just let the runtime do the Right Thing.

As someone who's spent a few decades working in C++ codebases I'm not sure how this is a Rust specific problem.

Last large-scale C++ codebase I worked on took 60+ minutes to compile when you touched the precompiled header.

In fact, I've found it to be common to have binary-only library dependencies in C++ which makes it harder to tell what those downstream dependencies do.

I think you just run into it faster in Rust: I tried making a simple REST HTTP server (like 60 LoC) using actix-web and it pulled in over 200 sub-dependencies and took something like 10 minutes to compile.

This toy program written using Boost.Beast or something similar would compile much faster. Of course I would have spent all day setting up the library instead of waiting for it to compile...

The worst part is how easy it is to add derive and proc macros that kill compile times. I have a ~3k LOC project that blows up to over 45k LOC after `cargo expand` due to a dozen or so macros and the compile time is really starting to hurt iteration speed. Sadly, macros are by far one of my favorite Rust features.

I ended up paying for Clion just to get better debugger integration and a quick action for cargo expanding a file in the UI so I can copy paste the macro results into my codebase. I'm hoping that will improve incremental compilation times until I buy a Threadripper workstation.

Does the expanded code take longer to compile than the equivalent handwritten code? Is it just the expansion itself that takes a long time?

In the latter case, maybe cargo needs an intermediate macro-expansion cache (instead of just a crate-level build cache)

Often, you would write a lot less code if you were doing it by hand.

I adore CLion, it's brilliant for coding. At the moment I'm using it with Python and Git.

> this inevitably leads to deep dependency trees, slower compile times

I've never thought that "having more friction when adding dependencies" is a valid strategy for preventing overuse of third-party code. Unlike other "trust the developer" scenarios, dependency bloat is a minor issue at best and is very easy to detect and diagnose. IMO you should cut down your dependencies when they actually become a problem that's bigger than the one they're solving.

> and less knowledge about what's actually happening inside most rust code bases

I actually think the package ecosystem (along with macros) is crucial for the breadth and accessibility that Rust provides. Rust is a low-level language, but you have high-level libraries at your fingertips if that's the level where you need to be working. This means that instead of writing hot-paths in a low level language and application code in a high-level language (and dealing with the added complexity in terms of FFI, builds, tooling, developer knowledge, etc), it's very possible to just write the entire thing in Rust. Again in this case, I think "forcing devs to learn how everything works" is a weak argument for increasing the friction for adding dependencies. I'm pretty sure that everything on crates.io is required to include source, not just a binary, so it's trivial to dive in and learn how it works if you're so inclined.

I'm not a rust/cpp guy but I have a feeling that people skilled enough to need rust or cpp can probably use public libs for rapid prototyping and then recreate a smaller version if needs be.

Compile times in Rust are really the only problem with this. Rust with LTO on is really good about removing unused code, so it doesn't hurt you during runtime

Whenever you run anything there's millions of lines involved across the OS API's directly or indirectly, so in practice it doesn't matter what's happening behind the scenes. In languages with fast compile times like Go and Java people add dependencies without even caring. And I think that's how it should be.

All our Java backends at work are like 200MB and it doesn't matter and nobody cares. It still compiles in less than ten seconds, and all that code barely affects startup time.

> All of these other things could be done for C++. But, importantly, they haven't.

I think programmers often under-emphasize the importance of culture, so I want to boost this line.

Lots of "debates" around programming languages are, I think, better understood as differences in beliefs about what a language should prioritize. C++ could add an allocation checker that provides stronger memory guarantees by outlawing certain operations (or requiring they be marked a'la `unsafe`). They don't add it because, I think, they view other additions to the language as having a higher "return" on allowing people to create high quality programs.

It seems to me that a lot of this comes back to first-principles about what makes a good programming language. These ideas are, to some degree, set before a line of code is written. The rust folks thought memory and concurrency safety were very important when they were designing rust - even though they couldn't have formed those opinions by using the kind of language they were making (because nothing quite like it existed yet).

I also think about "infamous" qualities of languages (generics in go, the GIL in python). These ideas exist within wider cultural beliefs about what makes a good programming language. In a sense, it's inevitable that most of the people working on cpython aren't too concerned about the GIL, because they chose to work on a project that has developed around the GIL and believe its limitations are worth the gains from being able to rely on its coordinating function.

The implications of this are, to me, that when advocate for languages to adopt features we'd like to see we should remember the wider cultural context. It seems silly to expect the current golang team to implement generics "the way the community wants" because they've spent years building a fully featured language that functions without generics. Culture runs deep and it's not an intellectual or professional flaw to value improvements differently than our peers.

>nothing quite like it existed yet.

Cyclone has been around some 2002 according to Wikipedia. I don’t think a similar language (RAII + annotations + a mostly-mandatory validation step in the compiler) has had a well funded and sustained marketing campaign until rust, though. RAII also predates Rust by a couple decades.

Rust is also not very similar to itself from its early days.

Oh, I didn't know about Cyclone - that's cool! I wasn't trying to say that rust isn't influenced by languages that came before it.

I mean that, when they decided it was time to make a new language, the team would have had some understanding of what the language would be like. That understanding couldn't come from using the language itself (it doesn't exist yet). Instead, teams do cultural work to communicate & come to shared understandings about what the language they are making "should be like." That shared understanding survives past when the language is working and influences decisions to improve the language.

It's just worth considering and talking about project culture! It's common for there to be many possible improvements to a language and have the culture of the project (or wider community) be what tips the scales.

> 2) Rust's ecosystem makes it easy to find and integrate solutions not offered by the standard API

This made me think back to something Rich Hickey said in his "Are we there yet?" talk. In short rust has memory management, and so can better function as a library language, similar to how Java did back in the 90's. Rich's quote below.

>> And it is a big problem. I think the lack of garbage collection really impeded C++ in one of its design objectives, which is: it was supposed to be a library language. All the original design stuff and any time you heard Stroustrup talk about it, it is like, "C++ is going to be a library language". But it only ever ended up being a parochial library language. Every shop had a library, but there were not a library, and still are not a lot of libraries, that go between places, because of this problem.

https://github.com/matthiasn/talk-transcripts/blob/ccc4a0172...

C++ multi-threading is easy by not sharing data. Just have each thread produce a result and have a thread that accumulates the results via std::future.

I've converted many mono-thread C++ algorithm this way.

> Others are rightly pointing out that only #3 is fundamentally unique to Rust. All of these other things could be done for C++. But, importantly, they haven't.

Except they have, but some people aren't keeping up with C++ news, only bashing it.

1) can be easily done with bounds checked enabled STL, like modern C++ compilers do in debug builds, or force enable them in release builds.

2) Conan, vcpkg, and better than cargo, because they support binary dependencies.

>because they support binary dependencies

Without an ability to tweak build flags. Which makes them useless. And since they are not really crossplatform either, you better stick with portage, which is still far behind cargo.

So useless, that they get used daily by millions of C++ developers, and was one of the top requested features for vcpkg.

Sword fighting gets tiresome after a while, and not all businesses enjoy shipping source code anyway.

I tested most cpp pkg managers (including vcpkg and conan); They suck. They suck so much that I reached the conclusion that using git submodules was the way to go, and saw “just copy the header file” in readmes of modern projects. The pkg managers are hard to install and configure, don’t integrate well with the tooling, don’t support most stuff, have out-of-date documentation, and are just buggy.

Cpp API design also seems to suck. Doing a simple curl request is bloody complicated.

Yet millions of devs are adopting them every day, they might be really masochistics, like myself every time I rebuild from scratch my Rust compile time check, that with compiling everything from scratch takes 18 minutes on average laptop, versus 5 minutes in C++ (thanks binary libs).

Cargo keeps failing my litmus test for usability of compiled languages.

I just look forward that something like Cranelift will fix it.

> If you introduced to C++ a Result type

I don't know too much about the details of rust, but: std::optional? Whether people will use it is of course up to them, but my feeling is that they will.

Not the same, Optional is like Option in Rust, it’s about existence, not success or failure.

I believe std::expected is the correct answer but it’s unclear to me if it actually made it into the standard yet or not. I thought it had but can only find proposals...

AFAIK its not yet into any standards, possibly because the semantics are really complex (don't know how rust specifies them) - useful discussion here: https://www.reddit.com/r/cpp/comments/c75ipk/why_stdexpected...

Isn't Optional used in a lot of cases where empty means failure?

That's not enough.

Hacking failure reporting into an existence API will make your code confusing, and will leave you no way to report multiple types of errors.

It might be, but that doesn’t make it semantically the correct thing.

As always, it depends. But traditionally, most languages that use this strategy have both, and they aren’t interchangeable. I can see this happening more when you only have optional, though.

There are proposals for std::outcome and std::expected. I think the internal workings for one of them have also been passed to the C committee for standardisation.

from a safety perspective, std::optional is not much different than std::unique_ptr. both will happily allow you to invoke UB if you don't explicitly check for validity.

std::optional is like Rust's Option. But even then, without compile-time pattern-matching you leave the error to be checked at runtime, which defeats the purpose of this type. On top of that, it wouldn't be C++ if there wasn't some UB involved.

operator*() and operator->() on an empty std::optional is Undefined Behavior. This is mind-boggling. It defeats the whole purpose of using a wrapper type to enforce bulletproof error checking. It's the worst of both worlds: you have to use a cumbersome type, and it still fails like a nullable pointer.

yes, it is a somewhat baffling feature. the only practical use I can imagine for it is to avoid calling the copy constructor with an invalid instance of the type and/or to avoid a heap allocation. otherwise, it would be much more natural (and no less safe) for a c++ programmer to just use some flavor of pointer. in the case of pair<bool,T>, one would expect an optimizing compiler to skip the constructor call for an invalid instance anyway if the object was only accessed after checking the bool (unless the constructor has side-effects, yikes!).

I think most of this is just because C++ doesn't have very good pattern matching and didn't want to introduce new operators. (FWIW, using operator*() and operator->() feel very natural, although they are unsafe without using operator bool() as you have mentioned.)

> If you introduced to C++ a Result type, or a package manager (I assume people have done this already), most C++ developers wouldn't end up using them. Most libraries wouldn't be using the same vocabulary. It would be arduous to even move the standard library over, because it would be a breaking change. Much of the tooling out there would probably never get specialized support.

CMake kinda makes me think there's definite interest in this kind of thing in the C++ community. It's janky and horrible, but it has tackled the module/package/library issue a little bit (not in an elegant sense but its a little more standardized)

There’s no real thing preventing #1 from being done in C++, but it is incredibly hard to remove bad patterns from a language that allows them. This is both problematic because these languages don’t provide first class support to remove unwanted features, and because huge amounts of existing code will use patterns you’re hoping to excise. In order to successfully do this you have to do a huge amount of both linting and dependency checking, and the resulting requirement that you not use any existing libraries makes using Result in C++ about as hard as doing a progressive rewrite into Rust.

> or a package manager (I assume people have done this already),

Conan, the C/C++ Package Manager [1]

[1] https://conan.io/

> but to the simple fact that it's "C++ without baggage".

That's how I've always viewed it. Or rather a better replacement for C++.