Rust and the limits of swarm design | Hacker News Reader

Rust and the limits of swarm design

esr.ibiblio.org

53 pointsadiabatty54 comments

> This isn’t a question that comes up so often with respect to (say) Python because Python has an effective form of curation – blessing things into the standard library, at which point their alternatives generally disappear. In effect, Python modules are quality-filtered on the taste of the BDFL and the devteam.

I love Python but this statement is absolutely coming from someone who does not write Python on a regular basis. Many stdlib modules are greatly outclassed by third-party alternatives - urllib2 vs requests, datetime vs arrow just to name a few - to the point where no one actually uses the stdlib modules in production.

Sure, but even then, I think his point is still applicable. It's urllib2 and requests, not urllib2 and requests and 20 other libraries. The fact that an inferior library is being kept alive by being in the stdlib is maybe unfortunate, but there's still only a small number of choices overall. That makes it much easier to do research and see within the first few search results that the majority of the community recommends one over the other.

His point is absolutely not applicable. Rust is a _very_ new language. The community will eventually converge on specific crates for specific purposes in time.

The libraries you mentioned augment the stdlib rather than replacing it.

requests was indeed asked to join the stdlib. However, the schedule required for distribution would have made it more difficult for them to innovate in their current way. [0]

[0] https://github.com/kennethreitz/requests/issues/2424

Actually, the kind of features ESR asks for are covered by github, IMO. It's easy to review code quality from there, the number of stars and forks reflect project popularity, the list of committers (how many they are and who they are) can help having an idea about long term maintenance expectations, etc. The cool thing is that it works for almost any language : github is "social coding", thus it's also social validation.

But this article raises an other question : if you want 10 years long stability, is it wise to rely on third party code? If you plan on maintaining your own project that long, it's probably acceptable to take time to write your own implementation and maintain it. Or you could even use a third party dependency, then progressively replace it with your own API compatible implementation.

10 years is "that long", now?

How many libraries you use today that you already used ten years ago? Is there more than a handful?

I don't think it's about "libraries that you use". It's about the codebase of a widely used software. Most of the software I use existed 15 or even 30 years ago. It has been updated, of course, but not necessarily redesigned.

>"A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying."

This is a real problem. Here's a blog idea: "Consumer Reports for libraries"

(As an aside, the phrase "objectively terrifying" bugs the hell out of me. Terror is fundamentally subjective - it's a primal emotion. Please let's not let "objectively" become the new "literally".)

FWIW, this is not as bad a problem as he makes it out to be -- the reason folks don't seem to grasp it's a problem is because from inside the community the answer is usually "there are 2 crates for that, with the following tradeoffs". You have to ask, however; and it's hard to find out which crate you want to use without asking someone.

This is still a problem, however, but it's one the community has been aware of for a while. https://github.com/rust-lang/rfcs/pull/1824 is step 1 in fixing it.

"Inside the community" isn't a very strong argument in my opinion, because there is a strong bias. Folks "inside the community" are developing Rust packages because they really, really enjoy it. This is much different from having a mature language where folks who aren't as gung-ho about it develop things (out of curiosity, because work made them, etc.).

I understand there's a recognition of the problem, and I think it's great that it's on the radar.

My point about "inside the community" was solely to say that it's not as bad as he makes it out to be. You usually get recommendations and help picking one within minutes of posting on a Rust forum. (Or by googling it). Survivorship bias is irrelevant here.

This is not an ideal situation to be in, but it is better than he makes it out to be. And of course, we're already working on fixing it.

i think he is exaggerating a bit. Many languages have a model where most of the functionality lies in packages and not in the standard library. And it's working. You just have to get used to it and try to develop a taste in choosing the right package. But it's not fundamental and not the limit of swarm design, because you can use statistics to assess maturity.

I wouldn't say it works, rather that people will put up with it. When I write JavaScript/node.js code, I have no idea whether the dependencies I'm using will be supported 6 months from now.

It's not so easy to just "choose the right package". Often all the packages for a given function are someone's hobby.

Consequently JavaScript code needs constant, expensive maintenance. Whereas Python code can rely more on the standard library and big frameworks and tends to keep working for years.

Programming language designers should see themselves as builders and curators of great standard libraries, the language itself is almost incidental to that goal.

Libraries tend towards standardisation regardless of whether they're being curated by the language designers.

Python is actually a good example of this. Look at the library ecosystem built up around scientific computing. None of that is in the standard library, yet there is a common set of packages being used.

Of course. But these projects exist in addition to a rich standard library.

C++ takes the approach mentioned of having a rich standard library (Boost) that is a separate project from the core language. It (IMO) does not work very well. Many C++ programmers don't use Boost and generic C++ libraries are unable to use Boost as a dependency.

> Many C++ programmers don't use Boost and generic C++ libraries are unable to use Boost as a dependency.

That is because a boost installation litters your hard drive with several hundred thousand tiny files. I can't in good conscience require end-users to install that monster to compile my 10Kloc project.

Yeah, but C++ doesn't really have a good, cross platform package manager. In Java land, for example, the lingua franca is Maven (even across different build systems).

As a result many libraries are close to de facto standards because any Java dev can easily get those libraries, regardless of being on Win, Mac, random Linux distributions, etc.

Scientific libraries are a complex problem, writing yet another web framework isn't. Furthermore most sci/num libs are heavily linked to C/C++ and FORTRAN code, where most of the work is done.

Web frameworks are rarely part of standard libraries. The closest I can think of is ASP.NET for C#.

Go has a http server, so does python and javascript.

So a http server is a web framework now? Okay, let's go along with this.

Node.js is popular, but it's not 'standard'. It's not part of any ECMAScript standard libraries, which would be the requirement if it was to be standard.

I don't know the situation with Go that well, so if it has a http library in the standard I'll take your word for it, but there do seem to be some popular non-standard http servers like Caddy.

As for Python, this is probably an example of where the impact of standards is not that great. How often is the standard http server used compared to popular third party alternatives? Are there any web apps you know of that are using the standard http server in production?

What makes the libraries blessed by the language designers so much better than a library blessed by any other trusted party? Rust is an open source language, and only the core team works on it full time - even then, any one of them could leave at any time and the community at large would need to pick up the slack. This isn't any different than the situation around any of the amazing community libararies in the Rust ecosystem. Heck, libs blessed by the language designers aren't even always the best choice - just look at json manipulation in .NET; you get multiple different stdlib ways to work with json, yet the community still gathers around the much better JSON.NET package.

In terms of value proposition, I think the author is just hankering for some .NET style support agreements in order to hedge his maintenance cost bets - and I'm sure there are or will be plenty of consultancies willing to provide this service, so long as the demand exists. That open source language users are entitled enough to expect this service for free from language maintainers... I'm not sure that's completely justifiable.

If a library is supported and maintained by the language designers, then its lifetime will be at least as long as the lifetime of the programming language. Or if the library is replaced, then it will be done through some mechanism of deprecation, rather than sudden abandonment.

I don't see the standard library and third-party packages as being in opposition. Far from it. A rich standard library encourages a rich ecosystem of third-party packages.

Maintenance costs affect the authors of open source packages even more than they affect commercial users. They are working for free and in their spare time. A rich standard library makes a language attractive to people who want to write packages.

Omg. You just made me think about lifetime markers for crates... your project requires a minimum lifetime of stability in a crate or something. Kinda funny.

I don't think he's arguing that the language designers are the ones that need to be curating crates, he just brought up Python as an example of some sort of curation happening in a similar ecosystem. It's the community at large that needs to be more active about identifying and signaling which crates should be preferred to provide particular functionality.

I don't think you even need to just pick one crate for a particular functionality - narrowing the field down from 23 to a handful with well-understood strengths and drawbacks would be a good start.

That's not a great reason to move languages though.

For instance basically nothing comes in the standard library for C. If basically nothing is offered in the standard library of Rust then why switch? It's valid point.

If we want a lot of adoption to Rust, people need to have an easy way to find quality implementations in crates or add them to the standard library so they are easy to find. Either way, you need one of those two things to happen because people are lazy.

> then why switch?

Languages are a lot more than the standard library.

I don't think "it's working".

Perl users brags about the numbers of packages in CPAN. For me, that's a nightmare. There are a dozen packages for everything. Each time I pick up one (more or less at random after wondering for hours which one I should pick between all those, which are irregularly accurately described), I discover a few weeks later that there are bugs in this package, or that it is deprecated, or that it has been reimplemented for better performance with a slightly different name, or that the procedural access to the features disappeared in next version, or that the package Y version 0.12.3 is more stable than the package X version 2.1 that I chose, or that oh in fact this package function is TODO so I have to switch to another package because it's been marked TODO for 13 years, or that I have to ditch the package I use when my program grows because it does have the extra features I now need and I should have picked up another one from the start, etc.

So I try as much as possible to stay away from packages which are not in Core. The problem now is that even those 'standard' modules are in fact not so standard for Perl distributions which may omit some of them...

And then you have to fight between the packages provided by your OS/Perl distribution and the ones provided by CPAN...

I sometimes end up rewriting everything in C, where there is one standard library, and then de facto standard/widespread libraries with not many alternatives, so it's hard to pick a wrong one.

-------

Same problems for CTAN (Latex). Plenty of packages but most of them are incompatible with some others. In the end you may not be able to use them if you happen to need the 'wrong' combination of features.

It seems like the core thrust of this article is that someone isn't around to perpetually support the libraries the author would like to use; i.e., he/his company is unwilling to fork out the money for someone to keep the 'blessed' libraries fully up to date -- so now that's the implementer's problem! I can understand wanting stability and a promise of timely updates for all one's dependencies, but it doesn't seem like the core language team is the right group to foist that responsibility upon.

In the end, it doesn't matter who it's "fair" to assign responsibility to; a language you can't rely on is a language you can't rely on. All esr is pointing out is that there's a problem. Personally I think the Rust folks are pretty aware of this problem, and are managing just fine; Rust is definitely still in the early stages of language community growth, and locking things down too much would be a mistake.

I suppose I'm not arguing from the perspective of fairness, but moreso about incentives + experience. If we applied the author's argument against other fields and languages, e.g. machine learning and python, then the cpython maintainers would be right in including TensorFlow in the python stdlib (or whichever competitor dethrones in and sticks around for long enough).

That doesn't sit well with me, at least. I'd want the top ML developers looking at TensorFlow, and the best language implementers doing the same for cpython, not the latter trying desperately to improve code they have no expertise in. From that perspective, what matters is community: is there a knowledgable community supporting codebase 'x', where x could be the stdlib or some third party library. Some other commenters in this thread proposed better library curation/discovery tools in place of a formal commitment of long term support by the core team, which seems ideal! Using such an approach, you rely on the language for the core-language bits, and on third parties for the third-party bits, which seems on the surface to be healthier than arbitrarily choosing the only constant group (core language devs) as your supporters of everything.

Reminder that before you take anything ESR says remotely seriously, you should read http://www.catb.org/~esr/writings/dancing.html from beginning to end.

It actually pains me to say this in this case but, let's take arguments on their own merits. (Though I think it does not hurt to be aware of that stuff he wrote.)

He's not a Christian, so we should ignore what he says about computers? That's preposterous.

That _really_ isn't a TL;DR; of that article.

Haha nice try, but one won't be drawn into a theological debate about e.g. how many neopagan angels can dance on the head of an atheist pin. The point is that religious beliefs, no matter how banal or how fringe, are completely orthogonal to technical judgment. They are separate spheres. You've cannily left unclear the precise nature of the spiritual pablum you're peddling, but one can still see through your clumsy attempt to invoke religious prejudice.

> Which brings me directly to what bothers me about the crate system and the sociology behind it – I don’t see any pruning. Worse, I don’t see much understanding of the need for it. A lot of Rustaceans don’t seem to grasp why, when the question is “where do I get feature X?” the answer “oh, there are 23 crates for that” is objectively terrifying.

Actually, I think the community does recognize that this is a problem they are running into as they scale up. Previously there was the discussion of the Rust Platform [1] as a pseudo-stdlib which was received with a lot of mixed since the Python approach isn't actually all rainbows and sunshine. More recently, there has been effort to try to improve the discoverability of relevant crates [2].

[1] https://aturon.github.io/blog/2016/07/27/rust-platform/

[2] https://github.com/rust-lang/rfcs/pull/1824

I don't think Rust has a problem of having too many crates yet, and hopefully crates.io will improve discovery before it becomes a problem. They're making first steps to improve it already, e.g.: https://github.com/rust-lang/rfcs/pull/1824

I can't say I agree with the author's first article (which he mentions in this one), but I think he has the kernel of a good argument here. That's not to say I agree with it entirely.

At some point "design by everyone" turns into "design by committee" by another name. This criticism of Rust has manifested itself in other ways, too. For example the amount of "unsafe" used in various crates (the idea being that "the community" will collectively minimize "bad" use of "unsafe"). In my view there's a tendency for some of Rust's advocates to simply assert that it will work itself out without presenting good reason to believe so.

Do you have any numbers on the amounts of unsafe? I would be interested in a decent methodology to figure this out. I've seen people assert that this is true in the same vague way you're referring to for the other perspective.

I haven't any numbers and I wasn't asserting anything about the quantity that exists.

Cool. If anyone else has ideas here, get in touch.

My company has to approve all software on the development network. Getting newish versions of boost is bad enough. But keeping something like tokio or even simd and all its dependencies approved and in sync and then transferring them to the air gapped dev network is just this side of impossible. Is my company an outlier or is this common in bigcorp?

Maybe we could have something like what Debian has to manage channels? Differents stages (experimental, testing, stable) and a group of official maintainers? I know that's not easy to create that kind of communities and that you née a lot of ressources (benevolent people). But that works great for Linux distributions.

>My use case – foundational network infrastructure with planning horizons on a decadal scale – needs stability guarantees that Rust is not yet equipped to give.

Can you be any more pompous?

He may be talking about NTP -- that is core infrastructure code that had been poorly maintained for many years, and he's been involved in the rescue effort. Points off for not saying so, though; without context, your reaction is pretty natural.

The thing is he is attacking Rust not based on the safety the language provides, but more on a feeling that the package, crates, network doesn't fit his belief of how things should be curated. He states that he thinks this is why BDFL is good model, but gives no strong evidence supporting that view.

Rust has curation in its crate system. It's all in really good crowd sourcing numbers. Crates.io, number of downloads, with trends (i.e. Even if it was popular, is it still?). Then GitHub has great ones: stargazers == community interest level; watchers == # of people interested in tracking the project, possibly b/c it's a critical dependency; forks == # of people who've reviewed the code; contributors == overall success in the community; commit graphs = how much work has and is going into the project; issues can help you understand if there are any critical underlying problems with the maintenance.

When I'm evaluating a library I look at all of these numbers to understand how comfortable I am in pulling a dependency on the library. IMO this is far better than the curation of some system level tools, which just become defaults, and don't always offer users the ability to evaluate different options easily, cron stands out in my mind as one that there are better options than the default.

Anyway, his loss... the Rust community is awesome, in my humble opinion.

His specific concerns might not be entirely valid, but it's not an attack. It's more like an expression of a deeper fear of the language not taking off and time wasted.

> The thing is he is attacking Rust not based on the safety the language provides, but more on a feeling that the package, crates, network doesn't fit his belief of how things should be curated. He states that he thinks this is why BDFL is good model, but gives no strong evidence supporting that view.

One might easily argue that folks who criticize C and C++ for its ecosystem are also "attacking" it, by your reasoning. It detracts from the rest of your comment in my opinion (which isn't humble at all).

And "the Rust community" is not a monolithic entity. For example, I find "the Rust community" in aggregate here on HN, other than a relatively small few, to be rather caustic, aggressively defensive against criticisms of Rust, and a bit too handy with downvotes for disagreement of legitimate arguments against some of Rust's features, syntax, grammar, etc.

'Attack' is a bit of hyperbole, so it's fair criticism, but I do feel that the article he wrote had much less to do with issues with the language, but a very monocular and subjective view of how ecosystems should be maintained.

He does have some good ideas in there about how to possibly to make crates.io help you more easily identify the best crate for your usecase, but I think the solution to that is a fairly detailed algorithm weighing the some list of values like I put above. Otherwise you end up never allowing new options to blossom.

Trying to read how what I put before wasn't humble, and perhaps your right on that point as well. The perseption of humbleness is definitely subjective, and the added color I threw in there probably in fact did remove any humility from the overall statement, which of course are my own personal views and I highly encourage people to question and criticize them.

I meant my opinion wasn't humble, not yours.

ESR can always be more pompous.

Reply on news.ycombinator.com