Hacker News Reader

Comment by raphlinus | Hacker News Reader

In my post, I specifically call on proc-macro support (syn and quote) plus rand (not all of rand though, just the "give me a random number" functionality that comprises 99% of the use cases but 10% of the implementation complexity) to be added to the Rust standard library. But I feel these are particularly justified because they're already present, just not accessible. Overall, I think Rust's "batteries not included" approach is a good tradeoff.

View on HN

I don't necessarily disagree. I have my own pet things that I'd like to see in std too. I've long wanted to see lazy_static in std. Funnily enough, it looks like `once_cell` might wind up being a nicer approach to achieving a similar end, but without using a macro. So if we had added it to std many years ago, we might find ourselves with an API that we regret! It's tough.

With that said, if `syn` finds itself in a spot where it needs to do a breaking change release for a new language feature, then that would be tricky. It sounds like syn's architecture is pretty flexible (non-exhaustive enums), but it's not clear to me that it could support all possible language additions without breaking changes.

I think I get the tradeoffs but while my company would be a perfect fit for rust, we do all our development on an airgapped network. Custom registries are a thing now, but picking and choosing which packages our IT will consider trustworthy and then taking the subset of that with a license our legal will approve an then prunimg things with dpendencies that are now missing is a huge task that needs to happen at regular intervals and probably leaves some pretty big gaps in functionality. Its a shame, I like the language.

At least from a licensing perspective, you should be all set. Virtually everything in the Rust ecosystem is permissively licensed.

Trustworthy is another thing altogether though. How do you handle this process in C and C++? Both of those languages have fairly spartan standard libraries as well.

> Virtually everything in the Rust ecosystem is permissively licensed.

I just did our license audit and 100% of our shipped deps were Apache | MIT. If you can clear those two licenses with legal, you should be good to go for virtually any crate.

Might be worth releasing my one liner for this if anyone else finds it useful.

Do it! This is HN after all, no better place to gain free karma and appreciation for your command line prowess

That does sound pretty useful!

Please do.

The spartan C standard library was what my comment regarding 'small languages' was referencing. I don't have much experience with many of C's contemporaries, but for an opposing view take for instance the standard library of Ada ( I use the term 'standard' here to connote the library that is mandated as per the standard of these languages ). It is definitely orders of magnitude larger than that of C, and takes a fundamentally different perspective. Ada's standard library, while dated by modern standards, looks more like it was designed to address the specific needs of its domain. Mind you, it could be argued that C's stdlib was also, if you restrict its domain to 'OS development'. My point is that for a language like Rust, that does not have a tiny stdlib like C, its standard library should address the common use cases required by its developers. Node.js is another bad offender here. If you look at the most popular npm modules, you'll see things like 'body-parser' and 'async', which clearly show gaps in the functionality that Node's stdlib is catering for.

Which is only true if we forget about POSIX support that everyone kind of expects in every platform that offers C compilers.

Actually they don't, C standardization just dumped that standard library into POSIX instead, which I why all major OS end up supporting POSIX if they want to make C developers feel at home.

And you usually see them cursing the platforms that don't care about POSIX support.

And ISO C++ has repented themselves from following C's footsteps and have been improving the standard library since C++11, mostly by integrating boost libraries.

I don't understand what point you're trying to make. All I did was ask a simple question: if trust in Rust is hard, how do you handle trust in other ecosystems? The details on exactly how big or how small the standard libraries are (or why they are that way) are less important for this particular question. Consider the size of Python's standard library vs C or C++ or Rust. The size of C or C++ is much closer to Rust's size than Python's size.

Also, while some of your historical context is interesting, I don't really appreciate your editorialization, which I often find is off the mark personally.

My point is that although many only think about ISO C libc when talking about C's standard library, the reality is that with a few exceptions, libc always goes alongside POSIX across the large majority of platforms with a C compiler.

So in reality POSIX complements libc as C's "runtime platform", even though ISO C never considered to make libc that big.

My historical context is how I experienced how things went, throughout the media we had available at the time.

I surely welcome factual corrections when I fail off the mark.

Everyone benefits from learning proper history facts.

> My point is that although many only think about ISO C libc when talking about C's standard library

I wasn't. Even with POSIX, it is spartan by today's standards. Look at the standard libraries of Python and Go. POSIX doesn't have JSON (de)serialization, HTTP servers, XML parsing and a whole boatload of other crap. So I don't think there is anything wrong with my characterization.

> So if we had added it to std many years ago, we might find ourselves with an API that we regret! It's tough.

That is the ultimate problem of batteries includes vs a small standard library distilled into a specific example.

People used to tout Python's "batteries included" line as positive almost universally a decade and more ago. Then better replacements were developed, and those batteries started looking less appealing.

There's really no escaping that while including any base functionality. Either you include it and portions will be stale later as better interfaces and paradigms are developed, or you don't, and you risk slower adoption and harder usability as people need to figure out solutions for common tasks, even if that solution is as simple as find the crate that provides it. Because eventually that becomes a problem not of finding the crate, but finding the best crate out of the multiple that exist, which itself causes fragmentation of the common developer experience (and thus makes it harder to share knowledge and have a good community).

> not all of rand though, just the "give me a random number" functionality that comprises 99% of the use cases but 10% of the implementation complexity

Are you suggesting including the core `rand` crate with some traits and a basic pseudo-random generator in the stdlib, and allowing other crates to use those traits to implement the many other[1] RNGs?

I'm honestly not sure this is a good idea, it seems the rand crate has undergone a few iterations, crystalizing this inside the stdlib might lead to problems?

1. https://rust-random.github.io/book/guide-rngs.html

I admit, design of a proper random number library API is tricky, and experience from deploying rand will no doubt be invaluable. The point I was trying to make is that some use cases require sophistication such as being able to choose different algorithms, but a lot of the times rand shows up in a build time histogram, it's just because the user wanted some pretty good random numbers; a much simpler API would suffice.

I agree that in many cases a much simpler API would suffice, but it strikes me as odd to suggest adding a convenient but insufficient-for-security API to the same standard library that protects HashMap against DoS.

Having a very simple API is not at odds with security, though. A simple API could automatically seed from the OS and then run a tiny loop with sha2 or sha3, for example.

SHA2 is not designed to be used as a RNG. SHA3 might be a little bit better with its "streaming" modes, but there are many far better and faster ways to get random bits.

It doesn't have to be the absolute fastest, it just has to work.

What is far better and faster than SHA3?

And while SHA2 wasn't designed for that use, it's easy to make simple and provably correct constructs that turn a secure hash into a secure RNG.

You’re speculating to prove a point.

I would definitely not want my std to be designed this way.

1. It was just an example.

2. You definitely don't want your std to be designed with a simple API to a fast, secure, and popular crypto primitive? Pretend I named your favorite one, to avoid bikeshedding issues.

ChaCha20 is widely used for CSRNG.

Agreed, that would work fine. But for a generic RNG function, I feel like the difference between 3 and 7 cycles per byte isn't too impactful.

The request here is for a simpler API. I and many others would be more than happy to get a function that returns a random u64. No traits, no crippling commitments to a specific API design.

The API you describe, IIUC, can’t be serviced faithfully on the OSes targeted by std while being backed by real entropy.

Which is fine. Not everything needs SecureRandom. I'd wager more code using random numbers is for A/B tests, sample data generation or video games than people implementing crypto.

I get that security people don't want non-secure RNGs to exist out of fear that someone might try implement security related functions out of them but why should I care if I just want to choose between 3 types of enemies this wave?

But the std hashmap implementation[0] already depends on a PRNG via the rand crate[1], and thus OS random (for the initial seed). So it's in the standard library even if you can't use it directly. Which is what makes me think this is an API issue, not an implementation one.

[0]: https://doc.rust-lang.org/std/collections/hash_map/struct.Ra...

[1]: https://docs.rs/rand/0.7.0/rand/rngs/struct.ThreadRng.html

It does depend on this yes, but the random seed isn’t part of the public API. An RNG API in std would need to pick an algorithm and make that part of its stability guarantees.

In this case I'd quibble that it wouldn't need to guarantee a specific algorithm, just a minimal set of properties. If the user can't manually set the seed then they'll be no expectation that it's deterministic or follows any particular algorithm. So the precise implementation can change with versions or even platforms.

Javascript's Math.random and Crypto.getRandomValues works this way.

Id say that in Rust’s case, we’ve already seen several painful examples of Hyrum’s law.

http://www.hyrumslaw.com/

Why can't a simple API be backed by real entropy?

(past edit window) This was written with hostility I don’t feel. Sorry Raph!

No worries, it was read with offense I didn't take. I agree, designing a good random number API is in fact hard. I just think we can give developers a better out-of-the-box experience.

I would like getrandom[0] to be in the standard library. It simply abstracts over the platform random and is even used by the standard library itself.

[0]: https://crates.io/crates/getrandom

Inclusion of parts of rand might be a good idea, but syn seems to need breaking changes as for long as the Rust language is getting new features:

> Be aware that the underlying Rust language will continue to evolve. Syn is able to accommodate most kinds of Rust grammar changes via the nonexhaustive enums and Verbatim variants in the syntax tree, but we will plan to put out new major versions on a 12 to 24 month cadence to incorporate ongoing language changes as needed.

https://github.com/dtolnay/syn/releases/tag/1.0.0

Since stdlib and compiler releases are tightly coordinated, that shouldn't be a problem, no?

How do you accommodate breaking changes to the public interface of std?

Right, I see your point here. There are non-trivial issues that need to be worked out, which I'm sure is one reason it's not in the library yet. Another (mentioned elsewhere) is that it takes time to converge on what a good API might look like.

Well, with non-std I can always downgrade or upgrade the library.

With std I am bounded by the current language version.

You can only down or upgrade if uour dependencies don't depend on it too.

Having a defacto standard library has more issues IMO than a real standard library.

My core argument is that libraries usually do need breaking changes, and including them into std turns them into zombies that can never be removed. I think python has issues with this. And the syn example that I pointed out above shows that Rust isn't immune from this either.

Reply on news.ycombinator.com