Fast Rust Builds
matklad.github.io
matklad.github.io
pub fn read<P: AsRef<Path>>(path: P) -> io::Result<Vec<u8>> {
fn inner(path: &Path) -> io::Result<Vec<u8>> {
let mut file = File::open(path)?;
let mut bytes = Vec::new();
file.read_to_end(&mut bytes)?;
Ok(bytes)
}
inner(path.as_ref())
}
Nobody seems to care, but there's a crate that does this for you https://github.com/llogiq/momo
But the Rust compiler should be doing this automatically :(
Not saying it's good or bad, just how Rust ended up where it is.
I disagree it's a pejorative term as Wikipedia puts it though. Design by committee is in many cases necessary or even wanted (standard bodies).
# Your big function has specific types
function my_function(x::UInt, y::UInt)
[ ... ]
end
# Wrapper function to minimize monomorph. cost
my_function(x::Integer, y::Integer) = my_function(UInt(x), UInt(y)) * The output type must be explicitly named instead of inferred
* The let keyword for creation of variables instead of merely assignment
* The mut keyword to distinguish between mutable/immutable variables
* The pub keyword, instead of another mechanism
* Semicolons needed after every expression
That isn't syntax. It's a decision about analysis.
> The pub keyword, instead of another mechanism
The "other mechanisms" discussed are in fact not providing this feature, but pretending you needn't care, and I'd argue Rust shows that in fact you should care and pretending otherwise is a problem.
> Semicolons needed after every expression
On the contrary, semicolons are at the end of Rust's statements. If what you've got is an expression you don't need a semicolon. For example here's how log is defined on an f64 (a double precision floating point number)
pub fn log(self, base: f64) -> f64 {
self.ln() / base.ln()
}
I don't believe Julia or Python is "best practice" over Rust. I even empathize with the opinion that explicitness may be preferential sometimes. But you _must_ be able to see how Rust's syntax overload obfuscates the core business logic, right?
Sure, business logic is more readable in something like OCaml than in Rust. On the other hand, Rust isn't really worse than something like Java, and I like having more information than what you have in Python. I'll also add that Rust is often chosen for speed and reliability, and that part of that syntax is here to guarantee that. It's not directly business logic in the traditional sense, but it's something you expect from your code.
All in all, Rust is still not at the level of the "platonic ideal for syntaxes", especially not for most of the code I write (business logic for web apps, which wouldn't need Rust's speed). Still, in the realms of programming languages that are used, it's one of the best.
FWIW `__` doesn't mean "this is part of the implementation", it means "I'm designing a class for inheritance and I don't want child classes to stop on my attributes".
No. Python certainly does not have a concept of private code[0]. The Python community has a suggestion of things being implementation details.
Surely you can see that absolutely would not be good enough for a language which strives for static correctness? And even more so as the soundness of `unsafe` blocks often relies on the invariants of the type it's used in?
[0] well that's not entirely true, I guess there's native code.
fn f(x: impl Test) { ... }
fn f<T: Test>(x: T) { ... }
That pub? It's punching waaay above its weight! Having things private by default protects against semver hazards too, but it's also the major tool that enable writing safe code that uses unsafe { .. } blocks underneath. That's because a single unsafe block typically taints the whole module it's contained (such that even safe code needs a safety review). Being able to write safe abstractions of unsafe code is the pull of Rust so people need to get it right.
For example, setting the field len of a Vec to a value larger than what's initialized in its buffer will cause UB when accessing that memory; if this field weren't private, making a safe Vec would be impossible (likewise, a function that could set the len to any value must be private, etc).
So pub can only be used for things that are safe to export, and people should be careful with that.
Here's my least favorite syntax from there: that damn Ok(..). Ok-wrapping sucks. There were proposals to remove it, but they were rejected. There's a library that removes it [0], and people should use it. It also makes the Result<..> return type implicit in the function signature too.
[0] https://boats.gitlab.io/blog/post/failure-to-fehler/#but-abo...
fn read(path: &Path) -> io::Result<Vec<u8>> {
let mut file = File::open(path)?;
let mut bytes = Vec::new();
file.read_to_end(&mut bytes)?;
Ok(bytes)
}
func Read(path *Path) ([]byte, error) {
file, err := os.Open(path)
if err != nil {
return nil, err
}
slice, err := io.ReadAll(file)
if err != nil {
return nil, err
}
return slice, nil
}
[edited: I said there wasn't a io.ReadToEnd extracted from io.ReadFile in Go, but jatone pointed out it's called io.ReadAll.]
(defn Read [path]
(io/ReadToEnd (os/Open path)))
File::open(path)?.read_to_end()
Unsurprisingly APIs can be more readable when you don't ponder why they're designed the way they are and completely ignore their purpose.
The `read_to_end` API[0] was designed such that the buffer could be reused in order to avoid unnecessary allocations.
There are helpers for common tasks, though people may not know or notice them. In the case of `Read::read_to_end`, it's called std::fs::read[1], which actually preallocates the vector based on file size internally so might be faster than hand-rolling `read_to_end`.
So here's your code in Rust, for the last 3 years or so (since 1.26 was released):
fn read<P: AsRef<Path>>(path: P) -> io::Result<Vec<u8>> {
std::fs::read(path)
}
is the golang equivalent.
let bytes = std::fs::read(“path.txt”)?;
fun read(path: &Path): io.Result[Vec[UInt8] {
var file = File.open(path).orReturnErr()
var bytes = Vec.new()
file.readToEnd(&mut bytes).orReturnErr()
Ok(bytes)
}
Method syntax is reserved for functions. There are no method syntax macros. Since a return in a function terminates execution of this function, and not its caller, your example would not work.
let a: Result<u8, u8> = /* whatever *;
let b = match a {
Ok(v) => Ok(foo(v)),
e => e,
};
let a: Result<u8, u8> = /* whatever *;
let b = a.map(foo);
await cannot be a macro, it has to be built into the language, and so it doesn't have the ! because it's not one.
"orReturnErr()" seems quite against the Rust philosophy though — refusing the method call syntax for early return means folks can easily miss something important. Rust uses ? for error handling, ! for macros, and keywords (with no parenthesis) for other control flow. I prefer it that way.
file.read_to_end(&mut bytes)?;
By being explicit about this, Rust becomes easier to read, but (slightly) harder to write.
Because it's not unnecessary? `bytes` is a `Vec`, not a `&mut Vec`, those are different things, and Rust generally avoids magically doing things under the covers. So it doesn't auto-ref (let alone auto-mut-ref) parameters, only subjects.
Things<Like<This, That>, That> -- as in C++
Instead of a more legible (and very familiar to Rust authors)
Things (Like This That) That -- as in Haskell
Inherited from C++, we had also :: for namespacing (while a dot . could be perfectly be used for that), and tons of other stuff like <> for scoping generics.
Without some of that we could have:
pub fn read<P: AsRef Path>(path: P) -> io.Result (Vec u8) {
fn inner(path: &Path) -> io.Result (Vec u8) {
let mut file = File.open(path)?;
let mut bytes = Vec.new();
file.read_to_end(&mut bytes)?;
Ok(bytes)
}
inner(path.as_ref())
}
For doing better than that we could drop C-like syntax perhaps?
The trouble is that languages have a weirdness budget - there's a finite amount of novel stuff a language can have until it becomes too weird for mainstream programmers. So language designers have to pick their weird features carefully. Spending it all in novel syntax is unwise.
I think that at the time the Rust pull was safety for low level code and, as Rust was targeted to replace some C++ components, following C++ syntax was seen as a good idea.
The syntax of type application and function application is already different in rust: F<A, B> vs f(a, b).
Type application even has default and named parameters! F<X = A, Y = B> and F<A> if B has a default.
I just think this particular type-level syntax is very noisy, specially because types in Rust tend to be very nested.
And that was a good decision. If it were too far from the C++ syntax, I would not have learned it, because as a mostly C++ guy at the time, some similarities with C++ helped me keep interest in Rust. Also, I was unable to parse Haskell's "type parameters as function applications" syntax, which was purely confusing. The Rust syntax is an abomination, that's true, but it is a necessary evil.
It's more like this,
Thing<A> becomes Thing A
Thing<A, B> becomes Thing A B
Thing<A, B, C> becomes Thing A B C
The parens only appears when you want to further nest types
And.. okay, I reckon that the Haskell syntax for types would be unfamiliar. But, at least, it's not as cluttered
let x = Foo::new();
x.bar();
Foo::bar(x); pub fn read<P>(path: P) -> io::Result<Vec<u8>>
where
P: AsRef<Path>,
{
Compare:
def read(path):
pub fn read<P: AsRef<Path>>(path: P) -> io::Result<Vec<u8>> {
What it read? Anything that can be pointed with that PATH
What it return? Vec of bytes or a IO::Error
PLUS: This code is compiled tot he exact types that call it, Path not allocate, the result can be cloned, the errors are defined in IO, ....
--
Rust is very readable: Almost any line tell you what is happening and what expect.
Where it get messy is when it hide that crucial info. But is not many times it happens.
Your example is also less about Rust and more about type signatures.
No, is HOW Rust use type signatures. Rust is in the camp of "advanced usage", not like C#, Python and others, where a "type" is barely for classification. Instead, Rust use types to inform: Memory type (heap vs stack), Cost (cheap to copy or must clone and can be potentially very big), Permisions (read/mutable), Behavior, Share-ability (can go in threads), lazy or not (iterable), etc.
It use types for a lot, and communicate much information densely.
P.D: Not always perfect and I agree that the syntax is not as nice to me, but is incredible how much you can understand from most codebases just reading, even on docs (not need IDE assistance that much)
No, I don't really agree. The example was "def read(path)", which is intentionally vague and any typed language would clarify.
Your points can be/are valid, but they don't really change the fact that the above example was a cherry pick, and Rust's (over)use of symbols and special characters make it, again, about as hard to read as one could imagine. Just because you _can_ derive meaning from it doesn't change that.
From your post:
> What is path? Anything that can become path on the file system.
> What it read? Anything that can be pointed with that PATH
> What it return? Vec of bytes or a IO::Error
Certain idioms not make much sense at first, but a big part of be on Rust is learn them. Eventually, most of it make sense and are very predictable (is incredible how much adherence exist for them in the whole community!).
Blog post idea: express this snippet (with references, aliasing control, generics and all) in pseudo C++, Java, Python, etc. Then express “reading a file in gc language with exceptions” in pseudo Rust.
A proper language needs to allow the programmer to choose between implicit effects and explicit effects on a case-by-case basis. The reason ? is so tragic is because Rust tried to go with explicit effects only, but found that people wanted implicit effects, and ? is the closest that Rust can get. That's why it's syntactic noise: The programmer wants to get rid of it, they don't regard it as informative for this case, but they can't get rid of it.
(And checked exceptions can be perfectly well implemented with regular data, you don't need any special cases - in fact, you can do it with the same infrastructure that's already been added in a special-cased form for async/await support...)
Also, being able to call unsafe code inside unsafe functions is now considered a design mistake, and there is ongoing work to revert the default – requiring unsafe blocks even in unsafe functions: https://rust-lang.github.io/rfcs/2585-unsafe-block-in-unsafe...
[1]: https://docs.rs/anyhow/1.0.42/anyhow/trait.Context.html
This will become a lot easier once the actual full const-generics implementation lands. Because then even your "non-generic" implementations will be enabled to depend on compile-time-defined features of the instantiated type (such as size, alignment, value bounds etc.) without any sort of duplicate code generation. Of course, these in turn should delegate to truly non-generic code when feasible to minimize bloat.
This is so true, and so hard to convey to a lot of people, even in engineering. They like to abstract humans as machines that can just context switch at a whim, which is just not true.
It also models your source code in your build system (exposing libraries, binaries, etc).
I'd love to see a performance comparison with `rules_rust`.
There are also a number of other open source alternatives such as Buildbarn (https://github.com/buildbarn) and Buildfarm (https://github.com/bazelbuild/bazel-buildfarm) which you can host yourself.
Personally, I've found Buildbarn great to throw up in Kubernetes and build massive projects with.
I haven't really used Bazel much, but the features it brings are really huge: reliable pure compilation, so you can actually do globally cached and distributed builds reliably.
Maybe I should try it.
It's worth noting that there's a sort of uncanny valley in the amount of time something takes: when it's short, it's not too big a deal to wait for it. When it's fairly long (and predictably so), it's easy enough to context switch (I used to read a lot waiting for compiles to finish). It's the stuff in the 5–10 minute range that kills you.
What depresses me, is that while I can now do something else in a different window while a process runs, the issues with errors and requiring interaction still persist.
As a library author, I sometimes wonder if I should be introducing more feature flags into my crate to let people only use the bits that they want, in order to help compile times. The whole thing doesn't take very long to compile (~20 seconds), but maybe the added complexity of juggling feature flags might be worth it.
I also had a project a couple of years ago where I had to compile it in release mode to get reasonable performance for testing (Iw as processing GB's of data), which meant 2-3min incremental compile times. Not fun!
Some smaller libraries I've seen that just do some annoying-to-test but simple string manipulations have had similar compile times, mostly because they include specific versions of advanced logging or error reporting libraries for no obvious reason.
For example in the `procfs` crate, there's a dependency on `flate2` just in case a /boot/config.gz file exists (in order to see what options the kernel was built with). But if a consumer never needs to inspect the kernel options, this dependency (and related code) is just dead-weight.
The nice thing about rust is that this code will be removed from the final executable if it's unused, but maybe it's time to start thinking more carefully about compile times.
20 seconds once per machine per dependency version (i.e. there is no rebuild of the dependency until you change compiler flags or update the library) is OK.
Your 20 second library is not too bad when it's used on its own, but add 5 or 6 libraries that all "only" take a few seconds and you're waiting a solid five minutes compiling other people's code and their dependencies.
Even simple Rust programs quickly end up compiling several versions of several libraries because of the dependency tree, some of which can include large libraries like Tokio and Serde. I love libraries that make use of feature flags, they massively cut down on my compile time.
This isn't really any library developers' fault, it's a problem with the entire ecosystem. Cargo is not dissimilar from NPM where you can quickly end up with an exploded dependency chain of hundreds of megabytes in supporting code before you even realize.
Features are super neat, but it’s easy to see them overused. It’s also a bit hard to adequately test arbitrary combinations of features. It’s a different concern from a library, but we saw this with one of our applications. We had a situation where we were using features where we should have just been using env vars for feature flagging, and it caused a fair bit of hassle. We’ve since ripped out that usage, because we never in practice tested or deployed with anything other than “all features,” and even in the case where we want to use some functionality to upsell, enabling an env var and restarting is way easier than recompiling the project.
In particular, please for the love of god don't put the whole kitchen sink into the default feature. I'd even argue don't ever have a default feature at all. In practice, once a codebase gets above a specific size, it becomes completely impossible to exclude any features in a crate's default list because `default-features=false` only applies locally. Something 5 levels deep will bring those features in whether I like it or not just by not specifying anything.
Also, please don't do things like how slog uses a feature flag to control minimum log levels, or k8s-openapi uses it for version selection. These kinds of usages create extremely frustrating problems.
Honestly, features -- and especially the way default features work -- are the worst thing about cargo to me. I think they should be a lot more restricted than they currently are. If I had my way you'd only be able to use `#[cfg(feature = X)]` at module level to include or exclude code, never to select actual behaviour.
To me they're filling a hole that crates themselves should be filling. Certain language features aren't really rich enough to make really effective 'additive' crates though, unfortunately.
Curious, what that specific size approximately is? For rust-analyzer, getting all the deps to use features correctly is rather annoying, but isn’t particularly time-consuming.
If you depend on crates A and B, and B depends on A, and you specify `default-features=false` on A but B doesn't, you will always get A's default features included in your compile. There is no way to prevent this.
Personally, dynamically linking dependencies and using the `mold` linker has given me the most bang for my bucks. You can't dynamically compile all crates (actix_web, sqlx) unfortunately due to some assumptions macros make about crate paths (using Bevy scheme). And this exclusion extends to all crates pulled in as [dev-dependencies].
Consensus seemed to be that reqwest is the way to do this using rust, since the stdlib does not include an http client.
Adding the dependency on reqwest tried to pull in over 100 dependencies and after building for a while, it failed somewhere along the way because my rust version was too old. Now, maybe some of these were optional dependencies, but they all got pulled in automatically.
so, when TFA says
> Another obvious advice is to use fewer, smaller dependencies.
I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies.
That _is_ the tradeoff - Rust builds up from nothing and makes it friction-less to add a new dependency. Go starts at a higher level and the friction for adding new stuff is high. And some of the high-level stuff, like the GC, can't be removed.
How often are folks writing applications that don’t need to connect to the web these days?
For Go, perhaps rarely. For Rust, not needing to touch the web is extremely common.
Let's also keep in mind that Rust's stdlib does include IP, TCP, and UDP.
Maybe the lesson here is to update Rust if you haven't used it in a while?
In Go? Rarely, I would expect. In Rust, I'd assume connecting to the web is the exception rather than the norm. Most Rust projects tend to be lower-level infrastructure & systems programming.
This probably explains, and in turn is explained by, their different choices re: http in stdlib. They have much different aims and target use-case.
Every day? I think the only one I've done that in Rust it was through the program I was wrapping anyway.
Not really, it's just a 'go get'.
Seeing how they are different languages, there's not a single language feature that explains this dependency gluttony.
I'd like to open up the option that it is a necessary side-effect of large ecosystems: more devs using the language, more packages published, more quality in small packages. And then, when building a large-ish package, it only makes sense to use existing packages rather than reinventing the wheel.
And so, driven by nothing but good will and best practices, since you have a lot of prebuilt "assets" at your disposal ,your project ends up with 100s of deps and everybody complains.
Yes, for instance one of `reqwest`'s dependencies is `base64`. We can argue both that re-inventing base64 in every big library is pointless, and also that a language doesn't need base64 in its stdlib because C and C++ don't have it.
There's no right way to cut this cake - Many a language has included something in the standard library, and then struggled to fix a broken interface once everything depended on it. (Like JS picking UTF-16 for strings) Many a language has also failed to include something in the stdlib and caused every application to suffer and re-invent it. (Like C++ not really saying anything about Unicode strings, which means Qt and everyone else has their own little take on it.)
And the suffering is only bigger if you make dependencies artificially hard to add.
Rust's approach is more flexible, at the cost of building the universe from scratch every time you make a new workspace. Personally I don't mind it. At least it's not like the problems I've heard from Haskell land. Moving from futures to async in Rust was fairly smooth.
Doesn't seem hard to work out.
So maybe it's harder to work out than you think.
And yeah, I doubt I would support including base64 in std unless there were a compelling reason to do so. I can't think of one.
And yes, use UTF-8. Not contesting that.
Rust gets a bunch of stuff right that other languages missed. I don't think this one of them. I think Go's strategy of having a pretty-ok version of most of the table-stakes things developers do is probably the right one, just like I think sum types and option/result are better than error types and `nil`.
I share your irritation at people who drop into the middle of discussions like these to nitpick random language war arguments, but this is a thread that is actually about Rust dependency hell, and the comment I responded to, I think, sort of diagnoses why Rust has that "problem" (if you think it's a problem --- I do, but whatever).
For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
Again: these are downsides, but not dispositions. On balance there could be good reasons to have a terse, svelte, balletic std. I'm just saying, there's clearly a downside to the decision.
Right, I partially agree, and this is a more substantive critique. But it's a much harder position to put into a pithy HN comment that sounds obviously correct. ;-)
With that said, it's pretty common in my experience to wind up in exactly the same sort of situation in Go-land. Go's standard library might hold it off for a little bit, but I seem to accrue large dependency trees there too.
> There might be an upside to that (I have not detected it)
Aw c'mon. The comment you were originally nit-picking even called it out. I'm sure you've heard the quip, "the standard library is where things go to die." And Python's in particular is often brought up as an example. Go's standard library hasn't suffered as badly as Python's has, but there are parts that have died.
If you've used Python a lot, then the fact that you have to use 'requests' and not std's HTTP client is probably not a downside. You've already digested that ecosystem knowledge. But as stewards of the standard library, this is something we see as a downside.
> For whatever this is worth, by the way, languages that have Base64 in std appear to include Ruby, Javascript, Python, Clojure, Java, Scala, Go, Elixir, Nim, and Swift. OCaml, C++ and Haskell do not.
Yes. Our design philosophy around std learns from some of those languages. Python in particular has influence in two respects. First is what I said above: when you have to make an HTTP request in Python, the standard advice is "use the third party requests library" despite the standard library having an HTTP client. This should make it clear that following your advice is merely a necessary criterion; it is not sufficient. Second is that "std should remain backward compatible for approximately forever." That means we have a very high bar for including things. (I am saying "we" here because I am on the Rust team that decides these things.) API evolution is more difficult in std then outside std, because we can't just release a 2.0. So if we get the API wrong, it sucks. And eventually a superior third party crate appears and everyone has to "learn" to use the third party crate instead of std.
And of course, adding things to std also means a higher maintenance burden. There have to be people willing to champion and maintain these sorts of huge additions. And many of us (now speaking as the regex crate maintainer) are actively opposed to throwing it into std precisely because it becomes more difficult to evolve.
Even if we did all this and put whatever pet thing it is you want to use without pulling in dependencies (I guess that's "HTTP client" here), that doesn't really solve the general problem. Eventually, you're going to pull in a dependency for _something_. And that crate's author may have decided that depending on something else instead of rolling their own thing was the better choice. Or also the opposite: perhaps they want to split apart the internals of their crate so that others can reuse those components. I did that with the regex crate. So on the one hand, I'm contributing to the high dependency count number, but on the other, I'm solving actual problems people have by providing access to internal APIs that are separately versioned. How many other regex libraries do that? Not many. Because dependencies are too annoying in the C ecosystem, which is where most regex engines live.
Getting more hand wavy... In the Go world---and I speak as someone who has used and enjoyed Go for over a decade now---performance is not a #1 concern. Not like it is in the Rust ecosystem. This has huge API implications. Culturally, people expect Rust programs to be fast and they expect to be able to achieve congruent performance as they could in C. In this environment, API complexity isn't much of a reason to not doing something. Everything gets sacrificed at the alter of performance. I know, because I've been carrying sacrificial lambs to the alter for years in the Rust ecosystem. But in Go? Nope. API simplicity is valued higher relative to things like performance.
Here's the punch line: if you want to write a program that sends an HTTP request and has a dependency count of 0, then Rust isn't the right language for you. It probably never will be.
Also, to be clear, I am someone that works both sides of this debate. I can often be found advocating against the use of dependencies and decreasing dependency count. Precisely because I perceive and experience costs associated with them. I've done work to reduce the dependency tree size of core ecosystem libraries. But there are some fundamental trade offs that put floors on how long you can go. And you can also seen me as an advocate against using many of my crates if a "simple" solution works for you. Take aho-corasick for example. My implementation of it is several thousand lines because I sacrificed just about every lamb possible at the alter of performance. But the <10 line naive implementation of multiple substring search might be just fine for your use case. And if it is, just write that and don't bring out the big guns.
You're right that Go has some degrees of freedom that Rust doesn't have, especially on performance. But people do have perf problems with the Go stdlib, and they replace components (amusingly, one of the first things I built in Go did its own sockets and its own polling because Go's timers were too expensive for what I needed).
I just think it's better to have a good-enough stdlib --- with well-thought out interfaces where you're not going to have net/http and then net/http2 and then net/http3, and 16 different ways to run a subprocess and read the output --- that people can optionally outdo with 3rd party dependencies, than the situation I think Rust is in right now, where you basically can't do anything without a bunch of dependencies.
I'm definitely not dunking on Rust. We do a bunch of Rust here and I mostly enjoy working in it.
In no small part because it was born 20 years later: lots of modules were added a bit haphazardly to the Python stdlib early on and the tech ended up mostly dying, but for the most part they're still there in case you care to use them. Like sunau, aiff, or dbm. Others were added because package managers were not really a thing back then so you'd try to get useful packages into the stdlib so they could be used easily, but then the maintenance burden increases a lot (and all the users assume the stdlib is maintained anyway).
String encoding is pervasive, and if the standard library didn’t have strings or didn’t specify the encoding, there would be much grief throughout the ecosystem, because you can’t just patch it in at the library level, it’s a language-level feature. So string encoding is a choice that has to be made.
Base64, on the other hand, is just a bunch of functions, nothing special about it that means it has to be in the standard library.
When I say that something “has to be in the standard library”, I mean that it can’t be implemented outside the standard library. That’s certainly not the case here. You’re using an outright bad definition of “need” here—subjective opinion rather than objective requirement.
> because everyone needs it
This is factually wildly wrong. I wrote a fair bit more here but decided it wasn’t helpful. Précis: web stuff tends to load it indirectly (though amusingly most of the time actually not use it, so that Base64 code won’t actually end up in your binary), but it’s not terribly common outside of internet stuff to reach for Base64.
I’ll leave just one more remark about Base64: once things are in the standard library, breaking changes can no longer be made; the base64 crate is still experiencing breaking changes (<https://github.com/marshallpierce/rust-base64/blob/master/RE...>, 0.12 and 0.13 were last year and 0.20 is not released), largely for performance reasons.
Please don’t just call the thin-std approach “problematic” without acknowledging that the alternative is at least as problematic, just with a different set of caveats.
It also seems to be distinctly losing popularity as time goes on, as the alphabet is not restricted enough to be easy for a human to sight-read/copy, and a wider alphabet will have as good an ASCII channel compatibility.
As it stands, by being external, people can depend on base64 0.11, 0.12, 0.13, &c. and it all works fine, and over time they’ll tend to migrate to the latest version (dealing with the generally minor breaking changes involved) but it doesn’t matter so much.
The incompatible interfaces thing can be a problem in Rust and does require a bit of care, but in this particular case isn’t a serious problem; the base64 crate is almost always going to be implementation detail, not exposed publicly. With serde, yes, reaching a stable 1.0 was definitely more important for ecosystem compatibility.
Further: this seems like an easy problem to fix, much easier than "Go lacks match statements and generics" or "Python lacks meaningful typing altogether". But my guess is that culturally, it's just as hard to fix.
Besides that, you’ll normally be dealing with the latest version, and if you deal with an older version then you should normally consider upgrading it. I looked through a few projects of mine with base64 deep in their dependency trees, and it was all 0.13 except for one instance of 0.12 via a dependency that needs updating.
You’re stubbornly ignoring the problem of freezing base64 into the standard library, apparently obdurately refusing to even acknowledge that a thick std is every bit as problematic as a thin std, just with a different set of caveats.
If you freeze base64 into the standard library, it’s frozen. No more even slightly incompatible changes can occur, and so you’re stuck with something inferior for ever after. This is exactly the urllib/urllib2/urllib3/requests situation. This is the “standard library is where things go to die” problem.
Rust’s chosen standard library strategy is in no way wrong; it’s just a different set of trade-offs, and incidentally one that experience says is decidedly preferable for a language of Rust’s complexity (wherein there is routinely not one simple and obvious way of doing a thing, in contrast to languages like Go, Python and JavaScript which have distinctly lower hazards to freezing things into the standard library, yet even their standard libraries have wilted in places).
I don't think you've thought this all the way through. This last response of yours seems more reflexive than contemplative; for instance, it suggests that breaking changes are no big deal, because they're small breaking changes. That's not how breaking changes work! Indeed, I feel like everyone who's ever worked in a totally undisciplined package ecosystem (I'm not saying Rust is one of those) knows what the end result of this is: hours of brain surgery figuring out how to reconcile incompatible dependencies-of-dependencies.
nix is not a good example. Nix gives _additional_ APIs that std doesn't provide that are specific to Unix.
This isn't about std-having-base64 preventing the use of external third party implementations. This is an absurd interpretation of what chrismorgan is saying. They're saying that std's implementation is frozen forever. We can't change it. One of three things occurs:
1. We got the API right and everyone is happy in every use case. 2. We got the API really wrong and now we probably need to deprecate it. 3. The API is too simple to permit maximal performance, and now people need to know a decision procedure to choose between std and a third party library.
You seem to think (2)+(3) is okay, but I'm actually not a huge fan of either. (2) causes churn and ecosystem pain. (3) causes confusion and becomes the butt of a joke: "yeah you actually shouldn't use the base64 stuff in std since it's so slow, so just use FooBar's implementation from crates.io instead." It's not unlike the JSON situation in the Go world: I am routinely having to replace the standard library's JSON implementation with a third party one that's faster. The difference is that "good enough" in Go-land is much more culturally acceptable than it is in Rust-land. In Rust land, as I said in an adjacent thread, everything gets sacrificed at the alter of performance. When people come to Rust, they generally expect to get the fastest of everything. Otherwise, it's not good enough to supplant C or C++.
The result of this is that adding things to std is a risk. So when we add things, we need to do a risk analysis and evaluate its benefits against its costs. When you nitpick a specific example like base64, it's easy to lose sight of the bigger picture here. Bringing base64 into std wouldn't solve the problem you're complaining about. (If it did, I have no doubt we'd do it, because it's base64. Probably we can get that right.) What you're really advocating is a philosophical/cultural change that will dramatically increase the size of std, and thus increase the risk we take with respect to introducing things that people will ultimately wind up not using. We don't want to do that. We want the ecosystem to work that out so that they have room to evolve.
> knows what the end result of this is: hours of brain surgery figuring out how to reconcile incompatible dependencies-of-dependencies
This generally only occurs for public dependencies. base64 is very unlikely to be a public dependency. Folks who maintain foundational public dependencies are generally quite careful about pushing our breaking change releases. (`rand` is a good counter example, but it looks like they've slowed things down as of lately.)
I'm not a huge fan of taking forever to get to a stable 1.0 for core ecosystem libraries, so that's definitely a knock against our approach. I'd rather see folks commit to a stable interface for some years even if the API isn't perfect in order to minimize churn.
I can assure you that my position is not some reflexive HN drivel. I've been on the library team since the beginning. I've thought a lot about this and lived through the trade offs. And chrismorgan is right: this is a decision with trade offs. There is no obviously correct choice here. I probably hate bringing in dependencies as much or more than you do.
I don't follow this logic. If we just wanted to do everything the way C/C++ does it, we'd be using C/C++, and Rust wouldn't exist.
For anything else I don't have any major complaint.
[1]: <https://github.com/abseil/abseil-cpp/blob/master/absl/contai...>
https://docs.microsoft.com/en-us/visualstudio/debugger/how-t...
That is the thing, the standard doesn't require bounds checking for operator()[], while it does require for at(), but it doesn't forbid it either, it is up to the implementations.
Take expected (which missed C++ 20 but may land in C++ 23). The idea here is like Rust's Result type†, but C++ lacks a sum type so it's instead presented a bit like a smart pointer. You check it for errors and then just dereference expected to get your answer if there weren't any errors. Simple. If you can't be bothered to do error-checking, no worries, dereferencing expected when it's an error throws an exception, and your usual C++ exception handling applies.
But no, the committee says this doesn't look dangerous enough to us, so they required it to be rewritten to say dereferencing expected when it's an error is Undefined Behaviour instead of throwing an exception.
Imagine deliberately adding more Undefined Behaviour to a programming language (technically in this case the language's standard library) in response to work to compete with safer languages. It's an act of self-sabotage without question.
† Unlike exceptions, in Result your error is just more data, to examine, process, store and use now or later or not at all at your option. An exception changes control flow in the program when the error occurs, which may not be what you need at all. This is crucial to some designs.
As I understand it the runtime performance-at-all-costs school, which includes people from Google, don't want the C++ language to be deliberately unsafe they just don't prioritize safety over performance. This group won't allow C++ to have runtime bounds checking for the array subscript operator or its overloaded equivalents and I think they're wrong about that, but it's not making the problem worse. Stuff like ranges, views and iterators (even though C++ iterators are incredibly ugly) mean in idiomatic C++ today you're rarely reaching for this particular gun that's supplied loaded and pointing directly at your feet. In contrast what was done to expected just seems like self-sabotage. If it's too slow then don't use it. Or rather, as these same people would be the first to point out, measure to check your intuition that it's too slow and only then if it really is don't use it.
In all these years I have left RTTI enabled, used exceptions, tried to keep bounds checking enabled in STL types, and it was never an issue in production.
The question is whether that code is going to come from a monolithic Swiss Army Knife of a library (aka a standard library), a bunch of packages, or some code that everybody copies into their codebase. In which case the packages option doesn't seem so bad. At least they're explicitly versioned, tracked for vulnerabilities and can be updated at a granular level.
What seems to be missing is standard library as constellation of well designed modules/packages. I decided to just do it for myself (and anybody who finds it interesting). I don't think it's going to fly high but if somebody is interested the link is here [0]. I'm adding modules every few days.
The aim is to have something worthwile to show around the time node v16 goes lts. I have some experience taking care of high stake business critical systems. This is fun side project as a hobby atm. Modules being dropped there are reflection of problems I'm facing during my day-to-day work.
You'll see there some non conventional code. Large parts are inspired by functional programming, ocalm and simplicitly in general. Personally I find it exciting how well this kind of approach fits into production projects.
The nice thing about this approach is that you can have niche leaf packages/modules as part of standard library (constellation) - if you don't want to use it, it doesn't matter, it wont incur any cost. For example order book module or exchange matching engine - those can simply live there, be useful to people who care and have zero impact to people who don't care at all. I find it to be an interesting difference from builtin, shipped library.
Large package ecosystem is a sign of popularity. With time libraries should converge onto shared approaches. Interestingly practice shows wide spread of duplication. I think more effort from big players with wide reach could reduce it. Ie. not me doing stuff but Microsoft or others picking up the ball.
Other languages that don't face this, e.g. Python, have very large standard libraries which already include a dizzying array of features. E.g. take a look at some of the things you get with standard Python[0].
- sqlite3 (DB-API 2.0 interface for SQLite databases)
- bz2 (Support for bzip2 compression)
- sunau (Read and write Sun AU files)
- netrc (netrc file processing)
- curses (terminal handling for character-cell displays)
- mailbox (manipulate mailboxes in various formats)
https://leancrew.com/all-this/2012/04/where-modules-go-to-di...
HN discussion: https://news.ycombinator.com/item?id=3913182
The obvious compromise would be to let libraries bake as third-party projects for a good long while until there's a fair amount of consensus that they're the Right Thing To Do, then pull them into the standard library. Java's Joda-Time would be one example.
One recent build I did, a small library included a dependency to build its documentation. This was a full-fledged documentation project, which pulled in hundreds more dependencies.
It would be interesting to see a display of “cost” in projects indicating how much additional dead weight each individual dependency pulls in. This might lead to slight better decisions on dependencies to choose.
cargo-bloat is kind of along these lines, although it only measures the absolute size of each dependency (as opposed to "dead weight") and has some platform limitations.
It depends a lot.
Java has a huge ecosystem, but almost all of the core functionality is provided for you.
Like everything there's a 90/10 rule in that 90% of the 'things we need' really amount to only 10% of the total packages out there, so it actually makes sense that the people backing the platform provide for it.
Language, compiler, debugger, documentation, tutorials, standard libraries etc. - it's all part of the platform.
Looking at it that way, it becomes a little more obvious why even some cool languages don't quite breakthrough.
As another commenter pointed out, it may relate more to the size of the standard library than to the size of the ecosystem. Which your comment seems to emphasize as well.
Dude it's like one command to update.
They could at least package rustup, I have no idea why they don't.
Rust went for "evergreen" approach like Chrome. It focuses on making small gradual changes, quick and painless upgrades, and provides stability via backwards compatibility in the compiler. This allows everyone to only target one version: the current one.
Debian's philosophy is the polar opposite, so Debian's Rust version is doomed to be perpetually obsolete.
why not bash and curl?
Kind of like what smokeping does, just wanted to see what a from-scratch implementation that did just what I needed might look like.
Ended up being about 400 lines. Only dependencies were go-sqlite and go-chart.
Or... `rustup update`...
Raw dependency counts can be shocking for those coming from other languages where pulling in a dependency is an uncommon operation, but it's not a bad thing. I'd rather have my HTTP client library re-use popular, tested packages for things like decoding JSON, handling TLS, working with IP addresses, and other common tasks. I don't think it would be better to have every package roll their own versions of all of those common pieces.
You can browse the dependencies of reqwest here: https://crates.io/crates/reqwest/0.11.4/dependencies
Notably, several of the big dependencies are maintained by the same person who maintains reqwest. It's a common practice with Rust crates to split a project into reusable parts. Reqwest uses hyper (http client), http, http-body, mime, url, and several other crates that are owned by the person who owns the reqwest crate.
If we instead mashed all of those dependencies together into a single crate it would appease the people who don't like seeing high dependency numbers, but it wouldn't really make the product any better. It would make it harder to re-use the individual pieces, though.
> I just went back to go, which can do the crazy obscure task of making an http request without requiring 100 external dependencies.
Go is famous for including HTTP utilities in the standard library. If I was building a simple app that made HTTP requests and little more, I'd probably pick Go as well. Or Python.
Or you could have searched for a minimal HTTP request crate ( https://github.com/algesten/ureq ) designed to be small instead of using the most full-featured crate.
Or you could have searched for Rust libcurl bindings ( https://github.com/alexcrichton/curl-rust ) to use an old standby.
If your goal is to reach for something quick and easy with batteries included and minimal external dependencies, Rust is not a good choice. Doesn't mean Rust is "bad", but it's not the right tool for every simple job. You also can't just pick the most popular crate and assume it's exactly what you want.
Precisely this. Rust is not the same as Go or Java or Python: it's still a fairly low-level language. Standard practice is to build lower-level components (like http parsing and url crates) that are abstracted over by more 'friendly' crates (like reqwest). It's jarring, and certainly different from a lot of other languages, but it's a testament to the language's flexibility in my eyes.
Rust is a system language.
For example, one shock is that you can't print! or dbg! anything (without adding Debug/Display to each type).
The reasons?
Where it will print in a Airpod? yeah: Rust is made to work in places where NOT exist a place to print to. Or where NOT exist a filesystem, or HTTP because not exist TCP because not exist a network card.
So, in Rust all start in the deepest of the deepest bottom.
---
I have a good metric to select frameworks/libraries:
- If is too complex, NO except if is the only game on town
- If have crazy deps setup (ejem: OpenSSL, bastard mess of dep!) NOOOOO. (except if is the only game on town)
- If is too big, damm be good and solid (ej: PostgreSQL)
Whatever the lang, this have served me well, and when I have been lazy (I get a deps to OpenSSL and waste 1 week fixing CI builds with it then remember this and cut that mess) the pain quickly fix it.
So, next time you get a trouble like this (in Rust or whatever) ask what alternatives you can use instead!
I'd say, first of all, that it does matter, because very few projects with hundreds of (transitive) dependencies are going to be able to take security sufficiently seriously. And because minimizing the number of other projects you rely on is good engineering (see: left-pad).
I don't think the other two explanations are much better. Compare C as an example. With C you can use libcurl which is well-maintained, actually supports far more protocols than simply http(s), and uses far fewer dependencies to do it. It also runs on an insane number of platforms, so portability isn't the issue here either.
I think it primarily comes down to cultural differences. Rust is a newer, "cooler" platform, and attracts a very different audience than C does. Because of that Rust's users will be more apt to follow common modern programming practices: using a language's built in package manager and so on. (This culture both contributes to the development of languages like Rust and is fed by them.)
Remember the XKCD "Python" comic? [1] Python is what I would consider an older language in terms of culture, but it captures the idea of being able to just import a library that can already magically do everything you want. "Programming is fun again", you don't have to write boilerplate or worry about managing dependencies any more. The problem begins when everyone has that attitude, including the people writing libraries that will be used by others. Importing a library that can instantly solve your problem is just one line in your Cargo.toml away - who cares that it's got 200 dependencies!
I look forward when cargo offers that as an option, doesn't mean everyone has to offer their libraries that way, just having the possibility on crates.io would be nice.
I am also very thankful for what has been achieved thus far, so don't take this as thankless complaint rather as I would like Rust one day to be.
Yet another way is to just use bigger machines if you’re CPU bound. Github and Gitlab generally offer 1/2 core 3-8GB RAM instances, but SurplusCI[0] (a project I recently launched) has no such restriction. Up to 8 vcore 32GB RAM machines available.
Now, there are still ways to make the compiler choke: the mention of instantiations towards the end is a great example. But in my (non-professional) experience, the build times are pretty tolerable for what it's providing.
There are some alternate codegen backends that aim to fix this by compiling less-optimized debug builds more quickly.
Also read the Bevy documentation about improving build times.
Something I noticed since I've started timing builds is that codegen and linking usually takes less than 30% of the build time for an incremental build.
So in practice, switching to Cranelift and lld/mold will shave maybe 2 seconds off a 10-seconds build. Better, but still flow breaking.
If we want Go-like compile times, we're going to need some more fundamental improvements to the compiler.
I think this is probably right, and I think just about everyone "tolerates" the build times (that is, I see very few people stop developing in rust because of the build times).
But when, for example, I run `rustup update` to get a new nightly compiler and have to re-compile everything, it's annoying to have to wait 20 minutes. I'll tolerate it, and if a magic wand could make it instantaneous I would love that. But it's still annoying.
If you want to cut the CI run time even further. I'd recommend BuildJet for Github Action. We give you managed high-performance CI runners, which cuts both your CI time and price in half.
We plug right into Github Actions, so it's just one line change in your yaml file.