Hacker News Reader

Comment by vruiz | Hacker News Reader

> The core libraries are feature-complete for 1.0.

Can someone well informed about Rust give some impressions on the standard library? It is as comprehensive as in python or Go for example?

View on HN

Core team member here.

The standard library is _not_ "batteries included," on purpose. Given that we have Cargo, and it works well, tying package updates to the language version has quite a bit of downside, and very little upside.

That said, the Rust team itself maintains and provides a number of packages on Crates.io ourselves. Many of these were pulled _out_ of the standard library over the past few months.

The upside to a batteries-included stdlib is, of course, that you don't have to go fishing around for the best lib to do $WHATEVER_PARTICULAR_TASK, and you also don't have to wonder whether whatever lib you eventually choose will be abandoned by its developer next month.

At such a young stage of language development, I agree that it makes more sense to let the community develop libraries in order to foster competition and quality. But eventually (read: no less than a year or two from now) I think it will be up to the project maintainers to officially endorse certain third-party packages and commit to their maintenance. At a certain point, the advantages of a library that's well-known, well-maintained, and well-documented outweigh the disadvantages of de jure ossification.

It's been my experience that the standard library is _never_ the place for the best lib to do $WHATEVER_PARTICULAR_TASK, but different folks have different preferences. We'll see how it all shakes out!

I think Python and Go are two examples where the standard library has very much proven invaluable.

Without it, it's difficult to be confident in the portability of components, and it quite frankly makes the language less attractive for use.

Like the other poster mentioned, I'm happy for the RUST folks to take a "wait-and-see" approach, but at some point, I believe "blessed" components are going to be expected and strongly desired.

Python is a prime example of a rotten standard library.

Take urllib/urllib2 (use requests instead), unittest (use py.test or nose), os (too low-level, hence arcane usage) or time as examples (use pytz for anything serious), and of course Tkinter.

Take all of the modules that solve minor/niche tasks that could easily have been put in a seperate library (e.g. wave), that are usually a bad idea to use (e.g. pickle), that contain some copy/pasteable functions the authors deemed useful as comments (itertools), have documented bugs with copy/pasteable workarounds (csv).

Oh, and the way they do exceptions is a mess.

Oh, and don't try to read the Python standard libraries source code. It's ugly.

No, standard libraries should constrain themselves to providing a good foundation for library designers.

(I still like Python, and use it a lot.)

(Since you mentioned Go. One thing that I love about Go is that everything interaction with the underlying OS goes through the syscall package. Also, their designer went for more minimalism. And they didn't have to worry about design mistakes they made 25 years ago. Python is old. I'm not a fan of Gos compatibility promise: it means Go 1 will rot away too. But they're in a much better starting position.)

It all depends what you're using it for. If I'm writing code that's going to be around awhile and can tolerate some dependencies, I'll totally use, say, requests. But I use urllib2 from the repl all the time, especially if I'm on a machine that isn't mine. The fact that any Mac already has the tools on it to grab some JSON from an API, parse it, and do something useful with it from the command line, without having to download or install anything, is immensely useful, even if the API is a bit suboptimal. The same applies to quick and dirty things I'm shoving into a Gist to share with colleagues. Not all things have to be elegant to be useful.

(That said, those kinds of usecases aren't really in Rust's wheelhouse, so I still think in Rust's case having batteries not included is probably the right call.)

It's possible to achieve the best of both worlds. We could introduce the concept of a standard distribution instead of a standard library. A release of Python should ship with specific versions of requests, nose, and other packages that have a broad community consensus. Those packages could be individually upgraded later, but every install of Python 2.7.9, for instance, would have requests 2.5.1 or greater installed. That would avoid the stagnation packages see when they enter the standard library, and would maintain the benefits of universal availability.

Every language ecosystem should work this way.

I agree about urllib and unittest. However over the last decade I still find Python to be a one of the best "batteries included" language/platform.

That is not small thing either. It allows getting started easier, which in turns gets more people to use it.

Here is a list of modules I used and was happy there were in stdlib:

socket, shelve, cPickle, tarfile, urllib[2], Tkinter, time, ctypes, subprocess, asyncore, json, SimpleXMLRPCServer, wave (sorry, I did use it many time ;-) ), timeit, syslog and many others.

Most of the time I was happy to find them there.

While I'll readily agree that some parts of the standard library are rotten, that's not sufficient justification to say that there shouldn't be one.

I should also clarify my expectations about a standard library; to me, a standard library should have all of the basics covered (interaction with the underlying system, I/O, networking, etc.) and anything that benefits from better integration with the runtime (think data types such as those found in python's collections module).

If anything, I'd argue that the main problem with Python's standard library is not the library itself, but the lack of more focused curation.

Note that I never said that I expect all functionality to be available in a language's standard library; for me personally, Go's standard library has roughly the right balance.

The other thing that really needs to be in the standard library is protocols & interfaces that will be implemented by a number of userspace libraries. Go benefits immensely from having standard io.Reader and io.Writer types and most people implementing them instead of defining their own. Similarly, most of its web frameworks use http.ResponseWriter and http.Request instead of defining their own. Python's unittest module may be a mess as a test framework, but all the major Python unittest frameworks take a unittest.TestCase, which keeps tests portable among the different systems.

The worst case is exemplified by pre-STL C++, which didn't even have a string type in the stdlib. As a result, every project and library wrote their own, which meant that you basically had to choose a C++ ecosystem and develop for it rather than write libraries that are portable across multiple C++ projects.

... and a sufficiently large C++ application would use 10 different string classes in various parts of it, likely with different text encodings too, with lots of fun converting between them. Even a crappy UTF-16 based String like Java's is better than such a mess.

What's wrong with the exceptions? I never noticed that problem but agree completely on all your other points. (Especially itertools – why aren't the "recipes" defined in the module?!)

In my experience the batteries included is one of the best features of python. The standard library is fine for simple and small scripts, especially in restricted environments without pip or sudo rights. They are a lot of awesome python libs (like requests) but it is awesome to have the stdlib available everywhere and be able to depend on it.

Pip can install without root. Even if pip is not installed, it can bootstrap itself without root.

Python's standard library was only invaluable when its packaging situation was near-unusable. If Rust's isn't, the same thing will be true.

The opposite is also true; a good package manager can avoid the stdlib to bitrot ala Python, because you can easily swap out an old module by simply repackaging as a third party dependency.

On the contrary, the benefits of having a standard set of batteries, especially those upon which other might be developed, is fundamental to a sane ecosystem development. For instance, a standard framework for async I/O programming is important, because then people can write thousands of protocols that can be fully interoperable; if 2-3 different framework arises, each one can then develop its own ecosystem of protocols and libraries (not interoperable), and it's hard to think that each one would be as rich as the one in the former scenario. The same can be said for a HTTP library, a threading library, a XML/JSON marshaling library, a threading/concurrency library, and so on.

I think that crates.io still has a ways to go before the situation is ideal. Discoverability of libraries is the first pain point to look at. After that, I'd like to see crates.io automatically parse an uploaded package's docs (thank you, rustdoc!) and host them on the site itself.

I strongly disagree with that assertion; while I readily agree that some parts of the standard library are less maintained than others, as a consumer of many third-party components, I can say without a doubt that items in the core library are generally better maintained (from a security perspective) and easier to deal with.

The ease of installation has nothing to do with the desire for core components. Core components generally bring certain expectations/guarantees about security, reliability, and support.

That's just not true. Standard library components are not better maintained, not more secure, and not easier to deal with. There are too many examples to even count of each of those -- you can take a look at PEP 476 for just one recent example.

You're arguing with a general observation. The fact that it's not universally true is unsurprising. If you take the average quality of third-party libraries and the average quality of libraries in the standard library, I think you will find that the standard library is generally pretty good.

Whether it's better than the average is irrelevant, that isn't the benchmark. It needs to be better than or equal to all of the third party libraries for it to be worthwhile, otherwise, why does it exist? When you pick a library you don't pick all of them, you pick just the best one that meets the criteria you need, and the standard library can't beat the flexibility that other people will have to better meet those criteria without needing to partake in the process upstream.

(It's also not true that the standard library is generally pretty good IMHO, but that will start to veer us into subjectives [which I think the comment above my original post has already done anyhow]).

> When you pick a library you don't pick all of them, you pick just the best one that meets the criteria you need, and the standard library can't beat the flexibility that other people will have to better meet those criteria without needing to partake in the process upstream.

So, you're using three libraries that all depend on something like urllib - and they each use the library that fits their usecase best. Now you need to debug/review/depend on updates for 6 (7 if you also use the stdlib for something) foreign codebases rather than 3 + the standard library.

It's a trade-off between old/new good/best simple/complex (or complicated/complected). A standard lib that needs to maintain stability for 10+ years can never be "best" for all that time. But by being "good enough" it can often still be the best choice overall when the life cycle of a project is considered.

Sorry; I'll have to disagree. As part of a group that maintains Python packages for an operating system distribution, my experience has generally been more positive for the standard library compared to third-party components. Particularly when it comes to security issues.

> I think Python and Go are two examples where the standard library has very much proven invaluable.

C#. The amount of time I've wasted in Java programming teams while arguing over things like which of three quirky XML parsing implmentations[1] was the One To Use while the .net team powered off and built useful functionality...

[1] This was a few years ago, obviously.

C# seems to be in a similar position when it comes to JSON. There's both System.Runtime.Serialization.Json and System.Web.Script.Serialization built-in plus a whole bunch of third-party libraries (e.g. Json.NET seems to be popular). I'm still not sure which one to use.

So you prefer not having a choice over having a choice because choice will lead to discussions to pick the best solution.

Quite an odd view from a software engineer.

I like having as much choice as possible.

Having a standard library for a specific task doesn't forbid alternatives to be implemented and used. But it discourages them enough, which is a very good thing for consistency (especially in reading code), lowers the barrier to entry (eg: I don't have to learn how to parse json with the specific library the project I'm contributing to is using), reduces binary size (I don't link three different http libraries, mine and those chosen by my dependencies) and generically reduce the time wasted by humanity in useless duplications of efforts. When the standard sucks, well, good alternatives will appear.

Also, the sad state of Python packaging is the reason why stdlib bitrotted; technically, your package manager could take care of backward compatibility: if eg at some point in the future you want to switch from a json library to a (far) better one, more in line with how idiomatic language has evolved, you just need to repackage the old one as a third party package, and make sure the packager manager brings it down for the user automatically.

> So you prefer not having a choice over having a choice because choice will lead to discussions to pick the best solution.

I prefer to get things done.

Totally. Rust 1.0 is about a small, excellent, stable base. You can always add stuff, but you can never take it away...

The useful thing about a standard library is that it provides a common vocabulary for code, which is a great help for interoperability. For example: c++ doesn't have a 3D vector class in its standard library, so pretty much every 3D-related library invents its own & you end up having to write lots of useless glue code to convert from one to the other.

Yes, this is very true, and one of the reasons that we have a pretty big set of things like collections. Common traits are an excellent thing to put into a standard library.

I really love your usage of the phrase "common vocabulary" to describe the utility of the standard library. I'll be using it in every discussion of this sort from now on.

For what it's worth, while a lot of people complain about the Ruby standard library, when I want to reach for an HTTP library that is properly threadsafe, properly supports encodings (and in general is properly integrated with whatever version of the language the user happens to be using), Net::HTTP is a godsend.

People who complain about the ossification are often talking about the aesthetics of the API (since API tastes change, long-term stdlibs tend to feel outdated), or the level of abstraction (you have to do a lot to use Net::HTTP, at least in the past), and not about the functionality. I'd rather people build nice abstractions on top of Net::HTTP than tell me to use something else because the API is prettier.

Yeah, while that's kinda true, it's also incredibly hard to patch bugs. Contributing to Net::HTTP means dealing with a whole lot of stuff that you don't have to when contributing to a random project, and (as I'm sure you know, hahah) it's easy to use my custom patched random gem with Bundler, whereas I have to wait a year to get my Net::HTTP fix.

Yes, but the cost is that those gems have to worry about supporting a lot of different versions of Ruby. Writing a fix for Net::HTTP only involves making it work on master ;)

By extension, language builtins should also never be the place for the best facilities to do $WHATEVER_PARTICULAR_TASK. The line between builtins and standard libraries is extremely variable between languages.

There are languages with strong standard libraries: Erlang (both its normal libraries and OTP), Objective-C (Cocoa) and Mathematica are three that come to mind.

I think the stdlib is not the way to go. Give me an example of any ecosystem where the standard library is the defacto best solution?

But would agree that the only thing Rust might need are officially supported crates, while keeping the language itself fully separate.

The language is then completely free of the hastles of maintaining a stdlib that probably isn't used by most people anyway. But at the same time officially supported packages give developers and newbs a place to start, and the community some confidence that libx will be maintained in the future. I really think this solution is the best for seperation of concerns and maximizing developer productivity and efforts.

For things that the standard library can do, it is usually the best choice. It's code that's already sitting on the user's machine, why use something else?

And also, those are generally relatively straightforward things whose design are hard to get wrong (but there are exception, e.g. Python's urllib). Being a standard library and not a standard framework helps.

For what it's worth, I use Java's standard library whenever possible. It's quite extensive and the fact that the documentation is top-notch makes it a joy to use. That being said, there are still a lot of supplemental things you might want to do, which is what things like Google Guava or Apache Commons are for. But they are to supplement, not replace the standard library.

By the way, some parts of Apache Commons would be a good example of how not to design a standard library, I don't want to piece 10 objects together to do a simple utility function call.

>I think the stdlib is not the way to go. Give me an example of any ecosystem where the standard library is the defacto best solution?

Python, Go, C++'s STL, Java SDK, to name a few.

They might not be 100% perfect, but they are good and reliable, and always there for you. Hunting the latest "best" library that gets abandoned after a year (like in Javascript and Ruby often happens, and also Go too) gets old quickly.

Of course there could be a compromise approach, as you say.

A mininal standard lib for Rust PLUS a "blessed" set of Cargo packages that represent the batteries (Haskell "Platform" is like that, IIRC).

I love Python and Python's standard lib but it's hardly the defacto best solution.

urllib gets trumped by requests, ujson is much much better than the stdlib json module, hardly anyone still uses distutils over setuptools for example.

Woah, that's like a list of what not to do with standard libraries, particularly python. Have you seen the datetime module? httplib? Even go has it's weird databases module.

I think <stdio.h> is a good example. It gets the job done, its minimal and simple. That's what a standard library should be.

> I think <stdio.h> is a good example. It gets the job done, its minimal and simple.

It also routinely takes null-terminated strings without pairing it with a size parameter.

> Java SDK

Hardly. See Joda Time (it only took 10 years for java.time to catch up) and Apache Commons (the situation is much better now, but who hasn't turned to Apache Commons due to shortcomings in Java's standard library?).

In any case, I definitely agree with the frustrations of trying to find libraries for Ruby.

Again, people have seem to miss the point I was making.

I wrote that those batteries "might not be 100% perfect" -- and as a Java programmer back in the '00s, I know the specific problem points with Java time that Joda tried to solve (btw, it has inspired the time lib in the latest SDKs IIRC).

The thing is, the JDK APIs, with all their issues have been a genuine force in Java adoption, and something every Java programmer relies on. Of course there'll be some pain points, but it's nothing like having to hunt for a Collections or DB abstraction or String manipulation or XML etc lib each and every time you start a project.

Having an official batteries API doesn't preclude you from using external libs (like Joda) when they are better -- whereas not having one is a genuine loss.

If I have turned to Apache Commons that was overwhelmingly for things MISSING from the SDK, not for things the SDK already did.

And don't get me started on the Commons code quality, with BS reinventions of the wheel, lame FactoryFactoryProxyFactorySingletonFlyweightFacades and the like, and code that was blisfully ignorant of enconding issues...

"Best" always depends on your criteria. Very often, the standard library is a "good but not best" solution from a pure performance or features point of view. However, it often wins for ease of deployment (nothing to install), ease of maintenance (the project is not likely to be abandoned), documentation, and standardization.

IMHO, C++ in an example where the standard library (and its laboratory, boost) is often a very good solution for the few things it covers. It's standardized, well-documented, and not that bad from a performance point of view (of course, it's not perfect).

In addition, you expect your newly hired developers to know the standard library, but not every small libraries from github...

>Give me an example of any ecosystem where the standard library is the defacto best solution?

Almost all of them? Unless the stdlib is utterly utterly terrible the cost of the extra dependency is not worth the difference in quality between the stdlib and library which does the same thing.

> Give me an example of any ecosystem where the standard library is the defacto best solution?

In Go that's certainly the case and in my experience it's a great thing, only time will tell if it will end up suffering stagnation like maybe happened to python.

Or what happened to Java. And probably other ecosystems.

IMO a "softer", "recommended" packages approach is better longer term.

Ironic that you've both picked languages that are used quite effectively, if unsexily, to solve problems. Both Python and Java are at the "plateau of productivity" - there's not a whole lot of innovation going on, but they're stable, robust, and have a large ecosystem of tools that let you get work done quickly. A good portion of this is because of the stability of a large standard library that everyone can depend upon.

> The standard library is _not_ "batteries included," on purpose. Given that we have Cargo, and it works well, tying package updates to the language version has quite a bit of downside, and very little upside.

With a good package management system, a batteries-included standard library could just be a set of packages where you are guaranteed that a version of each backward-compatible to the one provided at language release (in SemVer terms, a package with the same major version number as the one packaged with the language release) would be available and maintained for the maintenance life of the language version. It wouldn't have to prevent either bug-fixes to packages out-of-band with language version upgrades, or even out-of-band new major versions of packages that support the existing language versions so long as the old major version was maintained in parallel as long as the language version was.

A language where the standard is "we download the packages we need" (somewhat like npm on node) feels like it has a lot less mental overhead to it because the default is "users expect to get dependencies easily", and you don't have to spend so much time trying to target older but "default" installations of things.

Which with a language like Rust capable of producing static binaries, feels like a better place to be in anyway since it's a lot easier to replace 1 executable then a whole net of shared dependencies.

It's true that that's possible. Gotta have those libraries actually exist first, though :) We'll see if we don't end up with something like this.

I've always thought a good compromise between these approaches, and for a while it kinda looked like at least Rubinius would go to something like it in the Ruby world (but efforts to modularize the ruby stdlib seem to have fizzled somewhat), would be to have a minimal stdlib but a broader set of basically LTS versions of exceptional packages, where they are expected to have important issues backported to a stable version alongside a version of the runtime itself.

This would still allow for pulling up to the bleeding edge, or simply not using, of some of those blessed packages if you choose to by specifying them in the cargo manifest, but would still provide guidance on basic useful packages.

I think it'd also be easier to boot stale packages in new releases (ie. the eternal security fail that is the yaml lib in ruby's stdlib).

For someone that is new to Rust how do I discover where the libraries are for the things I need to do?

In the past year I have needed libraries for HTTP, XML, JSON, CSV, arg parsing, image manipulation, PDF, RDBMS, a trie (and other data structures), async i/o, threads, files, ZIP, and others.

CSV parsing: https://github.com/BurntSushi/rust-csv

Arg parsing: https://github.com/docopt/docopt.rs

1. http://crates.io - central repository of rust crates (i.e. packages)

2. https://github.com/kud1ing/awesome-rust/ - manually curated list of good libraries

While others mentioned http://crates.io/ you might also be interested in http://rust-ci.org/ (has categories, includes build status and quite often hosted documentation).

Reply on news.ycombinator.com