The Rust Platform
aturon.github.io
aturon.github.io
Small vs large standard library:
A small standard library with most functionality in independent, community-maintained packages has given us API friction as types, traits (type classes), etc are hard to coordinate across maintainers and separate package release cycles. We ended up with lots of uncomfortable conversions at API boundaries.
Here's a number of examples of problems we currently have:
- Conversions between our 5(!) string types are very common.
- Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.
- Standard library cannot use containers, other than lists, for the same reason.
- No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation.
- Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances.
- It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen.
Empirically, languages that have large standard libraries (e.g. Java, Python, Go) seem to do better than their competitors.
How about: "std is where code goes when it's done".
As is, really done. The API's won't need changing.
And that's what we're doing.
> Keeping the standard library small helps with that since the bar is higher.
But weren't you just advocating for a large standard library?
Go prioritizes network programming and bundles the necessary components, like http & rpc servers and json.
The http libraries are extensible enough to allow for customization where it's wanted (like http mux) while still creating a canonical implementation that'a still viable.
Has Rust identified the core demographics of who they're targeting in order to provide the most applicable platform? Is the target everyone and all application type, therefore there is no default platform?
Edit: To put it another way, is there a set of packages that is either necessary for rust, rust development, or most development in rust? If std lib includes everything necessary, then who are you targeting with the default platform?
> Has Rust identified the core demographics of who they're targeting
> in order to provide the most applicable platform? Is the target
> everyone and all application type, therefore there is no default platform?
CSV, logger, json, fileutils, tempfile, pp, and on and on are used all the time...
There are a number of counterpoints in Python, in fact, which epitomizes the "standard library is where code goes to die" thing. Adding modules to the standard library in Python is, more often than not, overall a bad thing for the module. Python has not historically been awesome with standard library quality, either; see Java-style logging and unittest (I mean naming, not "Java idiomatic," which I think is fine for both).
This comes down to release cycles for the language, mostly. So I think API stability is a bit of a red herring when discussing Python, at least.
I tend to appreciate languages where I can remove the entire standard library and "start over," like C. (Yes, you can.) This can be good for a number of things: porting, embedding, frameworks, and so on.
[0]: http://artem.krylysov.com/blog/2015/09/29/benchmark-python-j...
[1]: https://github.com/search?q=simplejson+ImportError&type=Code...
I try to keep my dependency list as tiny as possible, and use what makes sense for my development and for future maintenance. Also, look at the result, in Python 3, json module beats simplejson.
But speed isn't the only thing that matters, it seems ujson would use more memory (https://news.ycombinator.com/item?id=9326499).
Anyway, my point isn't the specific example. That you and I even have this discussion at all and that there are hundreds of thousands of caught ImportErrors on that specific example on GitHub is my point regarding standard library stability; folks seem to think the standard library is the end-all (wherein we wouldn't be having this conversation at all), but Python has shown it is anything but when not carefully maintained. I think Rust is wise to approach this with caution.
Honestly, I'm not extremely familiar with Rust, but it seems it elected the C approach where you can gut the language. A+. Good. How it should be for a systems language like that, because now it can be ported, embedded, and so on.
That fact that thousands of files catching ImportError does not necessarily implies folks are questioning stdlib's stability. That merely means some people are deliberately choosing to prefer simplejson over json. The benchmarks demonstrated json module before Python 3 could be slower than simplejson, but json module since Python 3 has beaten simplejso in terms of speed of execution. Furthermore, there are old Stackoverflow threads on usjon vs simplejson vs json regarding performance. All the above would naturally suggest folks who choose to prefer simplejson over json is due to the concern of speed, rather than opinion on stability.
Also, stability is the wrong term for the problem you are describing. Agility is probably the better word. Python release tends ot be backward compatible (of course except Python 2 vs Python 3 and a few other modules like asyncio). Python core developers try not to break applications. If anything, non-core libraries will break compatibility more frequently without having to face larger opposiitons; I can break simplejson if I were the maintainer of simplejson. The consequence is maybe a couple angry GitHub issues and a few blog posts, unlike Python 3 which still gets a lot of angry media coverage till this day.
The problem with stdlib is absolutely about agility. The core community is extremely small. It can take many weeks and sometimes months to get your commit merged. The reason I like to keep stdlib around is good citizenship. I would love to have requests in the stdlib, but in a more agile and more frequent release. Python isn't the only player. OS distro are also responsible for the slowness. There's been discussion on python-dev regarding more frequent release and even potentially breaking up stdlib could be an option for the Python community.
Even more code just use the standard json package without any fallback. The ease of development or deployment is clearly worth more to them than what small speed advantage they can get from going with the external dependency.
The calculus will be different for Rust, of course, with different build and deployment system.
Standard doesn't mean completely done. Standard should be able to accomodate things like HTTP2, as Go has done, whether that means expanding the API or whatever.
I haven't seen anybody say otherwise.
New standard library APIs are stabilized at a steady clip with every new release: https://github.com/rust-lang/rust/blob/master/RELEASES.md
1. Easy to use
2. Have standardized syntax for flags across languages
3. Be easy to maintain
4. Be well documented so that I can use them a year or 5 years from now.
My applications often have complex syntax. For instance I work in frequent pattern mining, the basic syntax for some of my programs looks like:
program [global options] -o <output> --support=<int> \
<datatype> [data-type-options] <path-to-data>
<algorithm> [algorithm-options]
<filter-chains>
<logging-and-serialization>
I write programs in more languages than just Go, and the programs in Go need to work the same way the programs in other languages work. That means GNU option syntax, which is the superior syntax for my needs in any case.
http://www.slideshare.net/jpetazzo/docker-and-go-why-did-we-...
It does all the wrong things; singletons, no testability, cgo for implementations, side effects and you have to use every database differently based on their individual semantics.
Virtually everyone I've ever spoken to either uses a high level wrapper around the sql library or a no-sql solution.
That's the definition of 'stdlib is where packages go to die'.
It's not that the API is unusable, it's just basically not used by the community because there are other better things out there...but you're stuck with it forever, because it's there and some people do use it, and changing or removing it would be a breaking change.
Anyhow, we're just speculating. Does anyone actually collect metrics about the usage of different parts of the stdlib for any language?
Without hard data to back it up, you couldn't really make a strong argument either way.
>Virtually everyone I've ever spoken to either uses a high level wrapper around the sql library or a no-sql solution.
How does that reflect the quality of the std lib implementation? All the high-level wrappers I've seen still utilize database/sql, they just provide convenience methods on top of the existing functionality. Are people using NoSQL databases because database/sql is so bad or merely because that technology fits their project's requirements?
>That's the definition of 'stdlib is where packages go to die'.
steveklabnik's example of Ruby XML parsing libraries is a better example of this, if only because the std lib implementations are almost completely ignored by all other gems. Go's database/sql is actively used outside of the std lib to great affect, whether in wrappers and ORMs or in implementing other SQL databases (like Postgres).
SQL has its flaws, but it is testable. The testing approaches available vary depending on the implementation. For example, can write unit tests for SQL Server (using tSQLt, to give one example: http://tsqlt.org/ ).
Plus, there are plenty of ways to test standard SQL, you can easily do so through stored procedures.
Read this if you think I'm wrong: https://github.com/bradfitz/go-sql-test/blob/master/src/sqlt...
Great abstraction, right? Love how you have to mangle the raw SQL right? no. It's a bad abstraction.
When you test the implementation and you have to know the details of the implementation to test it, that's like, the definition of a bad abstraction.
It is what it is.
...and ultimately, we're stuck with it because its part of the standard library.
/shrug
> - Conversions between our 5(!) string types are very common.
> - Standard library I/O modules cannot use new, de-facto standard string types (i.e. `Text` and `ByteString`) defined outside it because of dependency cycle.
We have one string type defined in std, and nobody is defining new ones (modulo special cases for legacy encodings which would not be worth polluting the default string type with).
> - Standard library cannot use containers, other than lists, for the same reason.
> - No standard traits for containers, like maps and sets, as those are defined outside the standard library. Result is that code is written against one concrete implementation.
Hash maps and trees are in the standard library already. Everyone uses them.
> - Newtype wrapping to avoid orphan instances. Having traits defined in packages other than the standard library makes it harder to write non-orphan instances.
This is true, but this hasn't been much of a problem in Rust thus far.
> - It's too difficult to make larger changes as we cannot atomically update all the packages at once. Thus such changes don't happen.
That only matters if you're breaking public APIs, right? That seems orthogonal to the small-versus-large-standard-library debate. Even if you have a large standard library, if you promised it's stable you still can't break APIs.
If you have 100 different libs that are basically "standard" (who doesn't have `mtl` in their applications at this point), now you have to coordinate 100 different library updates roughly at the same time. If you forget even one of them, then you've broken everything.
I think the argument for a large Prelude/standard lib is similar to Google's "single repo" argument: Easy to catch usages and fix them all at once. Plus you're making the language more useful out of the box. People coming from python can understand this feeling of opening a python shell and being productive super quickly form the get-go.
Arguments for small std lib exist, of course. But Giant standard libraries are more useful than not.
EDIT: I think the failure of the Haskell Platform has a lot more to do with how Haskell deals with dependencies, and the difficulties it entails, than with the "batteries included" approach itself.
see: javascript
Writing "see: JavaScript" doesn't really help without context. Without context, one does no know if you meant "JavaScript in browser" or "JavaScript via Node.JS" or "I simply don't like npm".
I'm not claiming there aren't any problems; however, "problems" are situational and one person's "problem" is another person's meh.
I just think it's irresponsible to not provide detail when making such claims.
We have a policy of no breakage for stable libraries post 1.0 (as does Python, and Go, etc.). So no, we can't.
The size of the standard library has nothing to do with it.
The higher level the library (e.g. containing content specific to an application domain), the more magical-seeming libraries can be added to the ecosystem. The counter-risk is the standard library growing in undesirable directions that you can never change because you can't remove stuff.
The interstitial glue that lets third party libraries integrate with one another and be usable by your app: that's the single biggest reason for having a bigger standard library than a smaller one. It has very little to do with including the batteries in the box.
If you think it has something to do with including the batteries in the box, you'll be lured into the trap of making it easy to fetch the batteries from across the internet (that's almost the same, right?). The trouble is, the internet has 100 different batteries to choose from, and not only have you offloaded the choice onto the user, but the batteries use mutually incompatible terminals and you have to jerry-rig interfaces between them. Let a thousand flowers bloom, say some people: trouble is, waiting for the biggest flower can take years, and people pick different ones in the early days. A bad choice is better than indecision.
Low effort updates are even less what large standard libraries are about. Large standard libraries are much harder to update, not easier: there's much more surface area, so it's far easier to break an application - and since every application uses the standard library, you could potentially break them all. Easier versioning and updates are a strong argument for extracting out things into third-party libraries.
What Rust has right now is pretty amazing. What needs to happen is a way for devs to easily break the dependency cycle and include multiple versions of the same crate. Something that has plagued Haskell. I dunno what the answer is, trait only crates, struct only crates?
If people want to 'curate' (shop) a set of packages, they can make a meta package that exports its deps.
There is literally no reason to ship libs with the compiler aside from the basic verbs and nouns.
With verioned and properly name-spaced imports, one could use different curated libs.
I go to those pages and I am given 0 information on how the thing actually works.
It's also one of the features I sorely miss when using Rust. Luckily, Rust's stdlib is starting to tend towards being more practical with recent additions like system time.
Linking the language and the libraries together is a mistake.
The stdlib datetime class is terrible and desperately needs to be wrapped. Arrow is a good wrapper. I don't know what you're on about with counting bytes.
Now we have pandas, yay. But I don't see why one would use arrow. If you're patient enough, could you explain why you would use it? The website doesn't seem to be very convincing.
In the enterprise space it is quite common that we only get to use what it is in the computer and access to anything else is strictly controlled by IT.
So if it isn't in the standard library or some internal library mirror, we don't get to use it, as simple as that.
To get a jar into that mirror, a request needs to be sent to the legal team describing the license and business case use, after approval the IT team will add the said jar to the mirror.
The same applies to version upgrades of already approved jars.
This is a typical scenario I had already in a couple of projects.
The thing I like about python is it gives tools for library writers to build things without going too low level.
Application writers will always write with better libs, but don't have to worry about third party lib compatiblity on platforms because of the stdlib serving as a virtual machine (most of the time)
If I'm not mistaken, the stdlib contains urllib and urlib2, but not urllib3.
The fact that there 3 "urllib" packages show that the Python way is not so good.
Having a good standard library also makes deployment easier. (In Go, OTOH, I tend to care less, even though its standard library is quite good, because thanks to static linking, deployment is always easy, no matter how many third-party libraries I use.)
There's also `inlinable_string`, `string_cache`, `tendril`, `intern` if you need inlining for performance.
The bigger problem is with other things like 2D/3D points which can be (f32, f32), [f32; 2] or a custom struct.
The standard library also includes Path/PathBuf and OsStr/OsString. And third-party libraries also use [u8] for bytestrings.
It'd be nice to improve handling for user-supplied text where you can't assume UTF-8. For instance, git2-rs provides the contents of diffs as [u8], because it can't assume the diffed files use UTF-8. That led to this commit today: https://github.com/ogham/rust-ansi-term/pull/19/commits/a0da...
That felt like a lot of boilerplate to abstract between str and [u8]. Is there a better way to solve that problem?
(As much as I'd love to just say "use UTF-8", that would break on many git repositories, including git.git and linux.git.)
Right. And you want people to explicitly convert between them.
Having to convert between string types isn't a problem. String encoding is hard, and you're going to have to pay that cost somehow.
This doesn't have anything to do with large vs. small standard libraries, because all of these string types are defined in libstd.
I'd love to see additional support for handling bytestrings in libstd, to make it easier to write code that handles both &str and &[u8].
All things you can delegate to third parties, but not without lots of cross-talk and confusion until things settle down to winners and losers, which may be a long time in the future. Indecision can be costly.
Consider standard library profiles, with progressively higher levels of abstraction supported. It's the right decision for creating a good ecosystem. C and C++ took decades to build consensus on the more complicated libraries, and C++ eventually grew a pseudo-standard library in the form of boost to centralize efforts, simply because it is more efficient that way.
Thanks to GHC's extensions, Haskell has become a ridiculously powerful language in exactly the same way C++ has: by sacrificing elegance. The principled approach would've been to admit that, while type classes are good for a few use cases, (say, overloading numeric literals, string literals and sequences), they have unacceptable limitations as a large-scale program structuring construct. And instead use an ML-style module system for that purpose. But it's already too late to do that.
(0) It means that libraries in general won't play nicely with each other, unless they're explicitly designed to do so.
(1) It can't be fixed without throwing away code.
I do agree that lack of modularity features certainly doesn't help though.
> The principled approach
You're dead wrong. The principled approach here is dependent types and full-featured type-level functions. Fundeps are a hack that let you implement a small subset of such functions (while type families gets us a bit closer to the ideal).
> they have unacceptable limitations as a large-scale program structuring construct.
Such as?
> And instead use an ML-style module system for that purpose.
How about we just use C macros for parametricity?
ML-style modules have their uses, but they aren't nearly as elegant as a clean type-level solution.
It's not about how much you can do (otherwise, just use a dynamic language, you can do everything, even shoot yourself in the foot!), it's about whether the result makes sense, and how much effort it takes to make sense of it.
> You're dead wrong. The principled approach here is dependent types and full-featured type-level functions. Fundeps are a hack that let you implement a small subset of such functions (while type families gets us a bit closer to the ideal).
You wanna play the dependent type theory card? Type families as provided in Haskell are incompatible with univalence.
type instance Foo Bool = Int
type instance Foo YesNo = String
Case analysis only makes sense when performed on the cases of an inductive type, which the kind of all types is not.
> Such as?
The insistence on globally unique instances?
> How about we just use C macros for parametricity?
What does this even mean?
> ML-style modules have their uses, but they aren't nearly as elegant as a clean type-level solution.
See here for how modular type classes, as proposed for ML, would actually prevent the issues caused by Haskell-style type classes: http://blog.ezyang.com/2014/09/open-type-families-are-not-mo...
Type classes allow huge flexibility while maintaining type safety, to a much greater degree than fundeps allow.
> it's about whether the result makes sense
Which they do. Perhaps you have some examples of when type families confused you or made you perform an error?
> Type families as provided in Haskell are incompatible with univalence.
TFs aren't dependent types. However, they are on the right track. Fundeps are farther away from the right idea. Could you explain to me what's wrong with your example? I'm not up to date on HoTT, but it seems like there's nothing in principle wrong with pattern matching on elements of *. That seems like an important feature of type-level functions.
>The insistence on global unique instances?
Why is this a problem? It makes sense from a theoretical perspective (we don't associate multiple ordering properties with the things we call "the integers"), and it's very easy to use newtype wrappers to create new instances if needed.
> What does this even mean?
ML modules are flexible, but backwards from a theoretical perspective. Parametricity is something that should be embedded in the type system, not the module system.
> See here
Interesting example. However, I doubt that the syntactic cost of using such a system is less than the syntactic cost of enforcing global instance uniqueness and using newtype wrappers.
Um, aren't functional dependencies an add-on to multiparameter type classes? I don't see where the opposition is.
> Which they do. Perhaps you have some examples of when type families confused you or made you perform an error?
I already gave an example above. I defined two type instances that violate the principle of not doing evil: https://ncatlab.org/nlab/show/principle+of+equivalence
> TFs aren't dependent types. However, they are on the right track.
Dependent types are a good idea. The way Haskell attempts to approximate them is not. Parametricity is too good to give up. With the minor exception of reference cells (`IORef`, `STRef`, etc.), if two types are isomorphic, applying the same type constructor to them should yield isomorphic types.
You know what type families actually resemble? What C++ calls “traits”: ad-hoc specialized template classes containing type members.
> Fundeps are farther away from the right idea.
Functional dependencies are a consistent extension to type classes, which don't introduce a second source of ad-hoc polymorphism, unlike type families.
> Why is this a problem? It makes sense from a theoretical perspective (we don't associate multiple ordering properties with the things we call "the integers"),
What if I want to order them as Grey-coded numbers? In any case, the integers are far from the only type that can be given an order structure, and many types don't have a clear “bestest” order structure to be preferred over other possible ones.
> and it's very easy to use newtype wrappers to create new instances if needed.
Creating `newtype` wrappers is easy at the type level, but using them is super cumbersome at the term level.
> ML modules are flexible, but backwards from a theoretical perspective.
ML modules are plain System F-omega: http://www.mpi-sws.org/~rossberg/1ml/ . Where's the backwardness?
> Parametricity is something that should be embedded in the type system, not the module system.
It's type families, as done in Haskell, that violate parametricity! Standard ML has parametric polymorphism, uncompromised by questionable type system extensions.
> Interesting example. However, I doubt that the syntactic cost of using such a system is less than the syntactic cost of enforcing global instance uniqueness and using newtype wrappers.
I can't imagine it being more cumbersome than wrapping lots of terms in newtype wrappers just to satisfy the type class instance resolution system.
You're right, I meant "type families".
> I defined two type instances that violate the principle of not doing evil:
We're not doing abstract category theory; we're writing computer programs (well, I am). Have you ever run into a problem with type families in that capacity?
>if two types are isomorphic, applying the same type constructor to them should yield isomorphic types.
Agreed, but there's a difference between type functions and type constructors. TFs are (a limited form of) type functions. Value-level constructors admit lots of nice properties that value-level functions do not, and I see no reason to be uncomfortable with this being reflected at the type level.
> What if I want to order them as Grey-coded numbers
Use a newtype wrapper. Even if a language allowed ad-hoc instances, I would consider it messy practice to apply some weird non-intuitive ordering like this without specifically making a new type for it.
> Creating `newtype` wrappers is easy at the type level, but using them is super cumbersome at the term level.
And using ML-style modules is easy at the term level, but cumbersome at the type level.
It's a tradeoff, and I suspect that newtypes are usually the cleaner/easier solution.
> ML modules are plain System F-omega
I hadn't seen the 1ML project. That's pretty cool.
> It's type families, as done in Haskell, that violate parametricity!
How so? I really don't understand your argument here, if you just take TFs to be a limited form of type function.
I like being able to reason about my programs. For that to be a smooth process, the language has to be mathematically civilized.
> Agreed, but there's a difference between type functions and type constructors. TFs are (a limited form of) type functions.
By “type families”, I meant both data families and type families. Case-analyzing types is the problem, see below.
> And using ML-style modules is easy at the term level, but cumbersome at the type level.
Actually, ML-style modules are also more convenient at the type level too! If I want to make a type constructor parameterized by 15 type arguments, rather than a normal type constructor in the core language, I make a ML-style functor parameterized by a structure containing 15 abstract type members.
> How so? I really don't understand your argument here, if you just take TFs to be a limited form of type function.
“In programming language theory, parametricity is an abstract uniformity property enjoyed by parametrically polymorphic functions, which captures the intuition that all instances of a polymorphic function act the same way.”
You can do this in Haskell with DataKinds (you just pass around a type of the correct kind which contains all the parameters). Admittedly, it is quite clunky at the moment. I did this to pass around CPU configuration objects for hardware synthesis a la Clash, as CPU designs are often parametrized over quite a few Nats.
> parametricity is an abstract uniformity property enjoyed by parametrically polymorphic functions
Whenever one introduces a typeclass constraint to a function, one can only assume that the function exhibits uniform behavior up to the differences introduced by different instances of the typeclass. There is no particular reason to assume that (+) has the same behavior for Int and Word, except insofar as we have some traditional understanding of how addition should work and which laws it should respect. The same is true for type families. It is not a problem that they introduce non-uniform behavior; we can only ask that they respect some specified rules with respect to their argument and result types.
Case-analyzing types in type families is no worse than writing a typeclass instance for a concrete type. Would you say that the fact that "instance Ord Word" and "instance Ord Int" are non-isomorphic is a problem? After all, the types themselves are isomorphic!
Of course.
> Would you say that the fact that "instance Ord Word" and "instance Ord Int" are non-isomorphic is a problem? After all, the types themselves are isomorphic!
It's already bad enough, but at least the existence of non-uniform behavior is evident in a type signature containing type class constraints. OTOH, type families are sneaky, because they don't look any different from normal type constructors or synonyms.
That is fair.
I think we're on the same page at this point. You have made me realize that ML-style modules are useful in ways I did not realize before, so thanks for that.
Question: How would you feel if the tradition was to do something like
insert :: Ord a f => a -> Set f a -> Set f a
That is, "f" is some type that indicates a particular ordering among "a"s. Then, "Set"s are parametrized over both "f" and "a", and one cannot accidentally mix up Sets that use a different Ord instance.
Here's a quick example:
https://gist.github.com/wyager/a021f7e5d9f23643bc90a9866b5c0...
signature ORD =
sig
type t
val <= : t * t -> t
end
functor RedBlackSet (E : ORD) :> SET =
struct
type elem = E.t
datatype set
= Empty
| Red of set * elem * set
| Black of set * elem * set
(* ... *)
end
structure Foo = RedBlackSet (Int)
structure Bar = RedBlackSet (Backwards (Int))
(* Foo.set and Bar.set are different abstract types! *)You know that's not what parametricity means, right? Like, at all?
Here's a challenge.
`foo :: forall a. a -> a`
Now, by parametricity that should have only one inhabitant (upto iso). Use your claimed break in parametricity from type families and provide me two distinct inhabitants.
I can count at least three: `undefined`, `const undefined` and `id`.
> Use your claimed break in parametricity from type families and provide me two distinct inhabitants.
Does this count? Here Oleg constructs an inhabitant of False using just some means to case-analyze types (GADTs or type families): http://okmij.org/ftp/Haskell/impredicativity-bites.html
Strongly disagree.
(+2) 3 == (+1) 4 implies neither (+2) == (+1) nor 3 == 4. So Fst goes out the window.
(even 5) == (even 7) does not imply 5 == 7.
>type constructors are injective.
But type functions aren't, and that's what we want.
I agree that it's misleading to have type functions look like type constructors syntactically.
that said, constructing an inhabitant of false a _different_ way (when we can already write "someFalse = someFalse") is not particularly interesting, and again doesn't speak to parametricity in any direct way.
Hi. As someone that knows type theory and knows homotopy type theory and also knows Haskell well I would pose the following question to you: what purpose on god's green earth would be served by introducing univalence directly to haskell?
(Oh, and furthermore, you realize that fundeps have precisely the same issues in this setting?)
Contrariwise, don't you find it _useful_ that we can have two monoids, say And and Or, which have different `mappend` behaviour?
Now, can you imagine having that feature and _also_ respecting the idea that set-isomorphic things should be indistinguishable? How?
Generally, when I want to reason about tricky data structures, what I do is:
(0) Define a set-isomorphic auxiliary type that's easier to analyze, and whose operations are easier to implement, but have worse asymptotic performance.
(1) Prove that transporting the operations on the auxiliary type along the isomorphism yield the operations on the original tricky type.
I need univalence for this argument to hold water.
> (Oh, and furthermore, you realize that fundeps have precisely the same issues in this setting?)
Type classes are already Haskell's controlled mechanism for adding ad-hoc polymorphism “without hurting parametricity too much”. I consider it healthier to reuse and extend this mechanism (which is what functional dependencies do) rather than add a second one for exactly the same purpose (type families).
> Contrariwise, don't you find it _useful_ that we can have two monoids, say And and Or, which have different `mappend` behaviour?
Sure. In ML, I'd just make two structures having the MONOID signature. Haskellers have this wrong idea that the monoid is just the type - it's not! A monoid is a type plus two operations. Same carrier, different operations - different monoids.
> Now, can you imagine having that feature and _also_ respecting the idea that set-isomorphic things should be indistinguishable? How?
Yes. Acknowledging that an algebraic structure is more than its carrier set.
No, you don't. Univalence is the axiom that transporting operations across such equivalences _always_ works. If you're doing equational reasoning directly it doesn't arise.
Furthermore, all you need to do is to establish that the _type operations_ regarding one type respect the equivalence to the other type as an additional step.
As you say "a monoid is a type plus two operations" -- so fine, we can treat the monoid And as the type bool and the dictionary of operations on it, and all this still works out.
Sure, but the strategy I outlined is risky (as in “may lead to getting suck and having to undo work”) in a language where this isn't guaranteed to work.
> As you say "a monoid is a type plus two operations" -- so fine, we can treat the monoid And as the type bool and the dictionary of operations on it, and all this still works out.
Yup, but Haskell doesn't let you define types parameterized by entire algebraic structures. It only lets you define types parameterized by the carriers of algebraic structures.
> Empirically, languages that have large standard
> libraries (e.g. Java, Python, Go) seem to do better than
> their competitors.
Fitness for purpose is relative to the other options.
My point (or rather, the point of the parent comment that I'm agreeing with) there's a lot more than just the presence and characteristics of a standard library that determine how widespread a language becomes
For example check Turbo Pascal libraries, including Turbo Vision, already on MS-DOS.
C took off thanks to UNIX's adoption, like JavaScript on browsers nowadays, it became the language to use for anyone working on the enterprise on those new shiny UNIX boxes.
In Europe it was just another systems language to choose from, back when CP/M and other 8 / 16 bit systems were common.
Also there were C, Pascal and Modula-2 compilers available for ZX Spectrum.
And on my part tiny of the globe I can guarantee that everyone only cared about x86 Assembly, Turbo Basic and Turbo Pascal, with Clipper for business stuff.
I only got to learn C in 1993, after having been a Turbo Pascal 3, 5.5 and 6.0 user.
I first used C in 1983 on MS-DOS, I didn't use UNIX until a couple of years later. I bought Turbo Pascal 1.0 when it was released but already had a C compiler at that point.
So I got to see the language world in a different way, given the local choice of languages as I mentioned.
EDIT: Reformulated the answer
We only got to buy the compilers that were available on the computer local store, not always 100% original or find some magazine and order international via post.
BBS access was only available to a few fortunate capable of paying the high connection rates and the modem in first place.
We got to do with what was available to us and could afford to pay.
Some of my first Assemblers were taken from the Input magazines and typed in, because there was nothing else.
It wasn't any harder to buy stuff in Western Europe in the early 80s than it was in the US.
Average salary would be around 300 euros when converted for today's currency
We had just gotten out of a dictatorship.
To me, C did have a standard library: Unix. It's a runtime system too! Due to the nature of the original C bootstrapping process it just happens to be possible to remove this standard library, and Windows was evidence of this.
There is another interesting potential counter example: Lua. It's minimalistic standard library is part of what makes it so attractive for embedding, eg. in game engines. However, Lua's embedding API is so good, you could almost say that it comes with a large standard library too: Your existing C code!
I guess my larger point is that languages rarely are able to stand completely on their own. They need some sort of valuable body of code to justify people to choose the language and libraries together. It might have been the case 40 years ago that you'd reasonably choose to build something "from scratch", but today, if you start on an island, you need to build a bridge, lest you remain on an island forever. Better to start on the mainland.
It's one thing to build a layered system with a small core. It's another thing to completely ignore the fact that the libraries and community _are_ the language, in the only ways that actually matter.
> It's another thing to completely ignore the fact that
> the libraries and community _are_ the language, in the
> only ways that actually matter.
Things have been getting better lately because of Luarocks but its still an uphill battle.
foo:gsub("%s*$", "")
foo:gsub("^%s*", "")
return setmetatable(self, mt)
There are many criticisms one could make of Lua, but I don't think those two particular criticisms are legit. They're classic bikeshedding.
s.rstrip()
s:gsub("%s*$", "")
Lua's philosophy seems to be to include the absolute minimum that is unacceptably painful to omit. This is a perfectly reasonable tradeoff for Lua's primary use case: embedding.
With respect to strings in particular, most systems that Lua is embedded in has its own string type, or inherits one from a framework. This is an unfortunate reality of the C/C++ world.
Returning to my point about language standard libraries: The lack of a traditional "standard library" is a feature for Lua, but only because Lua has a strong FFI and C API that acts as a "bring your own standard library" mechanism. It's less about needing a standard library, and more about admitting a language is only one piece of the puzzle. For a language to flourish, you need to have some story for interfacing with the rest of the world in a rich way.
See, its not that simple :) http://lua-users.org/wiki/StringTrim
Anyway, I wasn't trying to say bad things about Lua with my examples. Its just that if you go to any large Lua project out there there is a very good chance you will find some "utils" module in there with yet another reimplementation of a lot of these common functions. Ideally we should have people reusing more stuff from Luarocks than they are right now.
Fully agree. We just ended up with ANSI C + POSIX, because the standard bodies refused to put everything into the same bag.
In the early days, most C compilers were anyway shipping partial UNIX APIs on top of their K&R and ANSI implementations.
C would not have gotten where it is today if it wasn't for the rise of UNIX in the industry, fueled up by free UNIX clones.
I think one reason (of many) that C++ has replaced C almost completely for new development is the STL. Of course, the STL fundamentally depends on the language feature of templates, which you can only approximate in C, but considering that Java and Objective-C, among other languages, lasted pretty long with no generics and only non-type-safe containers, I think C could have benefitted greatly from basic things like resizable arrays, hash tables, trees, better strings, etc. in the standard library. Now it is probably too late for it to matter (which most people consider a good thing).
It has slightly awkward but very simple API, and it's very fast.
Why? What does that gain?
The standard platform provides Data.Map for maps, Data.Vector for arrays, and Data.Sequence for fast-edit sequences.
It's not even clear what a "built-in" array or map in Haskell would even look like, or what semantics it should have. Especially in a pure functional language, you need to be clearer about what your intentions are. A regular mutable packed array won't work most of the time.
This is precisely why shipping things in the standard library is a bad idea. It ends up full of cruft that no-one uses because there are better alternatives.
> This is exactly the main problem with Haskell.
> A stunning language with a lousy standard library.
Rust does this well with iterators, for example.
Data.Vector provides a very efficient stream implementation for vector operation fusion, but it's unsuitable for iterators/streams that interact with the real world. Pipes, on the other hand, combined with FreeT, provides good, reasonable semantics for effectful streams.
As with many other things, Haskell forces you to be honest with what your code is actually doing (e.g. streaming things from a network) and this means that there's no one-size-fits-all implementation we can stuff everything into.
I haven't looked at pipes, but I'm guessing it doesn't all fuse away either.
Given that we're in a rust thread... are you familiar with rust's iterator fusion [0]? Basically there are three components: iterators (something like a source), iterator adapters (where all manipulations happen), and consumers (something like a sink). LLVM will compile all the iterator adapters into a single manipulation such that the underlying stream/vector/whatever only goes through it once.
I personally like it much better than Haskell's. With rust the fusion is guaranteed to happen, although it makes the types a little verbose and tricky to work with, but with, e.g., Haskell's Text's stream fusion I was never really sure that it was working, or if I could do something to prevent it. It seems like in Haskell it's more of a behind the scenes optimization that you hope kicks in, rather than designed into the types. Or do I misunderstand? I only dabbled in Haskell.
If I had some Haskell code that was like
map f . map g . filter x . map y $ stream
It would almost certainly get fused into a single low-level loop without extraneous allocations. However, I can also do something like
foo = map y $ stream
bar = map f . map g . filter x $ foo
baz = map z $ foo
And now what do you do?
Haskell's fusion is also more general, because it allows you to do pretty much arbitrary syntactic transformations.
Unfortunately, this means it's somewhat fragile and is easy to prevent from functioning. Rust can guarantee fusion because you're restricted in the kinds of things you can do with iterators.
On the other hand, Haskell's Pipes restrict you from doing things like re-using an iterator, and I'm not sure what the optimization story is there.
https://github.com/matthiasn/talk-transcripts/blob/master/Hi...
And they work! It's not stream fusion, but the composed functions being applied to whatever container or stream of values are applied per-value, so (map (comp xf1 xf2)) applied to [1 2 3] applies (xf2 (xf1 1)), (xf2 (xf1 2)), and so on, with similar allocation savings to stream fusion.
All five of those string types do different things. This isn't a problem; we just have increased expressivity. We couldn't fix this by having a more coordinated standard library. 5 is also a very manageable number IMO.
>It's too difficult to make larger changes as we cannot atomically update all the packages at once.
That's what Stack is for, no?
For a language to remain relevant in the long term, a system must be capable of evolving. Going out there and blessing potentially conceptually incorrect packages such as Mio is therefore not a good idea. The notion of "platforms" best resides in userland, where collections can compete and evolve.
By keeping platforms / package collections in userland we can get organic collections of stuff that make sense to bundle together. Imagine a "server platform", an "event loop platform", a "calculus platform", a "programming language platform". It would be detrimental to creativity to have all these collections live in the shadow of the "rust committee blessed platform".
But so yeah, I'm not opposed to the idea of platforms - I just don't think blessing userland stuff from the top down is the right play in the long term.
Tl;Dr: package collections sound like a cool idea, but if you care about the long term, imposing them from the top down is bad
It sounds like the proposal in the OP avoids this problem by having versioned platforms that are independent of the Rust version. So if something turns out to be a bad idea, it can be stripped out of later platform versions and replaced with something better without disrupting users of the older platforms.
I don't see why that couldn't be worked into this idea.
Completely agree. Don't underestimate the amount of effort it costs them though.
That's exactly what we want to do. We don't have the domain expertise in many of these areas, but we want to enable those ecosystems to help develop their own platforms. It's one of our goals to make sure any infrastructure we develop for the "Rust Platform" is usable throughout our community.
Write standalone tools, but don't create a "platform". Don't make the use of the language dependent on your tools.
C++ does not have a "platform". Nor does it need one.
Then, of course, the other installers will gradually break and be abandoned. The effect is that users must run the "Rust Platform", unless they have the resources to build their own distro.
Is there a monetization scheme behind this? Does someone aspire to be the Canonical of the Rust ecosystem?
What other installers are you referring to, exactly? The old rustup.sh which couldn't support multiple toolchains installed alongside each other? multirust which didn't work on Windows? rustup is a massive improvement over both, IMO.
> Is there a monetization scheme behind this? Does someone aspire to be the Canonical of the Rust ecosystem?
This seems like a complete non-sequitur. I have no idea what you're trying to suggest here.
Of course not. Spreading conspiracy theories isn't appreciated.
Helpfully, many of these platforms reinvent basic things like strings [1] and reference counted smart pointers [2] in incompatible ways.
Wouldn't it be better if there were just one platform?
I think the ability to opt out of the rust-platform metapackage is a great feature.
[0] https://en.wikipedia.org/wiki/C%2B%2B11#C.2B.2B_standard_lib...
I work on multiple medium-sized projects that disagree. If you're not writing GUI code, it's quite possible to write 99% platform agnostic code without the help of a 3rd standard library supplement, especially with C++11.
In a high-level language like Python there's always a right solution. It's just rarely well-tailored to the problem.
I'm not at all surprised C++ is still muddling around in the middle somewhere.
Sure it does, it is called C++ Standard Library and ISO C++ catching up to what other languages already offer.
My suggestion, look at how Stack (the tool) and Stackage (the platform) work:
https://docs.haskellstack.org/en/stable/README/
My email is what finally moved the committee and GHC devs on including Stack with the Platform and on some other decisions concerning the website.
Like I said, not all the downsides may be applicable to how Cargo works or what aturon has in mind, but please don't cite it as an exemplar of anything.
On the first front I guess it's alright (but stackage is better) and on the second front it's totally inadequate—maybe even harmful in that it probably made the problem just less painful enough to spur procrastination.
I think most of this lesson doesn't apply to rust, but still good to be aware. It certainly gave me a strong eversion to course-grained dependency management as a bandaid.
Were. Were hard. It's quite fine now with Stack. I know users of all kinds of languages that all miss Stack when they're working on their non-Haskell projects.
EDIT: To elaborate on why I like Stack:
+ Fully declarative. I don't run commands to edit my Stack environment, instead I modify the relevant stack.yml file. This means that the current environment can always be easily examined, committed to Git, etc.
+ Easy NixOS integration which means I can also describe the non-haskell dependencies for a project, and enforce that _absolutely_ nothing else on my system gets used. This is amazing.
+ I like that it uses Stackage by default, and that I can pin projects to LTS Stackage releases. This means that many users of my projects will probably already have most dependencies installed -- especially cool for small things like tutorials where users won't have the patience for long build times.
+ It reuses the already existing `.cabal` file format so it's easy to make libraries compatible with cabal-install, the other package manager in the Haskell ecosystem.
That said, nix-style builds don't actually address my issues with cabal-install. My (very personal) preference is to always have the current dependencies being used reflected exactly in a local file. I picked up this attitude from NixOS, but it's an extremely useful way of doing things -- the knowledge that as long as I don't lose that file I can always rebuild my current environment exactly is just too awesome.
I think that if you need to track version bounds as well (say you're developing a library) they should be kept in a separate file, and you should always have a master file that describes what you're currently using locally. Happily this is exactly what Stack does with my-project.cabal and stack.yml.
That said, I appreciate that others like cabal-install, so I do what I can to make my code easy to use with both package managers (the main thing is keeping aggressive upper bounds on the few libraries I maintain, which especially helps cabal-install users since they do dependency solving more).
The problem with Haskell Platform came down to a set of intersecting issues:
- A culture of setting very strict & narrow version bounds. (Based on known-good rather than based on avoiding known-broken)
- Tools (Cabal) that enforce dependency version bounds. If there's mutual/transitive incompatibiltiy in version bounds - the build fails, period. You had to figure out the problem and fix it yourself if there was a truly irreconcilable issue.
- Recommended installer on the website (Platform) was unnecessarily installing packages into the global package database, making you "stuck" with those versions for _all_ packages you attempted to build. Cabal was restricted to finding builds that abide by dependency versions narrowed to the ones provided by the global package database.
These problems led to beginners being confused by seemingly spurious build failures because Platform would fall out of date with the rest of the ecosystem. Cabal would be unable to find sets of compatible dependencies and say it couldn't build the package.
What non-beginners were doing to avoid these problems was:
1. Install the bare compiler, no Platform
2. Use package database sandboxes for each project
All of these (UX and technical) problems were solved without compromising dependency conflict enforcement via Stackage and Stack.
Speaking hypothetically, if Cargo behaves like Maven or Ivy and pulls in two dependencies who want conflicting versions (1.1 and 1.2, say) of a particular library and just picks a winner, then you'll never see something like this.
In the sense of being a sucking chest wound driving new people away, I agree.
>just include both
makes my guts wrench a bit though. I understand how you're (currently) getting away with it but yikes.
EDIT: Apparently coolsunglasses was referring to the Platform, not Cargo, with this comment.
From my perspective: one of the biggest issues in cargo right now. I know it's not the same as the HP problem, but current cargo is definitely not good at solving this.
Haskell Platform is a perfectly adequate example to take high-level inspiration for the core of this idea: use Cargo (which is like Stack/Stackage for rust) to help bootstrap Rust libraries when using rustup to install and upgrade (mostly equivalent to `stack setup`).
In Cargo jargon, a solutuon would be for metapackages could double as sources: `foo = { metapackage = bar }` to use foo from bar.
But we can do even better. In practice, while code will continue working
with an old metapackage version, people are going to want to upgrade.
We can smooth that process by allowing metapackage dependencies to be
overridden if they appear explicitly in the Cargo.toml file. [dependencies]
hyper = { metapackage = { rust-platform = "2.7" } }
I don't think we need to pick one way exclusivity. My specific use-case is making PRs for packages to work in kernelspace / in unikernels. The library night initially be packaged the easy way but then I'd use this. I don't want the PR recipient to also worry they might get out of sync with the platform as a side effect.
OTOH, compared to e.g. C++, where even strings aren't standard across multiple real large code bases (e.g. Gecko and Qt) having standard types especially for foundational stuff can be very useful if the foundation is right or right enough. Java having a single (though not optimal) foundational notion for Unicode (UTF-16 even though UTF-8 would be optimal) and a common efficient and spec-wise correct (though not design-wise optimal) XML API (SAX) was a key reason to develop Validator.nu in Java as opposed to e.g. Python (whose notion of Unicode varied between UTF-16 and UTF-32 based on how the interpreter was compiled!).
Still, at this early stage, I'd point to crates.io in general as the "batteries" instead of creating a layer of official approval on top. But I'm biased, because lately, I've been writing a second-to-market crate for a topic for which a crate already exists.
Only if it can't find the crate otherwise. Generally it just searches your library path and any directories you specify with -L, e.g. most cargo executables can be recompiled (the first compilation via cargo used to compile the dependencies) via `rustc src/main.rs -L target/debug/deps`
(And that doesn't change the main point, which is that you could still be explicit about what deps you're including)
As for dropping extern crate, it either marries rust to cargo, or it makes imports implicit. Neither is desirable.
Basically: people knowing will still have freedom, people not knowing will have an easier life.
And in the end, in 2016, you generally don't care if you download a few extra 100Mo. If you do, you will just find spend the time to find to have a smaller setup.
I suggest a trip to anywhere outside the western world.
- choosing a bare bone small manual install; - making the download in advance in a place where you can or during the night;
It's not like you have less option. You just have a clear default choice.
P.S. Thanks for all your work Steve, it's really appreciated.
Personally, I think that letting users (especially beginners) opt in to a larger set of starting libraries could be very beneficial for adoption.
People then gain experience with it and sometimes subtle issues emerge. Those issues can be fixed, either in Boost itself or when libraries move from Boost to stdlib (and sometimes the issues inform language features).
While C++'s standard library is not "batteries included" like Python's, it's been very gratifying to see it expand slowly and surely over the last few years.
One of the main constraints that would guide selection of crates for this platform metapackage is that they must have compatible dependencies. So package A depending on Bv1 and package C depending on Bv2 wouldn't work because they Bv1 and Bv2 would both have to be included, leading to a conflict.
But this information is (in theory) encoded in semantic versioning. Assuming proper semantic versions for crates, a target set of crates to be included could be specified, and then automatically the various sets of crate versions that do not have conflicting dependencies could be calculated.
These compatible crate/version sets could be automatically generated and published as metacrates.
Consider the following crate/version dependencies:
Av1 -> Bv1
Cv1 -> Bv1
Av2 -> Bv2
Cv2 -> Bv1
Av3 -> Bv2
Cv3 -> Bv2
By imposing an ordering on these compatible sets they could be automatically identified. compatible_A_C_0 is {'Av1','Cv1'}, compatible_A_C_1 is {'Av1','Cv2'}, and so on.
Obviously the semver could be wrong and unexpected incompatibilities could crop up. But couldn't these just be autogenerated and then voted on? Then the top best compatible sets will filter to the top and, de facto, the Rust Platform has been autogenerated?
> So package A depending on Bv1 and package C depending on Bv2 wouldn't
> work because they Bv1 and Bv2 would both have to be included,
> leading to a conflict.
Furthermore, it's more than just a constraint problem; there's also integration issues, testing the whole thing together, etc.
Are metapackages going to be available for others to utilize? If so, how will conflicts be resolved if packages require two different versions of the same package?