Hacker News Reader

Comment by JoshTriplett | Hacker News Reader

I used to describe my preferred family of languages as:

- C when I absolutely had to (kernel/modules/plumbing).

- Python for scripting and broad accessibility.

- Haskell when I had the choice and I knew everybody who would work on the project.

I was skeptical of Rust when it first came out, due in large part to the many different kinds of pointers it originally had, many of which involved significant manual memory management. But now, with a strong static type system, garbage collection, pattern matching, associated types, and many other features, Rust is looking like a serious contender to replace all three of those languages for me.

Still waiting to see if it develops a strong following, community, and batteries-included library ecosystem, but I need to start doing more experiments with Rust.

Disappointing to see yet another language-specific package management system (Cargo), though.

View on HN

> Disappointing to see yet another language-specific package management system (Cargo), though.

It's been a huge step up from makefiles in developing Servo. I can barely ever bring myself to go back to using make and git submodules now.

The workflow of breaking up your project into small, self-contained packages that people hack on independently and are all built with a package manager that natively understands the language and can build without a single line of shell is a huge improvement over vendoring dependencies and dealing with a maze of configure scripts or arcane make/(insert your preferred replacement here) files. It improved compilation time a lot too!

> It's been a huge step up from makefiles in developing Servo.

As a maintainer of a number of Rust libraries I highly agree! It has definitely won over a few skeptical collaborators of mine (who were fans of Make).

As others have said, having a package management system that deeply understands the language and tooling is awesome. Examples:

- Rust has a distributed-by-default documentation generator (rustdoc), cargo knows this and provides `cargo doc` to render a library's docs with it.

- rustdoc can run code examples in the documentation as tests, to check that everything is up-to-date, `cargo test` does this (along with running the in-source unit tests and any external tests).

- the Rust compiler allows for plugins, which are dynamic libraries loaded into the compiler and can be used for things like custom macros (aka procedural macros aka syntax extensions) and custom compiler warnings. cargo understands these, allows them to be 'imported' via the normal dependency mechanism, and specifying `plugin = true` in a package makes cargo do the right thing, e.g. building as a dynamic library (static libraries are the default) and compiling for the correct target when cross-compiling.

I'm sure all of this is possible with other systems, but it seems unlikely to be so nice to use.

It's very similar to Racket, and yes, it is nice to use!

Other systems can get you much of the way there (node, Python are the only ones I'm really familiar with) but I suspect you need a little language help to achieve the same kind of convenience.

Rust does not have any garbage collection, to be clear. All your other features are correct though :)

(We have previously said "opt-in GC" but that was a lie. See https://news.ycombinator.com/item?id=8312327 for more.)

Yeah, I should have said that more clearly. Automatic memory management (never having to call free), not garbage collection. Which is arguably more awesome: whenever the compiler can figure out at compile time when you'll stop using memory, it can statically decide to reclaim it there.

Yes! The fact that Rust has zero-runtime-overhead automatic memory management is a big deal. I've been using Rust for almost a year, and it's been a source of great delight for me.

I am a huge fan of no required GC, and of precise refcounting. But does Rust have anything on the roadmap to handle truly cyclic structures that don't lend themselves to a weak pointer approach?

One option would be to allocate objects out of an arena[0] and then destroy the entire thing at once when you're finished with the contained objects.

[0] http://doc.rust-lang.org/arena/index.html

Makes sense.

If anyone is interested, I think it could be interesting to see if the refcounting approach I developed for handling cyclic references in my library "upb" would work as a Rust library:

https://github.com/haberman/upb/blob/master/upb/refcounted.h

The basic idea is that you refcount groups of objects instead of refcounting objects independently. You compute the groups dynamically such that no cycle can span groups. This is sort of like an arena, except that no arena is ever explicitly created, and the collection can be more precise than an arena.

In my library, objects go through a two-phase lifecycle. First they are mutable, then they are frozen, after which no further mutations can be made. When an object is frozen, its outbound pointers are also frozen.

The nice property of this scheme is that you can perfectly compute these "virtual arenas" by computing strongly-connected components at freeze time. This makes the scheme optimal for frozen objects. For mutable objects, the groups are computed more conservatively and objects may not get freed as often as they otherwise would with a perfect scheme (ie. the same downside of an arena).

I would love to know how this scheme might possibly be modeled in Rust. Since Rust also has strong semantics around mutability and immutability, it seems possible that a very nice idiomatic Rust interface could be implemented around this scheme, giving more precise cleanup than an arena while still allowing circular structures.

This sounds like an interesting variant on a Train collector.

Maybe try to avoid these as much as you can - they make the code complicated in any language - and if you must, then manage the memory yourself?

I think what you want is a cycle collector, and to invoke it at appropriate points. I don't see why Rust couldn't have a refcounted pointer type with an invokable cycle collector, but maybe someone on the core team could chime in here.

I'm curious if other approaches are viable also, like I mentioned here: https://news.ycombinator.com/item?id=8321618

Interesting!

Curious to hear more about language-specific (though OS-agnostic!) package management systems. IMO composer is the best thing ever happened to PHP, Ruby gems are huge, Python eggs also make a very useful ecosystem.

OpenSUSE's Open Build System would be great to ship independent packages, but those are again heavily tied to Unices, hence leaving other platforms behind.

> Curious to hear more about language-specific (though OS-agnostic!) package management systems.

As far as I can tell, one of the main justifications for most language package management systems is "we also run on Windows/OSX, which has no package management, so we'll invent our own". As a result, users of systems that do have sane package management get stuck with multiple package management systems, one for the distro and one for every language. Even then, I find it disappointing that nobody has built a cross-platform package management system for arbitrary languages to unify those efforts.

The problem with language package management systems is they've been used for installing user facing software. As a developer tool I think it is the perfect way to go.

And you should add Linux to your Windows/OSX as being an issue, which Linux package management tool would you build packages for? All of them?

The end user package management provided by the OS should be for installing end user packages and the language tool for installing and publishing libraries and dev tools.

> The problem with language package management systems is they've been used for installing user facing software. As a developer tool I think it is the perfect way to go.

Precisely so.

The other justification is generally a clash of cultures: the people who maintain distro/OS package managers generally come out of the culture of sysadmins, who value stability over feature-richness, while the people working the language communities generally come out of the culture of developers, whose priorities are the exact opposite.

When languages try to hook into existing OS-level systems, the people on the language end get frustrated by the way the people on the package-manager end don't hurry to rush out bleeding-edge versions of packages the second they hit Github. To the package-manager people, that's no big deal, their orientation towards stability and predictability makes them comfortable with waiting a little for the coffee to cool. But to the developers, who want to get their hands on the Latest and Greatest Right Now!, it feels like slogging though molasses.

So the developers eventually end up blowing their stacks and stomping off yelling "Yeah? Well fine, we'll build our own package manager then! With blackjack! And hookers!"

Maybe OS-level package managers should default to stable, but let the user check a box to get the latest and greatest. Developers want a stable system like everyone else, but for the stuff we're hacking on, we have a legitimate need to get the most recent, so our software isn't obsolete by the time we finish it.

Most OS-level package managers also aren't designed to install more than one version of a package at a time. They don't tend to integrate with build systems as well, either.

I'd be happy enough if the OS-level packagers stopped modifying the package-level packages they packaged.

That's not so simple. A distro is a fine-tuned collections of packages which work more or less well together. Debian, for instance, comes in stable/testing/unstable/experimental flavours, depending on how daring you are. But even this isn't a universal solution. If you are deploying for instance a web application, you will want to deploy a locked down number of dependencies as well, regardless of what is present on the target system. And you may need deploy multiple applications side by side. Few system package managers have an answer to this.

So developers end up installing later versions manually. And in many cases it's no big deal. If the distro has Julia 0.2.1 and Emacs 23, I can upgrade to Julia 0.3 and Emacs 24 and it's not likely to damage anything. It'd just be nice if I could do it with the package manager instead.

But just because I'm doing that doesn't necessarily mean I want, say, the latest unstable version of the window manager.

Debian will let you do that. You can run, say, your machine on testing but get the latest Firefox from experimental if you want. This may, however, upgrade other dependencies on your system, but it's pretty much unavoidable.

> As a result, users of systems that do have sane package management

Given the diversity of OS in the IT landscape, which systems are those?

I met only one package manager that I don't need to fight in order to get what I want: Gentoo's Portage. With a local overlay and language specific functionality concentrated in eclasses it's trivial to add new packages, do version bumps, have fine grained control over installed versions, enabled features, etc.

The distro only contains a small selection of the packages (even if there are hundreds or thousands of them) and the language package system is usually the source the distro maintainers use to find the packages anyway.

Most of the unix package management systems are not really that good for development: they don't support sandboxing, or non-root installation/execution, or don't support the right kind of versioning, or don't support multiple versions of a library, or some other issue.

The only existing system that I might trust enough for this kind of thing Nix, and at least for Haskell it is a pretty solid alternative to their language-standard package manager. Unfortunately, its popularity is way lower then stuff like apt or yum which are pretty shitty for development compared to something like pip or cabal.

Lately, IMHO, Gradle in Android Development(applicable for Java development as well) is a huge improvement over managing dependencies with pom.xml(ant) and linking jar files manually. Besides, you can totally customize build.gralde too.

>Disappointing to see yet another language-specific package management system (Cargo), though.

As a packager in a Linux distro, I'm disappointed every time somebody tries to cram in PL-specific packages inside distro packages.

Interesting, how do you get around it, when those packages are depended upon by user facing software?

> Disappointing to see yet another language-specific package management system (Cargo), though.

Coming from Python I find Cargo very very smart and very well thought so far, it is not feature heavy, but everything has a very clear and useful purpose. For instance today I found that if I created a file .cargo/config I could override my dependancies to make Cargo search projects on my fs instead of grabbing them on Github, while doing developments it's a big thing I think.

> Disappointing to see yet another language-specific package management system (Cargo), though.

I don't think it is. You need support for Rust modules on various platforms Linux/Mac/Windows(possibly Android). No single tool works on all those platforms. Cargo does and it has minimal dependencies.

Not having to juggle three different configuration (CMake, Makefile, etc.) on different platform is actually pretty great.

> Disappointing to see yet another language-specific package management system (Cargo), though

So what is the solution to have portable packages for:

- RPM systems

- Debian systems

- tarball systems

- Pkg systems

- MSI systems

- Mainframe OS

- Embedded OS

- Aix/HP-UX/Solaris package systems

- ...

The goal of the [nix](http://nixos.org/) project is to solve this, and every time anyone brings up a package manager on HN, someone has to mention nix. The reality is that nix is really nice, but isn't any better than making a new package manager until it has wide adoption, so no one is using it.

Nix was brought up during the discussion that led to Cargo, but no Windows support is a deal breaker.

Wouldn't it still have been less effort to port Nix to Windows, than to write an entirely new package manager and then port it to every OS?

If that were the only downside, possibly. I don't really do Windows development, so I can't tell you how difficult porting Nix would be. There's large advantage to having a packaging system that knows your language well. It's going to have tighter integration than a generic one ever could.

I would probably make the same decision, but I hope in the end Caro is easy to wrap with Nix, which is a breath of fresh air particularly when needing to mix dependencies that cross language boundaries and share those build recipes with a team.

Previously, I wrote shell scripts and worried whether everyone on the team had rsync installed, or xmlstarlet, or some other less common tool. Now I wrap those scripts in a Nix package that explicitly depends on all those and distribute with confidence. It's fantastic.

Bundler and rubygems, for example, do various things that make good support within Nix rough. Two examples: 1. rubygems has no standard way of declaring dependencies on C libraries; 2. as far as I know there is no way to ask Bundler to resolve dependencies, create a Gemfile.lock, but not install any gems (I realize github gems must be downloaded to see the gemspec...)

Cargo has the second, and there's a plan for the first.

That said, the reason that you want it to do the installation is that a lockfile is supposed to represent the way to do a build successfully. Without building everything, you can't actually be sure that the lockfile is correct. In theory, it should be...

> reason that you want it to do the installation is that a lockfile is supposed to represent the way to do a build successfully

Sure, and I'd like to do that build within Nix (and someone else might want to do it with another packager), which gives a stronger guarantee than Bundler since it incorporates C library dependencies and more. Anyway, the specifics aren't relevant to this discussion, and it seems you have a grasp of the issues, so carry on!

It seems to be only for GNU/Linux systems, what about all other OSs out there?

I've been experimenting with Nix on Mac OS X lately and it works fine. I've heard that it works on FreeBSD as well. The big gap is Windows.

The good news is that you can integrate your language-specific tools with Nix as well, such as has been done for Haskell, node.js and other things. (I'm looking at it so that we can integrate our Dylan stuff with it.)

When this discussions happen on HN, I always see a narrow discussion of Mac OS X, GNU/Linux, Windows and with luck *BSD.

But the world of operating systems is so much bigger than the desktop under the desk.

Good work on Dylan by the way.

I'd love to have the time and the resources to deal with more OSes. :) 20 years ago, I had to keep stuff running on Solaris and lots of other platforms. About 20 years ago, I still did some work on VMS on actual VAX hardware! It wasn't that long ago, that we had the possibility of BeOS either. Comparatively, we have quite a monoculture (of POSIX) these days with Windows being the non-POSIX representative.

Maybe unikernels like OpenMirage will help make things interesting.

And thanks! The work on Dylan is a lot of fun and keeps me semi-sane by keeping me busy.

Nix is much more than just a package manager though.

How would (or how does) Nix deal with windows anyways? Are it's abstractions portable enough?

> Disappointing to see yet another language-specific package management system (Cargo), though.

That's a much bigger problem to tackle, and one that nobody has really thought much on or done much work with. Package management is a tough problem in the abstract, and it's a big challenge to create a meta-package management infrastructure, more so when there doesn't seem to be any money in it (at least not easily).

> Disappointing to see yet another language-specific package management system (Cargo), though.

No, I don't want my language to be bound to someone else's package manager at all.

Reply on news.ycombinator.com