Habitat — A new approach to automation | Hacker News Reader

Habitat — A new approach to automation

habitat.sh

204 pointsChico7554 comments

I've been waiting for this announcement for a few weeks. Since the home page is quite lean on an actual explanation, let me give a rough summary of interesting points as i understand it. I apologize in advance if I have anything wrong.

1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

2. It attempts to move some configuration management primitives into the build pipeline so that the deployment config has enough flexibility for real-world use.

3. It intends to replace single host process supervision with a distributed model based on gossip.

This project is most definitely ambitious, and I'm not sure the whole goal is realizable but some aspects are particularly interesting.

1. The build process seems quite useful for building containers, especially if it gets some traction. The main drawback of nix is that no one knows how to use it. If this gets the backing of the hordes of chef-aware system administrators, we may have a good solution for isolated repeatable builds, especially if they focus on deterministic reproducible builds[1] as well.

2. Distributed systems are notoriously hard to configure in a general way. It seems like every company has their own magic combination of config management, monitoring, and scripting for keeping systems like zookeeper running. Its pretty clear that we don't have the right primitives for sharing this work. I don't think a container management system has the right primiteves for this either, because I don't think you can solve for all distributed systems in a generic way. You really need custom logic for the particular software you are trying to distribute. I'm not convinced that a gossip-based process-supervisor is the best approach for this, but it could give us a place to start collaborating on these primitives.

[1] https://wiki.debian.org/ReproducibleBuilds

>1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

Except the builds aren't at all repeatable in a meaningful way, judging by the example code on the home page. The 'do_build' function runs 'npm install'. This means that 1) the build container has network access (i.e. it's nondeterministic by definition) and 2) the declared dependencies do not fully describe the dependency graph. So, I really don't see how much it could possibly borrow from Nix if it threw out the most important part of it.

They used npm install in their example code? Well that was a poor choice. Having dealt with npm and node apps in my own build system, I have not found a good solution for deterministically building node apps. Does nix have a solution for this? Thankfully the prepackaged software with reasonable source build systems don't do craziness like this. See the redis package for example:

https://app.habitat.sh/#/pkgs/adam/redis/3.0.7/2016061320525...

Nix has a generator that generates a Nix file based on an npm package including SHAs and locked down versions. Unfortunately because typical JS packages use a bajillion dependencies each recursively these Nix files can be tens of thousands of lines, but they work.

Nix also has similar generators for rubygems, python, haskell and some other languages.

Nix's cargo integration is even cooler since cargo is already quite deterministic, you just put in a SHA of the final folder cargo creates with all the dependencies.

The nix solution is npm2nix, which runs npm against a package.json file and generates a nix expression for all the npm modules you need.

This mostly works, but occasionally you need to override the generated derivations to add implicit dependencies that npm doesn't capture, or otherwise tweak packages that rely on quirks of npm.

So basically it "does not work"...

Well, let's be clear, if it doesn't work, you know at compile time, not at runtime. I don't care how broken things are at compile time if I'm guaranteed that everything will work at runtime. That's kind of the whole point of nix. It's not impossible to write code with errors, but you should not be able to build as long as the errors exist.

Also, it does work. It's just not completely automatic.

I think the point is more that it's completely under your control. The application build would be done however best suits your infrastructure and requirements.

Using npm doesn't demonstrate repeatability, but you could just as easily build a given application from fixed, verified source in this step.

Many (bad) build systems will download stuff from the net and you may not even know it. I've discovered plenty of software like this because I used GNU Guix to build them, which does not allow network access to builds for which the checksum is not known in advance.

It's absolutely essential that a package manager constrain the environment in which builds are performed. Without it, the packages are doomed to be built in a nondeterministic manner and potentially tampered with.

You always have the option of running a local npm cache/registry. We do for npm and artifactory.

Not exactly the same as vendoring dependencies, but it's a decent compromise in some situations.

Thanks for trying to explain. So many times when I visit these websites advertising developer-oriented tools I just click around trying to find an actual explanation of what the thing does and how it works, only to close the tab a couple of minutes later with no clear idea.

The article link points directly to the Habitat demo. For those of us that don't already know what Habitat is, a better place to start might be the about page: https://www.habitat.sh/about/why-habitat/

Habitat seems like an entirely new Chef product to enable what they're calling Application Automation.

Relevant quote: "[...] automation must travel with the application, rather than be provided by the platform. Everything the application needs, from build dependencies, run-time dependencies, configuration, dynamic topologies, deployment strategies, secrets management, security auditing, and so on belongs with the application, because we're doing it for the application. Runtime and infrastructure layers exist to support the application, but the efficient building, deployment, and management of the application are decoupled from those layers."

Incredibly ambitious.

Containerization is disrupting a lot of their current market, so they need to adapt in some fashion. Given the fact that the only hammer they have at hand is ruby systems automation tooling it is not a big surprise they are aiming at these targets.

Unfortunately, IMHO, the scope is a mile wide and an inch deep in terms of what chef brings to the table here. I can walk down that list and either think immediately of another tool that is dedicated to the specific task or the needs are so slight that a few ansible or bash scripts take care of the need. Sounds like they are aiming to occupy some space in the general "tooling & orchestration" arena, but it will be an uphill climb.

This is an excellent point - kind of goes against my earlier comment. Maybe they have to move this way because that's their only choice - if they don't move into automating this, containers will disrupt them (or already has)

Comments based on the most trivial skimming of the docs, so I'm probably wrong about everything.

> Incredibly ambitious.

It is, but ambition is necessary if you want to build distributed production systems. BOSH[0] does something similar in providing an all-or-nothing approach to configuring distributed systems built out of heterogenous software components spread across multiple machines.

Note here that the key is the heterogenous part -- if you have an undifferentiated flock of software, you can rely on a different abstraction to deploy them (anything from PaaS to a hand-rolled thing, though I don't recommend hand-rolling this stuff). But very few distributed systems are designed to rely on emergent behaviour. In fact that's usually what we're trying to avoid.

Where they appear to differ is that BOSH is quite strict about lifecycle -- you can't "truly" update configurations on-the-fly. You update the relevant YAML and then ask BOSH to update the world for you. It seems like a hassle, but if you couple it with git and Concourse pipelines, you get a complete, externalised history of every system configuration change you've ever made.

We already know that this top-down model of heterogenous distributed systems configuration management works better than trying to extend it from the machine up (as Cfengine and its various successors tried to do). BOSH has been deploying large systems for years and years. It was originally designed to deploy and upgrade Cloud Foundry without downtime, something it does with aplomb.

What I really like about Habitat over BOSH is the way it exposes the available knobs and dials. That kind of interactive discoverability is super helpful sometimes.

I was, however, struck by this claim:

> This architecture allows us to avoid the production cliff problem: the closer you get to production, the more difficult it is.

This is literally the problem that PaaSes solve for you. I've seen companies without PaaSes, with home-rolled PaaSes and with Cloud Foundry. In the latter case, the biggest obstacle is usually getting VPN access. And there are other PaaSes apart from CF. Habitat seems to be doing a kinda-sorta PaaS here (note the "Supervisor"), but without most of the feature plumbing that makes PaaSes operationally useful: routing, logging, service injection and the like.

Disclaimer: I work for Pivotal, we do the majority of the engineering on both BOSH and Cloud Foundry.

[0] https://bosh.io/

Found this elsewhere, interesting to note, Habitat looks to be built mostly with Rust

https://github.com/habitat-sh/habitat

This is Adam - Rust has been amazing to work with, and was the prefect choice for this platform. It's great.

You should reach out to the Rust team if you haven't already. I'm sure many would love a blog post on using Rust. I was thinking about using protocol buffers for something in Rust, it's cool to see an example of a real app using it, thanks!

We're totally going to do it.

Could you elaborate a bit more on why you chose Rust, and why you think it was the right choice for this project?

On the heels of this, I've been exploring Rust a bit lately but haven't been able to find any solid patterns / style guides for idiomatic Rust. Do any such resources exist? For example, the official rust book describes Arc, Mutex, etc but.. Are there more of these in the standard library? What purpose do they serve? What are some good idioms to apply when using them? I can surely make them work, but maybe I should be using some other thing?

You might want to join users.rust-lang.org.

Like most languages idiom is learned while working with the language. Something I've found for instance is to try and use Iterators to get around sharing rules as they can encapsulate sharing patterns in a lot of cases.

The specific modules you mentioned are for multi-thread support, those are only necessary for sharing memory between threads.

Anyway, I think this is off-topic for this thread.

Sure. Habitat's supervisor is a very low-level thing - it manages the lifecycle of your service, participates in the gossip ring, etc. That means a failure at runtime of the supervisor is pretty unacceptable, and similarly, you don't want anything (like a long GC pause, for example) to get in the way of the supervisor doing what it needs to do.

Rust gives us 3 things we needed, and it says so right on the website: fast, no segfaults, thread safety. Additionally, it's community is top notch. We've found the language to be great to work with, especially when you're refactoring. The type system makes it clear where it's possible for the code to fail, making it easier for us to reason about keeping the supervisor safe.

I wrote this while I was evaluating languages for the early habitat prototypes: https://medium.com/@adamhjk/rust-and-go-e18d511fbd95#.d35imt...

Best, Adam

Home Automation is what first came to mind when I read the title. With this in mind, reading the blurb on the website sounded like, oh they're talking about building home automation software...

"build automation, IT automation, deployment automation, etc" would have saved me some confusion. Or simply compare yourself to other solutions out there will help me understand what category this new piece of utility and name fall into.

Yep. Would made perfect sense for the name as well.

I wonder if this project is doing too many things. It includes a package builder and a supervisor. Is there any benefit to integrating the two?

For a project that's smaller in scope but addresses the part about packaging the automation with the application, Joyent's ContainerPilot [1] looks interesting.

[1]: https://www.joyent.com/containerpilot

The GitHub project: https://github.com/habitat-sh/habitat

I'm not sure what I'm looking at. What is being "automated" here? The headline and home page don't give much context.

The main points around automation seem to be[1]:

* Flexible versioning and configuration rollout

* Ability to deploy anywhere

* Ease of management

Concretely, they're touting their gossip-based configuration and topology management layer as a way to send configuration updates to service groups, and then letting the service groups apply the changes to all the group members. Contrast this with a more traditional way of propagating changes across a network by enumerating every single host/VM/container. I'd imagine another large component of the application automation would come in the form of tooling allowing the ability to deploy anywhere (bare metal vs. VM vs. container vs. IaaS/PaaS providers), but haven't seen that just yet. Maybe it's being covered in the live announcement.

[1] https://www.habitat.sh/about/why-package-automation-with-app...

So but am I automating devices in my home/building IoT-style? Or am I automating deployments of software? Or hardware? "gossip-based configuration and topology management" means nothing to me unless I know actually what the overall context is here.

From what I'm reading, I believe this is primarily geared towards trying to automate software deployments.

The two-word answer seems to be "software builds".

I am always amazed at people's capability to give an extremely complicated answer to an extremely simple question. "What is being automated here?" "Software builds."

I watched the videos, did the tutorial, read the docs, read the tech site stories but I still don't see why I would use Habitat over the current Docker workflow and ecosystem (including third party) for building, shipping and deploying applications. Both are a type of packaging that puts the application, its dependencies and configuration in a bundle that can run exactly the same everywhere. Docker has Dockerfiles and docker-compose.yml files, Habitat has plan files (seems like a shell script, Dockerfiles and compose files are much simpler). Docker has a central repository/registry "docker hub", habitat has depots. "docker run redis" == "hab start core/redis", etc... etc...

The docs/articles say habitat apps can run in containers, but why bother if your app can already be run in a container that achieves the same thing with even greater isolation (cgroups). Why put a self contained app in a self contained container application?

I'm not trying to put down Habitat, I'm trying to understand its role. As I currently see it, why would I choose this over the more mature Docker environment with a massive established ecosystem and user base?

Again I'm just trying to understand what I'm missing, if I'm missing something please point me in the right direction.

This looks really interesting. Im waiting for 8:30PDT for the full announcement though!

Im curious about how it works under the hood, it seems to solve 'all the problems' ;)

My skepticism comes from some of chefs past product releases, which have seemed rather, half baked.

Products like chef metal (provisioning), pushy (push jobs) or delivery...

This is in no way negative as I am a long time chef user and fan of chef.

Hmm, I very much like the concept but when I see the need for supervisors and a habitat service, it makes me wonder if this is really what is says it is.

If I need to pre-install all these services on my hosts before any of this will work, then how is that really different from uniformly deploying all my applications on an application server such as WebSphere? An EAR too contains all my configuration data, my bindings, my security settings etc. So...if you uniformly run WebSphere everywhere in your enterprise, it sort of accomplishers the same thing.

HOWEVER, if the habitat configuration is capable of talking to my F5 load balancer to create a resource pool, self register my app with my corporate DNS, create a DB schema and populate it with my tables, talk to my networking equipment and open all relevant firewall ports and do so when the application is deployed to a new environment, as well as undo all these actions if it's ever moved elsewhere, then you have piqued my interest...

There's no pre-installation - the supervisor is bundled with the application deployment itself.

Theres a live announcement now https://bignews.chef.io/

I love the Chef ppl, but...

Is it just me or are there a ton of other folks doing this? The idea of introducing yet another layer of stuff is scary, and there's lots of work left to do on Chef. In a lot of ways, they feel just like Nomad (coincidentally also on the front page)... I wish they would spend some time on their core products versus expanding into an overfilled market.

That's what I thought, too. Chef (software/ecosystem) needs a bit more love IMHO.

However it shows that Chef (Company) is still able to design, build and ship a new product, as an organization. That's a very positive sign. Until recently I thought when comparing Chef with Hashicorp: Hashicorp is so much faster in building and releasing usable stuff but they seem to feel the pain in growing too fast and too broad in terms of products.

So, in the end: It's a great thing, just also don't forget to put more love into Chef (software) and don't dissipate developers/time/budget :)

As nice as the 'console' on the page is, the fact the window gets cleared when you click to go to the second step makes it look like it lost what you did on the first page. To me that makes it look like it's broken - the steps should maintain the state and build on each other, not start over each time.

Is this like Docker?

I had the same question

Nope! You can use it with docker, and we think it makes the experience better. We love docker.

Is there anywhere that actually lays out the tech that powers habitat? ie:

This thing uses kubernetes or this thing uses mesos This other thing uses docker/rkt/containerd/lxd This is the set of things we wrote in rust, etc, etc.

Rather like nix + consul.

How does this compare with Heroku's buildpack system?

That jumping header is quote annoying, I guess the developer did not calculate a proper padding to accomodate for a sticky menu once it becomes fixed positioned

We updated the link from the live demo page (https://www.habitat.sh/try/) to the homepage.

Ah, another limiting abstraction layer to learn and to fight.

Bash is plenty good.

But all of the package recipes are just Bash scripts, which I think is just terrible, btw. Using a real programming language would have been a much better choice.

We struggled with this! I think we found the right balance, but time will tell.

> Using a real programming language would have been a much better choice.

The problem is that everyone disagrees on what a tolerable real programming language is. Some refuse to use Python; others Ruby; others Scheme; others Lisp.

At least every can tolerate shell more or less.

Can I also buy a sofa? http://www.habitat.co.uk/

Reply on news.ycombinator.com