Comment by tptacek | Hacker News Reader

Comment by tptacek | Hacker News Reader

Parent Full thread

There are multiple reasons, some simple and some subtle. We don't totally have the bin-packing problem Nomad is designed to solve. We do have "users have fussy and meticulously specified ideas of how they want their apps to deploy", which is not really what Nomad is designed for; Nomad is designed to decouple your pool of compute from your applications, and to make it low-drama to deploy new applications to that pool of compute. But our users come with some level of drama built-in; it's an impedance mismatch.

I like Nomad a lot. I think it's a better fit for what we're doing than Consul is (they're both great, but I think Consul is the bigger sticking point). You'd get different opinions from different people on our team, but I think Nomad's going to be around for awhile. But there are things we very much want to ship that we can't ship on Nomad.

I would 100% use Nomad again if I had a somewhat complicated app I needed to deploy and couldn't use Fly.io for. (I'd use Consul too, for that matter.)

Per the sibling comment: we are indeed writing a new orchestration system in Go, called `flyd`.

I'm curious what kind of things do you want to ship that you can't ship on Nomad?

The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs.

We also need precise placement. We have people who want to run 8 VMs in London, 3 VMs in Los Angeles, and one VM in Sydney. They also want to do rolling deploys that revert on failure. Nomad's scheduling is not that precise.

Databases too. Both Nomad and k8s have ways to do persistent workloads, but they rely heavily on the underlying infrastructure to "move" volumes between throwaway containers. We don't want that, we actually want something closer to old school VM provisioning than "orchestration".

Hiring? Sounds like a match for problems I find interesting.

Soon! We're polishing up a platform engineer role now. It'll appear on https://fly.io/jobs/

If you find these kind of DevOps problems challenging, I'm hiring. :D

> The giant feature is "start a VM in less than 1 second". Followed by things like suspending/resuming VMs.

Is this for scaling existing apps up (eg flyctl regions add ...) or for new deployments (flyctl deploy)?

I suspect the former (scaling existing) as artifact distribution and deployment parameters like health checks seem like they prevent subsecond deployments (even ignoring the time for initial artifact uploading/building). Although my view is probably quite biased by Nomad having a much better chance at solving the former than the latter!

(Also hi! I appreciate our chats in the past and feel free to reach out if you're willing to brain dump Nomad gripes on me. :) )

It's very different than Nomad! The basic model is direct "machine" management with very little orchestration. Machines are kind of like allocations, but more permanent.

Some machines operations are slow, some are fast. Create is slow, update is slow. Start and stop are fast. Future "suspend" and "resume" are very fast.

For the regions case, people use these by having a couple of machines per region they care about. Then they start and stop as needed. A stopped machine can be migrated between nodes, but we ensure it's ready to start quickly if we do that.

Some interesting things happen with this model. Create is slow, but we are able to make it fast _sometimes_ because we can target creates at hardware that already has the rootfs cached.

Updates don't work like nomad, we update machines in place for a rolling deploy. This lets us do things like pull an image in parallel, then stop the machine, update, and restart very quickly.

We're also doing things differently with health checks. We may not wait for an active health checks before sending a machine an HTTP request. Many workloads don't even care about health checks, they just want a VM running and streaming logs back as fast as possible.

There are features I could ask for in Nomad to do what we'd need, but they'd make it not Nomad. :D

> Both Nomad and k8s have ways to do persistent workloads, but they rely heavily on the underlying infrastructure to "move" volumes between throwaway containers

I did this with Nomad by writing a small API to "place" host volumes which Nomad can use for job placement constraints automatically. It works shockingly well for my use-case which I think is somewhat similar to Fly's? A smallish number of containers that require host persistent storage, and don't expect anything to move -- just disappear forever (persistent til' it ain't).

This is what we do now, basically. We have a Nomad Device Plugin and create local LVMs. We also have plumbing to move these between hosts. It works pretty well, it's just very limiting.

What we need is to be able to target machines to specific volumes. And for things like database clusters, we need to control restart order during upgrades.

> we are indeed writing a new orchestration system in Go, called `flyd`.

I know it's just a wild coincidence, but I couldn't help but be reminded of https://github.com/flynn/flynn

I'm friends with at least one Flynn person and I'm always wondering what they'd think of our weird architecture. There's a lot of interesting code in that repo!

I hope there will be a blog post about flyd! I love reading about what fly.io is doing.

Reply on news.ycombinator.com