ejholmes

A true build system really needs to build out a DAG of dependencies to be efficient and fast. Compose this with https://github.com/ejholmes/walk and you're good.

walk is built with Go, which also means it's a single statically linked binary with no dependencies if you download the pre-compiled binary.

If you don't like bash syntax, you can use Python, or Ruby, or Perl, etc. Any executable can serve as a Walkfile.

It's a completely fair point that make is installed on pretty much every machine by default, which is why we won't see it going away anytime soon (nor should it, it's still good).

This looks awesome! Definitely very similar to walk(1). Thanks for sharing!

Consider yourself lucky! If I had a penny for everytime I had to explain automatic variables...

1. Yeah, hash functions are deterministic, but your input needs to be determinsitic across machines too. For example, on a unix system, you may want to conditionally build if any files have changed. To do that, you could generate a deterministic hash of the dependencies with something like `find . | sort | xargs sha1sum | sha1sum | cut -d ' ' -f 1`. Including mtime in that would break across machines.

2. Mainly because doesn't translate across machines; it only applies to your machine. If someone checks out the repo on their machine, mtime is different. As soon as you move a build system to CI, you need something better than mtime, like content adressable hashes, if you intend to cash targets.

3. It executes the Walkfile twice for each target in the graph; once to determine the targets deps, and once to execute the target. This definitely hasn't been a bottle neck for anything I've used walk(1) with so far.

4. Correct! But even more so, replace "shell script" with "executable". The Walkfile can be written in any language you want, as long as it has the executable bit set.

That works, until your build system is sufficiently large or time consuming or not C/C++ like make was originally built for. For example, at my company we have a build system for building all of our Amazon Machine Images (AMIs). It doesn't make sense to re-build images unless they're dependencies have changed, but mtime just doesn't make any sense for a build system like this. Trying to coerce make into doing what we wanted was like pulling teeth.

Also, I'm always surprised how little mention there is of graphs when we start talking about build systems. If you're making a build system (for literally anything) the DAG is your best friend. This is how all major build system tools work (make, terraform, systemd (yes, it's a build system when you think about it)) and it's how we're able to parallelize execution so easily. It's important to be conscious of the fact that this is what your doing when your making a build system; connecting a graph.

Highly recommend reading https://ocw.mit.edu/courses/electrical-engineering-and-compu... for some theory on parallel execution for graphs, if your interested in things like this.

Make's underlying design is great (it builds a DAG of dependencies, which allows for parallel walking of the graph), but there's a number of practical problems that make it a royal pain to use as a generic build system:

1. Using make in a CI system doesn't really work, because of the way it handles conditional building based on mtime. Sometimes you just don't want the condition to be based on mtime, but rather a deterministic hash, or something else entirely.

2. Make is _really_ hard to use to try to compose a large build system from small re-usable steps. If you try to break it up into multiple Makefiles, you lose all of the benefits of a single connected graph. Read the article about why recursive make is harmful: http://aegis.sourceforge.net/auug97.pdf

3. Let's be honest, nobody really wants to learn Makefile syntax.

As a shameless plug, I built a tool similar to Make and redo, but just allows you to describe everything as a set of executables. It still builds a DAG of the dependencies, and allows you to compose massive build systems from smaller components: https://github.com/ejholmes/walk. You can use this to build anything your heart desires, as long as you can describe it as a graph of dependencies.

I think I remember hearing that they were on EngineYard back in the day, but never Heroku.

> A single level perceptron classifier with weights estimated from sonar training data set using stochastic gradient descent.

I don't have a clue what any of those words mean. Neat!

If we distribute OCI images via IPFS, problem solved.

This is great. I really wish the container boom started with a more Unix philosophy around containers; package "images" as a tarball, run the container with a wrapper like runc. The container inherits Stdout, Stderr, Stdin, etc. You can pipe the output to log aggregation, etc. Small, sharp, reusable tools.

Containers should just be the new static binary. Instead, we have Docker "Daemon"'s that can crash.

Awesome article and really well written. I am curious why people choose to go with overlay networking. To me, it seems like an extremely complex solution to a non-problem (source, I run a 700+ container cluster without any overlay networking).

Great article. I'd also recommend people take a look at ThreatStack for aggregating syscall events: https://www.threatstack.com/.

This has been one of my favorite new additions to Go 1.7. Incredibly useful for debugging persistent connection pool efficiency.

I'm assuming you're referring to instabilities in Docker networking. I can't point to anything specific, but that's because we've seen it as an unnecessary layer of complexity, that's trying to solve a problem that most companies don't/won't have.

DNS + a load balancer is proven and stable. We have 65,536 available ports on linux, and doing port mapping to prevent conflicts across containers is trivial (Empire handles this for us). Client side service discovery is usually unnecessary for most Docker use cases (stateless services), where a load balancer is a better fit (why push load balancing to clients? Would you expect your browser to have to load balance when visiting google.com?).

I'm sure there are cases where networking overlays are a good fit, I just haven't found one yet.

Having been running a large scale production system on ECS for well over a year now, I'd like to say that ECS is hardly a bad choice; it's incredibly stable, integrates with proven AWS technologies (CloudFormation, ELB, etc) and it just works without having to manage clustering. All it needs is a layer of usability on top, which you can get from things like Empire and Convox.

While native Docker clustering is cool in theory, I think the vast amount of complexity around networking makes it unsuitable for production deployments at this time. If stability is more important than bells and whistles, ECS is great choice. Docker's production story, outside of simply running containers, has a lot to be desired.

You know, if you move out of San Francisco and let your employees be remote you can save an insane amount of money on office space, pay employees less and allow them to have a higher quality of life. Tech's insitence on being in one of the most expensive areas is just bizarre when most employees can pick up a laptop and work anywhere there's wifi.

Having been running Docker in a production system now for 6+ months, the size of the images is really a non-issue if you're using the Docker layer cache effectively. We use all of the official language images and builds are under a minute, with deploys happening in ~5-10 seconds (machines generally only need to download a single layer).

One of the great things about stuff like Convox and Empire is that, because they're built on existing stable technologies and utilize AWS managed services, you're basically asking if ELB, EC2 and ECS can scale. The most unstable component in all of this is probably Docker. In our experience running it in production at scale, it's faults have been pretty minor (things like push/pull being agonizingly slow) and failure modes get taken care of by ECS managing desired state well.

Hey! Creator of Empire here. The demo stack we provide isn't suitable for production use as you found out already. Empire can definitely be deployed in a production ready manor where you can have internal and external services, which is how we have it setup at Remind. The docs are a little lacking right now unfortunately, but we'll be improving this in the future.

We actually have a production ready example stack using our stack provisioning tool called stacker: https://github.com/remind101/stacker/pull/43.

Creator of Empire here. Empire and Convox are kinda like brothers and sisters, both really similar in implementation and philosophy. There's a couple of subtle differences, like how Empire expects that you've already built a Docker image somehow, whereas Convox can just take some code and do it for you (hosts it's own docker registry). Convox also supports streaming logs, which is pretty awesome (although, coming soon to Empire too :)). Convox is a little more reliant on AWS, whereas Empire was designed to be able to work with other backends like Kubernetes for example (not implemented yet). I would say, use which ever one best suites your needs, they're both great, and like David said, we talk a lot and share ideas back and forth.

Our approach was to re-use as much existing technology as possible, which is not the case for most others. That's the "complication" I'm referring to here. Empire grew from the need for a production grade platform that was going to be stable.

Empire doesn't actually lock you into ECS. The scheduling backend is pluggable and could support Kubernetes/Swarm in the future.

Thanks for the kind words. As a shameless plug, we are hiring: https://www.remind.com/careers :)

Thanks Blake! I think somebody mentioned that we were standing on the shoulders of giants. I think most of your contributions around this domain qualify for that :)

We have a setup that has been working out well for us:

1. We build docker images on every commit, in CI, and tag it with the git commit sha and branch (we don't actually use the branch tag anywhere, but we still tag it). This is essentially our "build" phase in the 12factor build/release/run. Every git commit has an associated docker image.

2. Our tooling for deploying is heavily based around the GitHub Deployments API. We have a project called Tugboat (https://github.com/remind101/tugboat) that receives deployment requests and fulfills them using the "/deploys" API of Empire. Tugboat simply deploys a docker image matching the GitHub repo, tagged with the git commit sha that is being requested for deployment (e.g. "remind101/acme-inc:<git sha>").

We originally started maintaining our own base images based on alpine, but it ended up not being worth the effort. Now, we just use the official base images for each language we use (Mostly Go, Ruby and Node here). We only run a single process inside each container. We treat our docker images much like portable Go binaries.

True, "self-managed" is probably more accurate.

Exactly! One of our goals was to also make the scheduling backend pluggable, so we're hoping that the community will implement a Kubernetes backend in the future. There's a lot of similar concepts between both, but we ultimately chose ECS for it's ease of operation and the integration with existing AWS services like ELB.

We actually have a relay (https://github.com/remind101/empire/tree/master/relay) service that can be run alongside Empire that acts as a proxy to interactive Docker sessions. It's a bit of an experiment right now and something we'd like to solve better in the future, but it allows you to run containers with `emp run <command> -a <app>`.

Dokku is definitely an awesome project (pretty much anything from Jeff Lindsay is pretty good)! The primary problem is that dokku is meant for just 1 service and we have quite a few.

We'd love to see one standard too. Personally, I think it's good to have a lot of competing solutions right now (ECS vs Kubernetes, Docker vs Rocket, etc) and we'll see things settle in the next couple of years as containerization becomes more common.