How we reduced deployment times by 95%

Hey, great questions, some of the same questions we (the SRE/Platform Team at Plaid) had.

Plaid's rollback job still works the same way for the service using this new deployment, so thankfully, nothing new for engineers at Plaid to learn there.

We also have metrics in Prometheus to indicate which versions of code are running so we can easily verify what is deployed.

WRT the rate limit, we have a great relationship with AWS and Plaid pushes hard on limits which most often AWS is happy to increase for us, but this was a hard limit that could not be raised at the time, but I'm sure they are working on it.

Agreed.

I think Heroku did a number of things almost better in terms of a deployment / dev story? But getting past the pricing was super hard.

Compare to Deis, who built a Heroku analogue that you can run on your own compute resources. It was wildly popular for a little while, and then when it turned out there wasn't really a market for it, they simply had to leave it behind. (Today, that's known as Hephy Workflow, full disclosure I'm one of the maintainers of this FOSS project.)

Workflow and previous versions of Deis, as well as Dokku and friends also depend heavily on Open Source contributions of Heroku, so it's somewhat misguided to say that Heroku stays proprietary. They have been "leading the pack" when it comes to Open Source implementations of buildpacks, also Cloud Native form which you can find at buildpacks.io—and they set the original bar for a developer-friendly PaaS experience, which is still basically unrivaled even in the current era of Kubernetes product explosion.

We often ask about Workflow: "why isn't this particular technology several orders of magnitude more popular" and struggle to explain it, since it makes perfect sense to us and of course whenever people try it, they usually love it.

But Deis saw the writing on the wall, shifted gears to be some of the early Kubernetes experts when it started to become clear who was going to win; invented and open sourced "Helm" the K8s package manager utility, which is still very popular by comparison, one of the leading ways to package stuff for K8s. And then Microsoft scooped them up.

You could also read this as "don't run 4,000 of anything".

4,000 node processes? Wow. Just wow. Surely there is a better solution for this. Maybe this is one of those weird edge cases where you would craft a bespoke infrastructure tool, rather than trying to hack a bunch of off-the-shelf products.

In a way you're right, which is why we are exploring kubernetes. At the same time, ECS has worked really well for us as we've grown to this point. It's worth noting that we have some other motivating reasons to switch off ECS beyond deploy limitations, such as poor observability of ECS agent logs.

It's inevitable that early systems are going to bend and break at various points of the scaling process, and the interesting question is how to navigate that – fight fires for awhile or fix things immediately, hack together a stopgap solution or do the full fix, etc. We thought ours might be an interesting data point to share.

ECS can be good for really small things but, even when I worked in public radio, we discovered that ECS is pretty hot garbage without writing a bunch of stuff to deal with its quirks. Scaling worked kinda wonky, and ECS has its own jargon for terminology that already exists with Docker in general, making it unnecessarily confusing. Why they call a container a "task" when the term "container" already exists is beyond me. We ended up using Rancher to manage Kubernetes.

(Disclaimer: I work for AWS, but any opinions expressed here are solely my own and not necessarily the company's.)

I don't know about that. ECS works fine at large scale, but it's not going to replace 4000 tasks immediately. And doing so could potentially be a huge shock to your customers if you tried. (You have to take the time to gracefully drain existing connections, etc.) Nor did the author desire to implement a blue-green type deployment which would use AWS's elasticity to its fullest potential out of unspecified cost concerns.

It's not clear to me that Kubernetes would fare significantly better here. It's not without its own particular performance bottlenecks; and the issues about safely draining connections and blue/green deployments would remain the same there. The issues aren't really related to the particular deployment orchestrator, as much as the fact that 4000 containers is pretty large number by any measure for a single unit of deployment.

All a bank is, is a UX on top of low-level financial APIs like ACH. If the bank then exposed an API, you could just use it to build another bank (without all the effort the original bank went through) and so compete with them on lower margins (because you don't have nearly the capital costs to recover that they do.)

Basically, it's the same reason that phone companies would never have allowed MVNOs to exist without legal regulation forcing them. The MVNOs outcompete the infrastructure-building phone companies, because MVNOs don't have to build infrastructure!

Yeah, I mentioned this in a comment in a different thread. The duplicate containers in the task definition need to be marked as "essential" in CloudFormation to make sure our capacity doesn't degrade on container exits, and this means that one container exiting will also exit other containers in the same task. So we have a bleed-over effect where OOM in one request could cause N-1 other requests to fail.

The essential vs non-essential container designation is a little confusing. The standard use case for multi-container tasks seems to be that all containers are marked as essential, i.e. they essentially represent different services that are operating in concert on the same machine. This is definitely not the situation we're in, where each container is totally independent.

So it seems like we'd be a perfect use case for non-essential containers. However, (1) at least one container _must_ be marked as essential, and (2) non-essential containers which exit don't get restarted or replaced. This means we would still have a limited bleed-over effect (if the essential container exits, the other ones do too), and more importantly, we can't guarantee that our capacity will be robust to process exits.

Java had targeted hot reloads (going further than full reboot) for quite a while. See Jrebel for example.

Remove the instance from whatever routes requests to it, without killing it -- wait some predefined amount of time -- kill it.

I've built services that can gracefully shut down, and my typical design is as follows:

First, maintain an active request counter that you increment when you accept a request and decrement when the request has been fully delivered (including output buffer drained).

Second, implement a TERM signal handler. The handler shuts down the listening socket (so no new connections will be accepted), starts a countdown timer, and waits for the active-request counter to reach 0. When either the countdown timer expires or the active-request counter reaches 0, exit.

When we start processing a request, we take the JS Promise that represents the result of that operation and put it into an array in our internal "server state" object. When the server gets a shutdown request, it basically awaits Promise.all(this.inFlightRequests) before exiting. Depending on the LB/routing layer on top of your service, you might need to do this in a loop in case work comes in after you do your first await. In other languages you can join a thread pool, wait on a wait group, or use whatever other synchronization tools are provided.

Note that there's an ECS_CONTAINER_STOP_TIMEOUT parameter in ECS that sets a hard upper limit on how long containers have to exit before getting SIGKILLed, and it defaults to 30 seconds. If you want to allow requests to drain for longer than that, you'll need to update that parameter.

(For services with fast request processing time, the draining process is often a lot simpler. You can just route traffic away from the server, then wait a short period of time before telling the process to exit. It's not quite as precise, but works well for many use cases and requires no bookkeeping.)

While graceful shutdown is important, I think the higher priority should be ensuring that you can gracefully recover.

Because eventually, something is going to die while a request is in-flight. So your batch processing needs to be able to recover from the "somebody tripped on the power cord" scenario.

EKS has had networking layer issues in the past from personal experience and I've heard from good engineers that their CNI layer code (which is open source) is not very good. That would make me concerned on how well EKS (which is different from Kubernetes itself) can scale and what edge cases you're liable to hit.