Hacker News Reader

Comment by asdfasgasdgasdg | Hacker News Reader

I can understand the technical argument against the stateless containerized model. It's not neat. But what is the business argument? That's where this perspective falls down for me. If you're building a database that manages a disk directly, I can see the argument for graceful shutdown and handover. In that case you really need the program to be colocated with the data and if you're down then the data is unavailable.

I've built that once and it was hell to get right. But probably 90+% of programs I've ever worked on do not achieve any business advantage from graceful restarts (as opposed to drain and restart i.e. the stateless model).

View on HN

Kuberenetes is a leaky abstraction that by default allows the deployment of sloppy unreliable apps if you don't know what you're doing. If something is broken and it just restarts automatically and no one is paying attention, it will do that in perpetuity. Ideally, you would see those issues and iterate to make your app more stable, but it makes it easy to just forget about it. The additional layer of complexity is also harder to understand easily for most devs that haven't spent the time to learn kubernetes and even with devs that do spend the time it can still make understanding how everything works hard. Letting tech debt like unstable apps grow into mud balls can reduce a team's velocity in the long term and in some cases the short term as well.

Kubernetes is great for easily sharing and executing recipes, but it can make it tough for people that want to become chefs that are able to use reason and intuition to build and understand complex systems.

On the flip side, why bother spending time optimizing a process if nobody notices when it fails? Aren't there more important problems to be solving in at scenario?

Maybe people do notice, but it’s not easy to report back? Or they notice the system runs slowly? Or they just accept the fact that all systems (extrapolating from several banking, public service, and publishing sites) are crap, and need to live with it?

Tech debt filled mud balls where feature after feature is clumped on top of one another without worrying what came before it and what will come after it are not great to maintain. Sure if you don't plan on maintaining or building on the component and you don't care if it restarts, then let it restart forever. But if it's a component that you plan on adding features to for a long period of time and stability issues are ignored, those types of issues can compound over time and make it more difficult to reason about and maintain.

I agree, things that change frequently need more attention to make sure you're not spending more time than you need to make changes.

Acknowledging that I'm changing the goalposts a bit, I would consider a lack of understanding about the lifecycle of an application that causes development delays an abstract form of "noticing when it fails"

When you have two systems that need to talk to each other, and one system expects the other system to be reliable, it’s difficult to completely toss out all graceful shutdown and handover. Usually this logic is handled at the service discovery or load balancing layer with the deployment bits, so people don’t notice.

Kubernetes does a really good job at hiding this.

When it hides this, it also hides a couple other things with software — like failing containers, or apps that require churn or fail. I mean this is literally the reason why things like monit were built in the older days — things broke if you didn’t restart them when they broke.

In kubernetes, it also is easy to accidentally to change the system and make that behavior change (at a global level), that results in stuff like your app not being restarted as often, which results in elevated latency, and subsequent failures.

No system is 100% reliable, though. Some systems are close enough that we grit our teeth and pretend they are and wave away the inevitable outages that have nothing to do with our app and everything to do with an unreliable dependency.

In a way, you're better off with a 99% reliable dependency than 99.9999%. If you expect it to be down some of the time, you'll plan for it in your architecture and (in theory) gracefully handle the failure whether it's due to an insufficiency warmed container, a network issue, a bad deploy, or whatever else comes along.

I'd argue the biggest reason for a graceful restart would be reliability and latency.

But if you have the app behind a load balancer that can discover new tasks, rolling restarts for updates do not negatively affect those parameters.

This works pretty well for request oriented protocols with short response times.

It's not great for protocols with significant handshaking and potential for long connections. Things like chat services, shoutcast streaming, calling, imap, really anything realtime or notification driven would do better without forcing clients to reconnect. Especially since many will reconnect multiple times, as they may reconnect to servers which are running old code while the drain and restarts happen. If you need near instant cutover (which is pronably rare), in a drain and restart scenario, you basically need double the capacity, where hotload doesn't need anything extra.

So you pay a public cloud provider for 2x capacity for a couple of minutes while you do the rollout, and in exchange you get a much simpler design because the individual servers are stateless. I think that's better for the business.

Some services may require gigabytes of state to be downloaded to work with acceptable latency on local decisions, and that state is replicated in a way that is constantly updated. These could include ML models, large routing tables, or anything of the kind. Connections could be expected to be active for many minutes at a time (not everything is a web server serving short HTTP requests that are easy to resume), and so on.

Changing the instance means having to re-transfer all of the data and re-establish all of that state, on all of the nodes. You could easily see draining of connections take 15-20 minutes, and booting back and scaling up to be taking 15-20 minutes as well, if you can do it for _all_ the instances at once (which may not be a guarantee, and you could need to stagger things to be more cost-effective).

You start with each deploy taking easily over an hour. If you deploy 2-3 times a day and that your peak times line up with these, you can more than double your operating cost just to deploy, and that can take more than 4 figures to count.

Some of the systems we maintained (not those we necessarily live deployed to, but still required rolling restarts) required over 5,000 instances and could not just be doubled in size without running into limits in specific regions for given instance types.

If a blue/green deploy takes a couple minutes, you're probably not having a workload where this is worth thinking about that much.

This sounds like an interesting usecase.

Did you look at any of the node-based options like checkpointing the state to a file on the node, and loading that into your newly started pods? Or using read-many persistent volumes? (Not sure if you needed to write to the state file from every process too?)

(This doesn’t help with connections of course, that’s a bit more thorny.)

> checkpointing the state to a file on the node, and loading that into your newly started pods

For some types of my nodes, the majority of the state was tcp connections and associated processes. I don't think there's a generally available system that is capable of transferring tcp connection state (although, I'd love to build one! if you've got a need, funding, and a flexible time table), which would be a prerequisite to moving the process state. All of those connections need to be ended, and clients reconnect to another server (where they might need to do it again, if they don't get lucky and get a new server to begin with).

The other nodes with more traditional state had up to half a terrabyte of state in memory, and potentially more on disk, and a good deal of writes. That's seven minutes to transfer state on 10G ethernet, assuming you can use the whole bandwidth and sending and receiving that data is faster than the network.

Although, in my experience, we didn't tend to explicitly replicate disk based storage for new nodes, all of our disk based data was transient, so replacing nodes meant writing to new nodes, read from new and old, and retiring the old nodes when their data had all been fetched and deleted, or the data retention cap was missed.

I/O Volume meant networked filesystems would be a big stretch. You could probably do something with dual-ported SAS drives, and redundant pairs of machines on the same rack, but then that pair with both go down when that rack has an unforseen problem, plus good luck getting dual-ported SAS drives hooked up properly when you're in someone else's bare metal managed hosting.

(Yeah OK, maybe we had big performance requirements, but hotloading works just as well for stuff that fits on a single redundant pair, or even a single server in a pinch)

Interesting case study, thanks for sharing!

if a request takes 100 ms, and a portion of them are murdered halfway through then either clients retry for reliability in exchange for extra latency, or reliability takes a hit.

The best policy is for a deployment to lower the health check on the service, wait a few minutes per box, then bounce the process.

This is assuming the service is 100% stateless.

> But what is the business argument?

Complexity costs money.

Not sure I follow. The solutions proposed in the article - designing your software so it can be live-reloaded, having people who understand the system deeply etc. - all seem more complex and more expensive than just stuffing everything in Kubernetes and letting pods restart.

1. You always need people that understand the system. Not everyone and not all of it, but you want that overall, people are able to understand and figure out what is going on. You similarly hope to always have people who understand the customers, the environment you operate in, and so on. If you can afford to do your thing without having people understanding how it works, you probably can afford to do your thing without a an actually good solution (just scrape by, it's alright) and can go on with whatever.

I'd advance the theory that picking an off-the-shelf popular solution is going to be beneficial in that case because you externalize the costs of maintaining your expertise and knowledge to the rest of the ecosystem or industry, to other companies, and just never develop that fiber within your organization. It is, however, worthwhile to develop it for more things than just your tech stack. Everything having to do with on-boarding and dealing with legacy code is improved, along with broader dynamics if you try to be a learning organization that develops expertise in its people.

2. You always design your system in accordance to its deploy system, regardless of whether you realize it or not. If you ship binary artifacts and signed packages to customers who install them on their own devices, you will have a different development practice than if you do CI/CD with a single pipeline that always goes forwards. This will also be somewhat different if you work with open-source components that require paying attention to version schemes rather than just pushing a hash in a container image. If you use feature flags to help merging but also to control deployment, adopt A/B testing, and all these practices, you're intimately adapting your development approaches to deployment mechanisms that are available to you. It's not an extra cost, it's a cost you already pay today.

Reply on news.ycombinator.com