Hacker News Reader

Comment by throwaway012282 | Hacker News Reader

Sorry for the throwaway.

> I’ve been doing this “reliability” stuff for a little while now (~5 years), at companies ranging from about 20 developers to over 2,000

I've been doing it for 20+ years including running critical services in FAANGs.

> Use Docker > Use Kubernetes.

Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.

> Deploy everything all the time

On your local testbed, maybe, unless you already know a commit is broken. On production, absolutely not.

View on HN

> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.

And for every other company that isn't one of those 5, rolling your own deployment system won't be simpler, leaner or easier to debug. Nor will it, in all likelihood, be as stable, feature rich, observable, etc.

> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.

Most FAANGs developed their computing infrastructure before Kubernetes gained popularity. After all the investment into building it and fine tuning for their software services (like ads), they probably have a lot more reasons to not migrate to something different. Not just because their systems are "simpler, less bloated, easier to debug".

Interestingly, it's G in FAANG who came up with K8s in the first place.

> There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.

FAANG-appropriate solutions aren’t always appropriate for the rest of the world.

> Hard pass. There's a reason why most FAANGs developed their own packages management and deployment system and keep using them. They are simpler, less bloated, easier to debug.

That seems like an odd thing to say when Kubernetes came out of Google and they based it on Borg, their internal orchestrator.

FAANG create many things themselves as they're trend setters. They have the size to create them. Having worked on in-house solutions, it is incredibly difficult to maintain adequate documentation, training, and resources to keep it going.

Why, deploying everything all the time is fine, it's what FB, for instance, does. If you don't, there is a hazard of accumulating too much change per deployment, which becomes more dangerous to deploy, and more problematic to roll back.

But it does not mean "deploy every change to 100% of the fleet instantly". Use canaries, use slow, monitored rollout. With smaller changes, it's relatively painless and routine.

> Deploy everything all the time

More often than not, this ends up being a business decision about balancing risk with potential loss of revenue (you don't want to take your stuff down, but you also don't want to delay new features that could bring more business)

Fast growing companies trying to enter the market tend to favor "deploy more often, break more often" while companies with large, established bases tend to favor "more testing, more deliberate changes"

I would say the inverse is a better rule "you should be /able/ to deploy any time"

I'm nowhere near faangs but I agree with your view, coming from about 20+ years of development.

Reply on news.ycombinator.com