How we migrated onto K8s in less than 12 months

- it is not really any more lines - doesn’t break if dev upgrades to a different version of the resource (has happened before) - allows you to experiment with dev with other setups (eg additional ingresses, different paths etc) instead of changing a base config which will impact other envs

TLDR patch things that are more or less the same in each env; create complete resources for things that change more.

There is a bit of duplication but it is a lot more simple (see ‘simple made easy’ - rich hockey) than tracing through patches/templates.

In the software/tech industry it's common place to just accept that your app can't be down for any amount of time no matter what. No one checked to see how much more it would cost (engineering time & infra costs) to deploy the app so it would be HA, so no one checked to see if it would be worth it.

I blame this logic on the low interest rates for a decade. I could be wrong.

FAANG engineers made the same mistake, too, even though the analogy implies comparative competency or value.

If you have never dealt with, I have to run these 50 containers plus Nginx/CertBot while figuring out which node is best to run it, yea, I can see you not being thrilled about Kubernetes. For the rest of us though, Kubernetes helps out with that easily.

Because as annoying as getting the prometheus + loki + tempo + promtail stack going on k8s is —- I don’t really believe that writing it from scratch is easier.

K8s makes this really easy. Don't need to worry whether country X has a local Cloud data center of Vendor Y.

Plus it makes hiring so much easier as you only need to understand the abstraction layer.

We don't hire people for ARM64 or x86. We have abstraction layers. Multiple even.

We'd be fooling us not to use them.

First, when some team says "we want to use helm and etcd for some reason and we haven't been able to figure out how to get that working on our existing platform," start by asking them what their actual goal is. It is obscenely unlikely that helm (of all things) is a fundamental requirement to their work. Installing temporal, for example, doesn't require helm and is actually simple, if it turns out that temporal is the best workflow orchestrator for the job and that none of the probably 590 other options will do.

Second, once you have figured out what the actual goal is, and have a buffet of options available, price them out. Doing some napkin math on how many people were involved and how much work had to go into it, it looks to me that what you have spent to completely rearchitect your stack and operations and retrain everyone -- completely discounting opportunity cost -- is likely not to break even in even my most generous estimate of increased productivity for about five years. More likely, the increased cost of the platform switch, the lack of likely actual velocity accrual, and the opportunity cost make this a net-net bad move except for the resumes of all of those involved.

So am I reading this right that either downstream platform teams or devs wanted to leverage existing helm templates to provision infrastructure and being on ECS locked you out of those and the water eventually boiled over. If so that's a pretty strong statement about the platform effect of k8s.

I understand what you're saying, the thing that worries me though is that the input you get from other technical teams is very hard to verify. Do you intend to measure the development velocity of the teams before and after the platform change takes effect?

In my experience it is extremely hard to measure the real development velocity (in terms of value-add, not arbitrary story points) of a single team, not to mention a group of teams over time, not to mention as a result of a change.

This is not necessarily criticism of Figma, as much as it is criticism of the entire industry maybe.

Do you have an approach for measuring these things?

Easier said than done.

You can start by implementing this yourself and thinking how simple it is. But then you find that you also need to decide how to handle different environments, configuration and secret management, rollbacks, failover, load balancing, HA, scaling, and a million other details. And suddenly you find yourself maintaining a hodgepodge of bespoke infrastructure tooling instead of your core product.

K8s isn't for everyone. But it sure helps when someone else has thought about common infrastructure problems and solved them for you.

To me that seems like the simpler solution.

Unfortunately in my experience, this is true until it isn't. Once it isn't true, it can quickly become a painful blackbox debugging exercise. If your org is big enough to have dedicated AWS support then they can often get help from engineers, but if you aren't then life can get really complicated.

Still not a bad choice for most apps though, especially if it's just a run-of-the-mill HTTP-based app

Helm, for instance, is a great time saver for installing software. Often software will support nothing but helm. Ease of deployment is a good consideration. Their points on networking are absolutely spot on. The scaling considerations are spot on. Killing/isolating unhealthy containers is completely valid. I could go on a lot more, but I don't see a single point listed as invalid.

A common pattern you'll see though is skipping writing any sort of code and instead using a higher level dsl-ish configuration usually via yaml, using tools like Kyverno.

Maybe - people seem really gungho about serverless solutions here too

Personally I think k8s is where it's at now. The innovation and open source contributions are immense.

I'm glad we made the switch. I understand the frustrations of the past, but I think it was much harder to use 4+ years ago. Now, I don't see how anyone could mess it up so hard.

I’ve been struggling to square this sentiment as well. I spend all day in AWS and k8s and k8s is at least an order of magnitude simpler than AWS.

What are all the people who think operating k8s is too complicated operating on? Surely not AWS…

Not managing procurement of hardware, upgrades, etc, and a defined standard operating model with accessible documentation and the ability to hire people with experience, and have to hire less people as you are doing less is enough to build a viable and demonstrable business case.

Scale beyond a certain point is hard without support and delegated responsibility.

I wonder if as time goes on that skill to use hardware is dissappearing. New engineers don't learn it, and the ones that slowly forget. I'm not that sharp on anything I haven't done in years, even if it's in a related domain.

I can imagine the productivity of spinning up elastic cloud resources vs fixed data center resourcing being more important, especially considering how frequently a company like Figma ships new features.

Their ARR in 2022 was around $400M-450M. Say the infra budget at a typical 10% would be $50M. While it is a lot of money, it is not build your hardware money, also not all of it would be compute budget. They also would be spending on other SaaS apps like say Snowflake etc to special workloads like with GPUs, so not all workloads would be in-house ready. I would surprised if their commodity compute/k8s is more than half their overall budgets.

It is lot more likely to slow product growth to focus on this now, especially since they were/are still growing rapidly.

Larger SaaS companies than them in ARR still find using cloud exclusively is more productive/efficient.

They are almost certainly not paying sticker prices. Above a certain size, companies tend to have bespoke prices and SLAs that are negotiated in confidence.

Data centers are wildly expensive to operate if you want proper security, redundancy, reliability, recoverability, bandwidth, scale elasticity, etc.

And when I say security, I'm not just talking about software level security, but literal armed guards are needed at the scale of a company like Figma.

Bandwidth at that scale means literally negotiating to buy up enough direct fiber and verifying the routes that fiber takes between data centers.

At one of the companies I worked at, it was not uncommon to lose data center connectivity because a farmer's tractor cut a major fiber line we relied on.

Scalability might include tracking square footage available for new racks in physical buildings.

As long as your company is profitable, at anything but Facebook like scale, it may not be worth the trouble to try to run your own data center.

Even if the cloud doesn't save money, it saves mental energy and focus.

The _minute_ you start running containers in the cloud you need to think of "what happens if it goes down/how do I update it/how does it find the database", and you need an orchestrator of some sort, IMO. A managed service (I prefer ECS personally as it's just stupidly simple) is the way to go here.

For example, even servers (aka instances/vms/vps) with load balancers (aka fabric/mesh/istio/traefik/caddy/nginx/ha proxy/ATS/ALB/ELB/oh just shoot me) in front existed for apps that are LARGER than can fit on a single server (virtually the definition of horizontally scalable). These apps are typically monoliths or perhaps app tiers that have fallen out of style (like the traditional n-tier architecture of app server-cache-database, swap out whatever layers you like).

However, K8s is actually more about microservices. Each microservice can act like a tiny app on its own, but they are often inter-dependent and, especially at the beginning, it's often seen as not cost-effective to dedicate their own servers to them (along with the associate load balancing, redundant and cross-AZ, etc). And you might not even know what the scaling pain points for an app is, so this gives you a way to easily scale up without dedicating slightly expensive instances or support staff to running each cluster; your scale point is on the entire k8s cluster itself.

Even though that is ALL true, it's also true that k8s' sweet spot is actually pretty narrow, and many apps and teams probably won't benefit from it that much (or not at all and it actually ends up being a net negative, and that's not even talking about the much lower security isolation between containers compared to instances; yes, of course, k8s can schedule/orchestrate VMs as well, but no one really does that, unfortunately.)

But, it's always good resume fodder, and it's about the closest thing to a standard in the industry right now, since everyone has convinced themselves that the standard multi-AZ configuration of 2014 is just too expensive or complex to run compared to k8s, or something like that.

I had a different experience. Some years ago I wanted to set up a toy K8s cluster over an IPv6-only network. It was a total mess - documentation did not cover this case (at least I have not found it back then) and there was a lot of code to dig through to learn that it was not really supported back then as some code was hardcoded with AF_INET assumptions (I think it's all fixed nowadays). And maybe it's just me, but I really had much easier time navigating Linux kernel source than digging through K8s and CNI codebases.

This, together with a few very trivial crashes of "normal" non-toy clusters that I've seen (like two nodes suddenly failing to talk to each other, typically for simple textbook reasons like conntrack issues), resulted in an opinion "if something about this breaks, I have very limited ideas what to do, and it's a huge behemoth to learn". So I believe that simple things beat complex contraptions (assuming a simple system can do all you want it to do, of course!) in the long run because of the maintenance costs. Yeah, deploying K8s and running payloads is easy. Long-term maintenance - I'm not convinced that it can be easy, for a system of that scale.

I mean, I try to steer away from K8s until I find a use case for it, but I've heard that when K8s fails, a lot of people just tend to deploy a replacement and migrate all payloads there, because it's easier to do so than troubleshoot. (Could be just my bubble, of course.)

* Cert manager.

* External-dns.

* Monitoring stack (e.g. Grafana/Prometheus.)

* Overlay network.

* Integration with deployment tooling like ArgoCD or Spinnaker.

* Relatively easy to deploy anything that comes with a helm chart (your database or search engine or whatnot).

* Persistent volume/storage management.

* High availability.

It's also about using containers which mean there's a lot less to manage in hosts.

I'm a fan of k8s. There's a learning curve but there's a huge ecosystem and I also find the docs to be good.

But if you don't need any of it - don't use it! It is targeting a certain scale and beyond.

I use it (specifically, the canned k3s distro) for running a handful of single-instance things like for example plex on my utility server.

Containers are a very nice UX for isolating apps from the host system, and k8s is a very nice UX for running things made out of containers. Sure it's designed for complex distributed apps with lots of separate pieces, but it still handles the degenerate case (single instance of a single container) just fine.

I find k8s an extremely nice platform to deploy simple things in that don't need any of the advanced features. All you do is package your programs as containers and write a minimal manifest and there you go. You need to learn a few new things, but the things you do not have to worry about that is a really great return.

Nomad is a good contender in that space but I think HashiCorp is letting it slowly become EOL and there are bascially zero Nomad-As-A-Service providers.

(or, if you read it in a frustrated voice, The F**ing Article.)

It's only useful for the degenerate "run lots of instances of webapp servers running slow interpreted languages" use case.

Trying to do anything else in it is madness.

And for the "webapp servers" use case they could have built something a thousand times simpler and more robust. Serving templated html ain't rocket science. (At least compared to e.g. running an OLAP database cluster.)

With all that said, while I have no "hate" for the stack, I still have no plans to migrate our container infrastructure to it now or in the foreseeable future. I say that precisely because I've seen the source, not in spite of it. The net ROI on subsuming that level of complexity for most application ecosystems just doesn't strike me as obvious.

* Its secrets management was terrible, and for awhile it stored them in plaintext in etcd. * The learning curve was real and that's dangerous as there were no "best practice" guides or lessons learned. There are lots of horror stories of upgrades gone wrong, bugs, etc. Complexity leaves a greater chance of misconfiguration, which can cause security or stability problems. * It was often redundant. If you're in the cloud, you already had load balancers, service discovery, etc. * Upgrades were dangerous and painful in its early days. * It initially had glaring third party tooling integration issues, which made monitoring or package management harder (and led to third party apps like Helm, etc).

A lot of these have been rectified, but a lot of us have been burned by the promise of a tool that google said was used internally, which was a bit of a lie as kubernetes was a rewrite of Borg.

Kubernetes is powerful, but you can do powerful in simple(r) ways, too. If it was truly "the most amazing" it would have been designed to be simple by default with as much complexity needed as everybody's deployments. It wasn't.

I agree on how it's technically a waste of time to pursue fads, but it's also a huge PITA to have a platform that good engineers actively try to avoid, as their careers would stagnate (even as they themselves know that it's half a fad)

2. Fad-driven

I wonder why I don’t often see (1) critiqued on the basis of (2).

As someone using Kafka, I'd like to know what the (good) alternatives are if you have suggestions.

One of the most stable system archictectures I've built was on Kafka AND it was running with microservices managed by teams across multiple geographies and time zones. It was one of the most reliable systems in the bank. There are situations where it isn't appropriate, which can be said for most tech e.g( K8S vs ECS vs Nomad vs bare metal)

Every system has failure characteristics. Kafka's is defined as Consistent and Available and your system architecture needs to take that into consideration. Also the transactionality of tasks across multiple services and process boundaries is important.

Let's not pretend that kubernetes (or the tech of the day) is at fault while completely ignoring the complex architectural considerations that are being juggled

To be fair, small time units are difficult to internalize. Just look at what happens when someone finds out that it takes tens of nanoseconds to call a C function in Go (gc). They regularly conclude that it's completely unusable, and not just in a tight loop with an unfathomable number of calls, but even for a single call in their program that runs once per day. You can flat out tell another developer exactly how many microseconds the RPC is going to add and they still aren't apt to get it.

It is not rare to find developers who understand that RPC has a higher cost than a local function, though, and with enough understanding of that to know that there could be a problem if overused. Where they often fall down, however, is when the tools and frameworks try to hide the complexity by making RPC look like a local function. It then becomes easy to miss that there is additional overheard to consider. Make the complexity explicit and you won't find many developers oblivious to it.

Nah. Jira was horrible and slow long before the microservice trend.

So really eventually we'll all be optimizing around network interfaces as the bottleneck.

Is this generally correct? How well is this term defined?

If yes, then I'm not surprised this type of design has become a target for frustration. State is smeared across system, which implies a lot of messaging and arbitrary connections between services.

That type of design is useful if you are an application platform (or similar) where you have no say in what the individual entities are, and actually have no idea what they will be.

But if you have the birds-eye view and implement all of it, then why would you do that?

There are certainly ways to mitigate the SPOFiness of each of those cases, but that doesn’t make having them an antipattern.

This ties into a funny example: k8s manages my vm's via kubevirt, those then have a minimal k8s version installed that runs my jobs. The implementation simply mounts the extracted image to a virtual fs and executes it there, then deletes the file system.

But that's assuming you're running containerd or something similar. There are dozens of k8s implementations some as light as only providing you with manifests which then you have external schedulers which subscribe (called controllers) that execute on these manifests.

I agree that for a handful of pet-servers for a team with more existing Linux experience than k8s experience, this is a better starting point, because of the shorter learning curve. Just not kid ourselves that the end product has any less complexity, it's only a different skill set.