Hacker News Reader

> that new team should replace another team that's no longer needed, or more likely, involve a combination of hiring some experts and retraining your existing engineers.

The gotcha there is that it rarely goes that way unless you have a very clear direction from senior management, at least at big corps. In most cases, it's just another thing that gets added to the pile, and it's incredibly difficult to migrate entirely out of whatever the old solution was, so now you end up supporting both.

You should always have a next gen and a production version of your infrastructure as a code.

Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups.

It’s great if their own solution works well for them at less cost, but that system didn’t built itself and has non-zero maintenance costs.

> Note that they already have a team who has built their current compute platform, who built the pipeline to run containers/processes on VMs with auto scaling groups.

In that case I would expect Coinbase to write blog posts on how their setup is the absolute best solution to their problem, and not how they refrain from adopting the best solution to their problem because they claim they don't have anyone on the team that is able to pull that off.

> Presumably in a company like Coinbase there is already an infrastructure team that runs the AWS instances, helps build the AMIs, etc. This team could re-tool and hire some k8s experts to help them make the shift.

The key is that there is a lot of additional services and interface points to handle. As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding. Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.

: The control points that k8s exposes are not simple "drop in your provider here" bits of integration. You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them, and more access control to implement and verify.

My personal experience (4 years on GKE in production) has been the opposite; running on k8s has abstracted away a number of things that I’d otherwise have to engineer.

Volumes just get attached (using PersistentVolumeClaims), and automatically migrate to a new node of the original pod dies. Vs. having to do some sort of rsync between nodes to keep disks in sync.

Secrets get encrypted by k8s and mounted where needed. I would agree that RBAC is a bit tricky but I don’t think it’s harder than IAM provisioned with Terraform.

If you are not using a service mesh for your VMs then you don’t need one in k8s. (I don’t use one, and rolled TLS to the pod in less effort than it would take to maintain TLS to the VM). The reason you want a service mesh is to abstract TLS and retry mechanics from the application layer - i.e. make your service authors more productive. If you don’t use a service mesh then you are back to managing TLS per-service, which is where you are with VMs already.

There are definitely more services you _could_ run, but in my experience these are additive, I.e. they are extra work, but give you a productivity boost.

Anyway, YMMV and I haven’t operated a system as large as Coinbase, so I could be missing something. Interested in hearing others’ experiences though.

> As the Coinbase article noted, you need extra pieces on top of k8s (storage, service mesh, config/secrets, etc) that need care and feeding.

The problem with that assertion is that it does not make any sense at all. For instance, storage and config/secrets is already supported out-of-the-box with Kubernetes. Even so, complaining about storage with Kubernetes is like complaining about EBS or EFs or arguably S3 in AWS. And if you feel strongly about service meshes then you really aren't forced to use them.

> Even if the company moved 100% of their services into k8s there's now more work to be done for the same level of service.

There really isn't. For example, if they go with managed Kubernetes solutions then the only thing they need to worry about is to actually design their deployments, which would be very strange if they couldn't pull off. That's a ramp-up project for an intern if the solution architecture is already in place.

> You would likely still have the same core providers (ex: EBS for storage) but there is now more code running to orchestrate them

There really aren't. Kubernetes' equivalent to EBS is either setting up a volume or a persistent volume claim on a persistent volume. Just state how much memory you want and you're set.