Hacker News Reader

This makes too much sense to me to think people use Kubernetes for anything else. Why must one use Kubernetes for EVERYTHING unless there is some higher-order reason?

I use Kubernetes to host something like 6 of our own services, and it excels at that and is fairly simple. Databases and other things use different services.

We did the reverse of go. We started with purely stateless k8s and got really comfortable with the platform before moving to support stateful loads. For databases, many vendors have dedicated operators which introduces best practice deployments with less operational fuss provisioning the instances. Tooling like Strimzi for Kafka as an example here.

This may be the issue with stateful operators:

Do you really think they have ALL the operations coded properly for all operational conditions?

Stateless is so much easier for operations. It either is running, or not, A/B upgrades, yada yada.

Stateful has backups, restores, outages, patching, migrations, corruptions, fixes. For distributed systems it gets even hairier. Do they support your distribution? What if you have a multicloud or blended internal/cloud? Does the operator provide a turnkey restore from backup, and are you testing it?

Operators shouldn't be viewed as replacements for stateful operations knowledge, but it's probably what they'll be used for.

If you're using a large scale distributed stateful/database system, you need an operations team to support it, or pay for that.

Stateful has all of those problems anytime, anywhere. Even if you run a single stateful app in a single VM, you will still have all those issues.

You have a lot less problems in a single VM: There‘s (hopefully) a clear path on how to install software etc. and it is easy to understand. If your distributed PostgreSQL operator written by someone fails you‘re out of luck and it will be really really hard restoring your system. With non-kubernetes non-automated solutions not written by someone you may have some docs saying „Update the replication address and restart“.

The way to restore to a running system is much easier in old traditional vm approaches.

In K8 paths are clearer: Every bit of component that goes into a deployment is versioned. They deploy the same in every build. Its 100% certain.

The problem with stateful app scaling and HA comes from two things: the need to make files and database multi read-write and high available. Other than that, app scaling is pretty easy in K8 even for stateful apps.

I am wondering, did you ever look at tuning MALLOC_ARENA_MAX? This sort of constant consumption of memory is fairly well aligned to the default tuning of MALLOC_ARENA_MAX, which is 8 * nproc.

We've just tuned it for a java-based app which was also stuck in OOMkill hell, and this has completely resolved the situation (MALLOC_ARENA_MAX=2).

At Coherence (withcoherence.com - I'm a cofounder) we generally agree that GKE and other managed k8s are best used for stateless workloads. Rather than move stateful workloads back, leveraging managed services will yield the best results in the long term. In the case of Redis on GCP, something like Memorystore is going to be a better fit than managing a nest of node pools over time (think about version upgrades, resources differences across environments, etc...) However, the complexity of managing the different kinds of configuration across GKE and managed services can be a nightmare.

That's a problem we're hoping to help solve, where you define your application and it's dependencies, and we help run it in the right way to leverage managed cloud services across environments without passing that headache on!

Crossplane [1] is great way to create and manage resources across cloud providers, MSPs via kubernetes objects.

[1] https://crossplane.io

Config Connector [1] is also an option in this space for GCP, it supports many GCP resources and thus far our experience with it has been largely positive.

[1] https://cloud.google.com/config-connector/docs/overview

Have you checked out Managed Instance Groups? Used them a wile back, and worked as advertised :)

https://cloud.google.com/compute/docs/instance-groups#manage...

We do something similar with ElasticSearch. We use EKS (a k8s operator) but give each ES node a full k8s node using pod anti-affinities and taints. That way we can just select a sensible disk and instance size on our node pool and not worry about resource request/limit. It's been working very well so far.

ES handles node restarts or upgrades pretty gracefully though. I'd imagine for databases or "non-clustered" things you'd have to consider GKE's aggresive upgrade schedule. We use CloudSQL for some databases but our larger ones are still on GCE because we get more control of replication, CDC, and can use tools like proxysql to reduce downtime.

> Much lower total maintenance burden.

Devil's advocate but isn't having to maintain VMs (and then software deployed to those VMs) and k8s YAML/charts/whatever more "maintenance burden" than just one or the other?

I guess it would depend on how you managed your deployments/infrastructure but in general I would say no. In my experience stateless services are all managed differently because they typically need fewer resources and can be scaled more easily. Services that require state tend to have a more hands-on approach since they are usually in the critical path for many other services. Deploying a cookie-cutter service is something where k8s accels so it makes sense to use it for those types of workloads.

What do you define a server as stateless? As... not needing to talk to a filesystem or a cache server or a database?

Just something that takes in API requests (or a cron-like scheduled job) and makes other API calls/does plumbing?

At least in my experience, no.

We used managed services for our stateful stuff which significantly eased the operational burden there. Might be a different story if we looked at doing the absolute minimum cost optimization. However, at least for us, the extra cost of managed services is worth the price.

The yaml tends to be a "one and done" sort of thing. We touch it MAYBE once every 2 months if that.

Why did you run Redis on K8 in the first place? (One of the reasons we did not move to K8 was the default recommendation to not run Redis, SQL etc on the clusters.)

Our Redis use-case started off as ephemeral per-deployment storage for things like rate-limit counters; before evolving into durable per-deployment storage for things like service-metadata discovery.

Ephemeral Redis is well-suited to k8s — you can treat it as just a sidecar to your app layer deployment. Durable Redis is not. But sometimes the transition can sneak up on you.

> The point would be to get away from the need to manage the VMs ourselves, giving them over to GKE, while still retaining the assumptions of VM isolation (e.g. not having/needing memory limits, because the single pod is the only tenant of the VM anyway.)

Isn't this just moving the problem from per-pod resource constraints to per-VM resource constraints?

Yes. They could have just set the memory limit high enough to handle that workload. There's really no different.

With the pod, you still have the per-VM/node resource constraints, so it's an additional layer.

you can set hostAffinities on PersistentVolumes.

  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - <hostname>

this ensures that your workload will be rescheduled on the matching node

Thanks for your insights! One question regarding the approach to hosting stateful components you describe last:

what’s the difference between doing this as single node pools vs pod constraints like anti affinity?

It would make node pool operations like version upgrades more predictable since you'd know for sure which apps are running a given node pool

It can also make monitoring resource usage a little easier since you can just monitor at the node level

While I like Kubernetess generally, I agree that the OOM handling is not ideal, feels quite a bit like "exercise left to the reader".

For stateful stuff, give a try to Google's File Store. It has pretty good performance.

As for K8 managed VMs - thats a good idea. Resource management wise, security wise etc.