Hacker News Reader

Comment by xur17 | Hacker News Reader

We currently spin up dev clusters with a single node. $73/mo is going to basically double the cost of all of these..

This highlights a sorta-weird consequence of this pricing change: suddenly pricing incentivizes you to use namespacing instead of clusters for separating environments.

(As a security person: ugh.)

That’s interesting - I think you’re right. We might move our staging cluster into our main production deployment.

More likely though, AWS or OpenShift running on bare metal on a beefy ATX tower in the office. We want to have production and staging as close to each other as possible, so this is an additional reason and a p0 flag on reducing the dependency on Google-specific bits of Kubernetes as much as possible, hopefully also useful for our exit strategy as well.

Kubespray works well for me for setting up a bare bones kubernetes cluster for the lab.

I'll use helm to install metallb for the load balancer, which you can then tie into whatever egress controller you like to use.

For persistent storage a simple NFS server is the bees knees. Works very well and a NFS provisioned is a helm install. Very nice, especially, over 10GbE. Do NOT dismiss NFSv4. It's actually very nice for this sort of thing. I just use a small separate Linux box with software raid on it for that.

If you want to have the cluster self-host storage or need high availability then GlusterFS works great, but it's more overhead to manage.

Then you just use normal helm install routines to install and setup logging, dashboards, and all that.

Openshift is going to be a lot better for people who want to do multi-tenant stuff in a corporate enterprise environment. Like you have different teams of people, each with their own realm of responsibility. Openshift's UI and general approach is pretty good about allowing groups to self-manage without impacting one another. The additional security is a double edged. Fantastic if you need it, but annoying barrier to entry for users if you don't.

As far as AWS goes... EKS recently lowered their cost from 20 cents per hour to 10 cents. So costs for the cluster is on par with what Google is charging.

Azure doesn't charge for cluster management (yet), IIRC.

Have you used NFS for persistent storage in prod much? I know people do it, but numerous solutions architects have cautioned against it.

My experience with NFS over the years has taught me to avoid it. Yes, it mostly works. And then every once a while you have a client that either panics or hangs. Despite the versions of Linux, BSD, Solaris, Windows changing over the decades. The server end is usually a lot more stable. But that's of little to no comfort to know that yes, other clients are fine.

However, if you can tolerate client side failure then go for it.

(replying to freedomben): NFS has worked fairly well for persistent file storage that doesn't require high performance for reads/writes (e.g. good for media storage for a website with a CDN fronting a lot of traffic, good for some kinds of other data storage). It would be a terrible solution for things like database storage or other high-performance needs (clustering and separate PVs with high IOPS storage would be better here).

It's good to have multiple options if you want to host databases in the cluster.

For example you could use NFS for 90% of the storage needs for logging and sharing files between pods. Then use local storage, FCOE, or iSCSI-backed PVs for databases.

If you are doing bare hardware and your requirements for latency are not too stringent then not hosting databases in the cluster is also a good approach. Just used dedicated systems.

If you can get state out of the cluster then that makes things easier.

All of this depends on a huge number of other factors, of course.

> Have you used NFS for persistent storage in prod much?

I think NFS is heavily underrated. It's a good match for things like hosting VM images on a cluster and for Kubernetes.

In the past I really wanted to use things iSCSI for hosting VM images and such things, but I've found that NFS is actually a lot faster for a lot of things. There are complications to NFS, of course, but they haven't caused me problems.

I would be happy to use it in production, and have recommended it, but it's not unconditional. It depends on a number of different factors.

The only problem with NFS is how do you manage the actual NFS infrastructure? How much experience does your org have with NFS? Do you already have a existing file storage solution in production you can expand and use that with Kubernetes?

Like if your organization already has a lot of servers running ZFS, then that is a nice thing to leverage for NFS persistent storage. Since you already have expertise in-house it would be a mistake not to take advantage of it. I wouldn't recommend this approach for people not already doing it, though.

If you can afford some sort of enterprise-grade storage appliance that takes care of dedupe, checksums, failovers, and all that happy stuff, then that's great. Use that and it'll solve your problems. Especially if there is some sort of NFS provisoner that Kubernetes supports.

The only place were I would say it's a 'Hard No' is if you have some sort of high scalability requirements. Like if you wanted to start some web hosting company or needed to have hundreds of nodes in a cluster. In that case then distributed file systems is what you need... Self-hosted storage aka "Hyper Converged Infrastructure". The cost and overhead of managing these things is then relative small to the size of the cluster and what you are trying to do.

It's scary to me to have a cluster self-host storage because storage can use a huge amount of ram and cpu at the worst times. You can go from a happy low-resource cluster, then a node fails or other component takes a shit, and then while everything is recovering and checksum'ng (and lord knows what) the resource usage goes through the roof right during a critical time. The 'perfect storm' scenarios.

I use an SMB file share for my node pools - here's how to set up a non-managed cluster on Azure that does that: http://github.com/rcarmo/azure-k3s-cluster

What? Shouldn't you try to make the creation and deletion of your staging cluster cheap instead of moving it to somewhere else?

And if that is your central infrastructure, shouldn't it be worth the money?

I do get the issue with having cheap and beefy hardware somewhere else, i do that as well, but only for private. My hourly salary spending or wasting time on stuff like that costs the company more than just paying for an additional cluster with the same settings but perhaps with much less Nodes.

If more than one person is using it, the multiplication effects for suddenly unproductive people, is much higher. Also that decreases the per head cost.

You can dedicated nodes by namespace, at which point the isolation is pretty strong.

* Assuming you also configure strong RBAC, network isolation and don't let persistent volumes cross-talk

Nit: we don't recommend dedicated nodes for isolation. PTAL at https://www.youtube.com/watch?v=6rMGRvcjvKc And the guidance from GKE is at https://cloud.google.com/kubernetes-engine/docs/best-practic...

As also a security person (:wave:), you can use dedicated node pools and workload identity to isolate workloads in the same cluster.

Workload identity is a GCP-specific beta feature for mapping to GCP IAM, right?

yes

Yes and it's going GA.

https://cloud.google.com/blog/products/containers-kubernetes...

https://cloud.google.com/kubernetes-engine/docs/best-practic...

I suspect I'm in the minority on this, but I would love for k8s to have hierarchical namespaces. As much as they add complexity, there are a lot of cases where they're just reifying complexity that's already there, like when deployments are namespaced by environment (e.g. "dev-{service}", "prod-{service}", etc.) and so the hierarchy is already present but flattened into an inaccessible string representation. There are other solutions to this, but they all seem to extract their cost in terms of more manual fleet management.

Hey - I'm a member of the multitenancy working group (wg-multitenancy). We're working on a project called the Hierarchical Namespace Controller (aka HNC - read about it at http://bit.ly/38YYhE0). This tries to add some hierarchical behaviour to K8s without actually modifying k/k, which means we're still forced to have unique names for all namespaces in a cluster - e.g., you still need dev-service and prod-service. But it does add a consistent way to talk about hierarchy, some nice integrations and builtin behaviours.

Do you want to mention anything more about what you're hoping to get out of hierarchy? Is it just a management tool, is it for access control, metering/observability, etc...?

Thanks, A

Any reason why you put your link behind a URL shortener besides tracking number of clicks?

Since there are no character limits to worry about here unlike Twitter, better to put up the full URL so the community can decide for themselves if the domain linked to is worth clicking through or not.

Nope (other than that Google Docs URLs are looong), sorry.

Friendly docs: https://docs.google.com/document/d/1R4rwTweYBWYDTC9UC-qThaMk...

Code: https://github.com/kubernetes-sigs/multi-tenancy/tree/master...

Hey, thanks for asking! My interests in it are primarily for quota management -- in my experience, this is inevitably a hierarchical concern, in that you frequently run into the case of wanting to allot a certain cluster-wide quota to a large organizational unit, and similarly subdivide that quota between smaller organizational subunits. Being able to model that hierarchy with namespaces localizes changes more effectively: if you want to increase the larger unit's quota in a flat namespace world, for example, there's no way to talk about that unit's quota except as the sum of all of its constituent namespace quotas.

Thanks! We're not currently planning on implementing a hierarchical resource quota in HNC, but HNC is trying to define a definition of hierarchy that could certainly be used to create a HRQ. Give me a shout if you're interested in contributing.

Assuming you can do that, and your system is not using namespacing for its own purposes.

I know kubeflow can use namespaces for its own purposes, but otherwise I thought that was quite rare. Namespaces are intended to be used for exactly this usecase (isolating teams and/or workloads).

What kind of system have you seen where this isn't true?

I use namespaces to isolate names. So I can have a service called memcached in namespace A and also one in namespace B.

We've seen plenty of examples where people do this. Sometimes it's different teams (e.g. the ML team is namespaced away from the primary customer flow) and sometimes it's for different customers. Really, it sounds like we're in agreement about why this happens, the confusion is just whether or not that normally happens within a company in the course of doing business?

Gotcha. If you're looking for subnamespacing, HNC offers self-service subnamespace creation, have you looked into that? https://github.com/kubernetes-sigs/multi-tenancy/tree/master...

(My apologies if we've chatted about this before in another venue, I'm losing track of whom I've already talked to)

Yes, namespace alone isn't sufficient for isolation. Would you be able to look at our latest Multi-Tenancy best practices?

https://cloud.google.com/kubernetes-engine/docs/best-practic...

It's a living product which comes with Terraform modules. We introduced various features to enable doing Multi-Tenancy as well (and more on their way!)

I’m sorry if i am reading it wrong, but this guide to multi-tenancy seems to suggest not being multi-tennant and instead running a separate cluster per project. This seems more like scaling single-tenancy than multi-tenant (no bin packing oppty for instance.) or did i read it wrong?

Sorry if it was confusing. You need to read into more about how to set up in-cluster multi-tenancy.

We do recommend robust configurations for production setup (e.g. dev, staging and production) however you can certainly squash and skip it if not necessary.

Thanks for the feedback though. We'll consider adding such notes explicitly.

>You need to read into more about how to set up in-cluster multi-tenancy.

I am trying to do that. Where would you suggest? Throughout the comments on this post, when people suggest namespace, pod or node level separation you ask them to PTAL and read the link which suggests the single-tenant cluster-per-project approach (that is under the Multi-tenant cluster, confusingly.) The link you sent talks about cluster-per-project, which is not multi-tenancy as I understand it. Perhaps a different name would be less confusing (robust federated cluster administration?)

"This guide provides best practices to safely and efficiently set up multiple multi-tenant clusters for an enterprise organization."

This "multiple" multi-tenant clusters part isn't coming through. Please do jump into "Securing the cluster" section to cut corners and learn what to do in a single cluster. We're fixing the sections to avoid the confusions. Thanks for the feedback!

https://cloud.google.com/kubernetes-engine/docs/best-practic...

Or move to minikube and friends. If it's a dev environment you can usually get away with such things.

Kubernetes consumes a lot of CPU even when idle, due to the polling design, which makes Minikube is a really poor fit for developer machines. It's well known [1] to sit there eating 20-30% CPU, draining your battery and frustrating your life while doing absolutely nothing. This applies to all the Kubernetes distributions, including Kind and Docker Desktop. Not sure if the same applies to K3s, though.

[1] https://github.com/kubernetes/minikube/issues/3207, https://github.com/docker/for-mac/issues/3065, https://github.com/docker/for-mac/issues/3539, etc; there must be dozens and dozens of these

It does. My k3s setups (like https://github.com/rcarmo/azure-k3s-cluster) take up nearly 100% of the puny master node I allocate to them, and kill my Raspberry Pi SD cards as well.

Swarm, in comparison, is much friendlier (and you can use it for dev/test across multiple machines just fine)

Its still billed by the minute. If you run your dev clusters all the time 24x7 then they apparently are critical enough.

For a dev environment, why not host your own hardware? Especially if cost is a concern, it seems like a no brainer.

Generally curious, isn’t Docker Kubernetes an option?

It is - especially on OSX, it is very cpu and memory intensive thought.

Reply on news.ycombinator.com