Cuber: Deploy your apps on Kubernetes easily
github.com
github.com
K0s is a new project I like that is similar to k3s in spirit. Microk8s is a snap version from canonical with some neat out of the box stuff (like the simplest GPU support); but again, only available as a snap.
Honestly though just use kubespray for production, it's fine
Imagine a 3 hour Ansible-playbook execution, which randomly failed on different tasks, most of which got fixed by hitting <up><enter> until it applied...
We switched to RKE, which runs even on our bigger clusters (nothing mad, maybe 40 nodes) in a few minutes and works perfectly..
We had much better luck just scripting kubeadm over ssh orchestration tooling.
Sure it helps with deploying a serviceable production cluster, but when you start throwing at it things like disaster recovery, the automation scripts can grow quite large and complex, even leveraging kubespray.
If you can, use a managed service like gks.
k0s is better but also has a lot of bugs. it's the closest to vanilla kubernetes among all the distributions.
> like the simplest GPU support
linux users should be ready to install the nvidia device plugin. if they can't do that, they're never going to succeed in running a gpu accelerated application on their cluster anyway.
> like bootstrapping
in my experience, writing all the bootstrap scripts is painful. but now that there's chatgpt, so much of the drudgery as gone away.
My first impression is that it looks quite awesome! If this is able to remove a big build process of containers then that would be awesome. I don't want to build my docker containers locally, I don't want to 'manage' a docker registry, I don't want to build my containers in my cluster (e.g. Kaniko). However, everything says it will push to the Docker registry and I haven't been able to find any private registry or non-registry option. That would be quite a dealbreaker because I don't want to 'store' my images long term, I will only use them for the few moments where the pods are deployed with a new image and the few days/weeks where they might restart. But retention is not a huge requirement.
Just my 2c.
Plus has clean github integration for incremental deploys, etc
All that being said, Cuber definitely feels like a different sort of market fit.
but it could have been built on top of Argo right ?
I totally get an opinionated out-of-the-box workflow and manifest file. But i would rather see that on top of the Argo infra (with its millions of person hours of kuberentes fighting).
I could see this project being picked up by some dev who just wants it on the cluster then an SRE ports it to something like Argo. Ideally the dev should learn Argo in the first place but a world where every dev understands what Argo is doing feels pretty unrealistic.
Side note (kind of off topic but I’ve been thinking about Argo a lot lately) I’ve been a bit more bearish on Argo for non production environments because it encourages pets and not cattle. I think nowadays the correct way to do environments is a gitops production cluster and dynamic pushed based ephemeral environments that are spawned from that production configuration and templates accordingly. Actually maybe it isn’t off topic: that’s what I’d like to see to offer to developers: some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
How so? From the rest of your comment it sounds like you may be saying that target environments are the pets, but ArgoCD makes no assumption about that. It will deploy to any cluster you configure it to deploy to, and spinning up a cluster is a logically (very!) separate operation from deploying an app. Argo only deals with the latter.
If you want something to actually provision an ephemeral environment for you, ArgoCD is not the tool. It's more focused than that, and that's its strength. All it does is deploy k8s manifests out of git to whatever cluster you tell it to deploy them to. That cluster could be ephemeral or not, it doesn't matter to Argo.
So by virtue of the way argo works it encourages pets, because its setup is not very dynamic. Like you said, in environments when you want to be able to roll back, that is its strength. But my point I was making above is that really you want that for only one environment: prod when it all boils down to it. Every other environment not being a dynamic branch off of prod just creates maintenance and state issues down the line
A "pet" is a machine or environment that you can't easily delete and replace. Nothing we're discussing can't easily be deleted and replaced. What you're describing relates to some other property.
As for "dynamic", with Argo you can just edit the namespaces of Applications in a branch and deploy a new environment.
> it would require constantly committing to some repo
If you have an automated CI/CD pipeline, you already do this anyway.
It's not clear to me what kind of operation you're thinking of, but in the same sort of way that k8s is overkill for many scenarios, so is ArgoCD. But the issue is not that it encourages pets or can't easily handle dynamic environments.
>some tool that is easily configurable like this that spawns ephemeral environments and deployments based off a checked in production grade configuration template like you would get with Argo
Totally with you there, but you can build this on top of Argo by having your opinionated manifest language (like Packer or Cuberfile or something else). But there is no joy to maintaining the Kubernetes api interfaces.
There’s probably 10 tools I could customize to do this, packer being one of them, but no real patterns I’m aware of where people are actually doing it with argocd or fluxcd config as a base. Usually you’re either all in or all out on gitops, so I would be interested in seeing blog posts or libraries of people who have done hybrid approaches (push and pull deployments) and the trade offs they had to make. There is a decent amount of decisions that have to be made to build an abstraction like that, so anyone who has done it before I’m happy to stand on the shoulders of. It’s been on my list of unsolved devops/infra problems to unwrap for a couple of years now
In the end there are really three steps that need to happen for true “copy prod” ephemeral environments. You need to provision some stack of infra and then deploy some stuff on top of it and then pull some arbitrary, extremely customizable state into that deployment. I’ve been quite interested in Layerform recently for the first part of that, I think they are getting close to the right abstraction on the infra side. But then you still have to deploy (Argo, helm, flux, something) into that infra stack. Then you have to get the state into that deployment somehow. there is no answer for the latter two from Layerform (terraform is not the right tool for deploys or state backups once you do anything beyond a hello world app). Getting the state in has its own whole Pandora’s box (can I copy customer data from prod into another environment? How do I do so on a way that doesn’t violate SOC2/hipaa/whatever? How will my system’s data and configurations affect the third party systems it connects to?) that I think is simply just not generalizable. But the first two steps should be able to be generalized, and the last step is something that tools can be made available to make it easier with the knowledge that it’s basically business logic at that point
It would still take equal amount of effort probably, but you get the benefit of a larger ecosystem and more work on the underlying infra, while you can focus entirely on the abstractions.
Not really. It's a deployment tool, the UI is there to help see what's happening with deployments. It's not intended to be a primary interface to cluster management the way k9s or Lens etc. is.
> They just merge, and code gets deployed.
How does "code get deployed" though? There has to be something that's actually (1) monitoring branches and (2) running the equivalent of `kubectl apply` to deploy manifests. That's what ArgoCD does.
If you're saying devs are using k9s or kubectl to manually trigger the apply from a branch, that has all sorts of drawbacks, which is exactly what Argo is designed to address.
What do the authors ever mean? How so? The smallest k8s cluster that makes sense has 3 nodes. This means you need to run at least 3 VMs and pay for everything else you need (storage, secrets, egress). And on some cloud providers like AWS, you pay an extra hourly fee just for the privilege of using k8s (that amounts to around $70 last time I checked). So the above statement makes no any sense.
The key thing is that a container needs about 10-20% of the memory of a virtual machine. Generally, a large cluster allows for very efficient "bin packing" so that workloads with different requirements all get distributed so that every VM is loaded to about 80-90%, whereas you'd be lucky to have 10% across your fleet of VMs. (Don't forget to include all the non-prod instances!)
At the tiniest scales, Kubernetes stops making sense because it's too complex to manage.
However, it does scale down to a single node. E.g.: Azure Kubernetes Service (AKS) will cheerfully run with just one Linux box and has no extra hidden costs.
Kubernetes is a surprisingly reasonable option for an organization with 5-10 developers that has outgrown Heroku-like services.
I don't think I've ever budgeted more than $150/month for a managed Kubernetes controller that Just Works, and that was a relatively expensive one. Everything was preconfigured and we could get started in 20 minutes. The actual servers in the cluster were just ordinary cloud VMs with locked-down images, and they were managed automatically.
Actually running an app on Kubernetes requires two things: (1) a Dockerfile, and (2) a YAML file listing the Kubernetes resources for the app. The YAML file might be anywhere from half a page to 2 pages for a simple one. There's an O'Reilly book that explains what to put in them.
I've also deployed large applications using Chef and Terraform. These often require far more configuration than slapping something on Kubernetes, and Chef offers many more ways for things to go wrong.
However, I do have several rules for Kubernetes happiness:
1. If your app is simple enough to run on a Heroku-like service, do that instead.
2. If you do choose Kubernetes, buy and skim the O'Reilly book, or a similar book.
3. Pay a cloud vendor for managed Kubernetes.
4. Do not mess with the networking config. Almost every Kubernetes horror story I've heard involves network overlays.
I'm not saying anyone ought to use Kubernetes, but it's surprisingly reasonable and pleasant for smallish full-time teams if you follow the rules above.
Managing k8s at the smallest scales is actually easier IMO. If you're using a managed k8s (you should be) then there's not much to actually manage. I've been using single-node clusters to great benefit for my business for years. Let's me focus all my time on billable hours and product updates. Huge force multiplier for me as a solopreneur.
For small scale hosting who do you recommend? I have a few projects I want to get going and thinking it might help to be able to leverage my existing kubernetes knowledge. I always figured it would be too expensive to go with kubernetes for a small project.
Is there a source for this statement?
Its origins started in Google, which has thousands of developers, and it is roughly speaking a clone of their Borg system, which, again, was designed for that scale.
In my recent experience a ton of tiny startups which should be using Heroku etc are running Kubernetes for no other reason than engineer ego and "that's what the big boys are doing".
And you say Heroku, "etc", but the thing is that there is very little heroku-like alternative. What "etc" is there really?
The real alternative is a Linux VM with docker-compose.
Once you outgrow that, managed kubernetes is the logical next step, even for a small team.
The utilization numbers only show that a particular administrator doesn't know how to use technology X, but knows better how to use technology Y or that the defaults in technology Y happen to work better for a particular benchmark (and the administrator doesn't know how to use either X or Y).
I believe that the later is usually the case. Compounded by laziness and lack of real financial incentive to optimize every last bit of the existing infrastructure.
The situation will, however, flip on its head if you want to build a reliable high-performance system. In this case Kubernetes will be an impendiment. Nothing it does it does well or can be optimized / configured to do well. When push comes to shove, you'll have to either get rid of Kubernetes, or lose the race.
K8s basically gets cheaper the more you run (as cloud costs, setup costs, engineering, bin-packing, volume discounts stack up), whereas Heroku basically scales linearly.
Something like k3s makes a lot of sense for development environments though. It is much easier to create a production like environment with something like k3s than it ever was using VMs.
I do feel like there's room in the middle for something more flexible than Ansible and Puppet, but less complex than Kubernetes. If you're just running a few VMs, "upgrading" to Kubernetes doesn't seem like it's worth the hassle. On the other hand, deploying the same application, or applications to say 4 or 8 VMs, that starting to get i little annoying. I'd argue for a tool that augments Docker-Compose, one that understands distribution of applications, but doesn't deal with networking (the network team can handle that). I suppose a little like Nomad?
As a solopreneur, I've saved so much time and money letting k8s run my workloads while I rack up billable hours and build products.
Frankly it is not painless at all. Ingress is complicated and there is a ton of way to configure it, with all the forced tie in with the cloud provider's offering, be it l3 or l4 load balancing, various proxy services, etc.
It's not easy at all either on docker-compose. You have to roll your own proxy container and take care of TLS termination, cert renewals, manage specific headers and config, etc.
I am not sure what exactly I want to add to the conversation, only that either way, it's not "easy". I'd be glad to be proven wrong if you have a specific solution that makes ingress really painless.
Is there a source on such bold claims?
Pretty sure the 100% and 80% refer to using Cuber for Pushpad. So its probably more about using Kubernetes than about Cuber.
For my use-case, which does not require HA, is relatively B/W intensive etc; K8S on bare metal can be 11x cheaper (with the same amount of hands-off, click to deploy, no messing around) as GCP.
So, fundamentally, there can be enormous gains w.r.t price vs cloud, 80% cost reduction leaves an infinitesimally small margin though; if they really are providing all the same benefits as a public cloud.
It would be very absurd to claim something like - I was using i9-13900KS but realized I could run the same workload on my raspberry pi, but hey I also used this packaging tool in the process therefore I saved 80% costs because of the packaging tool.
i stopped reading the spec, after i have seen that the only label they attach is the lb. anyone who really uses k8s will not at this product longer than it takes, reading the readme or the first bit of code.
I'm not a fan of these business case buzzword sentences. There is lock-in, although less, and "up to" is meaningless. It could be 81% cheaper, or it could be more expensive.
I think "up to 80% cheaper" implies "No more than 80% cheaper".
Seems more complex than a yaml manifest - Is there a Kubernetes operator available for these Cuberfiles
If the generated configurations are suitable and the SRE hire is (as they should be) able to trace through how they're generated and audit the results, I don't see why said SRE would want to remove it unless/until it actively gets in the way.
When I first start working on pre-existing infrastructure, finding out what my users' current pain points are and figuring out how to work on them is always the priority, and removing a tool that's currently working, probably creating additional hopefully-transitory pain points in the process, seems like the opposite of a good place to start.
I can deploy an application on a machine running a specific Debian version. Go to some provider of bare metal or VMs, they usually have a "Get a Debian version soandso machine" button. Then I copy some files to a specific directory, then I run a setup.sh script that turns this specific version of Debian into a server running my application.
But how do I deploy "on Kubernetes"? Where do I get a button "Rent a Kubernetes"?
What kind of a question is this?
It means to deploy "on Kubernetes".
It doesn't mean to run a script on a server.
It means you need a Kubernetes cluster where you have deploy permissions. Then you can deploy on Kubernetes.
> Where do I get a button "Rent a Kubernetes"?
https://cloud.google.com/kubernetes-engine/ https://www.digitalocean.com/products/kubernetes https://azure.microsoft.com/en-us/products/kubernetes-servic... https://aws.amazon.com/eks/
(In no particular order)
Kubernetes is an orchestrator that you control via an API, there is no file system to put things on and you don't run anything directly. You tell Kubernetes where to get your software from (as a container image) and how it should run it, and Kubernetes will then make sure the specified number of instances of your software is running with the settings you specify, on a suitable machine, with any storage volumes and other resources you request, or tell you it can't.
> But how do I deploy "on Kubernetes"?
Depends on the way it's set up. With vanilla Kubernetes you put container images in a registry that Kubernetes can access, write a manifest YAML and send that to the Kubernetes API (e.g. using the kubectl utility).
> Where do I get a button "Rent a Kubernetes"?
GCP, AWS, Azure, DigitalOcean, ... though they're labelled slightly differently.
Think of Kubernetes as an operating system for a cluster of computers.
You can rent managed Kubernetes from any cloud provider.
* Comply with general Linux rules for creating applications, i.e. either provide a file containing ELF executable with the appropriate metadata s.a. execution bit set etc. or similarly, a script file with a properly formatted shebang line.
* Comply with Debian package manager rules, i.e. create a properly formatted DEB file that can, in principle, be installed by "deb" package manager (i.e. doesn't create dependency problems, pre/post-install scripts execute etc.)
It's similar for Kubernetes deployment: except on top of creating a DEB package, you now also need to provide a "chart", that is a document (or a collection of documents) that describe in a formal way how the application is to be installed.
In a way very similar to how you can side-step creation of DEB package by, say, writing a Makefile and using that to install, you could also side-step creating a chart by, eg. writing a script that creates all the necessary Kubernetes entities. It's also not necessary that you create a chart in addition to DEB: it's possible that you use a Makefile-like approach for the part that would be better managed by DEB package, but still provide a chart for Kubernetes (it also has a package manager, however poorly designed and implemented: Helm).
---
In a way similar to how you could rent a bare-metal box or a VM, you can rent a Kubernetes cluster. Prices very a lot, and similar to the VM prices will greatly depend on the components you choose to put in your cluster.
This being said, running applications on Kubernetes is anything but cheap: Kubernetes adds a significant management layer overhead, but that pails on comparison to the typical way to deploy applications where there's rarely any form of cohesion between components (i.e. Linux container images that support individual components rarely have much of an overlap that can be automatically deduplicated, which means the application deployment ends up having tens or hundreds of copies of the same popular libraries, like libc or pthreads etc.) But it doesn't end there: Etcd, the Kuberenetes' database backend is hugely overloaded by all kinds of traffic sent to it. Most of that traffic is of no interest to the operator. So, your network also takes a noticeable performance hit from running Kubernetes in it (a typical solution is to dedicate an entirely separate physical network for Kubernetes' "control plane").
Of course, it's possible to come up with an equally bad if not worse solution in-house, if you want the same features provided by Kubernetes, but it's also possible and relatively easy to come up with a solution that will be less resource-hungry, while performing the same function.
Bottom line: claims about 80% cheaper are pulled out of... well, we'll not discuss where they pulled them out of.
It is one line of code.
kubectl apply -f k8s/deployment.yaml
The whole event driven distributed design is.
There aren't many use cases that require more than 5k concurrent clients and the number is even higher without the HTTP overhead (a single bare metal server can handle with the proper language).
For most companies it's a waste of money.
k8s is hype tech designed to drive business to public cloud providers, because of its complexity and high hardware demands. 9 bare metal servers minimum.
And then you have those pods, which are collections of services (e.g. postgres + api server + some cache). But in the case of postgres, why wouldn't you have a dedicated cluster instead of multiple, isolated instances as it is the case with k8s?
I know, now the EDA faction chimes in with their dogma. Again most cases don't require even 5k concurrent clients. And most projects never will grow to the point of that many concurrent clients.
Addendum: And then I read the who is hiring thread and it's all k8s k8s k8s... frustrating
Pods are not intended to bundle those 3 services. Typically one would use higher level abstractions (Deployment for the API, StatefulSet for the database), each composed of multiple Pods that should be scheduled on different physical machines.
It’s no wonder you’ve had a bad experience trying to use it the way you describe. Following the intended patterns provides a standard way to achieve rolling updates, horizontal autoscaling, and a ton of other functionality that used to be bespoke code at each organization. The old way was often buggy, and onboarding new engineers took much longer.
> But in the case of postgres, why wouldn't you have a dedicated cluster
Does k8s not push you to mix postgres pods in the same host as your API pods? Unless you go out of your way to handicap it.
I understand the appeal of applying the k8s (or swarm, or nomad) model to stateless API containers, but mixing in stateful db containers and pretending they're all equivalent makes less sense to me.
Persistent storage has been a solved problem, especially when using a managed Kubernetes service. What's the problem with running stateful workloads on a cluster?
You can have all that without k8s. k8s doesn't equal auto scaling, "rolling updates".
You are a booksmart idiot.
Of course you can have all that without k8s. I’ve been operating cloud infra for over a decade, and I used to roll my own solutions to implement that functionality, so I’m quite familiar with that approach. Embracing kubernetes has made life much easier. I do read a lot, but my opinion is based on operating production systems. Whether you like it or not, k8s has become the defacto standard interface for distributed systems.
High level programming languages and libraries provide useful layers of abstraction in their own domains. Do you also reject those, and use assembly language for everything?
This guy, seriously.
Completely ignores everything I wrote and starts bullshitting.
Once you move to a type of project with complex demands around deployments, log management, failover management, secrets, environments, resource isolation, etc., then building something to solve this set of problems over and over again is very expensive when off-the-shelf battle-tested solution is available.
You can do more on a single server than hosting "landing pages" or "personal blog".
I host seemingly complex projects with simple methods, without needing any complex k8s crap.
That's what you people aren't getting.
You're so full into the complexity that you don't see the simple solution as something worthwhile.
WHAT about k8s is more battle tested than decades of doing the simple and efficient on a 1-3 servers?
You're brainwashed. Possibly even a shill of one of those expensive cloud providers.
You don't have to go full complexity with k8s. You can just use a managed k8s provider and ship a simple deployment with an anti-affinity policy to distribute across VMs on creation. Sure, there are other solutions [and this one is likely to not be the cheapest], but they're all vendor locked--k8s is about as close to a cloud agnostic compute as we're likely to get.
Once your service grows, you'll notice some rough edges around the docker swarm. When you start having dedicated SRE/DevOps roles, moving from heroku/swarm to k8s will make sense.