Managed Kubernetes Price Comparison
devopsdirective.com
devopsdirective.com
If anything, egress pricing is to low in the vast majority of cases, and it really shows when customers start using it.
Except those are not free, they're just charged for separately under "egress transfer costs". End of the day, the engineers in the networking teams at AWS still have to get paid and that service probably also needs to run at a profit. When seen under this light, the costs make more sense to me than the usual simple view of "but bandwidth transfer costs are cheaper everywhere else!!!"
https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat.htm...
But technically, your EC2 instance never has a public IP address. You can see for yourself by assigning a public IP address to an ENI attached to your instance and doing an ifconfig.
An Internet Gateway is a one to one NAT.
This requires every egress gigabyte to generate a lot of money to be worth it. If you serve high-traffic applications, like video, you have to be Netflix.
They're now rolling out a Kubernetes service which I'm in the beta of.
Been working on migrating a high egress service over to k8s for the last month or so. So far so good.
I'd recommend you take a look at Linode's offerings.
Our product is a log processing/analytics tool. Customers pay for a certain period of time for which we'll hold onto their logs for search and analytics, after which point the logs are expired and then deleted. So our ingress-to-egress ratio is very, very high, and egress is practically a rounding error on our AWS bill.
I've seen some stupid shit. Like security architects requiring employees to always but always be tunneled in with a VPN, without considering what the egress cost is of an employee surfing YouTube through the VPN.
At the end of the day, you are responsible for your architecture choices. As part of your architecture planning, you should design an architecture with maintainable costs, in particular, one where the cost of adding additional users doesn't outweigh the revenue those users bring.
Actually the cost of this is pretty small. I tend to spend a big part of my day tunneling through a VPN hosted on GCP and almost always have a mix of some sort playing in the background on youtube i.e. fairly high use. Total cost is ~£5/month. It's not free, but I suspect most companies have bigger fish to fry.
The article and Jupyter Notebook now reflect that change.
My (admittedly small) cluster of 3x 4Gb droplets, an external load balancer, and volumes enough for logs, databases and filesystems costs about 70 USD/Month. It's been absolutely rock solid too. I have very few minor gripes and a lot of positive things to say about it.
[1] https://www.digitalocean.com/docs/kubernetes/how-to/autoscal...
---
EDIT: Done!
I have updated the notebook with the digital Ocean offering using their General Purpose (dedicated CPU) droplets.
The major takeaways for DO are that they:
- Also do not charge for the control plane resources
- $/vCPU is less expensive than the other providers
- $/GB memory is more expensive than the other providers
- No preemptible or committed use discounts available
Also, I would need to do some further research to understand how to fairly compare the physical CPU cores on the bare metal systems w/ the vCPUs offered across the major cloud VMs.
Having said that, DO is enough for virtually everything I've ever worked on, and the user experience and price are so much better. They're a clear winner for almost everything I do these days.
Whether a conscious decision or not, I think offering what the 80ish percent (just a guess) actually use, and streamlining it, is the right decision.
Google started charging after they added an SLA.
Generally OpenShift heavily targets enterprises as "All-in-One" package. Some of that works, some doesn't, but honestly it's often more a case of the IT dept that manages the install ;)
Except installing OpenShift. That's horrific. Someone should repent for the install process, seriously.
Makes for higher possibility that $DAYJOB upgrades to OpenShift 4.x, but then, we would rather get rid of our (intra-group) provider and their openshift environments...
1. ships a built-in Ingress controller that has smarts like blocking subdomain/path hijacking across namespaces. Someone's dev namespace can't start adding its Pods to a production API's ingress path
2. the oc cli has first class support for switching namespaces without bash aliases
3. RBAC logic is implemented such that as a unprivileged user you can list only the namespaces you have access to with `kubectl get ns` vs "not allowed to list all namespaces" error.
4. first class representation of an "image stream" which given a container location will cache it locally, emit events like "do a rolling deployment when this changes" and a few other very simple but logical helpers
Plus all the other top-level features like Operators and over the air updates. I think seeing some very specific wins can help folks understand that the little things matter just as much as the big ones.
disclosure: work at Red Hat
OpenShift in general I don't think can be compared here, because most of the time OpenShift isn't a cloud product. There is OpenShift online, which I would love to see improved, but it's a minority of OpenShift uses. The pricing there isn't direct apples-to-apples either, since OpenShift adds a lot of value on top of "raw k8s" (there's really no such thing as "raw k8s" since k8s is a platform for platforms, and every cloud vendor adds stuff to it, but it's a useful simplification for comparison). OpenShift adds some non-trivial features to Kubernetes and they aren't just plucked from existing projects and bundled. Some major K8s features started out as OpenShift features and got merged upstream (Ingress, Deployments, etc). Red Hat innovates and improves the distribution rather than just repackaging it and slapping on a support contract. Red Hat also tests and certifies other products so you have confidence that they'll work together, which is important for decision makers that don't have the technical depth to evaluate everything themselves.
> is there any merit to IBM trying to sell it so hard?
Red Hat is trying to sell it hard, and technically Red Hat is a subsidiary of IBM, so what you say is not technically wrong. However, it's not really accurate either. People that work for "IBM" don't really care much about OpenShift, people that work for "Red Hat" do. It's an amazing product and is getting better every day, and we see it as a major contender in the Kubernetes and PaaS space. Internally we don't think of ourselves as "IBM." The two companies are quite separate and are being kept that way. Over time we will cross-polinate more, but that's a ways down the road IMHO.
If we `s/IBM/Red Hat` tho, I probably shouldn't answer regarding if there's merit to selling it. I do, but then I wouldn't be working on it if I didn't think that. I am not a fan of our pricing model, and hope that changes, but nobody in pricing cares what I think :-)
If you are an enterprise customer, I think OpenShift is a great buy. If you're a startup, it probably isn't (at least not yet. I'm hoping to change that in the future).
If your problem is "I have a BigCo with my own infra running RHEL" then it's almost definitely what you want. It follows the old saying of "nobody ever got fired for buying IBM." It gives your boss a vendor to yell at when things go wrong and its add-ons can provide an onramp for enterprise-type ops and devs who are still unfamiliar with CI/CD and containers.
Where it's not very good is if you're not a BigCo. If your org has one or more of the following characteristics consider other distros / managed cloud hosting first: a nimble ops team, existing cloud buy-in, a limited budget, a heavy "dev-opsy" culture (i.e. existing CI/CD/container processes), a desire to avoid vendor lock in, or less conservative management.
Here's what we do:
- We don't use "develop on OpenShift" features like Eclipse Che or on-demand Jenkins nodes to build a project
- We don't use OpenShift-specific resources if there's _any_ alternative in the vanilla kubernetes resource definitions (e.g., use Ingress instead of Route)
- Use outside CI/CD to handle building and packaging of the applications, then deploy them with Helm like any other Kubernetes cluster
- Use the OCP console like a crutch of last resort, preferring `kubectl` whenever possible
All of this helps avoid vendor lock-in as much as possible while still taking advantage of the secure-by-default approach to a kubernetes cluster.
Talking with Red Hat engineers, it sounds like the OpenShift-specific things are contributed upstream and, while they may not become available by exactly that name and syntax, essentially the same functionality does come into vanilla Kubernetes. Routes inspired Ingress resources, for instance. The official stance is for OpenShift users to prefer the vanilla resources because the OpenShift-specific ones are intended to be shims.
(not a Red Hat employee, just work for a company that is a customer)
Slow provisioning time, slow PVCs, slow LoadBalancer provisioning, slow node pool management, plus non-production ready node pool implementation.
Some more: rolling upgrades of k8s (said to not affect the uptime of the cluster) not being rolling in actuality, allowing upgrades when the service principal is expired thus preventing the nodes from being added to the LB, certain aks versions not being upgradable requiring you to recreate the cluster from scratch ...
I now have an OpenShift cluster that I do testing with, but if I didn't I'd probably use OVH k8s in dev because it does seem by far the cheapest.
Most of my direct experience with Kubernetes has been on GKE, but I have been meaning to work through https://github.com/kelseyhightower/kubernetes-the-hard-way to gain more appreciation for what is going on behind the scenes.
For contrast, you can manage a Kubernetes deployment using standardized yaml and kubectl commands, regardless of whether the application is running on localhost (minikube), on Azure or on GKE.
BTW, AWS Lightsail has decent GUI. Alas, it doesn't support containers out of the box. The best support for Docker image-based deployment is Azure App Service.
That's ECS, not EKS. Two different products.
The EKS documentation is at https://docs.aws.amazon.com/eks/latest/userguide/fargate.htm...
> For contrast, you can manage a Kubernetes deployment using standardized yaml and kubectl commands, regardless of whether the application is running on localhost (minikube), on Azure or on GKE.
Likewise for EKS.
> At the low end it’s worth considering Fargate distinct from EKS.
When you are using actual EKS (with or without Fargate), you certainly can use standardized kubectl commands.
The only "proprietary" things I see in your link is the specific AWS CLI commands used to set up the cluster before you can use kubectl, but both Azure and GCP require using the Azure CLI and gcloud CLI for cluster deployment, too. There's also setting up AWS-specific security groups and IAM roles, but you have to do those same things on GCP or Azure, too, and both of those have their own "proprietary" ways of setting up networking and security, so I don't see the differentiating factor.
I just have the CLI commands in my dockerfiles as comments, so once I get things sorted locally using docker I update the task with some copy / paste. I only update occasionally when I need to make some changes (locally do a lot more).
The one thing I'd love to get my DOCKER image sizes down - they seem way too big for what they do but it's just easier to start with full fat images. I tried alpine images and couldn't get stuff to install / compile etc.
I liked jpetazzo's post on the subject but there are plenty to choose from https://www.ardanlabs.com/blog/2020/02/docker-images-part1-r...
But if you are at any type of scale, the last thing on your to do list is “cloud migration”.
If you're not going to build the next Facebook, why would you need so much complexity?
If you're not going to build the next Facebook, why would you need so much complexity?
Said applications don't have to be applications that you're writing as part of your startup. It can be ElasticSearch cluster, Redis, tooling to run some kind of ETL that happens to be able to reuse your k8s cluster for task execution, CI/CD, etc. etc.
Once you don't personally have your hand in every application your company is running (in addition to the points the other comments have brought up).
I don't find them complex at all. You just tell the tools to be in a specific state and the tool applies the necessary changes. Server templates. Provisioning. Orchestration. etc.
That being said, I always recommend using a tool like Terraform to back up infrastructure and the likes.
The point I wanted to make is that my opinion is a bit different. Being able to declare how state should be instead of doing it imperatively/with configuration management is just something I enjoy and which I think does not cost much more in comparison.
That is why I wondered why not use it as a small startup?
They are mostly plain old PHP webapps, non-trivial amount of them Wordpress (shudduer), some done in random frameworks, some with ancient dependencies, some in node.js, one was ruby, etc. They are the equivalent of good old shared hosting stuff.
With kubernetes, we generally manage them in much simplified form, and definitely much cheaper than hosting multiple VMs to work around issues in conflicting dependencies at distro level or keeping track of our own builds of PHP.
We also run CI/CD on the same cluster. Ingress mechanics mean we have vastly simplified routing. Yes, we cheat a lot by using GKE instead of our own deployment of k8s, but we can manage that too, it's just cheaper that way.
Pretty much smooth sailing with little operational worries except the fact that Google's stackdriver IMO sucks :)
When you buy a large instance, you still need to set up the instance and tweak it to your application's needs. You then need to babysit this node.
It's called "use kickstart"
I wasn't saying that just use 24 core single instance. Perhaps I should have worded it better.
You can use GCP control panel to launch clusters, add nodes, launch containers expose and autoscale them without touching a single line of config file or a terminal if thats what you like. If you launch / manage your own cluster though.. It’s a pain.
How do you coordinate rolling deployments?
(I'm not saying you need Kubernetes to do these things. But if you've written something to handle them, it is very probably Kubernetes-shaped).
There's two things there - the complexity of engineering the cluster itself, and the complexity of using it.
The former is where the pain is. If you can remove that pain by using a reliable managed offering, it changes the perspective a bit.
The complexity of using it is also non-trivial - you have to learn it's terminology and primitives, understand the deployment and scheduling model, know how volumes are provisioned, aggregate logs, etc.
However, the ROI for learning that complexity can be pretty good. If you get comfortable with it (maybe a month or so of learning and hacking?) you get sane rolling deployment processes by default, process management and auto healing, secrets and config management, scheduled tasks, health checks, and much, much easier infrastructure as code. Which means if things do go really sideways, it's usually not that hard just to stand up a replacement cluster from scratch. With a bit more reading and some additional open source services, you get fully automatic SSL management with let's encrypt too.
All that said, I absolutely agree with you in principle. No one should overcomplicate their deployments for no benefit. It's just worth reflecting on what those benefits are. Kubernetes had a bunch of them - and whether they're worth the effort of getting to know it depends very much on the system(s) you're running.
Thanks for an insightful response - Don't you think that a lot of wizbang software developers just want to use latest and greatest buzzword thing, whatever that might be, to get some cool points? As I grow older, I see increasing lack of objectivity and decreasing attention to KISS principles.
Yes, absolutely, and it drives me up the wall. I've seen some incredibly unsuitable technology choices made, the complexity of which have absolutely sunk productivity on projects.
That said, it's easy to become cynical and associate anything that's become recently popular with hype-driven garbage. But that can blind you to some really great new stuff too. I tend to hang behind the very early adopters and wait to see how useful new tech becomes in the wild - the "a year with X" style blog posts tend to be really informative.
The point I'm trying to make is that people often think in absolute terms. The problem isn't thinking "React is good for SPAs" - it's thinking "React is good for all websites". I've come across a number of engineers now who genuinely think React is a good fit for e.g. totally static sites, "because it's fast".
I've come across similar persistent beliefs around NoSQL databases. Some teams wouldn't begin to consider an RDBMS for their primary data store, even while their data is heavily relational - because "SQL databases don't scale".
It's not that technologies are good or bad - it's the blind belief that a given technology is best for all situations. The hype drives the development. Not of the underlying tech itself, but in the teams using that tech.
If the same company had to hire someone else with your same skillset as they expand or replace a coworker, they would pay them market value. They would have to offer competitive salary. When that happens, you find new employees making more than equally skilled existing employees. It’s called salary inversion.
but, the company I worked for was not paying higher wages for the newer employees than myself and I got pretty regular, pleasant raises.
I bet they talk about the company “being like a family”, don’t they?
It remains the best company I’ve ever worked for. Small and my opinion and actions actually had real impact to core features of the company, so :)
Worth it, I think.
I did have direct influence into huge and important parts of that business, though and I really, actually was respected and given broad rein in what I could do. My coworkers and I also tended to have a great time working, and I had direct lines to the CTO. We talked often.
Helps if you also don't do due diligence on various things (like actually caring about logging and monitoring), so you scratch out those concerns. This is not even half sarcastic, people went pretty far with that.
You might probably still spend more time than you think on actual deployment and management due to lack of automation & standardization, but if you're sufficiently small on the backend it works.
When the number of concerns you have to manage rises and you want to optimize the infrastructure costs, things get weirder. Kubernetes' main offering is NOT scaling. It's providing a standardized control plane you can use to reduce the operational burden of managing often wildly different things that are components in your total infrastructure build - Things like infrastructural metrics and logging, your business metrics and logging, various extra software you might run to take care of stuff, making it easier to do HA, abstract away the underlying servers so you don't have to remember which server hosted which files or how the volumes were mounted (it can be as easy as classic old NFS/CIFS server with a rock-solid disk array, you just don't have to care about mounting the volumes on individual servers). It makes it easier to manage HTTPS routing in my experience (plop an nginx-ingress-controller with a host port, do a basic routing setup to push external HTTP/HTTPS traffic to those nodes that run it, get the bliss to forget about configuring individual Nginx configs anymore --- or use your cloud's Ingress controller, with little configuration difference!).
In my experience, k8s was a force multiplier to any kind of infrastructure work, because what used to be much more complex, even with automation tools like Puppet or Chef, now had a way to be nicely forced into packages that even self-arranged themselves on the servers, without me having to care about which server and where. Except for setting up a node on-prem, or very rare debug tasks (that usually can be later rebuilt using k8s apis to get another force-multiplier), I don't have to SSH to servers except maybe for fun. Sometimes I login to specific containers, but that falls under debugging and tweaking the applications I run, not the underlying infrastructure.
That's the offering - whether it is right for you, is another matter. Especially if you're in cash-rich SV startup, things like Heroku might be better. For me, the costs of running on PaaS are higher than getting another full-time engineer... and we manage to run k8s part-time.
There are tremendous benefits to K8S. It isn't just hype.
On the flip side, starting out as a monolithic (all in one VM) app will take significant effort to transition to micro services / K8S.
If I think I might end up at microservices/K8S, I think I might as well plan for it (abstractly) initially.
Additionally, new crazy things are emerging like service mesh (Istio, Linkerd) which make observability and security easier.
Of course, it all is getting quite complex. But it brings value in the end.