Kubernetes on Google, Azure and AWS Compared
kubedex.com
kubedex.com
Wat? IMO, the Azure portal is amazing to work with, especially compared to the old-fangled, inconsistent UI that AWS provides.
> Show them how some things on Azure need to be done in a clunky web UI
Years ago, this was true, but it hasn't been for a long time - the Azure web UI of today is fast, consistent and looks great.
> some things on Azure need to be done in a clunky web UI, other things need Powershell and other random stuff uses the CLI
I've been using Azure for years, and I'm not aware of anything that can only be done in the UI. I also don't believe there is anything that only works with Powershell, or only works with the CLI.
> how that effects the design of DevOps pipelines and automation in general. Yes, you can make it work, but why make life hard for yourself
Eh? Azure DevOps pipelines are great to work with, and there's a huge library of tasks available.
After reading this, it doesn't sound like the author has actually worked with Azure recently, so I'm really not sure why he bothered including it in this article.
AWS portal is bizarre, it's like each portion is written by a different company. In ECR for example, clicking into a repo is fine, but then going back doesn't do anything. Why? Even simple favicons don't look like they're from the same company. When you've got 40 tabs open you want an idea that that particular tab is from AWS or not.
Azure is nicer, but it's not reliable (see my other post on this thread). It's almost like Azure have done too much lazy-loading which persists through login sessions, you can't rely on a refresh to give you the truth - for all the problems with AWS, you can.
And yes I know this isn't CLI stuff, but it's important. Even on fully automated / scripted deployments it's sometimes nice to go into the UI and see what's going on. If it doesn't represent reality, there is a problem.
FWIW GC doesn't have any of these problems, but it is slow.
As another datapoint of one, I've been working with Azure for years, and have never found the UI to be unreliable.
I could go on and on. It's not like Azure is completely bad, but I far prefer the developer experience of AWS. It's not perfect, but IMO it's much better.
Regarding copying stuff, if you hover over just about any text element, a little 'copy' icon will appear. I like this feature.
Think I didn't try that? Doesn't work on CosmosDB databases. Yes, for a lot of items it's true that "copy" is easy.... but when it's not easy, it's crazy hard.
I'm guessing that your company doesn't have hundreds of Azure subscriptions (and the proportional number of other resources to manage, in them). When there are a lot of elements that you need to manage/ browse through/ look for, you start to appreciate the simplicity of the AWS ui, and the fact that it prioritizes "snappy" over "fancy".
Azure might be great for an SMB, but I think for large enterprises, AWS is still unchallenged.
Just about everywhere I'd want to filter stuff in the Azure portal, there's a search box or/and filters. Honestly, I find it to be really snappy for me.
Worst thing about it is that I was originally impressed with Azure DevOps - the onboarding was awesome, almost combined the best bits of GitLab / GitHub with none of the crap of Jenkins. It was only after going back a couple of days later it let me down.
Azure Devops is awesome though. Told my wife i'd found a modern Microsoft product I really liked, she nearly fainted.
I know a lot of things have changed at Microsoft, but and also the language I was using (ruby) was not the best match perhaps. But at a lot of steps:
- documentation was directed at VS ("you have this nice integration to do that in 30s. Other people go this way, poor some jugs of coffee you'll need it.")
- docs and tutorials are old and deprecated. Even articles written by staffers a few months ago were explaining stuff that disappeared from the portal interface, and no update was available. There seems to have been a huge system transition, and it was painful to be caught in the middle, especially when after that it means the new system has a risk of being beta quality for a while.
- anything remotely complex means moving to Kudu, when there's already another set of CLI tools to deal with.
Basically it felt like I needed to invest 6 months of my life to really get it, and then I'll be ready to use it professionally. AWS or even GKE don't have as much as a learning curve for people coming from the linux world in my opinion.
I'm mostly from a Windows and .NET background, but I'm reasonably comfortable in Linux too, and I've been surprised at the breadth of support for Linux and traditionally non-Windows languages, such as Python and Node.js.
As an example, take the docs for what must be one of their most used services, App Services: https://docs.microsoft.com/en-gb/azure/app-service/
Right on the front page there are quickstarts for .NET, sure - but also Node.js, PHP, Python, Java...
Some of their newer services are even Linux first, such as AKS!
AWS's UI might be inconsistent, even old fashioned - but it's extremely quick to navigate around and find things.
With Azure I'm either having to spend 10 minutes hovering over an array of icons trying to figure out which one will take me to the service I want, or I'm fighting with their panels which hide information I actually wanted.
When it comes to other basic functionality like updating billing details - Azure only lets that be done by the root account, which I really don't want to hand over to our accounts department, and they don't want me to know the card details they use for it. So we're at this stale-mate where I have to log in as root, navigate to the right area, then let them enter the card details.
I, too, think the Azure portal is amazing to work with. It feels much more coherent than the AWS portal. I like the Azure command line package as well, in preference to the AWS CLI. I've never touched the Azure PowerShell tools and have never felt like I had to.
Creating resources via the portal or command line is something I only do as a last resort anyways; we use Terraform for all of our cloud resource creation purposes.
There are some things I definitely prefer in Azure to AWS too; I find the AD-based authentication to be much easier to understand and implement compared to IAM. The SMB shares (File storage) are great, integrate well with Kubernetes, and are a lot faster than EFS.
Non-premium VM IO performance is abysmal though, and I really wish I could store SSH public keys in Azure AD.
This is the same advice I gave to the AKS product manager.
I don't find the website to be slow - they seem to have updated it ~6 months ago, and it seems to be really responsive since then.
I'm with you on time to create some resources though - starting up a new VM takes anywhere between 5 minutes and (on occasion) an hour. I'm working a bit with Azure Batch these days too, and it generally takes 15-25 minutes to start a small pool, which is really irritating. I wish they'd introduce a containerised version of batch, with jobs starting in seconds.
Also, the storage with azure is very, VERY slow. Spin up a VM and give it a go. ultra ssds might change things, depending on pricing.
AAD password policy can only be set with powershell
1. Creating a project timed out after 5 minutes or so. This was in the gui, but had no adblocker or anything similar. Refreshed page = no project. 2. Went to a different machine, logged in, no project still. 3. Went to original machine - refreshed - no project. Logged out and back in again - project was there.
Azure DevOps was awesome at first but I can not trust something like this when I work with it all day.
BTW never had a problem bringing up an instance quickly on GC. AWS sometimes just sat there for a while, depending on the instance type and region. Azure just stops because it feels like it. I'm sure GC isn't any better than the other two, it's just less utilised.
The UI is my biggest issue. Yes, we don't depend on it because we all script our stuff. But after spending an hour fapping at an SSH terminal it's nice to look at the UI and see stuff happen. Azure is the worst at this, with AWS a close second.
That reminded me of my favourite bug with AWS UI / Container Registry - you can order the list of container images any way you like, but it'll only order those it's loaded. So i'm looking for a new image called 'zzzzz' but it'll never appear, I need to search for it. Did anybody involved ever use this?
I won't say you always have a choice, but I'm happy to hook you up with my network if you want to passively seek out a new role. Email in my profile.
Sidenote: I have seen the ECR UX shenanigans you mention.
GCP's UX is so nice... e.g. compare the equivalent as command line option in the UI vs the automation script that Azure gives you.
AWS UI was usable. Mostly stuck to automation or the CLI though. Azure was... I didn't use it a lot, but I thought it was terrible.
With GCP, I find the CLI to be more intuitive, but even still, I find myself using the UI more often than not, it's that good.
I just wish the GCP console people would talk to the Gsuite admin people.
1. Web UI
2. Powershell
3. CLI
4. REST API
How does Hetzner compare here? In terms of both, general experience and costs.
We use Hetzner for this purpose. It's hard to compare it with the other providers since with Hetzner you are renting unmanaged servers, not turning up cloud instances.
But it is absolutely the best value you can get. Our bill would be at least 10x higher if we were using Google/Amazon/Microsoft.
Considering we spent a lot on servers as is, that makes it well worth it for us. It might not be if your product is very computationally light.
Hetzner Cloud is already a thing[1] and is a direct alternative to Amazon EC2, if you choose dedicated vCUP.
What exactly would that person be handling? We probably spend no more than a couple hours per week handling everything involved. That's the point of something like Kubernetes in the first place.
>Will your workload on Google cost more than $50k?
Our sever bill with GKE would far exceed $50k/yr (it would exceed $50k/mo!)
As a result, Hetzner is a no-brainer for us. As mentioned before, that might be different for you if you are doing something that is computationally light.
Funnily I spend more time administrating GKE than I do with applications on VPS.
I can say that this has pretty much gone as I'd expected.
Microsoft have 80,000+ developers. Their partner ecosystem is absolutely massive. I've watched them hire hundreds of developer advocates that talk at events. It's therefore quite hard to write anything online that's critical without at least half of the comments coming from a biased source.
It is interesting to see the gap between views in the comments here.
Edit: I watch this comment go up and down a lot as the vote battle between those with an agenda and those without click against each other.
Weirdly, I did criticise AWS a bit for their EKS offering in the blog but I've not had anywhere near the toxicity from people about that.
I downvoted due to your statement that implies that anyone disagreeing with your view has "an agenda" whereas those upvoting you don't.
People have different opinions and turning debates into an adversarial situation "if you're not agreeing with me you're against me" isn't helpful, in my opinion.
Some stuff is perceptual, other stuff is factual. The issue I personally took with your post is that you stated things that are blatantly untrue, such as:
some things on Azure need to be done in a clunky web UI, other things need Powershell and other random stuff uses the CLI
If you want to be pedantic about the sentence you took offense at it is actually true. There are operations that can only ber performed using Powershell. People have given examples in this thread. But that isn't the point.
What I should have written for that part was that most people google for solutions. I've done this in the past on Azure and you get a random assortment of answers. Click here, Powershell this there, do whatever. It's disjointed and takes you out of mental context. I could update the blog with more clarity but I don't think anyone will change their mind.
I've used Azure, GKE, AWS. It was played down a little in the blog, but I have used the non AKS parts of Azure pretty extensively. Have you tried GKE? My suspicion is that you've been stuck in a Microsoft world for a while.
Not everyone is a paid shill but some people have a lot invested in the Microsoft ecosystem and I can only assume that's the reason for some of the comments here. I do 100% believe the Microsoft PR team has posted in here at least once though :)
I've worked on both sides of these fences (AWS/Azure, Windows/Linux), and think that you are correct. The Microsoft partner programme is basically all about Azure these days, so if you are a Microsoft professional then Azure has been made part of your job, and nobody wants to think that they are doing something second-rate.
https://news.ycombinator.com/newsguidelines.html
> Please don't impute astroturfing or shillage. That degrades discussion and is usually mistaken. If you're worried about it, email us and we'll look at the data.
AWS works. A lot of the UI is dated but the new designs are nice.
GCP is the best. Google has done a great job on Dev UX both CLI and Web Console.
Care to elaborate?
Of course UX is also based on my bias (AWS and GCP).
Nonetheless I was surprised at the presentational bar for the azure UI. It felt very dated. In the end I got a hang of it but again, it still felt like using an old pc.
CLI wise I had a lot of errors spinning up resources. My support experience getting it figured out was sucky. I had to abandon azure as a decision quickly with that experience. Too risky.
Documentation is a mess : the general layout for google cloud is really a pain to read and navigate, but in addition to that you often have to jump between kubernetes doc and google doc, with some information being on both. Don’t do that please : either make it obvious that people need to get familiar with certain chapters of kube doc, or provide all the info ( me preference would actually go to the first option..)
It’s quite hard to guess what you can do on the gke web interface and what you can’t. You can feel it’s meant for people that really know kubernetes, and not people who discover both gke and kubernetes at the same time.
And for the life of me i couldn’t get the load balancer manage https. I’ve read this was possible, but never saw the actual page explaining how. I ended up using cloudfront but lost the ability to see end user ip in my logs in the process. ( also logging is a real beast to tame on its own, with no obvious way to know what is available by default, what is a paid option, what should be configured on the stackdriver website, and what should be coded )
If you use GKE ingress, you can access end-user IP in X-Forwarded-For header.
"I’m being serious when I say this: if the company I’m working for decided to migrate to Azure I’d find a new job."
"It needs to be fast and bug free so that I can build cool automation on top. Working on something like Azure, especially after having worked on AWS for years, would be extremely depressing."
If the article is supposed to be an _unbiased_ comparison between cloud hosted Kubernetes providers, I'd say it's a bit of a fail. For some it would be completely different experience because they have experience with Microsoft technologies. And those people might as well quit if their company moves to AWS or a non-Azure platform.
Azure services can be flaky, and slow, and the "blades" UI is a bold design choice that doesn't really work in practice. No amount of experience with Microsoft technologies can help you with those problems, unfortunately.
Does anyone know what the author is referring to with this claim? I don't see anything in the sheet to back this up. At least from a high level, all three options support network policies via CNI, and GKE and EKS use the same one, Calico.
"Each cluster receiving an IP range for nodes and another for the containers inside, which are directly routable across your private network, other clusters, and regions."
AWS and Azure don't have a flat global network. You have to setup VPN's and complicated overlay networks.
Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well.
Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance.
On workloads where I care about Network IO:CPU ratio, I'm using 8 core nodes and seeing 16gbps throughput between them consistently.
AWS tries to avoid letting customers create multi-region or global failure modes, like:
* https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15
* https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03
* https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11
* https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04
* https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07
* https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17
There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP).
> Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance.
Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ...
Edit: white-space only formatting changes
As for the instances I tested, they were m4 and r4. To get full 10GE out of either, I needed to use m4.10xl and r4.8xl, both roughly around $900 per month.
By comparison an n1-standard-8 is about $60/mo and the n1-highcpu-8 is $50/mo. I've tested the former and got something like 15.8 or 15.9gbps using iperf.
IMHO, that's pretty significant.
AWS has cross-region VPC peering, which is simple to setup and makes it feel “flat”. There’s no need for VPNs or overlay networks.
Also, while setting up peering is a 10-20 minute task there are lot of constraints, such as the VPC's have to have different CIDR ranges. Oops, if you go with the AWS default, probably all regions using 172.31.0.0/20.
Google got networking right with a flat global any region setup. If you need segmentation then just create a new GCP project. If it turns out projects need to communicate after all, GCP has VPC network peering.
Two different design philosophies, there are been a global GCP networking outages due to having the single flat network. A few this year. All providers have had their issues, but, global blast radiuses concern me.
Also in general GCP networking is better than aws ime (ever tried to have multiple projects in aws share a network?). Don’t know anything about Azure but I suspect msft still sucks at networking things so it’s not great.
I'm curious why you think that Calico's use of iptables will have performance impacts for large clusters, and what type of performance impacts you expect (bandwidth / latency / cpu / something else?).
From my experience, Calico performs rather well in large clusters (e.g. 2k nodes, appx 100 pods / node, several hundred network policies).
After an initial look I think he is referring not to the kubernetes specific networking support but the GCP networking.
From the spreadsheet alone GKE has support for cross region load balancers. The author also points out cross region networking.
I don't know the exact advantage the author sees with cross region load balancers since you can only deploy a GKE cluster to a single given region. The GCP LB's are really nice though in that traffic enters Google's network at the closest location to the user and then travel within Google's network, which is wicked fast.
I have not played around with federated clusters in multiple zones so don't know how that would play out either.
They may also just be referring to just general networking from the providers. With the first two points they at least make mention in the spreadsheet of it though.
tl;dr: not sure what the author meant but here here is some food for thought on it.
We've experienced multiple outages from forced upgrades. We usually can find out the reason through github, but it may not be a priority for them to provide the fix. If they do, it could take days or weeks for it to become available. Much revenue has been lost because they can't do something at the speed which we could do it.
This, at least, is wrong. Everything related to Azure's hosted Kubernetes can be done via the CLI without ever touching the web portal or PS.
Did I mention we are hiring devops engineers? If you are a kubernetes guru email me :-)
Figuring out how to manage kubernetes on different providers can be a little tricky.
I tried Azure and had some issues and didn't really like it. I haven't tried AWS or GKE for k8s yet.
Granted, I haven't really used Azure in six months.
- 3x ingress pods (1 for an internal load balancer, 2 for an external load balancer)
- 1x cert-manager
- 2x production version of internal app01
- 2x staging version of internal app02
- 1x fluentd
- 1x elastalert
- 1x kubewatch
- 1x prometheus node exporter
- 1x redis for sentry
- 1x sentry web
- 2x sentry worker
- 4x sourcegraph language servers (there are 9 language servers running across the 4 nodes, this node seems to especially like them)
- 1x thelounge
- 1x staging version of app02
- 2x production version of app02
- 1x kube proxy
- 1x kubedns
- 1x couchdb (small footprint just for testing some things)
the node is just over half provisioned and we are due to add another node to the cluster soon.
The node exporter runs as a daemonset on each node and provides node specific metrics like CPU, memory, disk IO, network IO, etc.
The node metrics comes from the node itself.
The helm charts include this: https://github.com/helm/charts/tree/master/stable/prometheus...
It's a bad idea to have that many if they're all Apache or whatever. But when they're simple Golang containers that handle APIs it's ok.
On K8s I typically see somewhere in the low 20s as the number of pods per node.
Openshift I'll see high 20s or low 30s as the average number of pods across most openshift customers. But we're seeing some crazy numbers for some larger enterprise customers. 100,400,2500 pods per node. This seems to be driven by the way openshift is licensing
The latter is an absolute nightmare to support, and they seem to have trouble organizing internally as well.
I think I'm running 70 16-core nodes.
It'd be nice to use larger nodes and have more overhead, but I suspect kubelet would have a hard time.