The naming of the software makes for very unfortunate conversations if you're talking to other Greeks about it.
Yes, it probably is overkill for your 10req/s app, granted. But I highly advise everyone to at least give it a try out of curiosity, because a lot of hard problems at medium scale and above just go poof with K8s.
No, they don't? I don't know why anybody would just assume that something as complex as kubernetes would just run flawlessly once you actually try to run it on thousands of servers. Must be something to do with google PR because people definitely don't seem to assume the same for e.g. hadoop or openstack. Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running?
>when I started my new job on a 22 node GKE cluster
>problems at medium scale
vs
>Make a guess at how many people large companies have to employ to actually keep their smart cluster scheduler running
K8s obviously is not a silver bullet and of course there's ops work to be done. I can't comment on whether operating K8s clusters in other contexts makes economic sense, but I know it does for our current team.
But the reason for that is not that it makes "hard problems go poof at scale". The reason is that you're using a hosted service where somebody else (in this case, Google themselves) takes care of the problems for you for a fee and - at small scale - you only have to pay them a fraction of a single operation's engineers salary for it.
So of course it makes economic sense for you to use a hosted service where the sharing economy kicks in, but your recommendation to use kubernetes because it solves hard technical problems at medium scale does not follow from that.
Even if they had, it doesn’t justify your aggression. Maybe take a break from the internet for a bit to clear your head.
GKE's services are, as far as I can tell from their pricing page[0], free. The compute running on top of GKE is charged at the GCE rate, the master node spun up by GKE is free.
Disclaimer: I work at Google Cloud, but nowhere near these offerings.
As to why they're taking the loss on the master node VM, I don't know. I had previously expected that it was a cost and was quite frankly pleasantly surprised that it wasn't - it seems like the most obvious sell. If I had to guess as to why it's not my best assumption would be that there's far more to be gained in getting companies comfortable with scaling and from angles that go beyond just the strict monetary benefit of them going from 3 compute instances to 300.
Much different use-case than most, though: automated machine learning pipelines via Pachyderm [1]. Would never have touched K8's if it wasn't for Pachd, and between their layer and GKE it's been quite painless. May have given up on Pachyderm if it wasn't for GKE; we're not cut out for maintaining K8's, and our first shot on AWS (pre-AKS) was not pretty.
tl;dr: K8's are great when it's all but completely abstracted away from you
Same reliability as AWS, similar performance, much cheaper.
Sure, in case all of Paris goes down, you’re down, too. But in day-to-day usage you can have good reliability on a shoestring budget.
Genuinely interested.
VMs are the most reliable thing of any cloud, and often more reliable then their own managed services because of complexity. GKE does live-migration of VMs and Azure/AWS are close to the same thing, so building on that foundation and doing your own "managed" layer via Kubernetes provides a better experience, while also giving you that spot instance discount that you wouldn't get otherwise with PaaS.
EDIT: No, really, k8s won't make a typical RDBMS suddenly able to run in three datacenters across three continents.
Honestly, at that point, what does k8s get you?
AIUI, Kubernetes is fine for stateless systems, but it is really no better on the storing-state question.
But the parent's comment is missing the point. K8s is not for failover. K8s is literally just a giant monolith of microservices for running microservices. It's not intended to provide failover for non-microservices, it's intended only to run microservices, and as a side-effect of needing to be able to scale them, it inherently provides some failover mechanisms.
$ docker build . -tag gcr.io/google-samples/hello-app:1.0
$ docker push gcr.io/google-samples/hello-app:1.0
$ kubectl run hello-server --image gcr.io/google-samples/hello-app:1.0 --port 8080
$ kubectl expose deployment hello-server --type "LoadBalancer"
FROM golang:1.8-alpine
ADD . /go/src/hello-app
RUN go install hello-app
FROM alpine:latest
COPY --from=0 /go/bin/hello-app .
ENV PORT 8080
CMD ["./hello-app"]
https://github.com/GoogleCloudPlatform/kubernetes-engine-sam...
I'm not saying that is a bad thing by itself, but there definitely is huge amount of added complexity behind the scenes.
To the best of my exposure, Kubernetes is a well engineered system.
You can try to go cross-country with either of them. One was engineered to survive extreme environments, protect you from the elements, and move really fast. The other was engineered to leave you exposed, operate in a moderate climate, and go much slower.
If you have problems with the former, it's time to call the technicians. If you have problems with the latter, you might fix it yourself with a pocket screwdriver.
https://docs.aws.amazon.com/codedeploy/latest/userguide/gett...
Distelli (now Puppet Pipelines) too:
https://www.youtube.com/watch?v=ZZlYADohS4Q
And then there are the PaaS options like Heroku, Beanstalk, AppEngine, Lambda, Serverless, and Apex.
That was my point. I wanted to point out that while some people have only/first heard about failover in the context of kubernetes, it is not something that is specific to kubernetes or even the problem that kubernetes was build to solve.
Of course it is not designed to be a failover solution specifically and using it (exclusively) as such would be ill-advised; I was just trying to be diplomatic while pointing that out.
Some ideas for things you could do in a web/website/internet context assuming you have a single point of presence:
One type of "HA" is network-level failover; haproxy (L7), nginx (L7) and pacemaker (usually L3) seem to be very popular options, but I think there are dozens of other alternatives. In terms of network-level failover, things get more interesting once you are running in multiple locations, have more than one physical uplink to the internet or do the routing for your IP block yourself.
For application-level failover and HA, one option for client/server style applications is to move all state into a database which has some form of HA/failover built in (so pretty much every DB). I think this is very common for web applications and also for some intranet applications.
Assuming you have a more complex application running in a datacenter, there is also a lot of very interesting stuff you can do using a lock service like Zookeeper or etcd or really almost any database. Of course, you can also make your app highly available without using an external service; there is a mountain of failover/HA research and algorithms that you can put directly into your app (2pc, paxos, raft, etc). Of course all of these require some cooperation/knowledge from the application. For some apps it might be very hard to make them "HA" without relying on an external system, but for some apps it will be trivial.
Note that when you move away from a web/datacenter context, to something like telecommunication or industrial automation, so something that doesn't run as a process in a datacenter but is implemented {as, on} hardware in the field, failover and high availability will have an entirely different meaning and will be done in a totally different way.
My biggest fear is helm becomes chef cookbooks
I have interacted with people who run production workloads via Helm but struggle to do very basic things with kubectl (like recreating a service).
This is an argument that can be used for any high level abstraction, however. I guess Chef is used since they have very good testing tools so Cookbooks are relatively more stable than other CM definitions. Until things go wrong with the service that is.
Success I have had with chef has been generating my own cookbooks that dont try to do everything for every OS
For example Azure App Service or equivalent from the other vendors. You don't have to manage a VM.
Most notably, swapping servers (slots) always caused downtime, accessing logs was flaky and slow, azure interface is just plain slow, continuous deploy was flaky (or it just didn't kick in until 20 min later?) and I had to manually restart servers many times
Care to quantify that? Because we don't use the cloud (in fact, in my current role, we don't even make use of virtualisation for production infrastructure) and we have no issues with productivity. Or rather, let's be a little more honest, we have no issues with productivity that would be solved by a move to AWS/Azure/$_some_other_cloud_provider.
Absolutely correct.
Disclaimer: I have no idea about your actual real-life situation. This is vague and hand wavy on purpose.
Whilst I realise you were being vague on purpose, the above statement represents no kind of business justification.
In my experience of moving applications to the cloud, there are always folks who don't believe there is any gain to be made. These folks have always been wrong (so far!)
I suppose I'm only commenting because I hate to see someone snub the potential of cloud. But hey I'm just some guy on HN.
While I agree that the cleanup can inevitably lead to productivity gains, I believe they are overshadowed by the productivity gains from the actual change that is being made.
As someone on Azure, I think the solution for small-ish guys like me (who still need like 8 VMs in production to run our site due to special requirements) is a greatly simplified container management system. Something that fulfills the basic Kubernetes features, but isn't such a steep learning curve.
I think Microsoft is doing this with Azure Container Instances, but I haven't tried it yet. (No doubt AWS is doing something about it as well.)
Or they're going to take care of all the k8 configuration crap through some neat Visual Studio Tools. In a VS preview I saw a "deploy to Kubernetes" button. I just want something that will give me more web API instances when my server is getting slammed.
I think once their managed kubernetes (AKS) hits GA, the learning curve will be slightly lower.
We've been running off aks for a bit, the only negative is how slow spinning up new clusters can take.
You don't need K8s, until you do, at which point switching can be a nightmare scenario.
If you need to do something that doesn’t map well to firebase, then my next stop would be AppEngine.
If what you want to run won’t map well to AppEngine, then start looking at K8S.
For me, this is usually wanting to run some kind of legacy system, or where a higher level of security is needed, or if you want or need to plan to run across multiple cloud vendors or in a mixed workload environment.
If AppEngine has been covered by google’s BAA when I started my current gig, I would have just used that, with k8s to handle the couple of oddball things that we are using that wouldn’t run as well there.
Personally, it’s not about scale so much as certain kinds of architectures or problems. You might have a system that needs to run a smallish number of nodes, but has reasons for needing what kubernetes offers without wanting to reinvent the wheel yet again with the existing cast of characters.
Firebase, AppEngine, etc. will all be blocked there?
Run your application on GKE and grow your company. When it comes time to move into China, add a cluster on a cloud provider there.
Really your next step in general is understanding containers and where they might be useful. K8s consideration is probably after that.
I guess the popular term is "cargo culting". Or maybe Heroku is not hip anymore although it probably is the best advice for many companies.
Developers really love working with Kubernetes. It doesn't take a lot to get started and if you're not doing anything too crazy it is a huge productivity boon IMO.
Where things get hairy is persistence, but that's the case regardless of Kubernetes or not.
This number of tools speaks for itself. Yeah, you would be better off with out of the box stuff due to lack of skills.
Edit: Put another way, give me APIs instead of config file hell.
Interesting proposition - if it helps attract better talent does that mean they needed k8s?
Look at this question and top answer from Quora. It asks how you would build a website.The answer dives into a host of engineered technologies that would suit something you expect to need to scale massively.
https://www.quora.com/If-you-were-to-build-a-website-right-n...
(it's strange that my question resulted in downvotes).
It is handy to not break the bank and use GKE autoscaling for the batch jobs. Surely we could setup some batch job manager over some raw VMs, or maybe spend some time to select and learn to use yet-another AWS service.
It is handy to go to the GKE "Workloads" page and see how many servers are currently active, using how many pods. Surely we could pull that off with raw VMs, but it would take me a few minutes extra / a brittle script to pull it off.
Do we _need_ Kubernetes? No. It makes us more productive? Yes. Am I confident that in the next assignment I'll be able to reuse my skills, regardless of what VM provider they use? 90% yes, pending EKS becoming usable.
Echoes of "you don't need C, just use Assembly, it's simpler and faster" from early '90s gaming industry.
Adding service discovery and container orchestration will probably not make your product better. Instead it will add more moving parts that can fail to your system and make operations more complex. So IMO a "containerized microservice architecture" is not a "feature" that you should add to your stack just because. It is a feature you should add to your stack once the benefits outweigh the costs, which IMO only happens at huge scale.
Most people know that "Google {does, invented} containers". What not so many developers seem to realize is that a Google Borg "container" is a fundamentally different thing from a Docker container. Borg containers at Google are not really a dependency management mechanism; they solve the problem of scheduling heterogenous workloads developed by tens of thousands of engineers on the same shared compute infrastructure. This however, is a problem that most companies simply do not have, as they are just not running at the required scale. At reasonable scale, buying a bunch of servers will always be cheaper than employing a Kubernetes team.
And if you do decide to add a clever cluster scheduler to your system it will probably not improve reliability, but will actually do the opposite. Even something like borg is not a panacea; you will occasionally have weird interactions from two incompatible workloads that happen to get scheduled on the same hardware, i.e. operational problems you would not have had without the "clever" scheduler. So again, unless you need it, you shouldn't use it.
I think the problem that Docker does solve for small startups is that it gives you a a repeatable and portable environment. It makes it easy to create an image of your software that you are sure will run on the target server without having talk to the sysadmins or ops departments. But you don't need kubernetes to get that. And while I can appreciate this benefit of using docker, I still think shipping a whole linux environment for every application is not the right long-term way to do it. It does not really "solve" the problem of reliable deployments on linux; it is just a (pretty nice) workaround.
Could something like mDNS be a lightweight solution to that problem?
And also I am genuinely curious how kubernetes would solve that. When you install kubernetes on all of these machines, don't you have to manually do the configuration for that either? So isn't it just choosing to rather configure kubernetes instead of your own application for each deployment? If it is that much simpler to setup a kubernetes cluster than your app, maybe the solution is to put some effort into the configuration management part of your product?
Managing many nodes is the reason orchestration software exists. Your suggestion to "put some effort into configuration management" is effectively naive, homegrown orchestration.
Build or buy? That's the same argument - except k8s is free and available from many providers.
https://www.youtube.com/watch?v=dxk8b9rSKOo
Services use an internal tool called Apollo to describe the service, deploy it to EC2 VMs, and auto-scale. Apollo inspired AWS CodeDeploy and AWS AutoScaling.
Services are reachable via load balancer sets similar to AWS ELB/ALB/NLB.
You reach a service by the LB-set's DNS name.
If you don't want to use AWS or AWS-specific deployment tools, you could use Puppet Pipelines to do the same on Google Cloud or Azure. Puppet Pipelines was built by people who previously built some of those internal Amazon tools and offers similar functionality but cross-cloud.
And if you want even fewer moving parts, just go PaaS or serverless.
www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html