Hacker News Reader

> What's neat about it is, you generally solve one problem, and once it's solved, it stays solved. Your solution is in a yaml file somewhere or in a command line option that you've persisted to a script or whatever. And everything accumulates! If you manage to get a storage cluster installed, boom, all of a sudden you have a ton of options available to you.

Can you expand on how this is different from traditional networking? I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network, and it wasn't like I had to keep tinkering with it. Once I had network storage, I could use it from lots of different things, etc. But eventually you need software patches, updates, etc, and that was where the ongoing pain would sometimes crop up. Is that so different here? From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort.

There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc.

Maybe, if you're lucky, you had some set of really good and reliable Sys Admins that figured out a robust way to script and configure the setup process of your original on-premise data center and they captured that in very good, well-maintained documentation. If you're even luckier, maybe those same guys still work for you.

If not, well, Kubernetes provides an open standard that a lot of different people know how to use and the documentation for how everything is setup is directly in your source control. There's nothing exactly unique about this, but it is an actual open standard. It hasn't been owned by Google for years. It's owned by a non-profit. It's open source. If your deployments work on a cluster provided by one Kubernetes engine implementation, it'll work on any of them. You can roll your own, use GKE, EKS, whatever Microsoft offers, and it'll work in exactly the same way wherever you go.

For one person, this means nothing. This guy is just doing it for fun. But for organizations that used to lose millions of dollars a year and sometimes their entire market position thanks to vendor lock in or had to experience months of downtime for a data center migration, they may find something like this useful. Or maybe you just don't want to rely too heavily on Dennis Nedry and want to be able to bring in anyone with a couple years of experience using an open source, open standard toolset who can come in and be reasonably expected to understand how your system works pretty quickly without needing ten years of tribal knowledge.

> There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc.

It sounds like you're making the argument that deploying k8s for a startup is an extreme case of premature optimization.

Nah, it's par for course for startups for the following reasons...

You can get tons of credits for your startup, typically hundreds of thousands from Microsoft, Amazon, etc. -- Eventually you run out of credits, so you switch providers. I did this 3 times at a startup and got three years of free infrastructure.

If you are selling enterprise software, then you use k8s so you can deploy at enterprise without having to integrate with all the wacky requirements.

You can build a saas that offers isolated service nodes on k8s infra pretty easy if you just give everyone a small vm with a k8s cluster on it.

If you use a scale to zero model, your infra costs are a lot cheaper. Simiarly the auto-scaling k8s capabilities are really nice and it's awesome to be able to easily build systems that can scale up massively. You never know when you r startup will get popular.

Depends on what you mean by "deploy".

If you mean assembling your own k8s infrastructure by spinning up machines, then you're right.

If you mean deploying your software on managed k8s infrastructure, then this an outstanding use of your startup's time. You will be able to find plenty of developers already familiar with the workflow, and you'll have a straightforward time growing and deploying your app on different providers. It cleanly side-steps many production pitfalls that burned our time 10 years ago.

It's very frustrating that on HN of all places there is still this constant belief that all startups are the same and are building basic web apps.

Many types of startups will benefit from K8s right from day one.

My first, original production kubernetes cluster existed precisely because it lowered our costs - ridiculously so.

Because all that above? It comes with a system that will actively help you binpack your workloads as much as possible into the compute you give it.

Your usual "throw separate instances"? Can get expensive quick, especially if you're not aggressively modifying instance sizes. As for "lol just use Heroku"... I call that kind of company "bankrupt", but that's probably because the difference between cost of infrastructure and cost of engineer time is wildly different in my area to SV.

Maybe. I guess you could argue that before you find product market fit and your survivability is at stake, its better to be as ruthless as possible and cut down on any infra costs.

All startups are not the same though. If the startup has obtained any kind of reasonable Series A, it totally makes sense to invest in kubernetes, and allow your engineering team to essentially pull in literally any kind of dependency, test it quickly, and find out if it works or not. It acts as a catalyst that would let you churn out new products really quickly, and as such its an invaluable tool to allow your startup to move quickly.

If you work in enterprise environments and are able to use kubernetes, you are set to really shock your organization with how quickly you can move. I've seen this same situation play out in a few orgs and its really amusing how stupidly productive it allows engineers who learn it to be, and how quickly they get things done and get more responsibilities, promotions etc.

Startups can use operators which are like extra headcount.

> There isn't any gain for home networking. The gain is for enterprise applications that can now migrate a software-defined network to a completely different infrastructure provider without having to change the way they do monitoring, log collection, storage provisioning, DNS migrates with it for free, etc.

This is an unbelievable amount of gate-keeping hogwash. I don't know who this person is that they think they can arbitrate what is a good usage & what cases this is too-powerful too-interesting too-useful to bother using it in.

There is so so so much fear & doubt & scare in this post. Screw this gate-keeping crap.

> Maybe, if you're lucky, you had some set of really good and reliable Sys Admins that figured out a robust way to script and configure the setup process of your original on-premise data center and they captured that in very good, well-maintained documentation. If you're even luckier, maybe those same guys still work for you.

"Only us good right & virtuous & amazing engineers can handle this! This is too pure, too amazing for mortals! They're wasting their time! They'll get the configuration wrong! They're bad people. Only professionals are qualified to play with Kubernetes!"

UGH ENOUGH. Stop this terrible attitude. This is so down-talk-y.

Please don't assume, please don't dictate your limited terms to the world. Let the world try. Let us not be cowed, & afraid, to use good tech, by these scare words.

As it turns out, it's just not that hard. It's a better environment, a better world. There are lots of home users using Kubernetes. It doesn't take a colossal investment. It's fairly secure out of the box, at least if you're not trying to run a multi-tenant home. It just works. This blog post shows that! It's really simple.

See? Look. Lots of projects: https://github.com/k8s-at-home/awesome-home-kubernetes . Lots. Good people, just trying. Not taking the poison words to be afraid, that this is too hard.

"For one person, this means nothing. This guy is just doing it for fun."

What a BAD ATTITUDE. Snarking & being mean, to people out there, trying to find better ways to do things, to create shared, meaningful value. With good, autonomic systems. With reasonably competent free-to-everyone utility scale / cloud computing. Don't accept such words as these. Do not be afraid to involve yourselves. Do not be gate-kept like this. Run Kubernetes. Run good systems. Stop being sold on second, third rung systems. Believe in yourselves. Don't exclude yourself, be afraid. You are not saving yourselves any hardship by choosing lesser technology. You can run K3S in <20 minutes, and you can start loading amazing manifests & Charts seconds after. This blog write up shows that. IT MAKES PEOPLE AFRAID to think how democratic technology could be. Please allow yourself a moment of un-doubt where you consider, maybe, this has amazing value for the home, that it's already possibly incredibly robust, please consider that applying some manifests might be super easy. Please consider that blog posts are the canonical way to share work, before Kubernetes, but now I can link you to a repo full of people sharing manifests & charts & works, that stand a decent chance of running on any cloud or at home. There's a lot of sophisticated under-the-hood boons to running Kubernetes too, but as for what the home-user gains: it's amazing. It's easy to understand, eventually, and shared.

Skills learned building one thing in Kubernetes parlay much better into doing other Kubernetes tasks, related or not so much. Kubernauts are growing & learning & exploring, in a far more communal, shared operational way than what loose daemon-monkey-wrenching spread-out weirdness came before. It's not hard, it's not weird, it's not for enterprise. It runs out of the box, nicely. It's for people who want a good means to think about & control a wide variety of digital things. People who want a reasonably consistent underpinning to their technical operations. The alternative, what we had been doing, is having a lot of different things that each had their own means to be thought of & controlled; dis-unified, chaotic, piece-meal, inconsistent. Come try a better more unified way of computing. Come try utility computing. It's for people, all people. Many of us believe it serves us all better.

>Can you expand on how this is different from traditional networking?

Not OP, but I can chime in here.

It's not that it's solved in this specific deployment, but it's solved in yaml files that you can re-deploy to any environment. Once I get my bundle of yaml files, I can just point it to any cluster (or namespace) et voila, it'll be deployed. It's "infrastructure-as-code"[0]. To be clear, Kubernetes doesn't have a monopoly on this approach, but it certainly follows it.

Another idea is treating your servers like cattle, not pet. I used to have Linux VMs in my house named after Greek myths – these are pets. I named and loved each of 'em. And yes, after being setup, they worked. In Kubernetes, they're cattle – and you kill your cattle. You don't name and grow to love and nurture a specific VM. You can wipe your deployments and reprovsion a new one in a blink.

>I used to run a FreeBSD router with a ZFS storage setup and various other stuff on my home network

>From some coworkers who are closer to the cluster, keeping up with K8s version changes doesn't seem like a small effort.

You're not wrong here though. Sometimes there are maintenance tasks you have to do on the whole cluster (make sure there's enough storage, everything is updated, certs are good, etc). But this is orthogonal to having your infrastructure-as-code in yaml files. The yaml files assume a healthy cluster/namespace, but besides that any cluster should be fungible. This is treating your servers like cattle, instead of pets.

[0] https://docs.microsoft.com/en-us/azure/devops/learn/what-is-...

> Another idea is treating your servers like cattle, not pet. I used to have Linux VMs in my house named after Greek myths – these are pets. I named and loved each of 'em. And yes, after being setup, they worked. In Kubernetes, they're cattle – and you kill your cattle. You don't name and grow to love and nurture a specific VM. You can wipe your deployments and reprovsion a new one in a blink.

Very poetic, I love it! These kind of internet conventional wisdoms are gold, I also like:

Batteries included but swappable

Free as in Beer vs. free as in speech

Are there other contenders?

From a sysadmin/config management side of things, and since I "dabble" with networking, I've always been rather jealous of the network appliances. All the serious network devices had a single configuration file. Often the lines in the config file were equivalent to shell commands. For any network device, you could put the list of all the commands in notepad, as a script. But, if you exported the config file, it would look like a nicer version of your "script".

With config management, be it puppet, chef, salt, ansible, you start to approach that. Except these only manage the resources you define. If your config management manages apache resources, and someone wandered in and installed nginx that's out of scope. If you've got a list of machine-local usernames that you want to exist, you might not thinking about ensuring that other usernames are absent.

With kubernetes, it gets a bit closer. Let me preface that standing and running your own kubernetes cluster is a separate level of complexity, but once you have a k8s cluster, you can dump out a list of all resources (namespaces, pods, volumeclaims, ingress controllers, services...), modify that resulting file and apply it. Most people work in chunks, having a yaml file for each set of associated resources, or template those into a helm chart. With that, you can put all associated resources into a release, which you can delete and recreate as you desire.

There's still lots of gotchas, especially around persistence. It's not uncommon to find an orphaned volume claim that prevents your release upgrade from continuing until you dig in with 3 coworkers watching you as you try to remember if it's safe to delete that or not.

What's different is that the applications and the infrastructure are all managed and stored in the same location. There's a lot less you have to keep track of. Sure, you can get it all done and have it stable the traditional way, but k8s makes it fun!

Now smash that machine to bits with a hammer. How long do you need to get the exact same system back up?

In my case, it's 15 minutes.

    1. Get a new server
    2. Connect ethernet and IPMI ports
    3. Add node definition to XCAT
    4. Set boot to network, arm node for installation, power-on
    5. Profit.
    6. Get a coffee.

It's 15 minutes for [1-150] servers. Slightly longer for 150+. Because, network.

Any further setup is done via Salt, if necessary. With a single state file.

Fun fact: I provision K8S nodes that way too.

Just use https://github.com/fluxcd/flux — „The GitOps Kubernetes operator”

Let's assume I have backups for both my kube configs and my standalone machine... and spare parts for the cluster(s). I'm trying to understand better the maintenance story, because my impression is that kube itself has spec changes, and I have to stay on top of all the dockerfiles I'm using to keep up with patches there?

I use Kubernetes for work. We utilize GKE in google cloud. There have been a few times things have required some editing of our YAML files, such as when we had to change the first line of our yaml from:

    apiVersion: apps/Betav1
    kind: Deployment

to:

    apiVersion: apps/v1
    kind: Deployment

Because in K8s 1.16, they stopped the backwards compatibility for Beta on deployments. But that was quite some time they left it in for backwards compatibility, and it was a simple sed script to change it. We have never changed our Dockerfiles except to change the FROM line when we update a base image. They often introduce new features, but try to keep things as stable as possible for existing setups.

In our case, we store our YAML in git, and can deploy a new cluster with all our microservices in about 10 min (most of that is delays in the google global load-balancer setting up a TLS certificate for us, the machines are up and running after just a few minutes).

There can be a bit more scripting needed to do updates of the nodes to newer builds with no downtime, if your pods aren't totally stateless, but compared to what it used to be at an old job, with C apps running on Linux, behind load-balancers that had to be removed, upgraded, and added, a rolling update is like black magic.

> if your pods aren't totally stateless

This is exactly the bit I care about: what do I do with "stateful" systems (my db storage, mail server data,...)?

If I just mount those from external volumes, I lose a lot of idempotency and I now have to worry about whether I am trying to attach that volume to PG 9.3.1 or 9.3.2 or 12.0 (some are fine, others might cause data corruption).

I know idempotent deploys are all the rage, but all those deployed apps are there to serve some data which is as stateful as it can be.

I'd like a system where those external volumes are automatic snapshots on LVM (or ZFS/btrfs) when attached, but that introduces a whole another level of what-now if you need to go back to an older, now slightly stale data set.

How does k8s solve this problem for me?

This question always gets asked. I'm hoping for an answer one day too!

> and can deploy a new cluster with all our microservices in about 10 min I know nothing about k8s yet, but try to solve the same issue with docker swarm and one thing I'm unable to get my head around is this: as soon as yaml files are stored in apps' repos and deploy is being initiated by commiting to them, how do you let the repos know about the new deployment? Or is there a tool that is present in k8s but is missing in Docker Swarm that knows about scattered yaml files' locations (git repos) and pulls them on the new deployment?

People generally keep their kubernetes spec files in Git. Be it either the kubernetes .yaml files they apply directly, or helm charts.

These files can idempotently be re-applied to a cluster to restore, or upgrade, the running applications.

When you lose your cluster and you restore to a new cluster, you will of course initially deploy the same version of kubernetes as you were using previously.

Then once you choose to upgrade to a new version of kubernetes, it can happen that some of the APIs you were using have now been deprecated/removed. This is a very slow process, so you had plenty of time to upgrade ahead of time before being forced to.

But let's say you ignore all that, and have chosen to upgrade to a new kubernetes major version and are now forced to upgrade all your yamls. This happens rarely, and recently only because a few BETA APIs have become STABLE and people are now expected to use the STABLE versions. So you go ahead and make those few changes, re-run your deploy and you're done.

The people claiming it's easy are, well, lying. Our deployments fail all the time for things like Google moved their Helm images to a new repo.

Kubernetes is still early tech and thus has more moving parts than a more well established stack. Keeping up to date is crucial and no small feat.

At least the first 10 minutes is waiting for my corporate laptop to boot up ;)

The big difference is that kubernetes YAML is in a change control system, and specifies what you want rather than how you get it.

I think one big big aspect, in “use the right tool for the job”, that is often overlooked, is how good you already are wielding said tool.

I use Terraform to manage the few personal cloud resources I have. It is overkill, but I already learned it on the job so it ended up being quicker than setting stuff up by hand. If not, I wouldn’t have bothered learning it. At least in this context.

Same when building an MVP, I’m not going to pick some shiny tools I have never used before, but tools I know I can get the job done with.

Sure, I do like learning new things and tinker around with them, but it is not always the right time to do so, and the list of things I’d like to get good at, engineering related or not already is long a enough for a lifetime so one needs to prioritize.

Not only how well do you use the tool, but how useful is it to learn the tool? Once you can use it well, what else can you do with it?

I recently saw a video via twitter that mocked using kubernetes for your blog by showing someone making a sandwich with woodworking tools.

Overkill, yeah. But if you stretch the metaphor to the way it translates in k8s, if you get good at making a sandwich in 5 minutes with the tools, you're fairly close to also stamping out dressers and end tables in 5 minutes. At the very least you can make them, which you can't say of a bread knife. And once you do it a few times you can do it with incredible speed.

>Once the cluster is stable and you are comfortable bringing it up and tearing it down >K8s wants a docker container registry, you really don't want to run that on your cluster in the beginning, but once your cluster's secure, why not!

I'm betraying my ignorance here, but how does this work? If you're running the registry in your cluster, and you tear down your cluster (and the registry with it), how do you rebuild the cluster without being able to pull images?

Usually the important images are hosted somewhere else (ex. you're probably not hosting the controller-manager which is normally pulled from k8s.gcr.io), but for the ones that were hosted in your cluster, the pods will basically fail (and start to back off) until your local registry comes up and you'll see pull related errors. The rest of the pods that don't reference images from your cluster-local registry will start up just fine.

Assuming that you rebuild your cluster and restore your registry from backup (or simply reconnect it to some object store that it was connected to before, like S3), your registry pod (in a Deployment/Stateful/DaemonSet) will come up, and then the cluster will be able to connect to it, and the pods that were failing to start will start succeeding and you're back where you started.

You can always run it in the cloud and at least theoretically be able to switch providers and take your cluster with you.

Expensive. I've got all these machines sitting here at home, why not use those?

Hard to beat the uptime of a cloud provider with on prem, if you literally mean “at home” even harder, my Comcast goes out a few times a day. Even in offices you don’t benefit from the economies of scale and redundancies in the power and network systems that cloud providers have.

If you're really trying to serve the outside world, then you want colocation. Cloud providers are way more expensive.

Colocation isn’t a binary on-or-off thing. As a crappy example, Kubernetes pods maintain the invariant that all containers within a pod exist on the same Kubernetes node. So you can avoid the most egregious waste of resources/time that comes from having no collocation.

To your point, when talking to your db or [insert shared infra here], then you may want to thinking about real collocation. But it very much depends on your application’s needs - and remember that cloud providers can offer guarantees about which AZ or region you’re in.

I did mean real colo. If I ever run a serious public-facing website, I have a colo facility picked out, just drive down there with my custom server. Only a few hundred bucks a month to serve data to a CDN, along with whatever services I want to directly expose.

With the magic of VPN, I can put my colo box on the same k8s cluster as the rest of my home stuff, in case I want to live dangerously.

Depends on what you’re doing and at what scale. Especially for smaller orgs it’s really hard to duplicate the level of service the big boys can provide yourself.

I just thought of another reason. Rolling your own cluster forces you to understand what Kubernetes is at a basic level. If you never get that experience, and just roll with cloud services, you aren't really getting that and will tend to get mystified by errors which, you'd have a much better idea of where they're at if you've rolled your own.

Kubernetes is very complicated, but the complexity can be managed if you approach it from the bottom up. Using a cloud service robs you of that ability.

> Kubernetes is very complicated, but the complexity can be managed if you approach it from the bottom up.

s/Kubernetes/Computers

The pitch for Kubernetes is kind of supposed to be “infrastructure as a service,” isn’t it?

No, that's the pitch that Amazon / Google makes. Kubernetes has a much more complicated value proposition that you have to grasp the hiring dynamics of Google to really understand. Google hires really smart people but smarts isn't necessarily skills. So it builds tech to leverage that workforce.

Other companies hire skilled people that aren't necessarily super-smart, because super-smart people can make a lot more money at Google. So other companies use Google / Amazon, they don't really care what underlying techs it uses, so long as they know they'll be able to hire people with the skills.

If you're rolling your own Kubernetes, you're putting your faith in your own smarts, because you're largely on your own. K8s rewards a certain kind of developer, and badly punishes someone thinking running their own k8s is going to be as seamless a process as consuming GCP resources.

Yeah but that’s my point, not everyone needs to actually set it up. The intent is for most to just be users and concern themselves with editing yml files to bring up their services.

It's certainly not for everybody.

> my Comcast goes out a few times a day

I’m really sorry to read this. Given that you’re a Comcast customer it’s probably because you have no other viable high speed choices, but please know that regular outages at home are not normal in other parts of the world.

Lol, this is great to know but I live in downtown Dallas within spitting distance of AT&T world headquarters and literally the only "high speed" provider available to me is Spectrum (I can't explain why AT&T itself can't provide service to someone in walking distance of their own corporate headquarters compound - do they even provide service to themselves?). It goes out on me all the damn time. Heck, I lost service a few weeks back because someone moving into the house next door accidentally gave the wrong address when signing up for a new service account and Spectrum canceled my account, assuming I'd moved out since apparently someone else was moving into my house.

I have never been madder at a service provider of any service. But there is nothing I can do, at least not for six more months until Starlink finally goes online in my area and sends me a dish.

This in what? The heart of the 4th largest metro area in the richest nation on the planet? Maybe this is legitimately not a problem in most of the developed world, but U.S. residential communications infrastructure is amazingly pathetic.

Mine has never gone out, but when I had AT&T DSL it dropped whenever it rained and it was impossible to fix. (the other mystery being why we only had 1.5MBit DSL in the middle of Atlanta.)

What’s he meant to do with this information?

Lament alongside the rest of us Comcast customers paying our "protection money".

> like the effing Borg and it's great!

Don't know if that was an intended nod but I liked it :)

https://storage.googleapis.com/pub-tools-public-publication-...

I really like K3s. Comes with an ingress controller out of the box which makes it easier to get up and running. Add Let's Encrypt, and its pretty trivial to stand up new services on their own subdomains.

Also worth noting that it's a lot less resource hungry than the official Kubernetes distro. Kubernetes documentation recommends 2 CPUs and 2GB of RAM per node just to have enough left over to actually run any workloads.

k3s is specifically designed for resource constrained environments (e.g. Edge, IoT, CI/CD) and can get away with as little as 0.05 CPU and 256M RAM (on worker nodes).

I don't recommend this as a way to save money on infrastructure but it does open the door to further use cases.

I'm struggling to easily configure new subdomains that easily using GKE. The managed certificates system in GCloud is straight user-friendly hostile. I'll have a look at that.