Hacker News Reader

Comment by honkycat | Hacker News Reader

People love to pooh-pooh "complicated" things like unit tests, type systems, Kubernetes, GraphQL, etc. Things that are solving a specific problem for LARGE SCALE ENTERPRISE users.

I will quote myself here: A problem does not cease to exist just because you decided to ignore it.

Without Kubernetes, you still need to:

- Install software onto your machines

- Start services

- Configure your virtual machines to listen on specific ports

- have a load balancer directing traffic to and watching the health of those ports

- a system to re-start processes when they exit

- something to take the logs of your systems and ship them to a centralized place so you can analyze them.

- A place to store secrets and provide those secrets to your services.

- A system to replace outdated services with newer versions ( for either security updates, or feature updates ).

- A system to direct traffic to allow your services to communicate with one another. ( Service discovery )

- A way to add additional instances to a running service and tell the load balancer about them

- A way to remove instances when they are no longer needed due to decreased load.

So sure, you don't need Kubernetes at an enterprise organization! Just write all of that yourself! Great use of your time, instead of concentrating on writing features that will make your organization more money.

View on HN

(author here) The post isn't claiming that you should be doing something differently right now.

It's claiming that there's something better that isn't discovered. Probably 10 years in the future.

I will be really surprised if anyone really thinks that Kubernetes and even AWS is going to be the state of the art in 2031.

(Good recent blog post and line of research I like about compositional cloud programming, from a totally different angle: https://medium.com/riselab/the-state-of-the-serverless-art-7...)

FWIW I worked with Borg for 8 years on many applications (and at Google for over a decade), so this isn't coming from nowhere. The author of the post I quoted worked with it even more: https://news.ycombinator.com/item?id=25243159

I was never an SRE, but I have written and deployed code to every data center at Google, as well as helping dozens of people like data scientists and machine learning researchers use it, etc. It's hard to use.

I gave this post a modest title since I'm not doing anything about this right now, but I'm glad @genericlemon24 gave it some more visibility :)

This article really resonated with me. We are starting to run into container orchestration problems but I really don’t like what I read about K8s. Apart from anything else, it seems designed for much bigger problems than mine, and require the kind of huge mental effort to understand which, ironically, will make it harder for my business to grow.

I’m curious if you’ve taken a look at Nomad and the other HashiCorp tools? They appear focussed and compositional, as you say, and this is why we are probably going to adopt them instead of K8s - they seem to be in a strong position to replace the core of K8s with something simpler.

I use Nomad a lot in my company and I really like it.

Our team tried to migrate to AWS ECS a few times and found it much harder to abstract stuff from devs / create self-service patterns.

That said it's not a walk in the park. You will need to scratch your head a little bit to setup consul + nomad + vault + a load balancer correctly.

Thanks. We're going to start small with just nomad, then vault, and as our needs grow we will probably adopt consul (we already use terraform so hopefully not a huge stretch) and maybe boundary.

This is thing I like about the HashiCorp tools. You don't have to eat the whole cake in a single sitting.

>You will need to scratch your head a little bit to setup consul + nomad + vault + a load balancer correctly.

I've been wondering, would it make sense to try to package all that into a single, hopefully simple and easily configurable, Linux image? And if it might be, why hasn't anyone done that yet?

There are some good ansible playbooks on GitHub for nomad, consul and vault. I personally don't use vault because it's overkill for the proeuct in working on at the moment.

To avoid the pain of managing a CA and passing out certificates for TLS between services, I use a wireguard mesh and bind nomad, consul and vault to these wg interfaces. This includes all the chatter of these components, as well as the services I deploy with nomad. It's configured such that any job can join the "private" wireguard network or "public" internet gateway.

It takes a few days to set up, but it's very easy to manage.

Do you have somewhere to point me to set things up in this configuration?

I’m a freelancer that hosts client stuff and I need something between “SSH into server” and “kubernetes.”

No, I never did buy the docker hype. Seem to be doing okay.

Have you looked into Fly.io or AWS Fargate?

I've only looked at the HashiCorp tools, not really used them. My understanding is they originated in a VM-based world (?), and I've worked almost exclusively with containers. I'm sure that has changed over time.

I will say that I looked at HCL and it looks very nice:

https://github.com/hashicorp/hcl

But somehow it's not as popular as a mess of YAML and Go Templates? That genuinely leaves me scratching my head. I guess it's because people pick platforms and not languages? (BTW, in 2009 I designed and implemented the template language that Go templates are based on, and I find their common application pretty bizarre, e.g. in some Helm charts I looked at from this thread)

Oil is growing a config dialect that looks a lot like HCL (although it's convergent evolution; I've never used it.) I think there is a lot of room for mixing declarative and imperative; as far as I can see HCL is mostly declarative (defining data structures).

Anyway I'd be interested in reading about HashiCorp stuff but for some reason in my neck of the woods I don't hear too much about it. Maybe that's because they're paid services and the open source Kubernetes seems attractive by comparison? Or is it more of a VM vs. container thing?

All of the Hashicorp products are primarily open source products. While there are enterprise features and cloud-hosted versions of some of them, FOSS is the foundation of the company.

10 years ago there wasn't a Docker (released in 2013), and AWS was a tiny side player with most established businesses operating their own data centers.

I think it's safe to say that if the next 10 years are anywhere near as disruptive as the last 10 we will surely be doing a lot of things very differently.

Things have already changed since the first release of Kubernetes. Specifically hosted Kubernetes, aka GKE/EKS/AKS, is a marked step forwards from running Kubernetes yourself, that I think doesn't get enough recognition. We'll see what the future holds, but my prediction is that the future holds more layers of indirection, and the future of running web services is on AWS Lambda/Azure Functions/Google Cloud Functions, and other fully-managed PaaS, like Heroku, with more vendor agnosticism. Running Kubernetes, in addition to the technical benefits, also enables a company to treat AWS/GCP/Azure as a commodity, and can credibly threaten to move clouds when the contract is up for renewal.

Back in 2003 we had Solaris Zones (now called Solaris containers). Same concept as Docker, but we didn't knew exactly why it was such a good idea, and the hardware was expensive.

What made Docker spark was being abke to use commodity hardware and ush to production with the same exact environment and behavior.

You could have done the same with Solaris, if you developed on a Sun Ultra5 workstation and published the application in a zone in the server. But 2003 was a different world and not everyone had a Spark box to develop nearby.

IMO hashistack nomad provides a better development experience. The complexity is gradual and it doesn't try to do "everything", it can stay focused on workload orchestration(whether its a container, vm, or even a process) and delegates coordination out to specific services better suited for it(consul for service discovery, vault for secrets etc...)

Totally agree. Simple things stick, complicated things die.

If you’re explaining, your losing.

You're mixing together useful complexity with useless complexity.

Plus at the very least, I'd be very careful about putting type systems into the same basket as Kubernetes. One is a basic language feature used offline and before deploying. The other is a highly complex interwoven web of tools that might take your systems offline if used incorrectly.

Without Kubernetes, you need Debian and it's Apache and MySQL packages. It's called a LAMP stack and for many production deployments, that's good enough. Because without all that "cloud magic", a $50 per month sever running a bare metal OS is beyond overpowered for most web apps, so you can skip all the scaling exercises. And with a redundant PSU and a redundant network port, 99.99% uptime is achievable. A feat so difficult, I'd like to mention, that Amazon Web Services or Heruko rarely manage to...

Complexity has high costs. Just because you don't see Kubernetes' complexity now, doesn't mean you won't pay for it through reduced performance, increased bug surface, increased downtime, or additional configuration nightmares.

This argument is often made and is ridiculous.

No one should or is using Kubernetes to run a simple LAMP stack.

But if you have dozens of containers and want them to be manager in a consistent, secure, observable and maintainable way then Kubernetes is going to be a better solution than anything you build yourself.

> No one should or is using Kubernetes to run a simple LAMP stack.

Yes they are. Some developer got all excited about the capabilities of k8s, and had an initial larger scope for a project, so they set it up with GKE or EKS, and it managed to provide just enough business value to barrow in like a tick that won't be going away for years.

Developers get all excited for new shiny tools and chuck it into production all the time, particularly at smaller orgs.

> initial larger scope for a project

So they weren't trying to run a simple LAMP stack then.

Trying and doing can be very different things.

I have seen smaller RoR, Django or lampstack apps being deployed on kubernetes exactly for reasons you mentioned. It is often pitch as a silver bullet for the future.

When Boss says "my idea is gonna be HUGE, so make this go fast", you can either spend 4 hours optimize some DB queries, or you can spend 40+ hours in a broadly scoped "conversion" project and have a new thing to add to your resume, and then spend 4 hours optimizing some DB queries...

One of the worst, bazooka in knife fight moment was when I interviewed for a company that was a medium scale direct to consumer streaming service that was built some custom resource to create one pod per user, their worst case scenario was where they would have around 1000 concurrent users.

I worked in a Python web-scraping team doing 3 million requests per day. Their solution to manage concurrency was also one pod per crawl, each with its own DB connection. It really struck me as crazy that everyone thought this was a good idea.

Strikes me as someone buying a massive SUV, to only use it to move one bag of groceries from the store every week. Some day in the future they might suddenly have a family of 6 to feed or have to haul a boat somewhere, so better get the big, overkill car.

This describes ~70% of adults in the southeastern US.

I have personally experienced the other end of that problem, and it applies just as well to the k8s discussion. When all the kids are gone it's hard to admit you're not a soccer mom anymore and adjust to grandma mode. It can be a hard thing to trade in the Suburban for the Lincoln. Oh wait, Lincoln's aren't the cool thing so much anymore. That whole world changed while I was driving Suburbans.

It tends to happen with startups I think because developers know the startup is likely not going to be around for long, or they will move on in a couple years anyway. So why not shoehorn whatever tech you want and have that skill in your resume? Of course now the startup will need to hire for that tech when you leave and the cycle continues...

Heh, as someone who came very close to doing this (i.e. using k8s for a LAMP-stack type app at a startup), it's not just "shiny object syndrome" driving people to do this. Here's what our progression looked like:

1. Start with basic LAMP app in git that's manually deployed to an EC2 instance 2. Add in CI / CD + CodeDeploy 3. Create a staging environment 4. Dockerize local environment to keep dev environments in sync and onboard easier (really, this part's a gamechanger for a small company) 5. Ok so now we have Docker for local dev environments but stage and prod are managed separately. Can we just run our Docker containers in stage / production?

When I researched step 5, the options were basically k8s or Docker swarm but Docker swarm didn't seem battle tested (for prod). k8s was clearly a nightmare for a small team to maintain so we started looking into GKE / EKS -- but EKS was still in beta. Thus we punted. We've actually started using ECS for a newer project and I'd likely go that route for step 5 instead.

I agree that you probably shouldn’t but if you think no one “is”, I’d point to my last job, an enterprise that went to k8s for a single-serving php service that reads PDFs.

I recently asked a friend who still works there if anything else has been pushed to k8s since I left (6 months ago). The answer: no.

Sounds familiar.

I love the way that the Kubernetes debate always immediately devolves into Kubernetes vs. DIY where Kubernetes is the obviously correct answer.

Two groups of people shouting past each other.

My argument is that much of web development now often involves insane amount of tooling infrastructure and complexity all for the sake of delivering on cloud, scalable, virtualized, dockerized and otherwise over-processed equivalent of "hello world".

And 99% (rhetoric number of course) of needed solutions would be served by a single real server (rent one on Hetzner or wherever) with 0 need to ever upgrade.

At a previous company we had Series C financing and enough people to take over two floors of an office building... and our entire SaaS offering for hundreds of white-label business customers ran fine on a couple of big load-balanced EC2 VMs and one big Postgres database with some read replicas.

Yep, HA clusters/load balancing has been around for a long time and will get you pretty far even in some pretty large environments. Hell my companies main LOB app is running off a single Microsoft SQL serve(I know, I know, we have a lot of tech debt we're working through atm and it's on the list to set up HA). The longest downtime we've had was when we had to take it down to migrate it to our new hardware cluster and that was because the data transfer took 20 hours over a 1 GB port.

Another interesting angle about the single real server is the reliability.. it'll eventually break from some physical cause, that happens, it hurts your uptime.

But sometimes with the big microservice/kubernetes solution, the configuration space is so big that miscommunications/mistakes could potentially take you down for more hours/year than the downsides of being single-hosted would. So now you invested all of that effort and for what?

One company I worked for wrote a distributed scheduled jobs system that was immune to single-machine failures, it went down like 4 times in a year, messing up my team each time. I was like "guys, if we just provisioned one machine with cron and no failover, it would have better reliability".

That doesn't look good on a resume though and definitely won't get you hired at a FAANG company.

I design and implement products. That is what my resume says along with the list of said products and some references. This is usually enough to land me a contract. Been on my own 20 years already and make money from some products of my own or creating those for clients. The last thing I need is a job at FAANG.

Besides if it warm the cockles of your heart so much you can always take any of my products and shove it in the container and run it under k8.

Alas, a lot of people are. One of the reasons there's such a backlash against k8s - other than contrarianism, which is always with us - is that there are quite a few people who have their job and hobby confused, and inflicted k8s (worse yet, raw k8s) on their colleagues not because of a carefully thought out assessment of its value, but because it is Cool and they would like to have it on their CV.

> No one [...] is using Kubernetes to run a simple LAMP stack.

Au contraire! This is very common. Probably some combo of resume-driven devops and "new shiny" excitement.

or if you have a small team that must manage hundreds of these lamps…

i started by simple wrapper around docker 8 or so years ago. over the years we’ve moved to k8s b/c it provided essentially the same api we had home grown. this reduced the LoC and moved the complexity of something like dynamic reverse proxy via nginx downstream into kubernetes ingress abstraction backed by nginx-ingress-controller.

That's not resume driven development though, that's picking the best tool for the job when you've outgrown the current solution.

Absolutely agree with you. I have seen the debate between accidental and necessary complexities very often. It actually depends upon stage of the organisation. In my opinion many devs in startups and smaller orgs try to accomodate the future expectations around product and create accidental complexities Accidental complexity becomes necessary complexity when an organisation scales out.

Another case of premature optimization, really

You really only need a single interview to find out companies care a lot about premature optimization.

They’re always asking if you’ve worked with the latest tech stacks, never if you can set up a really well optimized nginx server.

I see this as Resume Driven Development. Yak shaving is fun and at some point complex DevOps becomes necessary, but most CRUD apps benefit from a simple approach that allows high feature velocity. It's a balancing act between productivity and technical debt.

I worked for one startup lead by a Java Architecture Astronaut, and working with the byzantine patterns and build systems made adding even simple features a morale-draining slog. It killed the product.

> You're mixing together useful complexity with useless complexity.

Or, to channel Fred Brooks, essential and inessential complexity.

> You're mixing together useful complexity with useless complexity.

> Complexity has high costs

Complexity management is the central theme of building any large, valuable system. We would probably find that the more complex (and correct) a system, the more valuable it becomes on a relative basis to other competing solutions. The US tax code is a pretty damn good example of complexity intentionally taken to the extreme (for purposes of total market capture). We shouldn't be surprised to find other technology vendors framing problems & marketing their wares under similar pretenses.

The best way to deal with complexity is to eliminate it or the conditions under which it must exist. For example, we made the engineering & product choice that says we do not ever intend to scale an instance of our application beyond the capabilities of a single server. Consider the implications of this constraint when reviewing how many engineers we actually need to hire, or if Kubernetes even makes sense.

I think one of the biggest failings in software development is a lack of respect for the nature and impact of complexity. If we are serious about reducing or eliminating modes of complexity, we have to be willing to dig really deep and consider dramatic changes to the ways in which we architect these systems.

I know its been posted to death on HN over the last ~48 hours, but Out of the Tar Pit is the best survey of complexity that I have seen in my career so far:

http://curtclifton.net/papers/MoseleyMarks06a.pdf

> useless complexity

Which items fall in this category?

Most of the time, K8s

Useless is a strong word in this context. Which parts or features of k8s are useless complexities? It's not like some random junior dev pulled a bunch of features out of a hat and implemented them. There was a ridiculous amount of thought put into its features and I can't think of a single complexity that is useless or even one that is useful but could be done in a more elegant /less complex way.

Yes, it is complex, but there are lots of use cases where it is the most elegant and least complex solution. Yes, it definitely does not make sense to use it for a lamp stack deployed to one server, but there are use cases where it's a huge improvement (e.g. Spark on Hadoop is extremely complex and clunky when compared to spark on kubernetes).

Like you said, if you're good with a LAMP stack and one server, then K8S is probably useless complexity.

The issue is that most people nowadays have never worked with a bare metal Lamp server, so they grossly underestimate how large their company can grow before needing any distributed HA solution. I'd wager 90% of startups go bankrupt or are acquired before outgrowing a single Lamp server.

> And with a redundant PSU and a redundant network port, 99.99% uptime is achievable.

It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box. A couple of problems with that:

1. What happens when you have to reboot to apply a kernel update?

2. The geographic location of that single box is itself a gap in redundancy. This is one thing I like about AWS and the other hyperscalers, with their regions that each have multiple data centers connected by a private network, with load balancers and other things spanning the region.

>"It's really tempting to believe that with the right hardware, we can put everything on one powerful and inexpensive box"

This is what I have. 2 boxes only. One on my own premises in Canada and another on Hetzner in Europe. Each one can go down, no sweat. Also I do not strive for "live uninterrupted updates". All my applications and processes allow for short disruptions not to interfere with the main course.

Also I write native servers in C++ for backend and they process thousands of requests per sec without breaking much sweat. Way more than my business would ever need.

Same situations with my clients as well with the difference that for legal reason they rent computers or whatever goes as such from Amazon or Azure.

Correct me, but I'd dare say admins have been rebooting their machines and running services without geographic redundancy for decades. Uptime was still very high.

This is a question that I have spent a ridiculous amount of time pondering.

My conclusions so far are this:

Single node application systems are by far the most reliable and manageable from a business logic standpoint. At no point does spreading a problem across more than 1 computer make that problem easier to solve.

If you are concerned about latency, you need to get really abstract with your problem and ask what is even possible in information theoretic terms. If you are truly constrained to 1 serialized, synchronous context (i.e. a competitive counterstrike match or a stock exchange), there is little you can do to alleviate the root problem as your users get further from the server. You can certainly look at using some consensus protocol like multi-paxos, but then your transaction latency goes from microseconds (if you were clever) to milliseconds, representing orders of magnitude slowdown in the typical case.

The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system.

For the customers I work with, it is wayyy easier to build & sell a system that operates on a single box with sync replication + manual failover. Our customers are tolerant to production having a brief outage for a few minutes during the business day. Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.

> Single node application systems are by far the most reliable and manageable from a business logic standpoint.

Manageable? Maybe. Reliable? No. Most companies don't need it but if a pipe bursts above the server room and now all of your medical records for your hospital are unavailable you're going to have a bad time.

> At no point does spreading a problem across more than 1 computer make that problem easier to solve.

I don't think anyone claimed it was easier (yet). The main thing people strive for is:

1. Dynamically reshaping your compute needs to match any needs.

2. Surviving failures of nodes or crashes in your application.

3. Managing all of the BS that goes with obtaining this (logs, etc).

> Especially considering the fact that I have still never had to do this exercise in a production setting. The hardware we run this stuff on is so ridiculously stable.

I hope this isn't the case but this sounds like quite the death flag.

I’ve only seen one burst pipe bring down a computer (Pitney Bowes, 1995), but I’ve witnessed several S3 outages.

You should spend a week working on a infra / ops team then.

I worked with multiple ops/infra teams, and have seen standard server rooms and advanced server rooms. They’re not supposed to have pipes leaking on the racks, that’s ridiculous. I know many stable redhat servers that didn’t need to reboot for years. I also maintain websites on hetzner servers that haven’t needed to reboot in years, under pretty high load. On the other hand, I got a hostnoc dedicated server and that thing went offline three times a month. So yeah, you have to know what you’re doing regardless of whether you’re building a server room or a k8s cluster.

> I know many stable redhat servers that didn’t need to reboot for years.

That's not actually something to brag about, as it implies they weren't receiving important kernel updates, particularly security updates.

So, we do ~daily restarts on our customers' systems. We have a solid 12 hours per day to patch up a server before it needs to be in production again.

That said. We communicate directly with other systems on the same premises that probably haven't been rebooted in over a decade.

Building something that can run 10 years without downtime is not happenstance. It is very intentional and deliberate engineering.

https://en.wikipedia.org/wiki/Kpatch

Allows you to patch the kernel without reboot.

As mentioned above, they were, both RedHat and Ubuntu now support kernel updates without restart.

More modern data centers have liquid pumps available for low-cost cooling.

Completely aligned with my parent, day-to-day operations are often slightly more complex than making sure your server run your app properly, I'd argue that our services are increasingly dependent on others (in both directions, dependencies multiply and are more and more critical). That's also by interacting more with external entities that they bring more value.

> The best solution I can come up with is a synchronously-replicated append-only log store which is utilized in a primary/sync-witness/async-witness/... configuration. The first tier of resilience would be synchronous and provided by a set of witness nodes which must ack as a majority to progress primary. These nodes would ideally be within 1-2ms of the primary. The async witnesses could be in orbit and/or on mars. These are more about extreme geological disaster recovery. The witness nodes would also use a separate consensus protocol to decide when the primary needs to be taken down and replaced with a sync (or god forbid async) witness. They would be able to elect an emergency leader separate from the primary who would be authorized to stop the bad primary in the hypervisor, and edit any relevant DNS records to ensure traffic stops hitting the bad system.

This part was what I felt deserved a counter-point though. Consensus is indeed at the core of the issue once you want distributed fault tolerance. However, I think you'll quickly hit two things with your approach: 1-2ms of latency, I fear that it may come with highly correlated failures on the "first tier of resilience". Moreover, the "second tier" being much farther, keeping them in consensus will imply harsh trade-offs. If you use synchronous consensus protocols, you'll slow down drastically the "first tier" (assuming you want consistency), if you go for the asynchronous replication (not consensus, this matters...) then the second tier can't really intervene on leader election or failover without risking a partition on a false positive (and if you try to be conservative there your RPO will suffer).

If you're into these (fundamental) issues, I'd recommend Leslie Lamport's work (i.e. https://www.college-de-france.fr/site/en-martin-abadi/semina... or http://www.lamport.org/), the paper pointing a disappointing impossibility: https://dl.acm.org/doi/abs/10.1145/3149.214121 and its generalization (https://dl.acm.org/doi/abs/10.1145/167088.167119).

I appreciate the commentary. I feel most of this effectively boils down to "it depends". Clearly, there is no one-size-fits-all solution that can make everyone happy all the time, especially considering all the constraints & variables present across the entire vertical.

Where your datacenters are geographically located is usually a big first step in even starting these types of conversations. The nature of "maybe sync replication represents a liability or is feasible" might be a conversation about the geography of a region and statistical likelihood of certain disasters impacting multiple sites simultaneously.

Some customers cant ever afford to lose a single transaction no matter what, some just need it to be reasonably stable but incredibly fast (e.g. gaming vs banking).

Will definitely be spending some time reviewing Lamport's works again. Establishing the notion of stable time between all participants is a fascinating way to solve a lot of problems in distributed systems.

1. I reboot it and my users will need to wait. 99.99% uptime is achieved with 53 minutes of downtime every year, largely enough for the occasional kernel update here and there.

If needed, I can choose to apply the update in the middle of the night in the timezone of most of my visitors.

If it really is a sensitive app and I can't afford any uptime, I just add another inexpensive box, put a load balancer in front (a Cloudflare load balancer would work fine). And since I now have 2 servers, I need a way to manage them without having to manually log in to each of them each time. Enter Ansible. And that's it.

2. Now that I have two cheap boxes, nothing prevents me from having them in two separate data centers and two separate providers.

Point 2. is relevant, point 1. not really, unless one isn't willing to pay for the specific OS version that allow live kernel updates.

1. Planned maintenance in the middle of the night. Even Amazon and Heroku need to do it sometimes. At least we had a forced Postgres version update recently which was 30 minutes of downtime.

2. The latency within AWS is easily as high as US-EU backbone latency. Also, most web apps these days need 2000ms to load all the tracking and advertising crap, so 100ms location latency are negligible in comparison. Plus for most real companies, you'll have one website per country anyway. One server for .com and one server for .eu

Couldn’t agree more! The crazy, ridiculous Rube Goldberg machines that are being strung together from those AWS components to solve the most mundane problems are getting ridiculous.

"Write it all yourself"

- Install software onto your machines

Package managers, thousands of them.

- Start services

SysVinit, and if shell is too complicated for you, you can write totally not-complicated unit files for SystemD. For most services, they already exist.

- Configure your virtual machines to listen on specific ports

Chef, Puppet, Ansible, other configuration tools, literally hundreds of them etc.

- have a load balancer directing traffic to and watching the health of those ports

Any commercial load balancer.

- a system to re-start processes when they exit

Any good init system will do this.

- something to take the logs of your systems and ship them to a centralized place so you can analyze them.

Syslog has had this functionality for decades.

- A place to store secrets and provide those secrets to your services.

A problem that is unique to kubernetes and serverless. Remember the days of assuming that your box was secure without having to do 10123 layers of abstraction?

- A system to replace outdated services with newer versions ( for either security updates, or feature updates ).

Package managers.

- A system to direct traffic to allow your services to communicate with one another. ( Service discovery )

This is called an internal load balancer.

- A way to add additional instances to a running service and tell the load balancer about them

Most load balancers have built up processes for these.

- A way to remove instances when they are no longer needed due to decreased load.

maybe the only thing you may need to activelly configure, again in your load balancer.

None of this really needs to be written itself, and these assumptions come from a very specific type of application architecture, which, no matter how much people try to make it, is not a one-size-fits-all solution.

I'm personally glad that Kubernetes has saved me from needing to manage all of this. I'm much more productive as an applications engineer now that I don't have to stare at a mountain of bespoke Ansible/Chef scripts operating on a Rube Goldberg machine of managed services.

This x10. Each such setup is a unique snowflake of brittle Ansible/Bash scripts and unit files. Anything slightly different from the initial use case will break.

Not to mention operations. K8s give you for free things that are a pain to setup otherwise. Want to autoscale your VMs based on load? Trivial in most cloud managed k8s.

Instead, you can now admin a Rube Goldberg machine of Helm charts, which run a pile Docker containers which are each their own microcosm of outdated packages and security vulnerabilities.

> Rube Goldberg machine of Helm charts

I love k8s but I do want to say that I hate the 'standard' way that people write general purpose Helm charts. They all try to be super configurable and template everything, but most make assumptions that undermine that idea, and I end up having to dig through them to make changes anyway.

I have found much more success by writing my own helm charts for everything I deploy, and putting in exactly the amount of templating that makes sense for me. Much more simple that way. Doing things this way has avoided a Rube Goldberg scenario.

your argument seems to be "its ok if youre rube goldberg"

just wait till you have a predecessor

That's the opposite of my argument. I'm saying that the predominant style is Rube Goldberg, but Helm charts don't have to be written that way. Instead of writing an unreadable mess that is 90% template, just template the 5% that you need, and the whole thing is very readable.

thats what you are hearing. what everyone else is hearing: "_my_ code is self documenting, so obviously its more legible!"

“ For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.”

Or… you could not.

https://news.ycombinator.com/item?id=9224

So you have a version of Kubernetes that is as easy to use as Dropbox? Where do I sign up for the beta?

https://aws.amazon.com/eks/

That comment has aged brilliantly.

Thanks for that!

how is quoting this here relevant? nobody's saying k8s isn't successful or going to be successful—the argument is whether its complexity and layers of abstraction are worthwhile. dropbox is a tool, k8s is infrastructure. the only similarity between this infamous post and the argument here is that existing tools can be used to achieve the same effect as a product. the response here is "that'll never catch on" (because obviously it has), rather it's "as far as infrastructure for your company goes, maybe the additional complexity isn't worth the turnkey solution"

"You don't need Kubernetes, for a Linux user you can already build a custom solution quite trivially by setting up a custom package repo then build and distribute your application using apt, then configuring SysVinit to monitor your services, whilst using Ansible to configure iptables rules in combination with a simple load balancer you can manage yourself, then use syslog to monitor logs across all your machines whilst hand-waving away secrets management as a problem with 'serverless'"

Yes, you could. Some people do. Others don't, because even if you need a small portion of the features a turnkey solution is likely a better choice in the long run than hand-rolling your own mix of 15+ different technologies to achieve the same thing.

The difference, is that Dropbox is user-facing software, while Kubernetes is software engineer-facing. Dropbox has to be usable by tech-illterate people. Tech-illiterate people have no idea what a Kubernetes is.

There is value in creating a vertically integrated solution in a space, similar to what Dropbox did, so if you find yourself building many of the pieces of Kubernetes internally, it's worth considering if adopting Kubernetes wouldn't be a more efficient use of resources.

Confounded why sshfs wasn't chosen.

> Remember the days of assuming that your box was secure without having to do 10123 layers of abstraction?

Yep, I remember when I deployed insecure apps to prod and copied secrets into running instances, too.

Remember how the ops team kept installing Tomcat with the default credentials?

This was the funniest point in that comment to me.

Read the intended way, it's borderline wrong.

Read as "remember when people assumed security without knowing" is basically most of computing the further back in time you go.

I can setup a Kubernetes cluster, a container registry, a Helm repository, a Helm file and a Dockerfile before you are finished setting up the infrastructure for an Apt repository.

My experience is the opposite - an APT repo is just files on disk behind any webserver, a few of them signed.

Setting up all the infra for publishing APT packages (one place to start: https://jenkins-debian-glue.org ) is far easier than trying to understand all the rest of the things you mention.

I mean, Kubernetes is just some Go binaries; you can have it up and running in literal seconds by installing a Kubernetes distribution like k3s. This is actually what I do personally on a dedicated server; it’s so easy I don’t even bother automating it further. Helm is just another Go binary, you can install it on your machine with cURL and it can connect to your cluster and do what it needs from there. The Docker registry can be run inside your cluster, so you can install it with Helm, and it will benefit from all of the Infra as Code that you get from Kubernetes. And finally, the Helm repo is “just files” but it is less complex than Apt.

I’ve been through the rigmarole for various Linux package managers over the years and I’m sure you could automate a great deal of it, but even if it were as easy as running a bash script (and it’s not,) setting up Kubernetes covers like half this list whereas setting up an Apt repo covers one item in it.

Yeah I don't understand where all this fictional .deb and APT "complexity" is coming from. Everything uses standard abstractions that are decades old at this point..... oh no, you have to make some directories! You have to put a manifest file in the right place! Oh my god, now you have to run a command!

Exactly, an autoscaling cluster of multiple nodes with everything installed in a declarative way with load balancers and service discovery, all ready in about 10 minutes. Wins hands down.

no. you cannot.

Now make it not-brittle and prone to falling over, without using hosted k8s. ;)

... but then you could pay a fraction for bare metal cloud hosting instead of paying out the nose for managed K8S at Google or AWS.

Its complexity and fragility are features. It's working as intended.

So instead of knowing about K8s services, ingests and deployments/pods I have to learn 15 tools.

Ingests are not much more complicated than an nginx config, services are literally 5 lines each pod, and the deployments are roughly as complicated as a 15 line docker file.

If you're familiar with Linux (which should be considered required-reading if you're learning about containers), most of this stuff is handled perfectly fine by the operating system. Sure, you could write it all in K8 and just let the layers of abstraction pile up. Or, most people will be suited perfectly fine by the software that already runs in their box.

I work in a small company, we don't have a sysadmin, so mostly we want to use managed services. Let's say we want a simple load balanced setup with 2 nodes. Our options are:

- Run our own load balancing machine and manage it (as said, we don't want this)

- Use AWS/GCP/Azure, setup Load Balancer (and rest of the project) manually or with Terraform/CloudFormation/whatever scripts

- Use AWS/GCP/Azure and Kubernetes, define Load Balancer in YAML, let K8S and the platform handle all the boring stuff

This is the simplest setup and already I will always go for Kubernetes, as it's the fastest and simplest, as well as the most easily maintainable. I can also easily slap on new services, upgrade stuff, etc. Being able to define the whole architecture in a declarative way, without actually having to manually do the changes, is a huge time-saver. Especially in our case, where we have more projects than developers - switching context from one project to another is much easier. Not to mention that I can just start a development environment with all the needed services using the same (or very similar) manifests, creating a near-prod environment.

I think the argument there is that it's only simple because the complexity of k8s has been taken away. I don't think anybody has claimed deploying to a k8s cluster is overly complex; running it well, handling upgrades, those are huge time sinks that need the requisite expertise.

Much like Multics was "simple" for the users, but not for the sysadmins.

Taking the complexity of k8s away was just gonna happen. As someone who built everything from scratch at a previous company, I chose eks at a start-up because it meant that the one-man-systemsguy didn't have to worry about building and hosting every single cog wheel that is required for package repos, OS deployment, configuration management, consul+vault (minimum), and too many other things that k8s does for you. Also, you can send someone on a CKA course and they know how your shit works. Try doing that with the hodge-podge system you built.

Training is a great point, and I think that's why major clouds are going to be stickiest (in terms of using them vs migrating to new things).

The central problem of most companies has been finding / affording people who can maintain their stuff.

If Amazon / MS / Google can make it simple enough that skilled people can be quickly cross trained, and then have enough architecture knowledge to be productive, that's a huge win over "require everyone to spend 6 months muddling through and learning our stack we built ourselves and partially documented."

That's the point though right? A good (couple of) sysadmins can run a k8s cluster that can be leveraged by dozens (even hundreds) of dev teams. Instead of every team having to re-invent the wheel you get a common platform and set of deployment patterns that can fit most any use case. Of course if you don't have multiple different teams (or every team is running their own k8s cluster) then that is definitely a problem. But just because a handful of teams make an ill-advised investment in k8s when they could do easily with something much simpler doesn't mean that k8s is "too complex." Too complex for that use case sure, but for the vast majority of k8s deployments I would wager that it does add a lot of value and subsume a lot of the inherent complexity of running distributed, fault-tolerant, multi-tenant applications.

You run a small company, I'd argue that you aren't "the average user". For you, Kubernetes sounds like it integrates pretty well into your environment and covers your blind spots: that's good! That being said, I'm not going to use Kubernetes or even teach other people how to use it. It's certainly not a one-size-fits-all tool, which worries me since it's (incorrectly) marketed as the "sysadmin panacea".

Set up servers at linode and use the linode node balancer?

> Being able to define the whole architecture in a declarative way

With k8s (and other 'cloud' stuff) you seem to need to know a whole mess of a lot of the tool's stuff up front, vs a "progressive enhancement" way of doing one thing, getting it working, doing something else, getting it working, etc.

> most of this stuff is handled perfectly fine by the operating system

No, you have to write or adopt tools for each of these things. They don't just magically happen.

Then you have to maintain, secure, integrate.

k8s solves a broad class of problems in an elegant way. Since other people have adopted it, it gets patched and improved. And you can easily hire for the skillset.

Okay, so let's add a couple of things.

How do you do failover?

Sharing servers to save on costs?

Orchestrate CI/CD pipelines, preferably on the fly?

Infrastructure as Code?

Eventually you a point where the abstraction wins. Most people will say "but AWS...", but the reality is quicker, easier to use, and runs via multiple providers, so I think it's going to keep doing well personally.

Not the OP here.

We aren't really comparing apples and oranges in all cases that have been talked about in the larger thread. Some of the comparisons seem to be between "self hosted LAMP stack" vs. "kubernetes as a service on AWS". These are vastly different things. We should compare "self hosted LAMP stack" vs. "hosted in cloud LAMP stack" for example or "self-hosted kubernetes" vs. "self-administered kubernetes on EC2" vs "kubernetes as a service on AWS". All of these will have vastly different characteristics, pros and cons depending on your company and teams' realities.

Failover is something that a load balancer does automatically for you. Your services just need to provide a health check. Now where you actually run those nodes is a different thing. These might be slow to procure servers hosted at your provider. Or these might be manually set up EC2 instances or terraformed EC2 instances. Dunno what everyone uses as load balancers nowadays but a previous place for example had F5s and we had our own vsphere farm.

Sharing servers: I don't think this is a good idea at all except if you mean internally and if you do that then there's good and bad ways (see above on vsphere farm. If one project caused another to starve performance wise because of what was running on the same physical machines it was easy to resolve. If this was virtual servers at a traditional hoster, good luck. AWS is probably somewhere in between with EC2 and especially their storage.

Dedicated CI/CD pipelines: This is an awesome one to have and can cost an arm and a leg. I enjoy this very much at my current place w/ EC2 CI agents that scale with the number of devs currently working and dedicated "complete copy of Prod" dev environments (basically a kubernetes namespace for each dev/QA person/e2e test run to play with as they like).

Infra as code: Does not require kubernetes at all but can be implemented with kubernetes. If you already used docker to run stuff anyway for example and you can "abstract away" the kubernetes complexities to your SRE team and/or AWS, go ahead and use kubernetes. But be aware that if nobody at your place actually know kubernetes because you just relied on the hosted version of it, you're at the whim of their support people when something blows up in Production. You may not be big enough to have your own SRE team to take care of this but then you might also just not really benefit enough from kubernetes complexity and a simpler arrangement could have been easier for the people you do have to actually understand.

I think you've missed the point I was making.

Essentially if you work back from the desired state of having IaC, CI/CD, test environments per MR, you likely see something like k8s as a framework that helps you achieve that.

Of course, if you start from "I just need a LAMP stack" you might have a very different conclusion. But when you reach the same endgame ( actually I need an environment for every MR ), you've probably incrementially built something more complex and bespoke.

This will explain why there are dozens of us who are quite happy with the product. The only real question is, do you already know it and do you find it much harder to ship a deployment to a managed k8s cluster vs systemd unit files?

If not, it might be an abstraction worth having. If you don't already know how though, then you might have better things to be doing with your time.

This really depends on how many boxes you have.

I have been professionally working in the infrastructure space for a decade and in an amateur fashion running Linux servers and services for another decade before that and I am pretty certain that I would screw this up in a threat-to-production way at least once or twice along the way and possibly hit a failure-to-launch on the product itself. I would then have to wrestle with the cognitive load of All That Stuff and by the way? The failure case, from a security perspective, of a moment's inattention has unbounded consequences. (The failure case from a scaling perspective is less so! But still bad.)

And I mean, I don't even like k8s. I typically go for the AWS suite of stuff when building out systems infrastructure. But this assertion is bonkers.

Why? You still need to manage all that for your server even if you are running kubernetes on top of it.

I can’t imagine anyone with root access to a kubernetes server is any less dangerous that root on a simple webserver.

No, I don't, because I can yawn dramatically and I can go to any cloud provider and get a k8s cluster with generally consistent and at worst a moral-equivalent set of standard building-block cloud tools already set up. It won't cost me much, it will work mostly-predictably out of the box, and there's support right there for when it fails. Like, that's what k8s is there for. I use AWS pretty exclusively so this doesn't appeal to me, but what does is doing the moral equivalent and having ECS just...there. (Or even better, Fargate, if I can't solve the bin packing problem by myself.)

I haven't "managed a server" outside of my house for a few years now, and I quite like it. I theoretically have had root to ECS clusters, but I've never logged into them. Why would I? Amazon is going to be better at it than I am. Not only do I have more important things to be doing, but I'll do a worse job of it than they will. And to be clear: I consider myself pretty kinda really good at this stuff. But not good enough to make it a competitive advantage unless it's what I want to sell, and I sure as heck don't.

And the article's point, that whatever comes next will probably be better and might even be The Real Thing--I think that is wise.

> Why would I? Amazon is going to be better at it than I am.

Until it's not. Then suddenly you're trying to decipher cryptic cloud provider error messages in a service that made a false promise to you that it's abstraction was so air-tight that you'd never have to learn the underlying technology at all.

Then suddenly, you do need to know the underlying implementation, and quickly.

Yup! I used to feel exactly as you do, and I make it my business to understand what is below the abstraction besides because some old habits die hard (and because I just like this stuff, tbh). But I started working at places with the kind of conservation and pre-testing that make that much less critical. Those organizations also that pay a great deal of money for the kind of support to make knowledge a habit of curiosity and personal fulfillment rather than save-the-worlding.

I haven't needed to do something like that in production, as opposed to pre-production deployment suss-out, since (and I went and checked my enough to be sure) 2017. Though, to be fair, I've been working in devrel since last August, so call it four years of rooting around in the trenches, not five. ;)

> I have to learn 15 tools.

kubectl, kustomize, helm, istio, graphana, various flavors of ingress controllers, overlay networks, service meshes, storage controllers, etcd, etc.

from tfa: https://landscape.cncf.io/

you still have to learn 15 tools, just now they are hidden behind the scenes, and you still have to understand the underlying systems to reason about your containers.

this isn't for or against k8s - i'm a right tool for the job guy - but as a tool kubernetes doesn't solve problems, it encapsulates them and shifts them around.

Plus all the cloud tools are immature with terrible error handling and logging.

So after an enjoyable time crafting a 30 level deep json file you get a failed helm deployment with a error message like "timed out waiting for the condition".

15 mature well documented tools are a lot easier than 15 kludged ill thought out Kubernetes definitions.

Any serious Kubernetes environment is not 5 lines per pod, its the hell of rbac and pod security policies and all sorts of overly cryptic cruft.

There was some project where one wrote all of that (essentially what Kubernetes does) in like 8k lines of bash script. Brilliant, yes. But there is not way I want any anything similar in my life.

I am not the biggest fan of the complexity Kubernetes is, but it solves a problems there is no way I want to solve individually and on my own.

I think the point of the blog post in the OP is that it should be a bunch of bash scripts with very few interdependencies, because most of the requirements in the grandparent comment are independent of each other, and tying them all together in a tool like kubernetes is unwieldy.

Have you ever tried to package things with .dep or .rpm? It's a f** nightmare.

A place to store secrets and provide those secrets to your services.

"A problem that is unique to kubernetes and serverless. Remember the days of assuming that your box was secure without having to do 10123 layers of abstraction?"

I remember 10 years ago things were not secur, you know when people baked their credentials in svn for example.

lol. as someone who has packaged stuff I can tell you that this K8S is orders of magnitudes more complicated. Also, once you figure out how to package stuff, you can do it in a repeatable manner - vs K8s which you basically have to babysit (upgrade/deprecations/node health/etc) forever and pay attention to all developments in the space.

   FROM python:3.8
   RUN apt install libsomething
   ENV RELEASE=production
   COPY . .
   RUN poetry install
   CMD ["poetry", "run", "server"]

What would be the rpm/deb equivalent of those 6 lines? Would it work on MacOS?

let’s unpack this for a while.

what is python:3.8? is this reproducible?

what is apt? where is the install coming from?

What about poetry?

Yeah it’s cool for shits and giggles but when this thing breaks you’re going to be in a world of pain

for the rpm equivalent:

python setup.py bdist_rpm

wat?

Oh my. I'm not sure that I'd use Python to make a point about easy interop with distro package managers. It quickly descends into a nightmarish hellscape if you have more than a few dependencies or, different versions of python, or god forbid: C extensions.

bdist_rpm isn’t equivalent to the Dockerfile above. It can be made reproducible with a few changes (locking the upstream image to a hash, locking the apt package version), but that’s likely overkill. Because when it breaks you’re not in for a “world of pain” at all, you just have a failing CI for an hour.

I take it from the lack of an answer to the question that the equivalent non-docker packaging would be much more complex.

Checkinstall makes packaging pretty easy for anything you aren't trying to distribute through the official distro channels.

https://help.ubuntu.com/community/CheckInstall

.deb packages are literally just a compressed archive with a folder structure that mostly mimics your folder structure on the hard drive. You've got some pre- and post-hooks where you can write some shellscript to do fancy stuff, and a signing process to ensure authenticity. Autostart is a SysV init script or systemd xml file away. How is that a f* nightmare?

This is supposed to be an argument against Kubernetes?

Nope, just an argument against the "you must write all of this yourself" line. :)

You're making their point for them.

Some of these are decent points, but a couple are misleading.

The security one is the big one. Things were just not as secure (and did not need to be as secure) “back then”. K8s has a lot of complexity, and security should definitely be simpler so it’s harder to misconfigure, but not doing anything is not viable.

Saying “Package Managers” is fine until you realise they solve only part of the problem. The mainstream ones are good tools to update package (and dependencies) from version X to Y. When you’re running a distributed system, it’s often not that simple if you want to be reliable. Coordinating a slow global update of your application from version X to Y (safely) is pretty tricky and I’m not aware of good self-contained solutions to this.

That escalated quickly. Unit tests and type systems are not complicated at all, and are applied by solo developers all the time. GraphQL and Kubernetes are completely different beasts, technologies designed to solve problems that not all developers have. There really isn't a comparison to be made.

I agree. GraphQL is conceptually straightforward, even if certain implementations can be complex. Any developer familiar with static typing is going to get it pretty easily.

I’m far from an expert, but ISTM that Kubernetes is complex both conceptually and in implementation. This has implications well beyond just operational reliability.

Almost every team I've worked on has needed to deploy multiple services somewhere, and almost every app has run into escalating round trip times from nested data and/or proliferating routes that present similar data in different ways. While it's true to say not all developers have those problems, they're very common.

That's a very SaaS-centric way of looking at software development.

Unit tests and type systems are useful across the whole stack. Systems developers, application developers, embedded developers, mobile developers, even sysadmins and IT people - they all have a use for these basic principles of how to design a piece of software.

GraphQL and Kubernetes, on the other hand, are solutions designed exclusively for web services deployed into the cloud, and they're primarily useful in situations where there are many different teams each working on different services, with differing release schedules and engineering priorities. These situations might seem very common in large companies, but I don't think they represent common aspects of software development in general.

Sure, but k8s isn't the only way to do any of those things, and it's certainly a heavyweight way of doing most of them.

It's not a question of k8s or bespoke. That's a false dichotomy.

I see way too many young/inexperienced tech teams using k8s to build things that could probably be hosted on a couple of AWS instances (if that). The parasitic costs are high.

I see way too many young/inexperienced tech teams STILL using an unmaintainable process of just spinning up an EC2 instance for random crap because there is no deployment strategy at the company.

Yup, k8s is at least standardized in a way that’s somewhat sane.

Before k8s every org I worked for had an absolute mess of tangled infrastructure

Not sure why this is being downvoted.

"We can do it ourselves!" attitude by people who are unskilled is the source of many legacy hell-webs sitting in companies all over the world that are desperately trying to be maintained by their inheritors.

Not responsive to the argument. k8s is maybe a "deployment strategy", but it's certainly not the only one. Or the best one for all circumstances.

"Large scale enterprise" is the key here.

Kubernetes was made by Google. Google is not your startup, it has millions of servers serving billions of users, of course it needs complex systems, and it has thousands of people to maintain them.

In a small company, you probably don't need much of what's in that "need to" list. Rent a server, maybe a second one for redundancy, install your packages, run your app, and if you did things well, you can do quite a lot with a single machine.

But a lot of people think they are Google, and get ready to scale to a level they will never reach, and do it badly.

I think that where most of the pooh-poohing comes from, the use of overly complicated solutions for your scale.

It's where Thomson-Unix way win as KISS and still work for small to large scale.

Comparing Kubernetes to type systems is like comparing a shack to a gothic cathedral. Type systems are incredibly stable. They have to be proved both sound and complete via meticulous argumentation. Once proven such, they work and their guarantees exist... no matter what. If you avoid the use of the `unsafe...` functions in languages like Haskell, you can be guaranteed of all the things the type system guarantees for you. In more structured languages like Idris or Coq, there is an absolute guarantee even on termination. This does not break.

Whereas on kubernetes... things break all the time. There is no well-defined semantic model for how the thing works. This is a far cry from something like the calculus of inductive constructions (basis of COQ) for which there is a well-understood 'spec'. Anyone can implement COIC in their language if they understand the spec. You cannot say the same for kubernetes.

Kubernetes is a nice bit of engineering. But it does not provide the same guarantees as type systems. In fact, of the four 'complicated' things you mentioned, only one thing has a well-defined semantic model and mathematically provable guarantees behind it. GraphQL is a particular language (and not one based on any great algebra either, like SQL), Kubernetes is just a program, and unit tests are just a technique. None of them are abstract entities with proven, unbreakable guarantees.

Really, comparing Kubernetes to something like system FC or COIC is like comparing Microsoft Word to Stoke's theorem.

The last thing I'll say is that type systems are incredibly easy. There are a few rules to memorize, but they are applied systematically. The same is not true of Kubernetes. Kubernetes breaks constantly. Its abstractions are incredibly leaky. It provides no guarantees other than an 'eventually'. And it is very complicated. There are myriad entities. Myriad operations. Myriad specs, working groups, etc. Type systems are relatively easy. There is a standard format for rules, and some proofs you don't really need to read through if you trust the experts.

Your post reads like a teenager yelling "you don't understand me" at parents who also were teenagers at one point. You really think that those are new and unique problems? Your bullet points are like a list of NixOS features. I just did all of that across half a dozen servers and a dozen virtual machines with `services.homelab.enable = true;` before I opened up HN while its deploying. I'm not surprised that you can't see us lowly peasants from your high horse but many of us have been doing everything you mentioned, probably far more reliably and reproducibly, for a long time.

> Your post reads like a teenager yelling "you don't understand me" at parents who also were teenagers at one point.

I don’t understand teenagers any more, and I’m barely 30. I don’t think this analogy really works.

I agree with your point though.

You don't have to understand teenagers to understand that their problems are the same that they have always been, except in different settings.

Yep, we used to setup these things with a bunch of different systems using our collection of Ansible playbooks. The playbooks are complex, so as to handle all kind of edge cases. Furthermore, since they are developed over a long period, the coding convention are not uniformed; it's quite hard to teach the new hires how to use and contribute to the playbooks.

We probably replaced tens of thousands line of Ansible code with a few thousands line of K8S code. We found the new code easier to maintain: because K8S is much stricter than Ansible, it's harder to deviate from the norm. Granted, we might be biased because K8S is all new and shiny, but so far we haven't regretted moving to K8S.

OK, true ... but if you do all that yourself, then "they" can never fire you, because no one else will know how the damn thing works. (Just be sure not to document anything!)

Unit tests and "type systems" have very little in common with Kubernetes and GraphQL.

Also, GraphQL is not complex. You can learn the basics in an hour or so.

Tangential to your point, but how did 'unit tests' end up in your list of complicated things? They are conceptually easy to understand, and they are certainly not only for large scale enterprise users. Granted, it takes years to learn how to write nice tests... maybe that is what you mean?

>> "complicated" things like unit tests, type systems, Kubernetes, GraphQL, etc.

Those are not even in the same ballpark in terms of how complicated they are. Unit tests and type systems are not complicated at all. GraphQL not really either. But Kubernetes is very, very much.

"People like to ..."

Perhaps the reason people question these complicated things is because they are, whether intentionally or not, being marketed to an audience on HN that includes small scale non-enterprise users.

I shall paraphrase others here: A problem does not exist for you simply because it exists for LARGE SCALE ENTERPRISE users.

What I would add to that is there is nothing particularly noteworthy about a large organisation's IT work simply because it is a large organisation or making billions in ad revenue, unless one is also working in a similar organisation. If some organisations are writing the next "Multics", it really should not be interesting to everyone. A single person who can do all the individual tasks you listed is likely to think critically when presented with "news" of organisations where no single individual can do those things. Its like how many Initech Corporation employees does it take to screw in a lightbulb.

I find some of the most interesting work is found in projects started by individual programmers working alone. luajit for example.

The Kubernetes marketing team has definitely gotten to you. The investment in DevRel is really paying off if people are unironically arguing that you _must_ use K8s or you're wasting money and time.

I'd be very curious to find your proposed cost savings after accounting for those teams of engineers tasked with maintaining a company's K8s clusters. There is no free lunch.

Even at smaller scale, dealing with any distro + k8s + helm can be simpler than doing ops on an bare ubuntu install.

If your dependencies are not too many / well supported you may get good results with nix.

Most people who are afraid of doing all of those things haven't actually done them, and would probably be shocked to find that they aren't actually that complex. Kubernetes actually makes them more complex, but simpler in the aggregate. In other words, most of the apparent value in kubernetes vanishes when you do a real bake-off between kubernetes vs rolling your own infrastructure. There is still SOME benefit, but it's usually exaggerated.

Another problem with kubernetes is the flexibility it gives you. Look at five engineering teams using kubernetes, and you'll see five wildly different setups. Within that maneuverability, in a project that ostensibly makes things "simple", hides the devils that will bite you when you least expect it.

Comparing type systems to kubernetes seems like an incredible category error to me. They have essentially nothing in common except they both have something to do with computers. Also, there are plenty of well-designed and beautiful type systems, but k8s is neither of those.

I do all of the above with my own jars I've written all by myself and I feel as if it was 10x faster than just scratching the surface of Kubernetes.

Resume driven development is very real if people can't write their own load balancer.

Recently I had cause too try kubernetes… it has quite the rep so I gave myself an hour to see if I could get a simple container job running on it.

I used GCP autopilot k8 cluster… and it was a slam dunk. I got it done in 30 minutes. I would highly recommend to others! And the cost is totally reasonable!

Running a k8 cluster from scratch is def a bigco thing, but if you’re in the cloud then the solutions are awesome. Plus you can always move your workload elsewhere later if necessary.

This is also my experience, k8s can get harder but for simple stuff it’s pretty dang easy

I’ve got a different experience with Kubernetes because from what I’ve seen in fails to provide most of the features you described out of the box. Or when it does, they have major issues of how to safely deploy and maintain them over time. I assume you mean all those services can be installed and configured on Kubernetes, once you have Kubernetes itself up and running. But that’s not the same thing.

You have to do those things WITH Kubernetes.

It doesn’t configure itself.

I focused on fixing Kubernetes problems at my last job (usually networking). How is that supporting the business (hint: it didn’t so management forced us off Kubernetes)

No piece of software is a panacea and shilling for project that’s intended to remind people Google exists, is not really putting time on anything useful either

My issue with Kubernetes and DevOps is companies that combine DevOps with development. As a developer, it is already hard enough to keep up with new frameworks. Now these companies want their devs to do DevOps, two vastly different expertise. Not sure how common it is in industry but I know enough developers who are now halfassing DevOps.

In my book DevOps is a set of practices that aims at improving the collaboration between Devs and Ops. I know that the term is now often used to label a role (or even a job description), but I think something important is lost in the switch.

According to what I put behind the concept, implicating Devs is at the core. You built it, you run it!

> In my book DevOps is a set of practices that aims at improving the collaboration between Devs and Ops.

I think that's what sold me on DevOps.

> You built it, you run it!

This adds too much responsibilities for devs but also makes it hard to find good enough developers who can also manage deployments and infrastructure. I have never seen happy and competent Dev+DevOps person. There is just too much cognitive load for a same person to do these two things right at the same time. The Hello World of Kubernetes deployment is easy on Cloud but anytime you need to do something a bit more complex, learning curve increases tremendously.

What seems to work is that each team having one or more dedicated DevOps person. Or I have seen a dedicated DevOps team in large orgs managing infrastructure for many other teams.

"Poeple like to ..."

Perhaps the reason people question these complicated things is because they are, whether intentionally or not, being marketed to an audience on HN that includes small scale non-enterprise users.

I shall quote thyself here: A problem does not exist for you simply because it exists for LARGE SCALE ENTERPRISE users.

Best luck finding some engineer can understand and do all those stuff today. It's possible, but it's hard. Everyone comes to the table with "Hey terraform and helm/k8s": D: D

If you don't write that yourself, you still have to understand how someone else wrote it so you can configure and use it properly, and understand how to debug it when it's not working.

> Just write all of that yourself!

At least then you’ll have a shot at actually understanding it. I can’t trust kubernetes when anything goes wrong because the system just isn’t very transparent.

The advantage of kubernetes is price and being a standard making staff hiring easier.

F5, autosys, spunk, etc are all much better products to do the tasks you mentioned but cost $$$ vs kubernetes.

> Things that are solving a specific problem for LARGE SCALE ENTERPRISE users.

So, just like mainframes?

Seriously, though, I think that was the point. The future needs to be a much less complicated tool.

I like Chef Habitat. It pretty much does all this. It's how I've kept away from k8s this long.

Unit tests, GraphQL and type systems aren't complicated, or at least don't need to be.

Right. But that doesn’t mean that Kubernetes is the right solution for this set of problems.

Not all of those things are actually needed at small to medium size.

We did all that on AWS, and do it now on GCE. Load balancers, instance groups, scaling policies, rolling updates... it's all automatic. If I wasn't on mobile I'd go into more detail. Config is ansible, jinja, blah blah the usual yaml mess.

It's not K8S or nothing. It's K8S or Nomad, which is a much simpler and easier to administrate solution.

While this is not false, I don't think many of the posts critical of K8s hitting the front page are advertising for Nomad, or focusing on drawbacks that don't apply to Nomad.

This is partially true. If the only feature you care about Kubernetes is container scheduling, then yes, Nomad is simpler. The same could probably be said about Docker Swarm. However, if you want service discovery, load balancing, secret management, etc., you'll probably need Nomad+Vault+Consul+Fabio/similar to get all the basic features. Want easy persistent storage provisioning? Add CSI to the mix.

Configuring these services to work together is not all trivial either (considering proper security, such as TLS everywhere) and there aren't many solutions available from the community (or managed) that package this in an easy way.

thank you lol. Hello World engineers will never stop criticizing K8s.

Yes, but you don't need many or any of those things to launch a Minimum Viable Product.

So Kubernetes can become invaluable once you need to scale, but when you are getting started it will probably only slow you down.

If you want your MVP to be publicly available and your corporations ops/sec to be on board with your plans then Kubernetes is an answer as well. Even if your MVP only needs a single instance and no scaling. Kubernetes provides a common API between developers and operations so both can do the job they where hired for while being in each others way as least as possible.

Pre-MVP, development and ops are likely the same people.

With Pre-MVP you mean installing it on your laptop right? It all just really depends on your companies size and the liberties you are given. At a certain size your company will have dedicated ops and security teams which call all the shots. For a lot of companies, Kubernetes gives developers the liberties they would normally only get with a lot of bureaucracy or red tape.

I have a Windows server and use dot net.

I press right click - publish and for prod, i have to enter the password.

Collecting logs uses the same mechanism as backups. They go to a cloud provider and are then easy to view by a web app.

Never needed more for after hours than this, perhaps upgrading a server instance from running too many apps on 1 server.

> solving a specific problem

The problem to me is that Kubernetes is not solving a specific problem, but a whole slew of problems. And some of them it's solving really poorly. For example, you can't really have downtime-free deploys in kubernetes (set a longish timer from SIGTERM to increase the chance that there's no downtime).

Instead I'd rather solve each problem in a good way. It's not that hard. I'm not implementing it from scratch, but with good tools that exists outside of kubernetes and actually solve a specific problem.

Why can you not have downtime-free deploys? You tell your applications to drain connections and gracefully exit on SIGTERM. https://pkg.go.dev/net/http#Server.Shutdown

If your server is incapable of gracefully exiting, that's not a K8s problem.

> Why can you not have downtime-free deploys? You tell your applications to drain connections and gracefully exit on SIGTERM. https://pkg.go.dev/net/http#Server.Shutdown

> If your server is incapable of gracefully exiting, that's not a K8s problem.

Also whatever load balancer/service mesh you have can be configured for 503 rerouting within DC as necessary too.

> You tell your applications to drain connections and gracefully exit on SIGTERM.

The problem is that k8s will send requests to your application after SIGTERM. So you have to wait some amount of time before shutting down to allow for that.

This was at least the case last time I used k8s, and it seemed like it was due to the distributed architecture, so something that was more than a mere bugfix away.

K8s has like, probably the most complete support for readiness/no downtime deploys in the whole damn industry so it's surprising to hear that...

https://cloud.google.com/blog/products/containers-kubernetes...

You can actually renew/upgrade your whole cluster with no downtime if you care enough to tackle the annoying bits that cost a few minutes in YOLO mode.

Reply on news.ycombinator.com