Hacker News Reader

Comment by throwaway787544 | Hacker News Reader

The question of whether to use K8s or not is like wondering what kind of saw you should use to cut wood. There's different saws for different purposes. But even with the right saw, you still have to know how to use it correctly. Better to use a hand saw correctly than a table saw incorrectly. (you can use a table saw incorrectly, but best case the work ends up crap, worst case you lose a finger)

After building infrastructure for dozens of teams, I'm quite convinced of the following:

- if your people aren't very skilled, they won't build anything well. most software engineers i've seen professionally working in the cloud are handymen trying to build a wood cabinet.

- if your people can't build well, it doesn't matter what technology they use. choosing between building a cabinet out of metal or cherry wood doesn't make much difference if they've never built a cabinet before.

- if the first two holds: then only use the technology which requires the least skill to use well, and where the amount of maintenance is closest to zero. don't build a wood cabinet from scratch when you can buy flat pack. don't buy flat pack when you can buy an assembled cabinet, get it shipped, and carried into your office.

- if using the aforementioned technology requires 'building' or 'assembling', and that is not core to the customer-facing aspect of your product, then you should not be building, you should be buying. if your business doesn't involve assembling flat pack furniture, don't ask your employees to build their own desks and chairs from Home Depot or Ikea parts. buy the premade desk and chairs, use them to make your actual product.

- a software engineer knows as much about cloud architecture as a fine woodworker knows about framing. "it's all just wood" until the house takes 10x as long to frame and is 10x as expensive and still doesn't meet code.

- people will try to build things they don't fully understand and leave the company before anyone realizes the mess they've made. imagine your retail store is accessible by driving a car over a wooden bridge built by a handyman.

View on HN

The problem I keep seeing in startups is that a lot of startup tech founders and employees don't see a startup as an embryonic business that will grow over the long term into a money-making venture requiring calculated risks and priorities, but rather as a personal playground to do all the fun things they were not allowed to do at their old BigCorp.

Use cool framework or language X? Use Kubernetes? Microservices? Lambdas? No-SQL? Two fingers to BigCorp, I'm CTO/tech employee #1 at a new startup and I get to do whatever I want. If and when it goes down in flames, I now have these skills on my resume I can leverage into a better opportunity.

Ideally your first tech employee/CTO should be a seasoned hand who has been burned before, who keeps an open mind to new approaches but knows when and how to adopt it. But it's quite rare to have seasoned developers in such startups, they're not attracted by the risk and chaos of these environments as they have better options and startups generally can't afford them.

> Use cool framework or language X? Use Kubernetes? Microservices? Lambdas? No-SQL? Two fingers to BigCorp, I'm CTO/tech employee #1 at a new startup and I get to do whatever I want. If and when it goes down in flames, I now have these skills on my resume I can leverage into a better opportunity.

I was building a SaaS application (as a tech co-founder) and chose rails. It was newish to me (I'd done 1-2 projects) and I admit, I wanted to get better at rails. But it was also, after assessing the options and my (limited) knowledge of the future, the tech with the best chance of helping the company succeed.

I have definitely run into startups where there was overly complicated technology, but tended to be later stage and have raised some money. As a founder/employee #1, I have a hard time believing anyone would pick tech just to burnish their resume; they should be trying to ship so the company succeeds.

What am I missing?

You’re missing the fact that employees who are interested in their own career would be less interested in a Rails stack on their resume.

It was over half a decade ago. But the point wasn't about the particular tech, it was that folks who join a startup, especially if early, should prefer the startup to succeed over choosing overcomplicated solutions, and I don't understand why. A couple of possible reasons:

* they don't think the solution is overcomplicated

* they don't really care about the startup succeeding

* they don't know what the ramifications of picking the wrong technology will have

* they think the risk of learning something new is outweighed by the value to the company (or themselves)

I don't think startup employees are altruistic, but picking a tech you know will have a detrimental effect on your employer strikes me as shortsighted.

I don’t think people want their employee to fail. But given a choice between the most “correct” technology and the most “marketable” technologies. It’s short sighted as employee to choose the most “correct” technology. More than likely the startup is going to fail.

If given a choice between Vue and React, I might prefer Vue. But I would still choose React as a tech lead because I know it is more marketable and you would get more self interested developers and it would be easier to recruit for.

Thanks for giving me clarity into your thought process. I agree the choice between "correct" and "marketable" is a difficult one. Sometimes the most marketable tech is the correct choice. Sometimes the correct choice is the one you and your team knows, even if it is not the most marketable. Sometimes it doesn't matter, because when you are building an app, there are many paths to success.

Technical correctness is not the only factor, there are other kinds of correctness, as you point out with React: "you would get more self interested developers and it would be easier to recruit for".

>As a founder/employee #1, I have a hard time believing anyone would pick tech just to burnish their resume; they should be trying to ship so the company succeeds....What am I missing?

That most startups are Underpants Gnomes, the company probably won't succeed even if they ship.

Even if the company is not Underpants Gnomes and their fundamentals are sound, succeeding or not is still largely luck based.

Maybe the reference to Underpants Gnomes is too dated now? - https://youtu.be/a5ih_TQWqCA

Love the underpants gnomes reference and I agree that many startups will fail. But wow, working at a company you feel in your bones will fail seems to me to be a poor way to spend a career.

Not that every company I've worked at succeeded, but I hoped that every startup I joined as an employee would.

Now, as a contractor or consulting company, you still want the client to succeed, but it's much easier to be arms-length, take orders, and collect your check.

Given the failure rate of startups, "I now have these skills on my resume I can leverage into a better opportunity" may be the actual goal of most employees.

I've seen this frequently in large companies as well. That's why you see legacy systems with random tech.

Because most startups aren’t going to grow in the long term to profitable businesses. Either they are going to flame out completely or they are going to get acquired, the product discontinued and the founder is going to walk away with money and post an article on their blog about “our amazing journey”.

As an employee, it’s in your own self interest to do resume driven development. Your “equity” probably won’t be worth anything so you might as well work toward preparing for your next job.

As a founder, you can’t offer the compensation of a larger company. What you can offer is the chance to work on “cool cutting edge technology”.

At this point in my career, I’m more motivated by being able to describe successful outcomes than cool new technology. But I do understand the motivations. I’m also not an old curmudgeon though that considers the cloud is just a bunch of VMs.

> The problem I keep seeing in startups is that a lot of startup tech founders and employees don't see a startup as an embryonic business that will grow over the long term into a money-making venture requiring calculated risks and priorities, but rather as a personal playground to do all the fun things they were not allowed to do at their old BigCorp.

I don't think this is an accurate or realistic take.

You're actually stating that tech founders, when starting greenfields projects, actually take the time to invest in frameworks that offer advantages over outdated legacy systems adopted way back in the past which are only in place because there is no chance in hell a company will spend resources to pay off their technical debt.

There was a point in time where adopting Java+Spring, or even React was something only some silly risk-taker would do when playing around in their personal playground. But as the investment paid off so handsomely, they have become baseline options.

If you are free to choose the absolute best tool you have at your reach, why wouldn't you?

Not OP, but over the last year, I've worked in 4 startups. (A) reached a $100m Series C. (B) had a tech stack that was far too complicated for what it needed which opened my eyes to how badly teams can shoot themselves in the foot with K8s. (C) was at YCombinator graduate that increased their ARR by $1m in the last month. (D) was funded based on a pitch deck and a bunch of former FAANG.

(B) had 200 unique visitors a month and a stack built on Kubernetes with something like 8 microservices. It was running Apollo, Socket.io, and on and on; literally overcomplicating everything...for 200 unique visitors a month. Every single aspect of that system was over-complicated. CTO was part of the problem, but was complicit in letting tech team go wild; many of whom just up and left after playing around with the tech.

(D) on the other hand had the exact problem that OP described: they started going hog wild building multi-layered service-oriented architecture when the system barely has any traffic. Deployment is a pain. Code is copy-pasted everywhere because the alternative is to publish and pull packages and that's a lot of friction. Team can code, but barely understands the tech stack.

(C) built the dumbest architecture I've ever seen. Shit was stapled together. Webhooks were constantly failing due to quota pressure. Their Firestore queries were inefficient and their sloppy migrations led to many broken documents in the store. But guess what? They are actually growing in this economic environment because they solved a real problem without worrying about growth until they actually saw the growth.

(A) built his solution in Angular because that's what he knew and didn't try to jump on React or other hype.

At (C) (the YC startup), I had a conversation with the COO that I'll never forget. We were discussing an architectural decision which would have scalability and stability impact down the line. What he said was to the effect that "We have 10,000 users today and if we lost them all, there are still 7 billion people. We'll find another 10,000". He made it clear that they wouldn't be happy about it, but that it's speed above all else.

> If you are free to choose the absolute best tool you have at your reach, why wouldn't you?

The purpose of a startup isn't to play around with tech: it's to find product-market fit and zero in on the business value. If you think of a startup as a place to play around with tech, you have no place in a startup.

You should almost always choose the "dumbest" possible technology used in the "dumbest" possible way so that you can hire anyone and they can step in and be productive when you need to scale. It minimizes ramp up time. It minimizes deployment and operational complexity. It minimizes the ways the system can fail. Literally build the dumbest thing that solves the core business problem that creates value for the user and then figure it out from there.

I think it is strange that the only example you gave of (A) is that they chose angular over react because that's what they knew.

Everyone else failed because their backend systems were overly complicated.

I don't see how there is a lesson to learn from it unless we make a ton of assumptions about (A), absolutely none of which have anything to do with angular or react.

The last paragraph definitely makes sense, though I'm currently in the unenviable position of un-fucking a startup because the initial developers chose a setup that was far too dumb. It's literally going to take at least as long to fix as it took to initially build out. Ce la vie I guess.

To get to $100m you have to do so many things right and so many things wrong that it's hard to isolate any one or two things. But key is that the co-founder and CTO stuck to what he knew and focused on solving a business problem. Angular or React doesn't matter; could have gone the other way.

You probably mis-understand what I mean by "dumb" and why I put it in quotes. Things that are "dumb" are intentionally so; they are built on stacks using technologies and approaches that have low cognitive load. It almost feels boring because of how easy it is to build on the stack and extend the stack without shooting yourself in the foot. A very practical example is something like Google Cloud Run vs Google Cloud Functions with Cloud Run being more flexible, easier to deploy, easier to work with, and easier to reason about. It's stupid simple and rather boring.

But besides the fact, dumb things are generally easier to unfuck than complicated things. Complicated things with many dependencies built on complicated stacks are much harder to untangle and refactor.

>You should almost always choose the "dumbest" possible technology used in the "dumbest" possible way so that you can hire anyone and they can step in and be productive when you need to scale. It minimizes ramp up time. It minimizes deployment and operational complexity. It minimizes the ways the system can fail. Literally build the dumbest thing that solves the core business problem that creates value for the user and then figure it out from there.

Yes!!! Spot on. I always say “make it so simple an idiot can immediately understand it, or me when I’m hungover”. Choose the most boring, simple…yet reliable, stable, well supported tech you can. Focus on business problems and not if you can use fancy-new-tech to solve them!

> (B) had 200 unique visitors a month and a stack built on Kubernetes with > something like 8 microservices. It was running Apollo, Socket.io, and on and on; > literally overcomplicating everything...for 200 unique visitors a month. Every > single aspect of that system was over-complicated. CTO was part of the problem, > but was complicit in letting tech team go wild; many of whom just up and left > after playing around with the tech.

I do not understand how can anyone draw any conclusions from this whether or not the technology were at fault here for the implied failure.

The GP is referring to a well-known phenomena that the founder of HN described in http://www.paulgraham.com/before.html.

> We saw this happen so often that we made up a name for it: playing house. Eventually I realized why it was happening. The reason young founders go through the motions of starting a startup is because that's what they've been trained to do for their whole lives up to that point. Think about what you have to do to get into college, for example. Extracurricular activities, check. Even in college classes most of the work is as artificial as running laps.

Many people, and I'd argue most people, start startups for their careers and for fun. Actually focusing on the business and solving business problems is foreign, grunge work. It's no surprise that founders and early employees pursue exciting tech to distract themselves from the challenge of product-market fit.

because engineering is about tradeoffs and the right tool for the job. The vehicle with the “absolute best” carrying capacity is gonna be a semi-truck, but that’s laughable if you’re going to use one for a supermarket run for groceries.

Newer != better

I think that’s true, but also it’s not clear what tools a startup should use. Even if they stick to a single VM, managing that VM in a way that is reasonably safe and reproducible is somewhat important—the more often you have people manually SSHing onto the VM the more likely you will need to recover functionality from scratch while also making it almost impossible to do so in any timely manner (because the functioning state of the VM is the result of many thousands of manual changes, many of which are undocumented, incompletely documented, or incorrectly documented). So then you can start learning about automating changes on a machine, but now you’re into territory that is about as complex as Kubernetes (or perhaps not “complex” as much as “you have to figure out what combination of tools and practices will work”, while a cloud-provider’s managed Kubernetes makes a lot of these kinds of choices for you out of the box and many others are obvious).

Similarly, you might reasonably try to use a PaaS, and it might work at first, but then you need to do something as common as a background task and find that your PaaS doesn’t support things like that (you can try running it in a process in your PaaS’s container and just accept that the background tasks are likely to be interrupted periodically?).

I don’t think Kubernetes is the answer for a startup, but it’s not obvious what “the answer” is, and much of the discussion ignores the complexity in the alternatives. It’s not helpful to say “Kubernetes is too complex” without articulating a simpler alternative (unless you are hinting at a market opportunity).

You’ll need to name names and their particular shortcomings if you’re going to say there are no good PaaS alternatives. In particular, both AWS and GCP have cron-as-a-service, which can then be hooked up to AWS Lambda/GCP Functions. Now, I’m not saying this will work for every use case out there (eg there are time limits on both), but without getting into the weeds a little bit on why those PaaS won’t work (for your use case), there are viable alternatives to K8s’ complexity.

> You’ll need to name names and their particular shortcomings if you’re going to say there are no good PaaS alternatives.

I'm not saying there are no good PaaS alternatives, I'm saying that it's not easy to find the right PaaS alternative. You might think you have the right one, and then some key feature is just missing (in my example, the ability to kick off orchestrator-aware background tasks).

> In particular, both AWS and GCP have cron-as-a-service, which can then be hooked up to AWS Lambda/GCP Functions.

Right, but now you're dealing with cloud-provider primitives which doesn't seem significantly more straightforward than dealing with Kubernetes (instead of knowing Kubernetes, you need to know the cloud provider APIs).

> there are viable alternatives to K8s’ complexity.

I'm sure there are, but figuring out which PaaS is appropriate for your use case (or a combination of PaaS + various cloud provider tools + etc) is a hard problem, possibly just as hard as building on a managed Kubernetes offering.

Personally, I'm hoping someone makes an open-source, push-button Kubernetes distro that has the common stuff built-in: external-dns, cert-manager, ingress-controllers, network storage backends, central logging, prometheus/grafana, etc. This would give you even more of the happy path built in without necessarily coupling you to a cloud provider while also giving you plenty of "future-proofing" (if your pass doesn't have some feature, you often need to switch to another platform, but if your Kubernetes distro doesn't have some feature, you can just build it atop Kubernetes).

I think it's clear to people who specialize in that. I don't know why startups don't hire a DevOps/SRE/Syseng/Sysadmin/Architect contractor right off the bat. Hire a really experienced one for one month and they'll give you several different options, lay out the pros and cons, costs, staffing, time estimates. Take their suggestions and implement the one that fits your business plans.

For example, a single VM you ssh into works fine if you do daily VM snapshots and deploy from snapshot. If somebody breaks it, revert to last working snapshot. Blue/green in multiple environments work with it too. Super cheap, super simple, easy to recover, automated. Very boring.

> I don't know why startups don't hire a DevOps/SRE/Syseng/Sysadmin/Architect contractor right off the bat.

It's entirely likely that they are doing this, but the DevOps/SREs that they're hiring are prescribing the tools that they know and have seen work in the organizations that they're coming from.

I've been an SRE for nearly a decade, and I honestly couldn't tell you in detail how to set up a pet-VM deployment with automation and guidance about who deploys to it, on what cadence, etc or who operates it, etc. In my first DevOps job, we were doing something like this, but it was a hot mess--that startup hired the wrong sysadmin to bootstrap us, and the next guy they hired moved us to AWS ECS (running on Fargate) and that was a dramatically more effective solution for us. The point isn't that ECS > pet VMs, but only that we were able to find a successful path with ECS where we were unable to with pet VMs despite hiring someone who allegedly had that experience. For that organization, ECS (rather than pet VMs) was the Very Boring Solution (mind you, I'm not a particularly big fan of ECS either).

People often confound complexity with having to learn something new.

k8s IS complex, not because it's new, which it isn't anymore, but it's so complex that even the developers have lost oversight.

And not just that, it's expensive. It requires a lot of hardware just for the base, 9 servers minimum.

The more complex something gets the harder it is to get done right.

I would downvote you if I could because what you wrote is just ignorant and anyone who dealt with k8s before can tell that you never dealt with it.

> I would downvote you if I could because what you wrote is just ignorant and anyone who dealt with k8s before can tell that you never dealt with it.

I don't know why you're snarking so hard when there are dozens of managed Kubernetes offerings out there.

> It requires a lot of hardware just for the base, 9 servers minimum.

An HA deployment requires 3 nodes for the control plane, and as many or few workers as necessary. But yes, this is 3 more nodes than you would run without automated orchestration. Of course, the cost of these 3 nodes is amortized over all of the worker nodes, so if you're only running 3 worker nodes you're paying 100% for orchestration, but if you're running 30 worker nodes you're only paying 10% (this is assuming equally sized nodes, for simplicity).

> The more complex something gets the harder it is to get done right.

Right, but if someone else packages that complexity up for you in a piece of software (like Kubernetes or Linux) it's often much easier to get it done right than it would be to roll your own from primitives.

If 9 nodes is a lot for you, then I agree that your problems are small enough that manual solutions can be much simpler than using Kubernetes.

You also don't need an angle grinder to open a cardboard box, but that doesn't mean angle grinders are stupid and can all be replaced with a box cutter.

Yeah, I have seen similar experiences, hence why I am a boring tech kind of person.

I think there are two types of startups and it depends what you are actually doing:

- Startup that has product market fit (or is really close), and is ready to scale: choose boring tech that works

- Startup that is a playground, exploring or building new tech, and may never make money.

You can join either as an employee/cofounder/leader, and have a great time, but if you mistake one for the other, you will have a bad time.

The PROBLEM is that all the job ads require you to have those skills that are impossible to very expensive to acquire if you're not employed at such a company that lets you experiment with that stuff you mentioned.

I write monoliths and I have only experimental experience with cloud, microservice, k8s, lambdas (no-sql is a no brainer). I can not find a job. All the recruiters offering any Go job, demand knowledge of exactly that. But since I'm a single guy with a currently non-existant budget I chose the cheapest way, monolith with a single database on a single server.

How am I supposed to acquire those skills? I don't have the time, motivation or the money to throw at expensive cloud or write overly complicated microservices with events. I need to get things done, not play with tech that my clueless employer wants, because they watched so Kelsey Hightower video.

They are often not required. It's obviously great if you have worked with a lot of the technology used at the new company, but many companies don't apply these as hard filters. I've never seen a candidate that had a 100% technology fit, but not one cares about that.

I view the technologies listed in the job ad much more as information for the applicant to judge if they'd be comfortable working with such a tech stack. Often it's the only place to get a glimpse of what technology a company is using.

The last company I worked for was essentially finance bros who had a no-code investment solution but wanted to sprinkle ML on top to get clients. Suddenly it needed to be able to run air-gapped on prem. Oh also on Ali cloud in china. Oh also on GCP and AWS hybrid. Business promised the clients it was ready before we even started building. 90% of the team was under 25. We tried our damned hardest. Used K8s to make the whole thing platform agnostic. It worked but it cost a lot. Business people are the worst.

Sounds like the exact use case for OpenShift.

i dont think there's any use case for openshift

use case: "I would like to use Kubernetes but I also would like a monkey to occasionally come out of a cage and hit my balls with a hammer"

Open shift : an unstable trailing-edge version of Kubernetes with too much security to the point of wasting everybody's time. Or so says my team which is still recovering from that unmitigated disaster ...

Sounds like the are looking for Qovery

Isn't a single binary the best fit in this case? Telling clients they need to build up a kubernetes cluster (especially in finance) won't always be the easiest decision.

Well said. Your first 3 points summarize the entirety of the problem.

Kubernetes is basically best practices (with some quirks) of running containers reliably across a cluster of machines with various deployment types, logging, DNS/discovery, networking, storage, load-balancing, high-availability and other features built-in. If you don't need any of that then don't use it, but if you do then you probably won't build a custom infrastructure that does it better.

> Better to use a hand saw correctly than a table saw incorrectly.

To be fair, better to use a handsaw even incorrectly than a tablesaw incorrectly.

I concur.

> you can use a table saw incorrectly, but best case the work ends up crap, worst case you lose a finger

Funny, just yesterday someone mentioned how Rust is compared to a SawStop: https://news.ycombinator.com/item?id=31784253

And here's a test showing how unlikely it is that it'll chop off your finger: https://youtu.be/SYLAi4jwXcs

But that won't stop your workpiece from flying into your eyeball/jaw/neck/chest, or you from inhaling fine particles. Safety features are nice, but aren't a replacement for knowing how to use the tool safely.

And that's just safety. Knowing how to use the tool properly means using a featherboard, or not using both the miter gauge and fence, using a crosscut sled, and other things which ensure your piece won't be marred or jammed or bind or cut incorrectly. Proper use also leads to increased efficiency and less waste.

Same things apply to K8s. Even if you buy some managed product or safety feature, you still need to know how to use it right after that point, or your work will end up shoddy.

> eyeball

The rest of your points stand (though I suspect a saw particle colliding with your jaw is unlikely to cause damage as severe as cutting off your finger), but any responsible person will be wearing safety goggles when operating a table saw.

I have never used a table saw before, but from my amateur youtube understanding, safety goggles aren't perfect protection against kickback, right?

Like, if I've got a chunk of wood being hurled at my face at high speeds, I'm definitely going to prefer to be wearing safety goggles, but I can still imagine the wood coming in at an angle and knocking them away, or being hit with enough force to break the goggles.

You're unlikely to get hit by larger pieces of wood at high speed. Dust, small chunks, maybe a part of the saw may come flying at you, but i suspect they're way more likely than large wood chunks. But if you believe this photo was not staged, then safety goggles are pretty good https://external-preview.redd.it/qVMDLhaXQN8Vd9UI6d5fDswF396...

"Safety features are nice, but aren't a replacement for knowing how to use the tool safely."

In many cases, safety features in languages do indeed remove the need to understand a dangerous tool. For instance, if your language doesn't have pointer arithmetic, you don't need to know how to use pointer arithmetic safely.

In this analogy, raw unfiltered K8s is the table saw and the SawStop is AWS ECS or GCP GKE

I guess you've never met a competent generalist, then, which is funny because that's exactly who you need as your first employee.

Funny part is that „competent generalists“ (those Brent-style types) are very rare and most technology needs to be designed for incompetent specialists. Also funny is that these generalists typically build as if everyone has their own skillset which renders many systems unusable once Brent has left the building.

They're not just rare, the bigger problem is that most companies don't know how to deal with them.

They either can't pay for them, don't give them enough freedom, weigh them down in a big team, or any number of other issues. One person in a specific role (even if they're not very good at it) is far easier for most organizations to deal with than the 10x generalist that can get things done.

It can work well in very early startups, but there's limited reward there for it to be worth it.

>those Brent-style types

Which Brent are you talking about?

I get the sense the name is being used in a general way like a “Karen”, but I’ve never heard of a “Brent”.

From The Phoenix Project: https://allengeer.com/how-to-handle-brent-in-the-phoenix-pro...

Brent is the engineer building all the real tech that a company in the Phoenix Project is building, and when the MC (a manager, Bill I believe) realizes this he re-directs resources so Brent can do his job without interruptions, ultimately saving the company. Excellent read, it's the first book managers are told to read when they use SCRUM/AGILE incorrectly.

Nobody cares if what you built is sustainable and scalable if you can't do it fast enough to get your first customers, and there is plenty of time to make what you've built usable by the time you've left.

If you, as a founder, hire an incompetent specialist as your first employee, maybe you don't deserve to be successful.

Your first hire is either a "Brent", or you have to get very lucky in just about every way.

Expecting someone who's capable of handling most roles isn't always a viable strategy.

Personally, I consider myself such a person and I've been called in numerous times when people:

  - need to use Kubernetes because of project requirements, but all they have is a single node with 8 GB of RAM (K3s sufficed nicely)
  - need to deliver OCI images and Helm charts, yet have no idea how to handle all of the infra aspects here (Nexus to the rescue)
  - need to containerize any number of applications from old timey monoliths that ran in loosely defined Tomcat and JDK versions, badly
  - need to figure out why their apps run badly, despite there being almost no logs ("Log output is spammy!" they say), no APM ("What?") or proper monitoring for whatever they've written
  - need to figure out how container networking works and why them attempting to use localhost for inter-service communication isn't always the best idea
  - need to figure out how to manage configuration properly, especially in situations where they add some random configuration parameter 4 months back, forget about it and then attempt to ask me about what it means, because of course they didn't document it
  - need to set up log rotation, because they absolutely refuse to use log shipping and see no benefit in it, yet don't want to delete the old logs whilst wanting to be able to view the old ones, yet don't know how to configure compression for the older ones and thus run out of disk space on the server
  - need to set up a reverse proxy and manage its configuration, because all they know is Apache/httpd (which I think is a nice project), but haven't bothered learning Nginx which they now need to use because of said requirements
  - need help with their supposedly automatic database migrations actually failing because they failed to account for pre-existing data
  - need help with the project moving ahead at a snail's pace because they wanted to pad their CVs and picked Tailwind CSS instead of some pre-made component library and thus have to reinvent the wheel, as well as use TypeScript which they do not know how to use and are slow with as a result

Somehow such generalists feel few and far between, so relying on them to always be there to save you from yourself won't be reliable.

Furthermore, there will be lots of people with either limited knowledge or limited interest in learning new things. I've see someone say that "restarting services is not in my job description" when asked to restart some environment. Many out there will also be perfectly fine with writing apps like they had 20 years ago, without taking advantage of any of the benefits of 12 Factor Apps or similar principles. They will also reach for the local file system instead of something like S3 for object storage and application memory instead of using Redis for session storage. They will seldom be able to write apps that are horizontally scalable and as a consequence you'll have to rely upon a single monolith which should always be up and working, which, given their lack of ability to write proper unit tests, will rarely work out that way.

Unless you are very selective in your hiring, plan for people like that also being present.

> Furthermore, there will be lots of people with either limited knowledge or limited interest in learning new things.

cries

> Many out there will also be perfectly fine with writing apps like they had 20 years ago

seen this many times

> Unless you are very selective in your hiring, plan for people like that also being present.

Hiring is everything. All issues you have listed would be solved just by having a senior (with real senior experience, not 10+ years written in their CV) in the team.

> Hiring is everything. All issues you have listed would be solved just by having a senior (with real senior experience, not 10+ years written in their CV) in the team.

But that's the crux of the problem, isn't it? The people who are doing the hiring often won't be able to optimally pick out the people that might lead to the success of any given project, given how many facets to judging someone's aptitude like that there might be, and how limited their resources might be.

I guess one can also mention the "ten years of experience" versus "one year of experience ten times" conundrum, but at the end of the day, a lot of it is going to be hit or miss.

As an individual contributor, it's probably a good idea to fix what you can, document what you cannot (and why) and always look for environments where you fit in the best, so that in the end you can work with people with whom you are compatible.

I grew up on a farm, so I've got a very generalist attitude to things for the simple reason that if the thing breaks you have to take the covers off and fix it, or you get to walk home in the rain. And then, tomorrow, you get to walk back out to it in the rain and fix it anyway.

I've used a microscope, a JTAG cable and a MIG welder on the same project on the same day.

Here's the thing though - I like simplicity. I like stuff to be simple. I drive a 25-year-old Range Rover because it's simple, and because if it breaks I can fix it quickly with simple tools. I bring this simplicity to the stuff I build - lowest parts count, easiest to get parts, comprehensive manual.

Build your software stack as though you expect to have to fix it in a boggy field on a pissing wet Sunday afternoon with a hammer and a roll of sticky tape, because that's exactly what you're going to have to do.

Totally agree about keeping things simple. When [it] hits the fan on Sunday afternoon, you can probably root-cause the issue fairly quickly and get things running again. And, you don't have to wait for that *one guy* who went on vacation Friday to come back a week later to show you how things were configured.

Exactly. A "perfect" fix next week is not better than "drive it gently and bring it back on Monday so I can look at it properly".

My style as well. I would add one thing which I call “critical dimension analysis.” Try to identify the main customer driver for the domain you’re in (could be cost/performance/something entirely different), and invest at little more in that technical area (while still keeping things relatively simple). That will give you a simple yet efficient stack for the domain you’re in, which will be a great base for taking your product forward.

I’ve experienced a lot of people recommending and learning a trendy technology to get it on their resume, and then leaving a mess behind when it gets them a new job. Leadership can’t fully abdicate the decision.

Lots of whom are here on HN recommending trendy technologies.

As intoxicating as they are, we need to move away from using analogies to problem domains most know little about to attempt to gain further insight into our own.

I like what you're saying, but your woodworking metaphor falls a bit flat for me because it teaches me more about woodworking than about building IT infrastructure.

Can small companies afford the complexities of k8s? I'm not yet convinced. Its pros are compelling yet the learning curve is steep. Which suggests a smaller talent pool.

Can small delivery companies afford the complexities of flat-bed trucks? Well: are they driving them? Repairing them? Assembling them? Loading/unloading?

You can bet that using a passenger car instead can be more "affordable", because more people know how to drive them, repair them, assemble, load/unload, they're cheaper, etc. However, if what they're delivering can't be hauled by a passenger car, or the logistics wouldn't make financial sense, that again changes the equation. There's no one answer.

So is everything not K8s a passenger car in this analogy?

Solution space is vast. We're all trying and weighing alternatives as we have the time and capacity.

If you use a good managed k8s offering, the complexities are not that great. GKE with Autopilot is a good option. In that case you don't need to know much more than how to write the yaml for a deployment. I've shown developers at all levels how to do that, it's not a barrier.

+1. GKE Autopilot and sticking to the core ~4 Kubernetes object kinds (Deplyoment, Service, Ingress), was a really easy way for us to get started with K8s (and in many cases can carry you a really long way).

Though easy to run, if there are multiple workloads of varying resource requirements, there will be a lots of wasted CPU and RAM, just because there are minimum CPU and CPU to RAM requirements.

Agree. I think they are working towards providing ISTIO (gateway replaces ingress, TLS internal communication, canary deployments, shadowing etc).

If they can spin up GPU or high memory nodes on demand with Autopilot that would be amazing.

There is also a new generation of platform that runs on top of Kubernetes that are emerging. Like Qovery

Why would you run it in k8s as opposed to, say, ECS? Honest question here, why not run it on something simpler that requires less new concepts and achieves the same results?

Because ECS sucks. It's slow and unergonomic, like a badly designed k8s with fewer features.

k8s is nicer to use and simpler if you ignore the complex bits. Plus it's the standard.

My experience of ECS is the opposite of yours. It integrates nicely with the AWS ecosystem and was substantially easier to use and educate others on. Would not hesitate to use again on either fargate or BYO EC2. I will acknowledge scheduling is not quite as fast as Nomad but I never found it 'slow'

Personally, I'd rather create a k8s deployment than an ECS task, but I can see your point. If all you want is an integrated with AWS experience, then it makes som sense that ECS is just simpler overall out of the box.

I don't think the delta to make k8s integrated is that much work with EKS, but ability to mutate the entire infrastructe if and when you do scale wins out for me. I think the complexity, most of which you can ignore, is worth the flexibility.

Either way, since k8s landed, AWS itself has started improving too.

I've used both and I would still prefer ecs/fargate to build a rather independent application and k8s to build a long-term platform.

A compelling reason is the large ecosystem of tooling that runs on k8s. Practically anything you want to do has a well maintained open source project ready to go.

For example? What could you do on k8s that you couldn't do on native aws?

Take a look at Kubeflow.org for an example. There are several reasons that a tool like that targets Kubernetes and not native AWS. One of the benefits of k8s is how portable and non-vendor-specific it is. Basically, it's become a standard platform that you can target complex applications to, without become tied to particular vendors, and with the ability to easily deploy in many different environments.

To be clear, I'm not claiming you can't do these things on native AWS, but rather there are wide choice of high-quality projects ready to go that target k8s.

  - Countless Helm charts
  - Development tools like Telepresence
  - Many GUIs / terminal UIs
  - CI/CD tools like Argo
  - Logging and monitoring tools
  - Chaos engineering tools
  - Security and compliance tools
  - Service meshes / request tracing
  - Operators for self-healing services
  - Resource provisioning
  - etc...

For a typical deployment, ECS isn't simpler than a fully managed k8s system, and doesn't have fewer new concepts. The wealth of concepts in k8s only comes into play when you're doing more advanced things that ECS doesn't have abstractions for anyway.

In ECS you have abstractions like task definitions, tasks, and services, all of which are specific to ECS, and so are new concepts for someone learning it originally. In Kubernetes a typical web app or service deployment uses a Deployment, a Service, and an Ingress. It isn't any harder to learn to use than ECS, and I find the k8s abstractions better designed anyway.

If you're already using ECS, and are happy with it, then there may be no strong reason to switch to k8s. But for anyone deciding which to use for the first time, I'd strongly advise against ECS, for several reasons.

One is that k8s has become an industry standard, and you can deploy k8s applications on many different clouds as well as non-cloud environments. As such, learning k8s is a much more transferable skill. K8s is an open source project run by the Cloud Native Computing Foundation, with many major and minor contributors. You can easily install a k8s distribution on your own machine using one of the small "edge" distributions like k3s or microk8s.

While in theory, some of the above is true for ECS, in practice it just doesn't have anything like the momentum of k8s, and afaict there aren't many people deploying ECS on other clouds or onprem.

Because of these kinds of differences, all in all I don't think there's much of a contest here. It's not so much that ECS is bad, but rather that k8s is technically excellent, and an industry standard backed by many companies, with significant momentum and an enormous ecosystem.

> Can small companies afford the complexities of k8s?

Where exactly do you see this complexity in Kubernetes?

I have a couple of Hetzner nodes running microk8s and I have a couple of web apps running in them. All it takes to deploy each app is putting together the kustomize script for the app and afterwards simple call to kubectl apply -k ${kustomize_dir}. I'm talking about specifying an ingress, deployments, services,... The basics. I even threw in a couple of secrets to pull Docker images from private container registries.

And everything just runs. With blue-green deployments, deployment history, monitoring, etc.

It's far more complicated to setup a CICD pipeline, and don't get me started on the god-awful mess that is CloudFormation or even CDK.

Where exactly did you see all that complexity you're talking about?

You apparently learned all about microk8s and spent time configuring it. I assume you did not get it right at the first time.

It is like Usain Bolt comming over and asking what is so hard for you in running 100m in ~10 seconds when you never left the couch.

It's just like any other tech; you read the docs, try it out, do some troubleshooting, and then you know how to use the tool.

I bet you could get microk8s running correctly on your first try. Give it a shot! Here's a doc: https://microk8s.io/docs/getting-started

You could probably get some additional nodes in your cluster on your first try too: https://microk8s.io/docs/clustering

This isn't Usain Bolt. This is normal people doing normal work with normal technical tools, and then somehow people keep claiming they must be some kind of world-class genius to have done it. Try it for yourself before you claim that you'd have to be a world-class peak performer to have installed and configured some simple daemons on a few linux servers.

Of course Usain Bolt is an exaggeration.

But you try it out, set stuff up from tutorial then do some troubleshooting put production data there and you get articles like these:

https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-...

https://www.zdnet.com/article/a-hacker-has-wiped-defaced-mor...

https://thenewstack.io/armo-misconfiguration-is-number-1-kub...

I'll try to take a different approach here than I did in my other recent reply to you.

I hear that you're saying that there are possible configurations that are insecure, and that at least some care and attention needs to be invested to avoid problems. I agree. This isn't specific to Kubernetes, as you show in your second link. This can be a problem, and people have in fact suffered harm due to leaving their doors unlocked.

On the other hand, most important security doors in most professional environments are not left unlocked and unmonitored.

If you have already learned basic sysadmin fundamentals, then you have the skills needed to learn to deploy and use Kubernetes just as securely as any other network service. The way that you learn to apply your general sysadmin skills to Kubernetes is by practicing with it. You can supplement your practice with books and training if you really want to, but it's not necessary. If you happen to have other people who have already gone through this process as support, that can help quite a bit. If there are any other better ways that people learn things like this, I have yet to hear of them.

If you don't already have those skills, then the way to build and develop them is exactly the same process. You try stuff out, read some docs, poke at things to see if you can break them. You can supplement this with classes if you like, but they're not necessary. Peers and mentors are great if you have them, but they're not necessary.

What alternative do you have in mind? What about kubernetes specifically is so monstrously complex? People keep asserting this, but I learned it just fine like any other nontrivial software I've ever worked with professionally. My peers at work learned it just fine like any other software we've worked with. My friends and colleagues I keep in touch with from previous jobs have learned it fine.

I don't really understand what kind of complexity bar you're trying to imply is just objectively too high to be reasonable? Yeah, it's got more moving parts than like Redis, because it does way more than Redis does. Sed is simpler than python, but that doesn't mean that you need to be Usain Bolt to learn python.

> You apparently learned all about microk8s and spent time configuring it. I assume you did not get it right at the first time.

What? With Ubuntu, microk8s works pretty much right out of the box.

The only thing you need to learn is how to install it with Snap.

What are you talking about?

> I assume you did not get it right at the first time.

I did, not because I'm a rocket surgeon but because it is really really that simple.

https://ubuntu.com/tutorials/install-a-local-kubernetes-with...

What exactly leads people like you to complain harshly about how hard it is something you never even tried before?

You're literally wasting far more time complaining in a random online forum about how hard a technology is than what it takes to not only learn the basics but also get it up and running.

You also had a pentest of your setup so you are perfectly sure you don't expose something to the internet that you are not supposed to?

You also considered updates for Ubuntu and microk8s so you have strategy for updating your nodes with newer versions and security patches.

I can follow a tutorial to set something up - but then there is always whole world of things that is never included in tutorials.

Just like kubelet accepting unauthenticated requests by default: https://medium.com/handy-tech/analysis-of-a-kubernetes-hack-...

If you're concerned that you've somehow accidentally exposed something to the internet that you didn't explicitly intend to expose to the internet, you can just do a trivial port scan. You just run nmap, look at the output, and you're done in like 30 seconds.

What does this have to do with kubernetes? "Don't expose stuff to the internet that you don't intend for everyone across the planet to be able to access" applies exactly the same to literally everything you could run on your servers.

This isn't remotely "Kubernetes is uniquely scary and complicated"; this is basic fundamental network security, and if you're not already handling this, then you need to go brush up on your basic networking fundamentals, not blame it somehow on kubernetes.

Almost every network service I can think of defaults to accepting unauthenticated connections, or connections authenticated with some default credentials. This is the normal, expected, default situation with network services. If you make the decision to expose something to the entire world, in a professional context, it is your responsibility to know the specific reasons it is safe to do so.

Are you really trying to argue that "Some rando decided to bareback the entire global internet with no firewall, on a personal home server, and didn't bother to type 'kubernetes secure configuration' into google, therefore Kubernetes is super hard and complicated and dangerous"?

It's not like this is some obscure cryptic detail; it's explicitly called out in the documentation that any half-decent professional would read before deploying a production service: https://kubernetes.io/docs/tasks/administer-cluster/securing...

  Controlling access to the Kubelet
  Kubelets expose HTTPS endpoints which grant powerful control over
  the node and containers. By default Kubelets allow unauthenticated
  access to this API.
  Production clusters should enable Kubelet authentication and authorization.
  Consult the Kubelet authentication/authorization reference for more information.

Yes, untrained amateurs sometimes do dumb stuff. Sometimes companies leave their S3 buckets open to the world. Sometimes people expose mysql to the internet with credentials they ship to users. Sometimes people expose unauthenticated Redis to the internet. This does not mean that these technologies are somehow fundamentally too complicated for mere mortals, it just means that it's dangerous to ask amateurs to do something in a professional context.

Try setting it up on your own without Ubuntu doing the legwork. Set up a 3 node control pane, the deployment servers and storage.

You come off as very arrogant who believes he knows everything, some humility would suit you well, but I think all pseudo smart Germans are like that.

I've set up quite a few kubernetes clusters on my own, and relied on the clusters I've built for production services at both startups and big tech companies. I've done quite a bit with both local storage and network storage via Ceph.

I am not German, and I have never been to Germany. If we're trading wild speculation about personal details, I think you could use some ambition and self-confidence.

> Try setting it up on your own without Ubuntu doing the legwork.

Why? Do you also see any purpose in hopping on one foot to work instead of driving there?

I don't understand what leads people like you to try to move goalposts to pretend something is harder than it is or needs to be.

It is not untrained amateurs it is also people who do stuff from tutorial and think they know everything.

So my post is not about Kubernetes per se - but about narration "it is super easy 6 year old could do it", well no not everyone can do it and one has to spend time with any new technology.

Besides nmap in that scenario is not helping as well, beacuse I have to expose port 443 to serve my customers and Kubelets expose https endpoints. If someone runs simple nmap scan sees 443 open and concludes all is correct because he will be serving https websites - so your "you are done in like 30 seconds" seems like shooting oneself in the foot.

Hmm, interesting, I may have been misreading you.

I agree that 6-year-olds and other people without any production sysadmin or SRE experience are going to have a pretty bad time learning to build and deploy a Kubernetes cluster.

My point is that any professional sysadmin or SRE can learn Kubernetes just fine. Yeah, there's a lot of stuff, but there's just about as many moving parts as I expect for a system that handles what Kubernetes does. You also mostly don't have to pay complexity cost for many optional features you don't care about; you can get a minimal cluster up, and then grow it as you need more features.

I don't follow what you're saying about port 443. The kubelet API is not listening on port 443 by default. I'm as confident as I can be without checking that no kubernetes components listen on port 443 by default.

Speaking more broadly, I agree that someone with no SRE experience and no network security experience won't get much value from 30 seconds of nmap. What I was trying to say is that "accidentally exposed the kubelet API to the global internet" is something that I expect a competent sysadmin to be able to detect and notice with 30 seconds of nmap.

When I'm saying "deploying kubernetes is fine", I'm saying that anyone who has any business running nontrivial production services in a professional setting will not have any trouble learning to use and deploy Kubernetes. Deploying a cluster does require competence with sysadmin or SRE fundamentals, but not particularly more so than other systems that handle similarly-complex topics.

Also, any junior sysadmin or programmer should be able to learn to use an already-running kubernetes cluster to deploy basic services with no trouble and just a bit of time. I have trained quite a few people on this, and it really does go just fine.

> You also had a pentest of your setup so you are perfectly sure you don't expose something to the internet that you are not supposed to?

What are you talking about?

With Kubernetes you need to explicitly expose something. In code. And apply that change. And see it explicitly listed in the descriptions.

Outside of Kubernetes, you're already talking about a requirement that applies to all web services, regardless of deployment solution. Why didn't you mentioned that?

> You also considered updates for Ubuntu and microk8s so you have strategy for updating your nodes with newer versions and security patches.

What point were you trying to make?

Do you believe Kubernetes is the only software that requires updating?

Even so, with Kubernetes you can launch a freshly created instance in your cloud provider of choice, add it to the cluster, and go on with your day.

If you'd want, you can drain a node, shut it down, and rebuild the node from scratch.

Where exactly do you see a challenge?

A lot of people in this job just really hate learning anything, and would rather spend way more time spread out over months and years than just investing some time and learning how to use something new.

It seems like somehow some people get into this job by only learning tools that they can pick up without trouble over a weekend?

The concepts you need to deploy stuff on kubernetes really aren't that complicated. It's just a bunch of yaml documents in extremely-well-documented schemas. If you want to run a service with N instances, you just write a deployment with `replicas: N`.

There are a lot of details I'd choose slightly differently if I were designing my perfect ideal cluster orchestration system, but the whole point of open source is that everyone who wants to build a comprehensive cluster orchestration system can just get together and collectively build it once, so I don't have to design and own it all internally. It's got all the pieces to build exactly what I need to make good use of a ton of computers, in a simple, reliable, repeatable, consistent, standard way. It gives you trivial primitives to build HA fault-tolerant deployments.

There are very few good excuses left to ever have any reason to page anyone over "One server had a hardware failure".

It just baffles me that people can see this powerful, industrial-grade, comprehensive tool, and decide "Nah, that'll never be worth starting to learn".

I half agree with you here. It is actually quite simple, but everything in Kubernetes is very explicit, which is good, but also intimidating. If you've never worked with Kubernetes before then it's a lot of added complexity without clear benefits.

> If you've never worked with Kubernetes before then it's a lot of added complexity without clear benefits.

Unless you're someone who only had to work on a monolith deployed to a single box somewhere, Kubernetes adds zero complexity to the problem you're already dealing with.

In fact, Kubernetes simplifies the whole problem of running stuff on a cluster. Network, security, deployment, observanility... That's all provided out of the box. And you can rollback whole deployments with a single command.

Heck, even ssh-ing into a container, regardless of where it's running, became trivial.

How is that harder than deploying stuff to boxes somewhere?

>Unless you're someone who only had to work on a monolith deployed to a single box somewhere,

I think the point is that 90%+ of websites are fine with a few monoliths behind a load balancer. That set up can handle low thousands of requests per second in Rails/Django/etc. Maybe low 10 thousands with a more performant language.

And it's not just k8s. It's the whole microservice/SOA that comes with it. It ramps up the complexity of everything and is a constant time sink in my experience.

A few years, I would have said no. Now, I'm cautiously optimistic about it.

Personally, I think that you can use something like Rancher (https://rancher.com/) or Portainer (https://www.portainer.io/) for easier management and/or dashboard functionality, to make the learning curve a bit more approachable. For example, you can create a deployment through the UI by following a wizard that also offers you configuration that you might want to use (e.g. resource limits) and then later retrieve the YAML manifest, should you wish to do that. They also make interacting with Helm charts (pre-made packages) more easy.

Furthermore, there are certified distributions which are not too resource hungry, especially if you need to self-host clusters, for example K3s (https://k3s.io/) and k0s (https://k0sproject.io/) are both production ready up to a certain scale, don't consume a lot of memory, are easy to setup and work with whilst being mostly OS agnostic (DEB distros will always work best, RPM ones have challenges as soon as you look elsewhere instead of at OpenShift, which is probably only good for enterprises).

If you can automated cluster setup with Ansible and treat the clusters as something that you can easily re-deploy when you inevitably screw up (you might not do that, but better to plan for failure), you should be good! Even Helm charts have gotten pretty easy to write and deploy and K8s works nicely with most CI/CD tools out there, given that kubectl lends itself pretty well to scripting.

the devil is in the details, they all look easy on the surface but there are so so many traps you can step into casually that they really aren't solutions for the k8s being too complex problem.

I think the upsides to k8s are relatively minor (if any) for most businesses, and the downsides are significant (tech debt and complexity). It's sometimes sold as revolutionary tech, but it's really just an incremental improvement, and the next incremental improvement will come along soon enough.

DevOps is a never ending yak shave.

I think so, as long as you're using a managed cluster on a cloud that has automatic handling of Ingress and LoadBalancer. My first production experience with it was at a start-up with 3 initial engineering staff. We already knew how to build docker containers. A basic helm chart was easy to create and push to the container registry. All we needed to do was set the kubeconfig and helm install v0.1. Even if you're not sure how it all works under the covers, your tiny team is pushing to dev cluster several times a day, and your dev site is online (you can hide it behind Cloudflare Access for free). Extras like Cluster Autoscaling, Horizontal Pod Autoscaler, ExternDNS and cert-manager are easy to add when you need to. Anyone can fumble-install Grafana/Prometheus/Loki and have metrics and logs. At that point you have elastic infrastructure, dashboards, alerts, TLS, LoadBalancers, DNS... How long would a small developer team take to do this without k8s? Imagine trying to figure out ECS, Fargate, Lambda, ACM, CloudWatch logs, alarms and metrics? And how much AWS-specific code is in every single one of your services to get it to work on native AWS?

> Imagine trying to figure out ECS, Fargate, Lambda, ACM, CloudWatch logs, alarms and metrics?

In case anyone is curious, we went that route.

- ECS + Fargate + CDK took one of us two (8h/day) weeks for the initial setup. We've sprinkled a few more days here and there since then.

- Cloudwatch logs are "setup-free" (your containers' logs get sent there by default when using CDK constructs).

- ACM... we don't use directly. CDK will easily setup TLS-enabled (with AWS-emitted certificates) ALB (Application Load Balancer(s)) for you.

- Lambdas we don't use much.

- Metrics & Alarms are easy to set up but they generally suck. The custom language for computed metrics is clumsy and quite limited. The anomaly detection sucks. And it is expensive even by cloud standards (don't create metrics and alarms willy-nilly or you'll feel it in the next invoice).

- Our application code doesn't know much about AWS (we do use libraries for S3 and SES, but these are just easy to swap adapters).

- We ended up with ~3k lines of CDK definitions in typescript. These are very easy to read, and only moderately hard to write (you do need to look up the docs). However, I can say without a doubt that it's been the easiest infrastructure definition/description language that I've ever used.

I don't have enough experience with K8s to know whether that route would have been better or worse, but I can say this route hasn't been a pain point for us.

Good to hear. CDK is a huge plus.

The complexity of Kubernetes is overstated, so I think they can. However, it doesn't mean they should. Personally I would just start with a docker-compose.yml and run it on a VPS. In the starting phase you probably don't have much traffic anyway and docker-compose can nicely progress to Kubernetes when you need it. Another upside is that any developer can run the entire stack locally on their machine, which a huge upside as well. This means you can fully focus on producing working code and don't have to bother about infrastructure too much.

Then again, I've never worked in a startup so the above approach is purely theoretical. Curious about what other people think.

I agree that "the complexity of Kubernetes is overstated". Kubernetes itself is actually pretty simple, very reliable, and more mature than it seems. The complexity and challenges of Kubernetes come from all of the add-ons that may not be necessary in most situations.

Vanilla K8s is pretty good. But when you think about admission controllers, policy engines, service meshes, progressive rollout, etc, you are increasing the scope.

Start with k8s, and hold back the temptation to solve 10 other problems with 10 other projects from the CNCF sandbox. Once you have a good system running, really evaluate the complexity of each new solution with value it provides, and make a decision. Say no to most.

Yes, while I worked at a bigger company, our team of 15 developers and 5 SRE managed to build a few self hosted Kubernetes clusters for our microservices, a pipeline and also set up cloud development for fail over. We used Data Dog, Consul, Elastic Stack and a few SQL and NoSQL dbs.

I mostly self learned and was helped a bit by guys who started before me. The rest self learned, mostly by doing, solving issues, reading articles and tutorials.

> if your people aren't very skilled, they won't build anything well

If your people aren't very skilled, making them manage a ton of unnecessary complexity is not a good idea.

k8s is not a saw. It's a hovercraft. Expensive to maintain and very few people need one.

> a software engineer knows as much about cloud architecture as a fine woodworker knows about framing

A mobile or frontend engineer, perhaps. But the idea that implementing and architecting the same type of object (backend services) are mutually exclusive skillsets strikes me as ridiculous. Since architecture decisions are by definition highly consequential and difficult to back out of, you want your most skilled and experienced software engineers making them, but it is precisely their software engineering experience that makes them qualified to do architecture. I would never trust a so-called "cloud architect" who is not also a first-rate backend engineer.

The thing is, Architect is its own role and discipline, and Cloud Architecture is distinct from Software Architecture.

For example, you have to know why using PrivateLink is a bad idea, what kinds of backups to what locations/accounts you can do with an encrypted RDS database with a CMK, the 50 different ways you can screw up access to an S3 object, the limits of how many IPs EKS can use per node, why somebody should use the M5dn instance type, when rolling+maintaining your own complicated expensive service might be way better than using a cloud-managed version, what services you cannot use in what regions and zones. You have to be a budget wizard, keep up on the latest solutions, and know how all this dovetails with cloud security.

A Software Architect should not have all of that Cloud trivia rattling around in their brain, but it's essential for a great Cloud Architect. Conversely, a Cloud Architect doesn't need to be a great coder to design great Cloud architecture. (Similar to how a construction architect doesn't need to be a great framer to know how the code affects framing, and how a great framer doesn't need to know how to calculate bending moments and shear forces in laminated beams to plan what day they'll need a lift for the beam and how to attach it)

This is so spot on. Developers and “software engineers” seem to have a pathological habit of wanting to bring in gee-whiz technologies that they only have a cursory knowledge of and actually do very little to make the products and projects better.

TL;DR:

- if you want to build something yourself -> you need experienced ppl to do that

- if you dont have the right ppl -> you better pick a SaaS or PaaS, but your business will be at mercy of your provider

- there is a huge shortage of experienced skilled ppl

Kubernetes is not a saw. It's a crane.

Reply on news.ycombinator.com