Hacker News Reader

Comment by ic0n0cl4st | Hacker News Reader

This past year I’ve finally been able to hit the $200/hour rate as a k8s consultant.

I feel like I’m selling snake oil. I’m working on my 7th implementation and none of my customers need kubernetes.

Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using.

My largest implementation was a $3m/year, 9 year old company who could run their entire infrastructure on a couple dozen ec2 instances. However they are under the delusion of “hitting it big and needing to be able to scale 100x”.

Paid off my student loans this year, and soon my house, thanks to superfluous k8s integrations!

View on HN

I don't think this kind of attitude should be encouraged. Imagine other professions celebrating things like that.

Surgeon: Some of my patients request bladder surgeries while in reality their problems could be solved with a few simple diet supplements. But I just do the procedure and take the money. Just bought my 3rd Tesla, thanks to superfluous bladder operations!

I know selling people shit they don't need is celebrated as an achievement nowadays, but feels like it shouldn't be.

Tons of work done in various industries is "shitwork" that the person doing the work wouldn't normally do except they're being paid to do it. At some point you give up arguing with the customer.

The dieting is a perfect example - many health issues can be solved by losing weight but few people want to make the lifestyle changes necessary to do that.

Is your "shitwork" related to "bullshit jobs"? https://en.wikipedia.org/wiki/Bullshit_Jobs

> few people want to make the lifestyle changes necessary to do that.

I didn't have the will to make lifestyle changes until I had a serious bout of sciatica. I needed that external motivator of torturous pain to be able to commit to changing my eating and fitness habits.

Two and a bit years later I'm down 12Kgs and slim and healthy (but, ironically, still have back pain issues that requires minor surgery to alleviate).

"Don't worry, it's only for the rest of your life". If you can come to terms with that, you're ready to start.

Dude, sciatica was my catalyst for lifestyle change as well. I had a bicycle accident 14 years ago and damaged my piraformis. It was the most humbling and painful experience of my life.

But that's exactly what surgeons and accountants and mechanics and consultants and entertainers do. Most work is unnecessary. Once you embrace that, it is a lot easier to understand how this economy works.

I can confirm this. I was offered surgery for my 2 year old for a problem that is not life threatening and could possibly (but rarely) cure itself. In the end it indeed cured itself.

Sounds exactly like the medical industrial complex in the US. Opioid crisis and general over medication and over treatment just magically happened / keeps happening. Surely no doctor ever thought let’s give the customer what they are asking.

Most people keep themselves from recognizing the problems with what they are doing if they are making lots of money off it, they convince themselves what they are doing is worthwhile after all.

"It is difficult to get a man to understand something when his salary depends upon his not understanding it."

I don't think (I hope?) anyone is celebrating "cheating the rubes". But I respect GP's ability to be honest about it -- it's super useful to the rest of us. For instance if we've been skeptical of k8, seeing someone who's an expert in it say that makes us think ok maybe we're on to something.

Most people in that position would keep their mouths shut, and probably keep themselves from admitting to themselves what's up.

I've learned to not question what the customer wants. If I push back and think "you probably really want solution X over the solution Y you're asking for" more often than not it turns into a big fiasco. Every issue that comes up becomes my fault for suggesting X and not going with Y. Where as if I just do Y as they asked then they associate the issues with Y and not me.

Sure, that's great for self-preservation. Can be done similarly with "what the boss [or uber-boss] wants".

It leads to so much crap, but it's great for self-preservation.

What do you think cosmetic plastic surgeons do?

How many of the (mostly web-based) technologies that we all love to discuss on HN are any different?

Nothing I do in this world is life or death. Hell, my customers neither, mostly they sell or support advertising in one fashion or another.

Some founder I work for might make billions on products people don’t need, why shouldn’t I make $50k off of him? Isn’t that how capitism works?

I think so. If someone sees enough value in what you provide to be willing to pay you, you shouldn't feel bad about it. It's not like you're pushing them to adopt k8s. They already did that.

It's like someone bought a Mercedes. Nobody needs a Mercedes, they could drive a Toyota. But now they need an oil change, and that's what you do, and they'll pay you for it.

I'm not sure I need to engage my imagination very far. Why do you think that's not exactly what happens?

It already happens in medicine. Of course, a lot of it is "I'm afraid of getting sued so I'm going to cover my ass with all these tests." but, still.

When I was younger I had a pain in my abdomen I had never experienced before. My grandmother told me to go to their family doctor who was probably older than they were. He pushed on some things and tapped on some things and finally diagnosed me. "It's gas. Go home and have a big fart." Sure enough, hours later, a big release and the pain was gone.

These days I'd have been sent in for at least an MRI or CAT scan to go along with all the blood work that they would do.

Now, I know what some of you are going to say. "But it COULD have been something way worse!" Sure, it could have been. But this guy had been a doctor for a long time by that point and knew from experience that it was gas. Just like when I recently had to rebuild a really old server and was getting errors that suggested missing class files but, after 25 years of doing this, I knew that it was actually missing a PHP module.

Less is more and experience is golden.

There are plenty of stories of the opposite where the old doctor uses their intuition and it fails to a basic checklist. Instead of seeing temp 101, symptoms A+B+C = X they see (looks young and healthy, can't be X because X is only for old people). Turns out it is X

I mean one of the most famous stores is of experienced doctor refusing to wash their hand because they know better based on their "years of experience"

His reason for why they don't need it was they are 'under the delusion of “hitting it big and needing to be able to scale 100x”.'

To be fair, as a kubernetes consultant, he's probably not qualified to determine whether or not they will hit it big.

What he probably is qualified for is to determine if they were to serve 100x their current load, would they be well served by kubernetes.

You are describing plastic surgery, no?

I’m confused. Isn’t a kubernetes cluster just a way to deploy software onto a couple of dozen ec2 instances?

Sure it has some accidental complexity, but so does having half a dozen different software systems each managing their own deployment pathway onto ec2.

There seems to be some belief that kubernetes has to be some kind of crazy complex system to operate, when really it’s just an api for managing which docker processes get spun up.

Most tools are easier than people make them out to be. The average dev just isn't that great, and the average company overestimates the difficulty of these tools, while burying their devs with mindless drone work so they never get the time to study these things. The average dev isn't one which has studied CS/software dev and/or spends loads of free time still investigating tech off the clock, either.

> when really it’s just an api for managing which docker processes get spun up

On GCP, the goal can be achieved without kubernetes, by deploying containers on VMs and MIGs:

https://cloud.google.com/compute/docs/containers/deploying-c...

Comparing that to kubernetes, the latter is indeed some kind of crazy complex system to operate.

The same can also be done either on GCP with non-COS images or on AWS, with a little bit of automation: https://news.ycombinator.com/item?id=26274033

Right. I said k8s is a way. Not the way. There might be others. They might have their own trade offs.

> Comparing that to kubernetes, the latter is indeed some kind of crazy complex system to operate.

I think the fair comparison here would be that to google's kubernetes engine. Getting a kubernetes cluster up and running in GCP is really easy.

> Isn’t a kubernetes cluster just a way to deploy software onto a couple of dozen ec2 instances?

Sure. And a nuclear power plant is just a way to generate electricity.

I am always baffled by this take. I run k8s because it's by far the easiest way to do what I want (deploy a handful of services to a handful of servers), not because it's cool or massively powerful.

In the past few years, k8s simply got easy to get started with and it feels like no one noticed.

If it was easy to set up a nuclear power plant, would that be a great idea for anyone to set one up and run it?

You will run into problems eventually that you won't be able to solve (especially as you have more complex requirements). If this is just for a pet project you probably won't care. But if there's a lot riding on you being able to fix it quickly, you will not enjoy being in a position where fixing it takes hours to days.

It's like a car. Super easy to operate, until there's a strange sound, and then something falls off. Do you know how to fix it? Do you need to get to work in 30 minutes? Can you just "order an Uber" when it comes to K8s?

I’m not sure what justifies the analogy of kubernetes to either a nuclear power plant or a car you mysteriously have no ability to fix, nor why the existence of an ‘Uber for kubernetes’ seems so outlandish (I mean there are literally api calls I can make to cloud platforms that will summon new kubernetes infrastructure for me to rent, in minutes, which is pretty Uber-like).

It’s not rocket science, it’s just networking, config management and container orchestration.

All these analogies are equally true of just.. running Linux in the first place. Or writing software. Using things that require some knowledge requires some knowledge.

Over-simplifying it does not change the fact that it is literally the most complicated system that exists today to just run a web server. The most complicated system of anything is going to be more difficult than the alternatives.

Sure. If ‘running a web server’ is the extent of your problem then kubernetes is a very wrong turn (in fact, so is ‘spinning up a Linux box’ - nobody needs to be writing an nginx config file to just put some static content on the internet).

I feel like this is just an argument between completely different communities. Like a recreational yachtsman saying ‘I don’t see why anyone should need a nuclear power plant to sail a boat’ to a naval attack submarine captain.

Yes, but the other way around: a naval attack submarine captain (Me) telling a recreational yachtsman (potential K8s users) not to put a nuclear power plant in their sailboat. I think some of us get that it should only be used sparingly as a big investment, but there's so many more using it just because it seems like they should.

I'd assume anybody criticizing kubernetes isn't a huge docker fan either. I see both used much more as dev ops tools rather than real cluster management.

Particularly questionable in organizations that may not have any in-house tech expertise. Some contractor, who may be the only programmer involved at a given time, comes in and sets everything up the kubernetes/docker way. From now on you'll always need kubernetes-enabled devs to maintain your thing.

Happened to somebody I know. They believed they had a "cluster" with different "servers" but it was a single machine. Then I had to explain why they cannot just FTP into the server, upload plugin folders (like some tutorial asked) and expect it to work. Nor did I have enough time to help out since who knows how these dockers mount folders, what's the latest "best practice" or what the dev who maintains it prefers. There's no way they'll ever need more than one machine. I'm pretty sure the system in question does not even have a good cluster story in the first place, so at best you could scale the database (which usually can do clusters well without 3rd party tools).

"Basically all of their developers and devops engineers want kubernetes on their resumes or because it is shiny and what the cool kids are using. "

That's what I am doing now. Knowing this stuff makes you much more employable and raises your salary. It's a stupid world but resume driven development is a rational thing to do and very encouraged by hiring practices. For a long time I thought the company would appreciate making things simple and cost efficient but when I look at the latest hires it's pretty clear that the company wouldn't even hire most of the people who keep the business running because they are "outdated". You really have to keep churning constantly or you fall off the bandwagon.

On the other hand I am pretty new to "modern" server and web development and the complexity is just insane. Getting a simple API server and frontend going with comprehensive testing, CI/CD, containers, dependency injection and whatever is a huge effort. It barely leaves you time to think about writing efficient quality code because you are so busy with pull requests, writing test code and other stuff.

Firstly - congrats on making great income by providing valuable skills to a hot market.

When you say "none of my customers need kubernetes" is that because they are not using any of its benefits, or because they could do it another way and kubernetes was one of the options rather than the only option?

In my experience very few people are fully exploiting everything k8s has to offer, but many people are getting side benefits from greater standardisation of deployments, consistency for implementation of logging, tracing and telemetry, reproducibility of environments and to an extend talent retention and development.

I'm very interested to understand where the line is for genuinely getting zero benefits of using k8s.

You are the real job creator, as there will be more work after your migrations than there was before

Thank you for this!

What in your opinion is the optimal way to run "a couple dozen ec2 instances"?

I'm kind of in that zone.

I don't think "totally manually" works out very well once you are in even "one dozen EC2 instances", it's too easy to make mistakes, you spend too much time (with downtime) figuring out what mistake you made and what it was supposed to be instead. Generally too hard to orchestrate changes without downtime.

But is there anything in between totally manual and kubernetes?

I think that's one reason people are reaching for kubernetes, they actually are having challenges that need a solution, and can't figure out what the solution is other than kubernetes.

It’s unpopular, but I embrace (AWS) lock-in for small and mid-sized infrastructure. AWS’ various services solve 99.9% of any infrastructure needs. High-bandwidh streaming and heavy gpu processing is one of the few places my customers find to be cost prohibitive.

For compute I lean towards an immutable infrastructure. I was a Puppet / Chef engineer for 8 years, never again. It’s cleaner to configure apps via some mixture of pre-baked amis, docker files and some sort of parameter store. (Usually SSM or Vault).

Fargate is a cleaner path but so much more expensive.

Thanks for response! Sure, I'm down with AWS lock-in, for the sake of argument.

But how do you manage this EC2 fleet and collection of other AWS services? Changes happen -- new version of your local software, change your mind about some configuration, new version of a database, whatever.

OK, it's immutable. But I still need to launch a new EC2, and get everyone else that's supposed to know about it to know about it and start talking to it instead of the old one (or get public traffic to go to it or whatever), and then take down the old one once it's safe to do so. (I also need to figure out how to construct it as immutable).

Purely manually? Just manual operations in the AWS console?

There may be an obvious answer here I am missing not being familiar with the tools. What can be overwhelming here is figuring out the "right" way to orchestrate this stuff, there are so many options (or sometimes you can't find any options but suspect you are not looking in the right place) and so many partisans. One reason for kubernetes popularity is the succesful awareness marketting, I've heard of it and know it is supposed to solve these challenges I'm having and don't really understand what the options are and am overwhelmed trying to figure it out... but k8 seems popular, it must be good?

Your CI pipeline can build an AMI image with integrated Docker image, and a simple systemd script to start it on boot.

Deploy step would just be to upgrade AutoScalingGroup with the new AMI, in a rolling fashion with CloudFormation.

---

However, in most cases I'd recommend going with ECS + Fargate, where you can just deploy your Docker image, and don't have to care about the underlying virtual machines.

---

If you want to go K8s route, EKS + Fargate, (or if you want to go cheaper, EKS + EC2 Node Groups), are the best way in most cases, but you need to be aware that you will have to keep an eye on upgrading EKS versions, Helm charts, and other resources you add to the cluster.

I'd say Kubernetes is a good tool if you need it, but otherwise I'd start with something that doesn't require any maintenance on the ops side.

This. I'm not some Unix sysadmin type who wants to do everything manually. But the AWS features work out of the box for basic deployment, scaling, etc. and Kubernetes seems to just add unnecessary operational complexity without any value if you don't need any other features and are okay with AWS lock-in.

The canonical AWS answer here is: Elastic Beanstalk.

If you use a CI tool, it probably has a CD counterpart which can pipe a passing build directly to EB.

Or you can deploy manually from your repo with "eb deploy".

In either case, AWS will automatically bring the app up on new instances, and test them. If everything looks good, the load balancer will direct live traffic to the new instances, and then spin down the old ones.

Config changes use a similar process.

Why not use AWS's managed k8s and eliminate needs for your job? I can't actually think what k8s consultant would do with managed k8s.

Obviously a personal opinion, but Ansible solves this problem really well. It does get sluggish once we're talking about more than 50 machines, though.

To reliably end up with a working cluster:

- spin up plain operating system VMs

- spin up one more VM for Ansible instigator

- collect the IPs, write them down to Ansible inventory

- pull/upload your custom binaries

- run Ansible playbooks to set the VMs up

Sometimes it's beneficial to have a two-step process. One playbook to install rarely changing components. Then make a VM snapshot. The second playbook installs your custom binaries. Future installations can then start from the snapshot.

There are lots of ways to skin the cat.

Back in the day I worked on code to create custom Linux ISOs with network provisioning. Always wondered why more people didn't do this. You just need a kickstart script. With a simple provision script you can have a fully automated install. No messing about copying ssh keys or setting IP addresses, etc.

I rarely see instances where kubernetes actually makes sense over just using regular AWS services (load balancing, autoscaling, AMIs, etc.) directly. It seems like k8s just adds a mostly superfluous extra layer of abstraction with a rather user-unfriendly interface.

I'm in your position, but in my opinion, there are plenty of who are better in kubernetes even if they don't need it. There is complexity in the processes and even the scripting people use to glue things together, and there are enough places for the team to screw up.

I've been on the fence about this. If you don't mind, how did you get started?

Oh, I can tell you because I work with these asshats. They come in, deploy the technology, usually get promoted and then leave. The people left behind have to maintain their garbage and because promotions are only given to new feature work we're the suckers holding the bag.

Actually This is exactly how I accidentally started consulting on k8s. Three of my customers had engineers who advocated for k8s, did 50-75% of an implementation then left.

I have told all of my customers they don’t need this, but they are convinced this is the way of the future.

"Right side of history" is a powerful motivator, for better or worse. How did you get the first customer?

Right side of history? I dunno about anything like that. It’s just business, show me the money.

There will be a new trend next year or the year after, ad nauseum.

I’ve been consulting on and off for over 20 years now, I dont even remember my first 10 customers anymore. I get a lot of referrals from customers and from a consulting company I sold.

I was primarily focusing on terraform, which is the only technology I’ve actully ever enjoyed professionally, but there’s just so much stupid $$ in k8s.

Better delete this ;)

Is this one of those rare instances when honesty isn't the best policy?

Doubtful. The consultant who is willing to say that straight (and with the experience to know it) is going to be worth a lot more than $200/h, to the right clients.

Worth, yes, but valued? Now if I were looking to hire someone, I might come right back to this thread.

Reply on news.ycombinator.com