Hacker News Reader

Comment by nostrebored | Hacker News Reader

“Everything You Expect from a Cloud, Running on Your Terms“*

Except you own ops, management, extension, interoperability, access, security, scalability, redundancy… words cannot express how ridiculous all of the koober propaganda is

View on HN

The way I've described this for years: Kubernetes makes managing 1000 servers as easy as managing 20 servers, and makes managing 3 servers as easy as managing 20 servers.

Most products barely need 1 server.

And if you ever outgrow that it's going to be a huge pain. Or a hardware failure. If you start on Kubernetes early you'll be able to add more servers for capacity very easily. Not to mention out of the box you get failover and HA. And you can deploy managed services and have database deployments, or object stores, etc.

But 99% of the time you don't outgrow it and don't have SLA's requiring them to have failover or HA. This is why so many sites can get away with using PostgreSQL with it's complete lack of good/native HA.

You could always run a VM / VPS against a managed DB. Many small cloud / VPS providers, like Digital Ocean or Vultr, also offer managed DB services that are cheaper than AWS RDS.

HA and failover, to me-in these small deploys, is more about hardware maintenance rather than maintaining SLAs. Being able to shutdown the computer hosting your containers to scale vertically.

Single-site-2-absurdly-intense-days failover, yes. With weeks of cleanup afterwards.

What every company seems to want: multisite, multimaster immediate failover, no.

Also kubernetes buys you scaling. Compute. Disk. Database (with help). Etc.

Now I rail against companies for wanting that, and I think you're right. Your webshop does not need that. It has so many moving parts the redundancy will cause more outages than it solves. But you can do this, and so people will pay for it.

It is a technical accomplishment.

And with sufficiently good sysadmins, it can work well, and scale.

My way of thinking about it is this: you have your own hyper-flexible Heroku, but (monkey's paw curls) you can only interact with it by typing large amounts of YAML.

Oh, and all the documentation for that YAML assumes you've memorized as much vocabulary as a Foreign Language 101 class.

(And there is a mad god that says: if you try to use click-ops to get around this without knowing the vocabulary, you're going to have a bad time.)

But on the other hand: to put it in terms of the "3 servers" - the moment you think you'll have 3 servers, and any level of uptime expectations, you'll inevitably have to rebuild them, services and logging and all, from scratch often and quickly enough that you might as well have 20 servers with how stressful that rebuild will be.

k8s can be a saving grace there, and I recommend it to anyone with the time and interest in how cluster best practices work! But it's not a free option or a weekend skill-up.

All things people used to own 10 years ago. It’s not like the people doing that stuff have vanished.

Cloud’s big promise was speed to market and price, and let’s be honest, price is no longer there compared to a decent operation.

The one thing where clouds remain kings is speed for small teams. Any large enough company should probably Ask themselves whether running their own operation using ias would be a better choice.

My company is on prem, spending north of 1 billion per year. Cloud is actually cheaper when considering total cost of ownership. Thats salaries, opex, capex costs. Worse, our speed to delivery is generally worse.

Because on prem is inelastic, we are at sub 10% peak utilization of compute resources. If we added in the likely higher cloud utilization rate we are talking of 30%+ savings from on prem.

> we are at sub 10% peak utilization of compute resources

so... you bought way too much hardware?

That's not unusual. First off, sometimes 1000 extra dollars will get you a ton more compute you need so why not and second, on prem tends to be extremely inelastic so you buy a ton of compute because you never know when compute requirements change.

If we're talking on the scale of $1,000s then it's cheaper to run on-prem than in the cloud. It's really easy to spend $1,000 on managed kubernetes and have very little actual compute.

Peak Utilization is a tough one for on prem and is a decent argument for cloud. I was at a company that was also at <10% peak utilization most of the time. It was finance, so it was mostly doing nothing, except for the couple days a year where we shot up 10000x, so we had to build for that case. So yeah losing the data centers, and cloud was a cost savings.

So the "obvious" (but complex!) solution is a "hybrid cloud": use on-prem for the predictable, constant "DC component" of your demand, and use cloud for cyclical or unpredictable demand changes. That will keep peak demand decoupled from permanently provisioned capacity while saving on always-utilized capacity. Easier said than done, of course.

The route we ended up going, was hybrid cloud with a colocation for on-prem (Mainframes and a few servers), and then AWS for the cloud portion. Not sure on what the cost savings were. As I wrote the authentication service I knew what those numbers were, so I knew that on a normal day we would be sitting at 1% cpu usage basically all day except market open/close. And then fiscal quarter ends were a big bump. And then of course big news days. 99% of the time for 99% of the days were just nothing.

K8s helps reduce that complexity a lot.

My comparison was cloud vs renting metal, not buying your own machines.

You have to pick your battles. Most of this stuff isn't necessary to babysit until you're scaling your app tremendously. And by the time you're doing that I'm sure you've got the people to do these things.

And then why would you need koob at all? All that setup and learning on a platform you don't understand and won't need to manage and you will do it wrong, so a completely wasted set of time and money afaict.

IMO, there's no alternative private cloud.

Exactly. https://www.reddit.com/r/kubernetes/comments/u9b95u/kubernet...

It's crazy to me how many people deploy unmaintainable spaghetti mess in all other environments I've been in. "koober" environments are the most organized.

I find this to be an insane argument. I have never seen a k8s environment that wasn’t a garbage heap. But everyone is eager to tell you what a nice can they’ve found next to the rotting banana.

Onboarding new team members? A disaster. The design? All done by one dumpster diver with nobody to call them out on the mistakes because they have no idea what the hell is happening.

I’ve never seen a k8s shop where there weren’t a few principal engineers being roped from incident to incident because teams couldn’t manage their own.

With cloud providers, even ones we don’t use, people have a general idea of how the primitives work. With any reasonably skilled team you get back to talking about how your product works instead of talking about persistent volume replication for Postgres backups or the differing file system behavior or what cli everyone should be using to manage your entire company’s software or how to stop someone from deleting the entire of everything you own or or or

Ops cost to number of server to manage is logarithmic but cloud cost is linear, so there's an intersection it starts to make less sense for cloud. Also equipment depreciation is tax deductible whereas cloud bill isn't. A year of EC2 instance bill is comparable to buying the equivalent server

Also there are vendors renting out datacenters so you do less of hardware management.

Having worked at two companies spending 250M+ on cloud bills alone, they try hard to decouple from cloud but many things are vendor locked

Hybrid has been the answer to both. It shouldn't be a binary decision. stateless compute workload can fairly easily be offloaded to private cloud.

> Also equipment depreciation is tax deductible whereas cloud bill isn't.

Genuine question out curiosity (I have a master in finance, but never practiced it) -- aren't both the cloud bill and depreciation all tax deductible, eventually? the bill 100% in that year and the depreciation spread over multiple years?

> Hybrid has been the answer to both. It shouldn't be a binary decision. stateless compute workload can fairly easily be offloaded to private cloud.

Can you elaborate on that? I'm studying for saa-c03, and I was shocked by how expensive egress out of aws can be.

And particularly the upgrades every 3 months. Not just your nodes and masters, but every operator you use, and your manifests each time they deprecate a manifest beta version.

Ive found nomad to be a much simpler replacement for smaller scale deployments.

It’s a well known thing that if you run on ec2 they handle all those things for you (especially the security part)

I wonder how many people will miss the sarcasm. ;) Cloud means you don't need any ops people, right?

They do manage almost all of them for you. If you’d like we can get some of the EC2 engineers on our team to tell you how.

If you don’t think IAM is a better set of semantics for securing your infrastructure than nightmarish k8s rbac i really don’t know what to tell you.

Can you think back to your batch and honestly tell me that the companies that used k8s are in a better place?

Reply on news.ycombinator.com