Hacker News Reader

Comment by endisneigh | Hacker News Reader

The development of things saddens me. (re: this whole thing about serverless and edge).

It should be that it's becoming easier to have a robust database (or job queue, or full text search, etc) on your premises, not have more and more go to the cloud.

Though it is true that managing your own infrastructure can be challenging, it is not inherently so. One can easily imagine self healing and self managing infrastructure, physical hardware deterioration and replacement aside.

EDIT: I want to clarify that I'm not denouncing the cloud. My point is simply that it should be easier to also manage your own fleet if you so choose. This could mean using VMs, bare metal or what have you. Even managing a fleet of VMs is nightmareish today.

View on HN

> it is true that managing your own infrastructure can be challenging, it is not inherently so.

The biggest challenge when you have on-premise infrastructure is battling managers to get an additional server.

I have seen managers laughing at my team leader when he asked for a server to do some QA. We needed the server to perform some tests for a critical product. They even had the audacity to say "Hey Manager Bob, how long have you asked for a server?" Manager Bob answered "It's been month's now"

Granted I think that's an extreme example but I don't doubt it was a common occurrence back in the day. Now I also have worked on enterprises where asking fora server (Or RAM increment in a given server) is as easy as submitting a ticket and waiting for a FEW DAYS.

Cloud is more expense on the long run, but it sure gives you agility to do things.

There is a middle ground between the cloud and on premise. You can get a VPS or a dedicated server from a traditional host. If you have a running tab with them, which is common for operations like this, you get your servers in a few hours, sometimes minutes.

Yeah the cloud is a lovely weapon against the BOFH. They smarten right up when you shrug your shoulders and say "Oh, I guess I'll just buy some cloud."

oh, your luser account is 'letsdothisagain', great thannnnks. I'll take care of that right now.

You don't have to go all in. You can have your own servers and rent when you need additional capacity. Best of both worlds.

There is some development on that front too. For postgres, check out CrunchyData's postgres operator (https://access.crunchydata.com/documentation/postgres-operat...) where with a single config file you get a HA database with read replica(s), connection pooling via pgbouncer and multiple backup targets via pgbackrest with cloning and point-in-time recovery

You can also use pgvector with langchain to create a weird version of search if so inclined.

I personally like the Zalando operator better, you can add databases and users by updating the CRD, feature parity between the two on HA is pretty good

https://github.com/zalando/postgres-operator

Can you explain how can physical hardware concerns be handwaved away?

It's not just about the hardware itself, that's the easiest part.

It's about building and maintaining the electrical/cooling/network infrastructure the servers require. It's about having multiple locations of that in case your building's power or network goes down/there's a flood/somebody steals all your stuff/whatever else.

Or you rent a rack in a colo (or five)... But that's just a step away from going full cloud, you just do a lot of work yourself and get no guarantees and no flexibility. At that point, why not just go full cloud?

This is not the 90s - monitoring and alerting are far more robust, and supply chains far more just in time.

There’s no reason a modern fleet couldn’t detect failures in hardware and automatically order replacement from Amazon or whatever. This is not to say you won’t need personnel, but you would be surprised how much of the fleet management at AWS and GCP at least are brute forced (as opposed to being completely automatic). To their defense they have a far more complicated and diverse fleet than what I’m describing, which is sort of my point.

99% of customers just need a highly available setup for their stateful boxes (DBs) and some for their stateless services. And the great thing about this setup is you can extend the stateless bit (with the cost of latency) to the cloud more or less infinitely.

As far as the rest of your point about cooling and network - it's a good point, but for a workload that isn't a datacenter, in my experience the majority of outages aren't due to that - it's due to misconfiguration.

Take a personal house. How often does someone break into the average person's home? Pretty unlikely in a low crime area. Take internet. There are some areas in the United States where there are multiple 1Gbs or even 10Gbs internet providers, you could redundantly network. In any case it's not that the cloud shouldn't be used, it's just interesting the direction things are going.

Of course, if money isn't an issue by all means people should use serverless and cloud spanner and be done with it.

> This is not to say you won’t need personnel

Ding ding ding, we have a winner!

Paying premiums for BigCloud means not paying salaries, not worrying about an outage that could happen while your seniors are on vacation, not needing to set up resilient internal processes and controls for datacenter management, not worrying about datacenter management compliance, etc. etc.

Needing to hire datacenter personnel is a problem and it is a problem that BigCloud solved.

You're thinking about it the wrong way. Sure you're not paying salaries, but now you're paying for experts who understand the cloud and you're paying more for those people compared to datacenter people. At Azure I was surprised how many data center people didn't even have a college degree and how little they were paid. Ultimately the only thing that matters is the total cost. To take true advantage of the cloud you need to use cloud specific technologies - serverless being the greatest example. Cloud VMs are neat and all, but in a sense it's the worst of both worlds.

I'm not saying there isn't value in the cloud. I'm saying most people are overpaying. BigCloud didn't solve this any more than paying employees in general to solve your problems. With the era of free money coming to the end, many companies will come to this realization themselves in any case. BigTech clouds have 50%+ margins - something will give eventually.

Any serious company is still going to have to pay oncallers, and admins with or without cloud. It's not like using the cloud absolves you from maintenance (pretty much every company with a valuation more than 100 million has an infrastructure team, and uses the cloud. So clearly the cloud doesn't mean you don't have to maintain your own infrastructure). And now we come back to my point - why isn't the fleet easier to maintain to begin with? I'm not even talking about bare-metal necessarily. Say you use VMs. Still a huge hassle.

> BigTech clouds have 50%+ margins - something will give eventually.

Yes, the oligopoly, and the vendor lock-in.

Certainly won't be expecting the "advantage" of being able to outsource your sysadmins to somewhere else at the drop of a hat to be the thing that gives.

How often do people actually migrate clouds?

Vendor lock in at this level is not really an issue for most, trying to be agnostic will create more problems and time sinks

> Paying premiums for BigCloud means not paying salaries

You need all the same people to manage the same software stack, regardless of whether the VMs sit on hardware you own or at an AWS rack. Nearly all the work is on the software side.

The physical bit of maintaining the boxes is a minimal percentage of the work. At various startups we usually didn't have anyone hired to do this work because there wasn't enough to justify a person. Most simple changes (like pull and replace a hot-plug drive) can be done by the colo personnel and for larger work someone would drive to the colo maybe once a month. You only start to need dedicated hardware maintenance personnel at a very large scale.

The premium you're paying for cpu and bandwidth at your BigCloud is so enormous that it'll easily pay many more salaries than the people you need even if you reach the scale of needing dedicated hardware people.

Yep, in the old days (almost 20 years ago now), I worked at a startup with its own racks. We'd go up there once a month to swap some drives. In my 5 years there, I had to use remote hands exactly twice to reboot a machine. A few other times, one of the servers froze but we were able to reboot it with ILO. This was 3 racks, about 30 machines spread between them (definitely a bunch of empty space, too.)

Happy you, I was in the same situation but I was in the server room daily. Practically every day something went wrong.

Either you had thousands of machines (at which point is totally makes sense to hire a dedicated person with all the money you're saving) or the site had environmental problems (dirty power, bad cooling, etc?) causing the issues.

Hardware is unbelievably reliable, systems normally run for many years without any attention.

How many systems did you have? What sorts of problems?

I am genuinely curious about both the software and hardware stack if you were there every day due to something going wrong.

1. First of all, a good bare metal setup leaves much of the software fleet management able to be done remotely. So if you were there because of that then clearly this wasn't done right. I'm talking about SSH access at a bare minimum, and ideally out of band access as well.

2. I'd say the things that go wrong in a data center are plentiful, but should be predictable. Power, network or cooling related issues means you picked the wrong site or your vendor screwed you. That leaves us to the actual hardware. Sourcing good hardware is obviously critical. Modern Dell and HP enterprise machines should be giving you very, very reliable hardware assuming the beforementioned concerns are addressed. It is true that disks fail, and sometimes memory, but if you were there literally every day there is just a critical failure in your setup somewhere.

Even if you had a data center with literally a thousand machines. An uncorrelated failure happening every day is so unlikely that it's not really worth mentioning. Correlate failures could certainly happen. Bad batch of disks, etc, but still shouldn't result in you being there literally every day. I could imagine a series of bad days though, sure.

You are right, but this was also solved by "dedicated server" folks years before cloud was a thing.

On GCP, they will live migrate your running VMs to new racks and you won't likely know unless you look for it in the logs, super nice feature, I once got bit by AWS and a disk physical failure, had to migrate myself (really a nuke and recreate since we couldn't do anything with the VM)

> It's about building and maintaining the electrical/cooling/network infrastructure the servers require.

Except at the extreme high end of scale, you never do this yourself. Rent racks in a colocation. It's a very clean separation of duties. You handle all the tech work, they handle all the real estate upkeep.

> Or you rent a rack in a colo (or five)... But that's just a step away from going full cloud

What? Not even close. When you rent a rack in a colo you have 100% of the benefits of owning the hardware (much higher performance, much lower cost). Just because you let someone else do the A/C and generator maintenance (and other non-computer real estate logistics) doesn't make it anything at all like the cloud.

> physical hardware deterioration and replacement aside.

I mean... if you ignore the hard parts of self managing then self managing is in deed easier lol

Self managing isn’t hard just because of that. You could use VMs with your cloud provider of choice and my point stands.

I'd argue that has gotten hugely easier. The problem is we have gotten hugely more demanding as well, so it ends up still being difficult.

While there are lots of cloud providers also trying to be edge providers, I would argue that "The Far Edge" is a great opportunity to improve and develop infrastructure that is easier to run on your own hardware.

What worked for cloud won't work for edge in most cases. Having to do processing locally means processing on your own hardware.

We've seen this as the reason a lot of companies are adopting tools like NATS.io to be more cloud agnostic and/or be able to practically run at the Edge.

> This could mean using VMs, bare metal or what have you.

For bare metal would something like Equinix fit?

> Even managing a fleet of VMs is nightmareish today.

What about fly.io?

fly.io has had alot of reliability problems in the past, with lots of users on HN and elsewhere talking about a whole host of problems from disappearing VMs to bad backups to poor uptime.

I love their model, but execution lacks.

That said, if they can get a handle on reliability and uptime, its extremely compelling

Reply on news.ycombinator.com