Hacker News Reader

Comment by warent | Hacker News Reader

When consulting for a seed funded startup, I suggested they buy their own servers and colocate them. It could save them almost 100k a year.

Everyone looked at me like an alien speaking a different language, lol. Then was politely dismissed, even though I'm experienced in running hardware.

AWS, GCP, Azure, really managed to expertly pull off the greatest heist of all time. They're useful for sure, but somehow they've gotten everyone to drink this potent marketing kool-aid that if you're not using them then you're not serious about success. So cringe.

View on HN

They'd have to keep you employed to manage the hardware. What happens if you get run over by a bus (which I never hope will happen)? What happens if you decide to move to another city? There are a lot more "what ifs" by employing you to build and manage the hardware.

Saving $100k would not be worth the hassle/risk for most well funded startups.

Again this is the kool-aid speaking. The co-location manages most of the complexity and the probability of failure is basically zero for the first 4 years.

Hardware these days is really good.

People have this illusion that servers are extremely hard to maintain because big companies constantly have to maintain their 1000s, and because clouds are incentivized to sell this lie.

It would still be a huge cost saving and trivial for them to contract someone to maintain the servers for like 1 hour a month with emergency on-call. Of which there are numerous such people who are not me.

I used to build PCs. You can get the same hardware for nearly half the cost of pre-built PCs like HP or Dell. I did a lot of research on the best bang for the buck parts and the most reliable parts. If something broke, just pull it out and replace it. Easy.

I worked in an IT department as an intern. I thought this company should build custom PCs for all their workers. You can get a much faster CPU or much faster GPU for the same price! It's a no brainer!

Then I realized that if I wanted to build custom PCs for all the workers, I'd have to provide support for all of them. I'd have to be as good as CDW, who was supplying the company with PCs. This means next-day repairs and replacements. Drivers. Security. Firmware updates. For hundreds of PCs.

Needless to say, a lot of people under appreciate all the things that vendors like AWS or CDW provide. They think they can do it because they know how it's done. But they're ignoring a lot of other factors that can come back to bite you.

I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.

> I'm sure you're very competent at managing server hardware. But if I'm a seed stage startup, I'd politely decline your offer.

It’s not either AWS or “bring your soldering iron”. Self-hosting K8s on managed hardware is still insanely cheaper than AWS.

GitOps, infrastructure as code, DevOps platforms, CNCF best practice setups are entirely doable without paying the Amazon tax. If your lead DevOps quits, you’re right, it’s easier to find someone who does AWS.

The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs.

It’s not much different than outsourcing DevOps, which I think is what some hackers frown upon, because that’s their skillset you buy in town at a premium.

Running K8S is itself a giant tax. The complexity is absurd and the components are not as reliable as having a literal team of people who are responsible for ops and making a service feature complete.

> The last time I worked for a company that was invested in AWS, their AWS budget corresponded to five full-time senior developers in running costs.

This means nothing to me -- what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment? How flexible are the different systems for the future? What are the reliability/uptime requirements and the marginal cost of outage? How replaceable are the dev ops engineers who are running your K8s/managed infra?

People moving to cloud isn't irrational. There's a reason that it's pervasive.

> what is the TCO of the AWS deployment vs. the TCO of the non AWS deployment?

I can only speak of one PHP/TypeScript app with a poor performance profile: from $250/mo to $5.000/mo in running costs, not counting the migration. The team that made the app couldn’t maintain the Helm chart or the Terraform resources, adding to a DevOps team bottleneck that appears to limit how teams innovate (kind of how like creating new branches was expensive on CVS).

> How replaceable are the dev ops engineers who are running your K8s/managed infra?

I haven’t met a competent AWS DevOps engineer who wasn’t also competent to run the clusters; it’s just that they prefer not to.

> People moving to cloud isn't irrational. There's a reason that it's pervasive.

Yes, it’s made really solid. AWS makes sense to reduce operational risk at a large scale.

The price is not my main argument against using AWS, although you’ve gotta make good money to justify the expense.

My motivation is to own one’s own infrastructure. I like to work on things that are both important and contented enough that you can’t, as a principle, outsource the operation.

You can obviously buy servers from HP or Dell. I have about 80 in a colo datacentre.

Power, air-conditioning, security etc isn't my problem. That's what we pay the colo for.

I go there about once a year to install new servers, and about once at some other point in the year to replace a failed disk. I've taken various developers who are interested each time, so if I'm out of town there are several people who can cover, plus the colo staff.

You can also pay for people to do this routine work.

And then there's still the option of renting managed servers, so that company handles all the hardware.

To be fair, this sounds a lot of to do items that I don't want to do.

I think it's nice to have a completely different type of work a couple of days per year.

I've found there's generally developers who are into building custom gaming PCs or Raspberry Pis or whatever who are interested to see the servers and lend a hand installing new ones.

I actually am into hacking at home but I'd definitely not play around with my business' production environment. Wouldn't know where to start if sh* hits the fan.

Not sure why you're being downvoted. But I love building PCs too. There's no way I'd want to put together my company's server hardware though even though the basics are the same.

I do not want to be on call 24/7.

How are the basics the same? No one builds their own server.

That's why managed servers exist.

You can't get much more time efficient than data centre operators handling hardware.

In the last few years, build-your-own storage got annoying. Most server disks were SATA, mostly the same form factor and SATA ports are available in abundance. You could get various more complex SFF-xyzw connectors, but the adapters were straightforwardly available.

Now there’s NVMe and there are quite a few form factors. And NVMe is generally a point-to-point link to the CPU, so you either need a motherboard with the right number and types of connectors or you need a PCIe/NVMe splitter or switch. And the latter don’t seem widely available on the non-sketchy-hardware market.

So the situation is fine for system integrators who make their own PCBs, but not amazing for DIY builders.

Well, it depends on the colo.

I mean, it's not like a colo doesn't have issues. At one point our colo was running some cable and we discovered that their cables to our rack were wiggly...after our stuff went offline intermittently. That was hard to track down.

And you need to set up the LOMs, the management network (hey, you don't want that on the public internet), VPN, serial cables (backup to network LOM), maybe a load balancer. And you have to hope that when you reboot the box it doesn't fail POST. And you have to remember to turn off power saving in the BIOS (how many people forget to do this?). And you have to set up the RAID 5/10.

And you have to do this sitting on the floor, because not a lot of places provide chairs in the racks. Oh, do you have a rack-mounted KVM/monitor/serial console or are you using your laptop with a serial cable to do all this?

I mean, it's not hard, it's just a lot of detail work. And I suspect that these skills are going away.

These are great points, thank you for the insight. A small note I will add is that these days there are remote BIOS options available for servers like iDRAC

Do they need a reboot to take effect? I haven't touched a LOM in more than 8 years!

DRACs are a Dell specific term for what is generically known as IPMI or out of band management. They were quite well developed when I worked on servers well over a decade ago now. The good ones are basically as independent of the system they're attached to as possible and essentially give you the equivalent of physical access to a server.

I know what DRACs are (it's Dell's LOM). I asked whether the DRAC bios stuff required a reboot.

Just looked at the docs, it's apply and reboot which makes sense.

Oh, I forgot that you also have to manage the certs on the LOMs. Luckily browsers still allow you to bypass expired certs by default.

I wonder if you can still get into machines with the old SHA-1 stuff?

That's why serial consoles are better IMO.

Yeah, I'd consider the DRAC completely independent of the BIOS question. There are also OS tools - I believe Dell's is called OMSA - that can flip some of the switches in the BIOS (I remember this from having to mess with NUMA settings at one time), but they still needed to reboot.

As far as the crypto stuff, it's an issue. I fairly recently had to tweak some old 11th generation Dell Servers and tried out the DRACs, but Java really doesn't want you to use RC4 - I remember that being the dealbreaker. Nicely, someone put a conatiner up on Docker Hub that dealt with it https://hub.docker.com/r/domistyle/idrac6/

I have rarely needed to make changes, but if I recall correctly there were some settings that would take effect immediately, and others that required a reboot. I'd need to refer to the documentation.

You write with authority, but then admit you haven't used this stuff for 8 years.

8 years ago you could already set up RAID or change BIOS settings using LOM, and KVMs were obsolete. The slightly janky Java remote access tools have been replaced by HTML 5 ones, and at least HP and Dell provide tools to script setting up a fleet of similar machines.

When I set up new servers, I photograph the sticker with the serial number, MAC address and LOM password, mostly so I can be sure exactly where each machine has been racked. Connect power, ethernet, LOM. Check the DHCP server log on the LOM network to see there are the expected number of new MAC addresses present, then go back to the office to finish the setup.

I agree that there's a point where co-locating is a good thing to think about. But I don't think $100k is it. To put it in perspective, $100k is probably how much an intermediate engineer costs a company (not just salary).

In exchange, you give up the usual cloud conveniences like easier budgeting, easier scale-up (and scale-down), lower up-front costs (which is a big deal for most early startups), better SLAs, being able to use cloud object storage without paying transfer fees, access to support, fewer things that can go wrong, etc.

Buying your own server requires an upfront investment. AS 90% of start-ups fail, hiring your computing resource flexibly is far better, even if it comes at a higher cost. Once your company is mature and relatively stable, it is worth considering if hosting your infrastructure is more cost-effective.

But now you have a culture that has no experience or value with self hosted hardware. How are you going to fix that cultural problem? Hire a bunch of outsiders? How’s that gonna fly?

Bringing your infrastructure from Cloud to on-perm is a project; you would have to hire a dedicated team. Once you have proven your start-up is providing a genuine solution, it is a lot easier to get funding for this.

And you haven't had turf wars where people conflate the issues and costs so badly that nobody really knows what the truth is anymore?

Apparently I'm getting your share.

You're missing several points on the spectrum.

You can provision dedicated servers with Terraform.

They can still be cloudy in the way you deal with hardware maintenance and risk of failure, and how you authenticate with them and how you name them, and how you hook them up on VPNs. But they're physical units without the AWS bells and whistles or the AWS premium. Not everyone needs their own colocated rack, and manage their own UPS and network peering.

The main problem is OPEX versus APEX, not whether you can Terraform your infrastructure. It depends on your business model whether self-hosting is more cost effective than the Cloud. Cloud is more flexible and as a result you pay a premium, however, the cost differences are not as large as claimed in this article

I think you're missing GP's point: On-prem/self-hosting vs. cloud aren't the only two solutions. You can also simply rent a managed (physical) server, like many people did before The Cloud became a thing.

OPEX vs. APEX is an interesting aspect.

It's covered more elsewhere in this thread, and I appreciated learning about it.

I wouldn't say that renting managed servers is simple at scale.

But it is both simpler and less expensive when not at scale.

> I wouldn't say that renting managed servers is simple at scale.

…unless they are not physical servers but vservers. Hetzner, for instance, has such a "cloud" offering where you can dynamically spin up new vservers via an API/Terraform.

They represent yet another point on the spectrum but are not "fully cloud" yet if we take "fully cloud" to mean that, beyond hardware, the software stack is partly managed by a third party, too.

Put differently, to me the spectrum looks more or less like this:

- On-prem self-hosting (own data center, own hardware etc.) - (Off-prem) non-managed hosting (data center provided by third party, but you're providing the server hardware and put it in one of their racks) - Managed hosting of physical servers (data center + server hardware provided by third party) - Managed hosting of vservers (Hetzner Cloud, AWS EC2, …) - "Full cloud": Software stack is partly managed by 3rd party (e.g. managed Postgres, managed Kafka, managed ElasticSearch, serverless, etc.)

This is a very good point. In other words, it's a classic Capital Expenditure vs Operating Expense. For startups it's usually more efficient to have a higher OpEx in the beginning and focus on optimizing OpEx later.

I can recommend this article for those who are interested - https://www.cloudzero.com/blog/capex-vs-opex

So don't buy them, rent them. There are plenty of points on the spectrum between buying your own HW and EC2.

If your startup survive next 6 months it is already more cost-effective to buy a server. And if startup fails you still have a server to run your next startup.

Not necessarily.

https://www.itpro.co.uk/cloud/public-cloud/369521/singapore-...

Reply on news.ycombinator.com