Hacker News Reader

Comment by tommek4077 | Hacker News Reader

Hosting is under 70k a year. You can get good sysadmins for under 35k/year? Minus the actual costs of hosting?

420k a year is the total for the organization. It includes salaries, administration, etc.

Cost breakdown: https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSk...

Just because you are using AWS/GCP/Azure doesn't mean that you don't need sysadmins. They are called DevOps engineers then.

I scaled my SaaS pretty far with no sysadmin nor Devops, nor any plans to acquire either. I know enough to setup some cloud servers, but I’m not qualified to manage barebones Linux at scale. The hacker news go rent a Hertzer trope is rather tired. If it was better, why aren’t more people picking it?

The type of people that colocate or generally avoid the cloud for cost reasons in my experience don't overlap a lot with the "I write blog posts about everything I do" crowd.

Because often someone else is paying for it (organization/investor/government) or the money cannot be used for something else. Another common reason is as a way to circumvent internal politics.

In my experience, whenever someone actually had to "foot the bill" for the AWS premium, your argument that the cloud is cheaper overall came under a very heavy scrutiny, including justifying those extra "9s" that increase administration effort.

The budget lists system administration cost at $12,875.29. Presumably that's not a full-time sysadmin. And there's no line-item for a devops engineer.

So administration costs aren't zero today, but I suspect they'd be higher if they moved off the cloud.

25% for site administration and moderation.

judging by the datacenter names in that spreadsheet they are hosting at OVH and most of the money already goes into salaries: https://docs.google.com/spreadsheets/d/1Si3PMUJGR9KrpE5lngSk...

Based on datacenter names they seem to be hosted by french provider OVH, not google nor aws

You need at least three admins to have 24h availability. I highly doubt you can save money by selfhosting at Lichess size.

Hosting on the cloud does not magically make your availability issues go away.

No, but they make many availability issues Someone Else's Problem.

When EC2 goes down, I need to maybe wake up and log in to see what happened. I see that all of us-east-1 is down (again), notify everyone involved: "It's down can't do anything, it'll come back up when AWS engineers get to it" and go back to bed.

I can check up on it again during office hours.

If this was a colocated server, I'd need to drag my ass out of bed and drive to the site to see what's up.

Drive to a colo? Most colos have 'remote hands' where you can have an onsite tech do stuff for you. There's also OOB management tools you can use (assuming the hardware isn't toast).

AWS is really expensive if you have the skills to managed hardware yourself.

Remote hands are expensive, and in my experience in using them at DC's all over the world, they're also not very competent certainly not to the point of being able to diagnose and fix issues for you. The best you could hope for with them is normally having them get OOB working so you can do most of the work yourself, and maybe try to swap some hardware if you can provide them documentation on EXACTLY what to do.

Remote hands are often contracted from online marketplaces for such people. I do this occasionally as it's easy money. We do exactly what you tell us to do without any insight into why we are doing this. My objective is to be on-site for 15 minutes and get you back into your equipment. It occasionally takes up to 2 hours to just get past the datacenter security. The most useful docs are a series of pictures on your equipment with labels. It is also helpful to tell us how to safely disconnect uncommon connectors.

See that last sentence there. You have to pay to get those skills, that's why Amazon can charge for it.

Well, we're also at a position where "You won't be fired for using Amazon". Suggesting colo at all gets you the same looks and derision suggesting PostgreSQL used to.

It makes the availability issues largely someone else's problem.

If YOUR server(s) go down for whatever reason, you're getting up at 3am and driving to the data centre to put the fire out assuming you know how to troubleshoot the issue (lets face it, most Devs probably don't since its not their specialty), and have the means of fixing it (failed RAID card? Hope you've got a spare). Then theres the headache of when the data centre is on the other side of the world - get ready to cough up hundreds of dollars to wake up remote hands, and try to get them to do the work for you (probably pretty poorly).

If AWS goes down at 3am, you simply roll over, go back to sleep and have another look when you wake up.

Of course this does all depend on the software, cloud infra being built sensibly but the same also applies for "on-prem" solutions as well.

There is plenty hybrid approaches. You can have cloudflare loadbalaner and enough redundacy to roll over and go to sleep. I think you would have less outages with dedicated hardware than aws control plane. For some compute or IO intense workloads it would make sense to self host.

You can outsource administration for rented severs.

Pretty much any outsourcing shop you hire will be able to do basic Ansible and monitoring and DBA tasks.

Look at the link. They are co-locating at OVH (or running dedicated servers). Which is as close to self-hosting as I can imagine.

Four, actually, if they are to have off-work days.

> Maybe stop using aws or google cloud?

Or, maybe start. Would at least be worth looking at to see if [the equivalents of] spot or reserved instances, Lambda's, Cloud Run, or other serverless etc. might save some money.

People don't realize how efficient a colocated server is. There's a few websites sending about half a petabyte a month of data and are spending far, far less than these hosting costs.

> Maybe stop using aws or google cloud?

That would save them $179 per year.

Don’t forget the $0.56! It’s almost $180/yr of frivolous cloud spend they could shave off…

Are they actually using AWS/GCP for their servers? Lichess.org's IP address seem to belong to OVH.

Reply on news.ycombinator.com