Hacker News Reader

Comment by zyztem | Hacker News Reader

Desktop processor without ECC memory plus very slow SATA hard drives - not a setup for reliability and performance

I don't get it for 90% of startup this setup is more than enough to start and far cheaper then goog, Amazon, azure. Just the problem is most modern devops are lazy to learn the basics of setting a HA system with proper backup. Hertzner make it easy but one needs to know what they are doing. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).

most modern devops are lazy to learn the basics of setting a HA system

Absurd nonsense.

So what, I save $100 over Google but spend 3h a week futzing with configs?

I’ve run the numbers with Hetzner, GCP, AWS, and even more managed products such as Firebase, Heroku, etc.

Bare Metal compared to GCP or AWS will save you about 75% of your operating costs.

Bare Metal compared to Heroku or Firebase will save you around 90% of your operating costs (a large part of this being caused by bandwidth, which is massively overpriced at Google and Amazon).

With those savings you can usually serve 5 to 10 times as many customers.

That might not be worth it if you’re in SV, and pay your devs the same wages Amazon or Google do (as then you’ll just pay more than with AWS or GCP), but if you’re in places where you pay half the wages Amazon or Google do4 , you can actually save a lot with this arrangement.

I am really reluctant on setting up my own server cluster on bare-metal right now, can you recommend some specific open-source tools to manage and monitor these clusters?

Kubernetes on CoreOS is a good option. Tectonic makes the setup part relatively easy.

Ok, thanks a lot. Since we're on it, is there any open-source software available that is specifically targeted at easily setting up storage servers (maybe with an S3 like SDK)? I found myself pretty much forced to use GCS because of a lack of good alternatives.

I haven't used it personally, but Minio [0] is something I've heard mentioned more than once.

[0] https://github.com/minio/minio/blob/master/README.md

Minio is very nice, but sadly it only allows one set of credentials – I’d have preferred being able to separate the access permissions of each service, but that’s not supported.

You will likely save a lot of headache to just use AWS or GCE. Running your own kube is a multiple week project.

I was a total noob at this, and started setting up a kube cluster on 10th of May, and I managed to set up a working cluster by the middle of the week. It’s actually not as complicated as expected.

I still agree with you, because no one should take setting up and running kube lightly – it’s actually a lot of work.

This is good, it means kube is about 100x easier to set up than openstack.

On the other hand, Google will run your Kube for free up to 5 nodes.

There’s actually even a hobbyists guide for running kube yourself: https://github.com/hobby-kube/guide

And there’s a few guides for setting it up automatically in vagrant, or on aws, or on Juju-managed clouds.

Sadly, Kube is very barebones – only doing scheduling, if you add a storage provider such as portworx (only block storage) or rook (block and object storage) you also gain that, but it doesn’t do any user auth functionality.

That’s something I’m atm still searching for, something allowing me to easily register and remove users, add them to groups and do RBAC, but without using ldap as backend, and with a way to get it working with OpenID Connect and OAuth 2.0, as well as with Shibboleth, Kerberos, and an LDAP compatibility layer (but with app-specific passwords there).

You can give Proxmox a try too.

Total server spend on my last startup was like $150 a month on GCE. So saving 75% is useless if it cost me even 2h.

Fwiw I started on DO and migrated off to get better stability and block storage / proper load balancing. Which DO has since added.

You mist have fit into the free egress tier. Maybe something that could be cached by Cloudflare or similar?

The egress costs for AWS, GCE, and Azure are usually the big cost driver.

Was a b2b app with ARPU of say $100. Our total all in hardware cost for that same user was maybe $.10.

So saving on server cost literally did nothing. Our employee cost where 20x, 100x our hardware costs.

Not all startups aim for millions of free uses you will make money off of later.

Definitely. But if hardware is the only expense that scales per user – as is with most B2C apps, and even many B2B products – the difference between GCP and bare metal can make you last a lot longer on the same seed round.

(The show Silicon Valley actually showed that part quite well)

Except as I told you not always.

If we optimized and saved $100 per month on hardware at the cost of even 1 engineer hour, it would be a net loss.

I think a LOT of startups are very simple from the hardware level. You can go real far on just a few servers. And at that level, who care who manages them? Do what is easier.

Sorry to break your bubble a vm with this configuration+bandwidth will be $1000 or more. So it's 1/10th to 1/5th over Amzn,goog or msft. Also if you are a startup you need to understand technology driving it. Blackbox doesn't help, what's necessary for goog isn't first priority for startup.

> Sorry to break your bubble a vm with this configuration+bandwidth will be $1000 or more.

But why would you buy a VM with this configuration? 32gb of RAM for two boxes, with two i7s that have 4 3.40ghz cores? Lots of people have the impression that cloud is more expensive than owning your own iron, and it probably is, but the difference is made a lot larger if you don't consider TCO and you make no effort to actually adapt your needs to the cloud. Cloud's economics are based around horizontal scaling and provisioning on-demand. If you're hosting in the cloud, you don't get two big machines like this, you shard your app over a bunch of cheap machines and you spin up more/bigger instances to meet peak demand. If you're buying cloud hosting but you go into it with "have everything we need to meet peak load running 24/7," of course it's going to cost you an arm and a leg.

Most startups I know don't need anywhere near that full-time processing power. So what does it matter if it's 10% of the price if it's 85% more than I need?

Our Buildsystem (Jenkins) needs CPU and RAM. Couldn't bother less with SATA Performance.

Frontendserver also: Only needs CPU and RAM.

depends, if you do have a database heavy test suite, you probably need fast disks. our application really needs a good i/o performance. I tried to run 3-4 runs on the same machine with containerized cpu/memory, however as soon as 3 tests are running the i/o latency/performance goes numb, only thing that helped was either using different disks or faster disks.

Reply on news.ycombinator.com