Hacker News Reader

Comment by PaulKeeble | Hacker News Reader

If you end up in the lightsail world and you are not utilising Amazon's other services then its probably cheaper to do this with another provider. Someone like Contabo or Hetzner will get you VMs at substantially less cost with similar fixed hardware and just give you the box but for a lot less a month. At these low scales with a completely open source stack Amazon isn't good value in my opinion. It is as you grow and it provides the scale that it becomes valuable.

View on HN

AWS initially was never meant for this type of workload.

It started with epiphemeral instances, available for a fixed workload, billed by the hour, such as batch jobs.

It turned out people were willing to pay $100 a month for a box worth $30 a month.

Machines provisioned by API were not mainstream in 2006, the "cheap" way to host was shared hosting, and dedicated / cpanel boxes were more common than VMs. Actually, pretty sure "vservers" were a big thing at the time and kind of fit where cheap VMs do now.

AWS was a very different product and not really comparable to other things available at the time.

Also the elasticity and backing by a company that had seen the scale themselves, and can possible deliver for you. I remember being a then startup and boxes costing more than 2x the money we were somehow attracted to it and kept it as the option we would move if we ever had a revenue.. Large companies probably spent more than their AWS bills on software license fees (Oracle/ERP/SAP etc) and employee salaries so they don't care.

Linode started in 2003. Just because there was no DigitalOcean shoveling money into the free credits pit of venture capitalism doesn't mean that affordable VPS services didn't exist. According to https://www.programmableweb.com/api/linode, Linode started offering an API before 2008.

DO had one of the best ads, or a least one of the best targeted / positioned ads.

"You've been developing like a beast, and you app is ready to go live"

https://www.youtube.com/watch?v=vHZLCahai4Q

Or DigitalOcean, which also offers many of the big-cloud perks like managed db, object storage, load balancers and kubernetes.

There's one critical thing that DigitalOcean, Linode, Vultr, etc. don't provide though: multiple data centers in the same region ("availability zones") with automated cross-zone failover and private networking between zones.

You were pre-conditioned to believe that it's a feature. Traditionally, datacenter itself is supposed to provide HA for both the power and the network.

"Availability zones" exist for AWS convenience, not for yours. It allows for cheaper and simpler DC ops, removes the need for redundant generators, simplifies network design, and makes "under-cloud" maintenance easier. It's a feature for them, a headache for you, and a (brilliantly addressed) challenge for AWS product marketing.

I don’t know about you, but I’ve been in several DC power failures where the fault was in the transfer switch.

It sure is nice to have separate failure domains with low enough latency between them to pretty much ignore it in application architectures.

Colos will rarely lose power, but having your line cut due to a backhoe is pretty common. Even in top tier facilities I observed some loss of service every 6-12 months, add in some misconfiguration risk and Colo failure becomes a frighteningly common affair.

This can be mitigated through redundant service providers, careful checks on shared interconnects and other measures - but having "hard" failure isolation at the facility level will also get you there with less chance of someone doing something dumb.

This kind of thinking is how you end up in a newspaper article where you're in a building in new york babysitting a generator during a hurricane while everyone sane is serving from Atlanta now.

You're doing it wrong. Plan to lose sites. If you plan to never lose a building, you're just setting yourself up for pain by optimizing for the wrong kind of redundancy.

I disagree. AZs are completely independent data centers kilometres apart. For any businesses which may need low latency but still want full HA (e.g. finance systems), it's a blessing. This requirement cannot be fully covered with separate data center regions, something like an airplane crash would still hit everything.

BinaryLane in Australia have VPC across datacenters

Not gonna lie, I have been scared to deploy to DigitalOcean after that debacle [0] posted here last year. It's worth the premium to me not to worry about that at night. Yes, AWS could shut me down too, but the probability seems lower.

[0] "Digital Ocean Killed Our Company", Hacker News, May 2019. https://news.ycombinator.com/item?id=20064169

Yep, now that DO has managed DB instances it’s become my go to provider for most setups.

Droplets (VPS) have generally worked out cheaper than EC2 (with more resources) for me.

The bit that initially sold me on it a while ago was no more worrying about CPU credits, ie consistently maxing the CPU out at 100% then getting throttled, on the small instances (T2).

Also the web interface isn’t a mishmash of 1000 services with different UX for each section.

If DO isn’t cheap enough for people there’s also Vultr which works out even cheaper ($2.50 a month for 1cpu/512mb) if you want something similar (not baremetal).

If you are extremely cheap like me, only have to run cronjobs at certain times or only need your instances to be running at predictable times you can also use the [AWS Instance Scheduler](https://aws.amazon.com/solutions/implementations/instance-sc... ). At my last job we were running SAP on EC2 and we were able to lower our bill like 50% or so by only running the instance 9-5 Mon-Fri. Now I use the Instance Scheduler for running cronjobs every day on a T3a instance and it costs less than a $1 usd per month. You can also configure your cronjob to stop the instance once it ends, that way the scheduler will only need to start it and you'll save the most (./myscript && shutdown -h now)

Is the bottom of the barrel VM's really what AWS is aiming for though? Or is it the auto scaling, variable, HA workloads...

Depending on the amplitude of your load cycle, it may be cheaper to just stay fully provisioned all the time on another provider.

> Vultr which works out even cheaper ($2.50 a month for 1cpu/512mb)

I can see that price on the pricing page but when I go the Deploy new server page, I cant see that price, the minimum is $5.

New York and Atlanta DC only.

I wrote down pros/cons of various ~$5 and below VPS services in a Gist I have been maintaining for a couple of years: https://gist.github.com/frafra/4688b146ca6d55accb768c3557939...

Hope you can add Linode here. I've been a long time user and it's a pretty good & popular option imo.

Not if you need any CPU performance to speak of. Given the misleading "vCPUs" that we are sold, I took some time to benchmark several major cloud providers and the results were… worrying: https://jan.rychter.com/enblog/cloud-server-cpu-performance-...

BTW, I run my SaaS on real servers from Hetzner. I figured I won't need instant auto-scaling anyway, and if you provision with ansible it doesn't really matter if it's an EC2 instance or a real server. What does matter is the price and the performance: I get servers which are significantly faster than anything you can get on AWS, and for a much better price.

To be honest, I do not understand the drive towards AWS. It makes sense in micro-deployments (lightsail and/or lambda, when your usage is sporadic) and in large deployments that need dynamic scaling. But it does not make much sense anywhere in the middle.

Good article, thank you for it. I only heard good things about Hetzner from many people, incl. networking setup.

As an owner of an iMac Pro (the 10c/20t CPU variant) I have to tell you that you are slightly mistaken -- the Xeon-2150B is a pretty strong CPU even compared to the bursty high-end desktop i9 CPUs. Sure I mostly do parallel compilation and work with (and produce) highly parallel programs, and there it really really shines.

But even for the occasional huge JS compilation it still performs pretty well. (Turbo boost puts it at about 4.3Ghz is the max I've seen.)

The iMac Pro's only real downside is that it has only 4 memory channels so as lightning-fast its NVMe SSD is, and as efficient the CPU with its huge caches is, the machine can likely perform anywhere from 30% to 80% faster if it had 8 memory channels.

But in general, iMac Pro is a very, and I really mean very, solid developer / design / professional software user machine.

I mean, all of that doesn't matter much in a world where the Threadripper 3960x / 3970x / 3990x now exist but still. iMac Pro is still the best Mac you can buy (Mac Pro 2019 uses the very last possible high-end desktop Xeons and I don't think Intel will be making many more of these; AMD is definitely on their tails and I don't think Apple will produce that many Mac Pros).

That being said, I am looking forward to buy a Threadripper 3990x monster machine somewhere in the next 1-2 years with dual 4k or 5k displays. Hopefully the Linux community will finally get its act together and make proper sub-pixel font aliasing by then...

> As an owner of an iMac Pro (the 10c/20t CPU variant) I have to tell you that you are slightly mistaken -- the Xeon-2150B is a pretty strong CPU even compared to the bursty high-end desktop i9 CPUs.

I do not think I am mistaken. I haven't yet encountered a Xeon that can beat the desktop i9 in single-core performance. Looking at geekbench scores (https://browser.geekbench.com/mac-benchmarks), my iMac is 10.5% faster in single-core than your iMac Pro, and your iMac Pro is about 15% faster in multi-core. For development work, I will take single-core performance any day.

Use cases and optimizing for them, I suppose.

I do mostly parallel work and for me the Xeon has proved itself as a better CPU compared to several desktop and laptop grade CPUs I also tried.

Words of sense are rare to be heard.

There has been a lot of comparisons and benchmarks and there will be even more.

After some point monthly bills grow to the scale where it makes perfect sense to invest money and time into your own infra. Yet People will still be throwing money into the hype oven.

I use Linode, but it was $10 or more, while I've just seen there's also a $5 VPS called "Nanode". I added it to the list and I will test it more in the future. Thank you!

Your list is missing quite a few major providers in the "low end" VPS space, such as BuyVM, Ramnode, Virmach, HostHatch, and probably others I'm forgetting.

Thanks, I read your comment on the gist too; I added them, but I have no direct experience. My gist was not intended to be complete, but a mostly EU well known VPS providers. A proper git repository with a nice markdown table would be better probably.

I personally use RamNode (a very small, 128M instance as a backup DNS in their Amsterdam DC) and can't really fault them. They've been rock solid so far.

Heard good stuff 'bout Ramnode.

https://toys.lerdorf.com/low-cost-vps-testing

Ah ok, I am using a $5 DO server as file server in Singapore, I thought I might save a few more bucks by moving to vultr but its ok $5 is pretty cheap.

This will work much better as an online spreadsheet. Thank you for the effort though!

CloudSigma is another player in that space. (No affiliation)

You are right, I have never heard of that thanks. I just added them to the list.

Do you host your entire stack on DO, or split compute and DB (and eat the latency hit)?

They also offer spamming services, phishing & Trojan hosting, and will block abuse reports for weeks at a time :)

Put a different way: they won't assume your service is abusive based on a few reports, and will allow you some time to address complaints yourself.

Is there a better way to handle such incidents (serious question, no sarcasm)? I feel like being too receptive to abuse reports would allow anyone to take down your service by submitting fraudulent abuse reports.

Hello, author here.

We definitely considered using other cloud providers like DO, linode, etc. But it was important for us to go with AWS because we needed some of the other services that AWS providers like s3, Route53, etc.

Some of our static websites are in fact hosted entirely using CloudFront + s3 combination which is something I forgot to mention in this post :)

Hi. Also I think you're future proofing your setup in a great way. There's no limit to what you can build. Also you get the fully fledged networking stack I assume, with VPCs, Security groups and so on.

Very good read, thanks for sharing. Have an upvote.

I haven’t used LightSail myself, so that’s the standard disclaimer. But while you do get a VPC, as long as you are in the Lightsail world, it’s invisible.

Once you graduate to full fledge AWS, you can peer your Lightsail VPC to your full fledge AWS VPC.

I'm not an expert on AWS, but can't you use all those services without AWS?

If you mean can you use S3 and Route53 etc. while hosting your servers somewhere other than AWS, absolutely - people do that all the time.

However in the Route53 case there are a few automatic integrations that only work with other AWS services. Not that big a deal though.

I'm not sure whether you're confusing AWS for some specific component of AWS, or whether you're just stating that competitors exist for those techs

But for the former: AWS is a whole suite of tools -- the specified technologies (cloudfront, S3, etc) are individual (and mostly decoupled) tools within that suite

One of my many AWS pet hates is that they have a service called Amazon WorkSpaces, i.e. AWS.

Sounds like they are saying that you can have an S3 equivalent and DNS services without being on AWS. Which is true and most of the time less expensive.

You can if your company can afford to pay enough to hire an entire devops team in a short time.

Why would you host a static website using s3? What's wrong with a traditional file system? You already have the load balancer.

Hosting a static website using S3 can be done dirt-cheap. We were hosting our website on S3 for a while now, with a monthly bill of 1,20€ with CloudFlare CDN in front of our bucket. Another big plus is that you do not have to worry about server administration, load balancers etc.

For a static website isn't something like Netlify easier (and cheaper, as in free)?

100GB bandwidth/month for free, it's quite small traffic.

Just curious, why CloudFlare instead of CloudFront?

I'd assume because CloudFlare doesn't charge anything for bandwidth; so their costs are exceptionally predictable

With block storage (a disk), you are going to need some compute capability to host it. Presumably with S3 you don't need that (I know you can do this with Azure Blob Storage, hosting static content directly; assume it's the same with AWS)

It is the same. S3 alone can host a static website (albeit only over http if you want a custom domain name; you need to add CloudFront [or equivalent] to get https hosting with a valid cert under a custom domain name).

My website is very small scale. I'm already using S3 to store some other things so hosting the static site on S3 is pretty straightforward. And it costs me literally pennies a month.

with caching (eg. cloudflare) in front of s3 hosting for static websites, you have almost zero cost hosting.

because it's super cheap and ridiculously easy

hardest part is working out how to go fully-public fully-static

Also, Lightsail is hot garbage. It has the same crazy aggressive throttles as a T2 instance. You'll have roughly 5% of the original CPU if you run hot for half an hour or so. It is not comparable at all to other cheap VPS options.

Just looked at contabo.com

Am I reading this right? 4 cores, 8 gigs of ram, 200 gigs SSD for 4.99 euro a month? That is around $5.60 USD a month. That feels too cheap to be true. What's the catch over ec2 or Digital Ocean?

Admittedly I've only seen a couple of sources but it looks like their tactics are as old as their website design. Good old fashioned overselling.

yeah, that's literally 4x the cost of my box at Hetzner

mind you, it's very much a pet and not cattle

This is interesting. Do you have (or know of) a published comparison somewhere?

What kind of comparison are you expecting?

Go to hetzner.de, create an account, fire up one of their 5€/month VPS and see if it runs your application well. If it doesn't, upgrade the VPS (or whatever) till the max or until you decide that AWS or whatnot might be better suited (or cheaper).

It's nothing personal against you, but those comparisons aren't worth anything. They are highly specific to the test case, which might very likely not be like the stuff you are trying to run.

And I've just so often seen people on StackOverflow ask how to fix their AWS setup to run a simple WordPress site, it's unbelievable.

There's this old saying: "If the only tool you've got is a hammer, everything looks like a nail."

That pretty much works for GC/AWS/Azure, too.

You can't learn everything by just trying the service though. Particularly on issues like downtime, support, and how they handle spikes. Also some providers have been known to artificially elevate resources for new users until they're locked in.

I think hetzner has servers in Germany and Finland only, not the right choice for most people here.

Hosting in Finland and Germany is a great choice for a lot of the right reasons.

Additionally, because it is hosted in Germany, you need to send them a copy of your ID and I found that their support was quite amateurish. So for me the process was: Go to hetzner.de, create an account, it gets instantly closed because you have a UK driving licence but were born elsewhere (perhaps too complicated for them?).

Never had that from AWS or OVH.

This is how I roll out my services. Always wondered if I need AWS, but always lacked the use case.

For heztner, just read the list prices for dedicated servers:

https://www.hetzner.com/dedicated-rootserver/ or maybe https://www.hetzner.com/sb

For example, a dedicated "AX51-NVMe" server for 65 euro / month = 73 usd / month gives you: AMD Ryzen 7 3700X Octa-Core (8 physical aka 16 logical cores), RAM: 64 GB DDR4 ECC, NVMe: 2x 1 TB NVMe SSD, unlimited traffic. That's less than half the cost of lightsail per logical core, or less than 1/4 of the cost of lightsail per GB of ECC ram.

That seems like a very powerful machine at an incredibly low cost. The closest equivalent at DigitalOcean is RAM 64 GB, 16 CPUs, 1.25 TB SSD, 9 TB transfer at $320/month. If I understand correctly the DO instance will have lower performance because it's virtual, and not bare metal. That amounts to about 4-5x difference in price, which is tremendous, considering that I thought DO to be the one of the cheapest providers. Is this comparison correct or am I missing something?

The part your missing is that basically all dedicated server hosters are this cheap.

DO, Vultr, AWS, GCP, Azure are the odd ones out and extremely expensive.

The only reason ever not to use dedicated servers is if you're in the bay area and ops wages are so overinflated that you literally can't afford an ops or devops person.

For everyone else on the planet, the comparison between the wage of an ops person, and server costs, is always in favor of dedicated hardware with your own ops people.

The thing I don't understand is why would people consider AWS to have lower "ops" costs?

I can deploy my SaaS either to VPS servers (AWS, DigitalOcean, Azure, etc) or to physical servers. Both deployments use the same ansible automation. The only difference is that I additionally use terraform to set up the cloud instances.

In every case, if a machine fails it is my problem.

Where is the savings on ops/devops?

Hilariously, this whole thing is again showing up as issue on the frontpage: https://news.ycombinator.com/item?id=23798347

> Where is the savings on ops/devops?

I don't know either, I've always used european dedicated hosters.

But apparently the typical YC user needs to pay 300'000$/year for a dedicated ops person just to run an ansible script?

Any employee comes with overhead costs ($$ and time and risk). Lots of side projects get off the ground only because people don’t have to cover the fixed quantum of costs that hiring an employee brings.

“If this doesn’t take off, I’m on the hook for ~$100/month when it could have been under $10/month“ just isn’t that compelling.

Ops has a low baseline complexity, and scales more or less linearly with usage on low usage scenarios.

It's something worth optimizing, but the idea that it will break a starting initiative isn't realistic.

You are not missing anything. I wondered about this myself, then started using physical servers from Hetzner for my SaaS. Quite happy with the results over the last 4 years.

what did the interconnection between the dedicated servers cost? in the past, there was no vlan support, i.e. you need to buy the interconnection, which was not cheap.

There are VLANs, but I don't use them. I found it's easier to use vpncloud (https://vpncloud.ddswd.de), that way I can use the same setup for development/staging/production, and it doesn't matter if the specific provider used supports VPCs/VLANs.

Also, with vpncloud I know that my data gets encrypted and is private — not necessarily the case with various VLAN setups.

when I started with hetzner there was no vlan, you needed to pay an "extra pack" and an additional network card for interconnection, with more than two servers you also would need to pay for an router and of course the time it did take to setup. so not really that cheap in the past.

They're not quite comparable, the Hetzner equivalent to that would be the CCX41 plus some extra storage for around $200/month (but you get 11 TB extra transfer). In general I think DigitalOcean VPS's are 1-3x the price of Hetzner ones.

I don't know why Hetzner dedicated servers are so cheap (they're also cheaper than their equivalent VPS's). I guess they take a bit longer to set up but there must be more to it than that.

A dedicated server is a one month commitment, possibly with a setup fee, Hetzner also charges to do things like replace failed drives.

You also don’t get the benefits of a virtualized instance like live migration for host maintenance. You can run your own hypervisor - but you’ll probably want extra hardware like 10Gb switches and the appropriate cross connects and as well as paying the one-time server move fee.

>Hetzner also charges to do things like replace failed drives

Are you sure? If I look at the AX dedicated server page, I can read that the basic support is free:

"Basic support includes the free replacement of defective hardware and the renewed loading of the basic system (in so far as a disk image system can be loaded)."

They’ll replace the drive for free - but likely with another used disk with quite some hours of runtime under its belt. If you want a new disk you need to pony up most of the time.

Dedicated servers are very cheap.

Could you please elaborate why? I thought they should be premium, as they are faster, have more memory and better performance because of bare metal, etc.

The question should be the other way around. Dedicated servers are not cheap per-se, they're just being sold at the market price.

The question should be, why is AWS so expensive on all fronts (not just the hardware but also bandwidth)? And the answer is again because the market lets them get away with it.

AWS is so profitable it turned Amazon from a marginal business into a tech giant. As for Google and Microsoft they don't get out of bed in the morning for anything less than a billion dollar opportunity.

A large part of the cost of a VM on these clouds is paying their super high employee comp, for advanced software development e.g. CosmosDB/DynamoDB, but most of all, contributing to the high revenue growth that drives stock prices.

It's not paying for the actual cost of hardware. Smart people know this and don't run in the cloud unless they get a sweetheart deal. Consider that GitHub, for example, runs/ran in their own datacenter and used the cloud only for spillover capacity. Zoom also (although now Oracle cut them a sweetheart deal). Netflix built their own CDN etc.

You pay them monthly instead of hourly, also you have to manage them yourself.

How to manage a bare metal server running CentOS automatically:

Add this cron job:

yum -y update

It's not quite that easy, is it.

At minimum you need a data replication and backup strategy. You're exposed to things like drive and RAM failures on dedicated hosts, so you'd need to think about RAID at least, unless you're running a system that clusters at a higher level (but then you need multiple machines).

However this is basically a matter of learning or hiring/renting a sysadmin to do it for you.

MTBF stats for different classes of hardware at different hosts would be valuable to have, as it can be affected by things like datacenter temperature. But I never heard of such a dataset.

I don't have a comparison but Hetzner is pretty much unbeatable in price if you can live with having your servers in Europe. If not, OVH, DigitalOcean, Linote or Vultr are good alternatives with datacenters around the globe and prices below that of AWS.

>If not, OVH, DigitalOcean, Linote or Vultr are good alternatives with datacenters around the globe and prices below that of AWS.

Sure they're below AWS, but if you compare lightsail, they're pretty much identical.

https://www.digitalocean.com/pricing/

https://aws.amazon.com/lightsail/pricing/?opdp1=pricing

I know that hetzner is affordable, but is digital ocean really cheaper than AWS?

That would largely depend on your usecase.

I created a comparison myself, because I couldn't find one. It was for my specific workload, which is CPU-intensive, but could be useful for others, too. An important find was that what we are being sold are not "cores", but "vCPUs", which usually correspond to hyperthreads, on slow Xeons. Which means no performance to speak of if the machine is loaded, which it usually is.

My conclusion was to rent physical servers (I use Hetzner) and get great performance at a fraction of the price.

https://jan.rychter.com/enblog/cloud-server-cpu-performance-...

Great read, thanks! I would definitely consider using dedicated servers instead of VPCs. AWS seems to be ridiculously overpriced.

https://toys.lerdorf.com/low-cost-vps-testing#hetzner

Here you go

Would like to see this too

Reply on news.ycombinator.com