Ahrefs Saved US$400M in 3 Years by Not Going to the Cloud
tech.ahrefs.com
tech.ahrefs.com
https://news.ycombinator.com/item?id=35108429
https://news.ycombinator.com/item?id=35100866
What I am missing though in the monthly cost is the personnel cost. I would assume that a decently qualified system and network administrator in Singapore, capable of operating and servicing this setup, will cost upward of $10,000 per month. And only having one is probably not enough, due to SPOF and other considerations. To be honest, even for the cloud you probably need someone to administer the setup, but chances are high that there is much less to do.
Paying for, and finding qualified staff is from my experience the main driver for small and mid-size companies to go to the cloud. And for many this actually makes sense financially. Even though many create unnecessary complex setups or get trapped in vendor lock-ins.
Occasionally you need to replace expired components, and install new systems, but that's about it. There will be weeklong periods where no action is required.
Finding qualified staff is definitely hard. Especially nowadays, there are a lot of DevOps astronauts that have never swapped a bad DIMM.
For geo-redundancy (unless you’re at massive scale beyond an HN comment) a couple of cabinets per facility is an incredible amount of compute.
For hardware issues get support from your hardware vendor. When equipment fails (drive in a RAID array, power supply, entire chassis - whatever) they’ll dispatch a tech directly to the facility and take care of it. Depending on what you’re willing to pay it’s either same day or next (and warranty support is usually the first thing vendors will discount if you negotiate).
I’ve “deployed” to multiple data centers that ran for years without ever stepping foot in them.
That said I’m with you - I’m a little concerned Big Cloud is here to stay after 20 years of incredibly effective marketing and mindshare dominance. We now have an entire generation who’s sum total understanding of hardware is a (reversible!) USB-C connector.
Also there’s no software costs… if you’re going to compare with the cloud you need to include VMware/Microsoft suite licenses, maintenance, and operators to have even remotely comparable capabilities. That tips this comparison strongly in favor of cloud.
Using Dell (as one example) you’d really be cheaping out on your hardware warranty if it didn’t include next day hardware replacement (for three/five years) and as I noted before, when placing server orders in the past they’ve discounted warranty/support to practically nothing. It’s actually fairly difficult to get a warranty that doesn’t include it. This is a level of support where the vendor literally dispatches someone to the facility who gets you back up and running. With redundant power supplies, RAID, etc the individual machine often doesn’t even go down and they just show up and replace the redundant component with the blinking light.
AWS is 20 years old and in two decades hardware has gotten EXTREMELY reliable and performant - in managing hundreds of servers across multiple facilities over the decades I can count the number of hardware failures on my fingers. The FUD/scare tactics/marketing of clouds to scare people away from hardware have been extremely effective. As one example for the most "unreliable" component the BackBlaze hard drive stats tell the story - they have a roughly 1% failure rate across > 100k consumer level mechanical drives[0] in service for a long as 92 months. These "failure prone" consumer drives aren't even available from server vendors. This is why their pricing is at least 1/10th that of big cloud.
VMware? Microsoft suite? For these kinds of applications I don’t know of anyone using anything other than containers, K8s, OpenStack, etc and Microsoft licensing is never “free” regardless of the approach. More traditional “enterprise applications” with a bunch of VMs that would necessitate something like VMware are essentially outside of scope and a different animal completely.
I also don’t know of any organizations outside of the smallest startups that don’t have at least one full time cloud devops/architect to manage anything beyond the simplest of cloud deployments. Compared to someone using an orchestration platform and managing a K8s control plane the employee cost is often a wash. For the rest of the dev/devops teams shipping a container is shipping a container.
The pricing differences at this scale are unbelievably dramatic (as noted in the article) - with ROI on hardware purchases (leases) and hosting vs cloud often less than one year. That’s A LOT of spend left over for the rounding error miscellaneous costs often noted. Cloud egress bandwidth pricing markups alone can pay for it when you’re pushing the kind of traffic that would warrant this approach.
I’ve been doing this since before AWS existed - the cloud has its place but in almost every case when this debate comes up arguments for “big cloud, always” typically come from people who have never deeply and holistically compared these approaches (and certainly never actually implemented “hardware”). For most of the "cloud only" generation their only experience with hardware is that time their laptop died and needless to say server grade hardware in temperature controlled facilities on conditioned power is infinitely more reliable than a consumer laptop getting thrown around.
Not saying that’s the case with you personally and “hardware, always” is just as inappropriate as “cloud, always” but to never even consider hardware and hosting in many cases results in wildly expensive cloud costs.
Don’t even get me started on cloud cost controls. The pricing is so dynamic and so obtuse there is an entire cottage industry of consultants and startups that exist solely to help orgs try to understand their cloud spend and attempt to optimize cost. Hardware leasing, fixed price co-lo, power, and bandwidth results in cost controls that just aren’t possible with cloud. Almost every startup has at least one story of scrambling to figure out what happened when they blow through their monthly cloud budget in many cases literally overnight - or begging their cloud provider to write off or steeply discount a monthly invoice that was wildly beyond predicted cost (and in many cases they literally wouldn't be able to pay).
In multiple rounds of investment, etc in startups I’ve often had the experience during technical and financial due-diligence where they look at hosting costs and literally can’t believe how cost effective and performant it is vs the “all cloud always” companies in their portfolio.
At a level of scale to even consider it hardware and co-lo is practically cheating.
[0] - https://www.backblaze.com/blog/backblaze-drive-stats-for-202...
It's not like operating AWS doesn't involve any personnel cost, does it? Even though it was marketed as such, it has become clear that managing a public cloud such as AWS is at least as complex as managing a datacenter. Sure, you don't swap broken drives in RAID arrays but you have other complexities to deal with - and that complexity increases year by year.
But I think AWS "management" definitely starts paying for itself when you want the higher availability features such as managed SQL databases, load balancers, queues and other features. None of them are super complicated on their own but they add up quickly, and at small scale the cost of having experts in each service will outweigh the cost of the service.
It is also worth considering the opportunity costs of changes. On-prem you may have a few spare servers lying around but if you need another machine or a new piece of hardware it will take significant time. On AWS you can spin up a 4 GPU machine in seconds, if you need more webserver machines just press the button. This quick turnaround can be very valuable as you are developing a new product and don't quite know what the requirements are.
As you get bigger the costs definitely flip. The profit margin that AWS takes starts to outweigh the costs of having more experts (and as your team grows you may have already acquired most of these experts, so no training is required) and it makes sense to move things in-house onto dedicated hardware.
The same fantasy says that if you own your own hardware you’ll need lots of experts that are so expensive and so hard to find that it makes no sense to self host.
Cloud is crazy expensive and delivers much slower computing power per dollar.
Consider if you need kubernetes plus cloud scaling, or you just need one modern ripping fast 16 core machine with 128GB RAM.
buy enough to stay ahead of demand. If you have unused servers, who cares, at least you don't have allocated by unused AWS servers that pay for themselves in hardware costs every... four months? I don't remember how long it is.
I then went to try to convert other businesses over to doing it cheaply, but as an engineer I just wound up getting into fights with my managers who loved spending money.
It is all about Empire building and managers wanted those expensive VMWare and EMC vendor relationships to manage in order to fluff up their resumes. They want more people under their umbrella.
If you start hiring Directors of Network Engineering you'll find that you start hiring lots of expensive CCIEs and spending a lot of money on Cisco.
It is almost a problem of being too small, since you can't quite easily point out how having single SATA drives in all the servers will save millions over having hardware RAID cards, and having simple networking in your rack design will also save millions over what Cisco would like to push on you. So you overspend by $250k here and there and everywhere, and pretty quickly you're back up to cloud costs.
At a smaller level it isn't quite as obvious how obscenely expensive to your capex these kinds of managers are, and they know hot to talk in ways that make Managment believe that they're buying all kinds of Enterprise-class resiliancy and leveragable vendor relationships and all that kind of nonsense. Hiring smart engineers instead is something that most Managers don't know how to do either, which is what you need in order to get a decent level of automation in place in order to scale the ability to manage it all.
And don't get me started on high priced consultants that can be brought in to overdesign the most complicated solutions possible by never saying 'no' to any kind of requirement that IT management can dream up. I've seen VLAN diagrams that you wouldn't believe...
But anyway, if you can solve that human problem, through hiring the right people and having a corporate culture of "Frugality" (as much as I hate it, that focus Amazon has is what let them do IT cheaply and then resell it). But it isn't a technical problem, or a financial problem, or even really a staffing problem, it is fundamentally a culture problem.
And then the trick in my mind would be to do Frugality right without being as much of a dick about it as Amazon is.
Not sure why e.g. Netflix doesn't exit AWS considering their needs flatlined. Or maybe they have some special agreement with much lower prices than market.
I talk about it a lot on HN but if you really want to get the financial types excited about leasing there are all kinds of interesting options like Section 179 Qualified Financing that can often sweeten the pot further.
I think many companies simply get away by having unhealthy oncall rotations. E.g. ask people to oncall outside business hours constantly.
These exist and falls under the “Enterprise Discount Program”, but there are other PPAs (Private Pricing Addendum).
The idea with EDP is if you have qualifying on-demand use that not covered by a Savings Plan or Reserved Instance, you instead get the EDP discount applied to the line item.
Where I work is actively renegotiating its EDP discount. Right now we get 14% off for applicable on-demand usage.
There are also other programs. Such as the “Amazon Partner Network” “Workload Migration Program” (APN WMP) for SAAS companies, which are credits (lump sum, not per line item) that are awarded for moving customers to your SAAS products that are hosted on AWS infrastructure. These are big 6-digit chunks for us. Many other credits and programs as well.
Think of how many copies of the data you need (and how much processing it takes) to use:
- a L3 CPU cache.
- a 16GB RAM cache.
- an embedded C/Rust/Go SQLite database.
- a distributed, fault tolerant Cassandra/elastic cache cluster.
You can literally solve the same typeahead/autocomplete/user lookup/session management/etc.. features using the last three of these - but the CPU cycle, memory usage and latency costs (plus complexity) explodes.
I'm always impressed by people that handle 1 million connections on a single box while I'm watching teams trying to handle a thousand connections with a whole fleet of infrastructure and caching layers.
Then again, part of the issue is the wrong language and wrong designs patterns. Highly optimized Rust and Go someone obsessed on all weekend is going to shred your team's unoptimized vanilla Ruby or Java install you didn't even have time to write tests for and don't honestly care about because you're too worried about being laid off.
Tons of cloud customers running NodeJS or Java stuff don't fit this. Maybe they have a few optimized pieces with high throughput and majority unoptimized where resource usage is low either way. Or they might be ok spending 10X compute resources even on something high-throughput because landing new features/products matters way more than optimizing a small part of their budget. Like, there are businesses renting big 8-socket servers with TBs of RAM to support some monstrous corporate DB that can only run on a single machine.
That is what everyone knows,
so that is what's taught,
so that is what everyone knows,
so that is what's taught.
I'd gladly onboard projects to Erlang, Rust, Zig, Go, or anything modern. Would save our planet so much wasted energy.
For IO bound work there’s not much difference between a service built with Go or Node. It’s really when you need services that have soft/hard real-time requirements or continuous processing requirements that imo it’s necessary to hire the right people for the right stuff.
For anyone else who is making a web platform that doesn’t have much complex backend processing, they can just use ruby or node.
Personally I use Go for all backend tasks and node for any one off scripting work. Go spits out a binary I can run on any Linux based system. Node scripts are easy to write and I use them for fixing issues or for simple single responsibility tasks.
This doesn't sound like a problem to me.
I am free to switch vendor every 12 months without pain.
I like Cloud66 because:
- Pricing is fair. I spend 100$/month for my stack - They back up db and host it to a different cloud provider (AWS)
- It’s easy to import Db
- UI is clean and easy to use
- It’s easy to switch vendor. Just clone the app and it will install all in a new server.
I have switched from Digital Ocean to Azure in a couple of hours with 30 min downtime.
If you have an 8GB vm you can see it in action via https://docs.openstack.org/devstack/zed/guides/single-vm.htm... (takes about 45 minutes) or if you want the "full" experience fire up an education cluster from https://openmetal.io/use-cases/on-demand-openstack-cloud/ (it takes about 4 hours or so but it's also a "real" setup unlike the vm which for sure allows running instances via qemu but isn't going to allow exercising the complete suite as easily)
Most services are probably well served by AWS/Heroku and others and simply saving the engineering time on developing the product, or optimizing when it becomes a problem
AWS can be 2x as expensive as DO for a simple VM, which you can easily work around by acquiring Reserve Instances or using "spot" instances
but, the "AWS premium" becomes a problem: - 10x when dealing with bandwidth which you cannot do much except a very big enterprise package and getting some discount that even then it will be multiple times higher than the alternatives
The way I see it, AWS is great but you need to consider your 1) key infra requirements 2) consider cost effectiveness of different solutions/services 3) consider alternatives i.e.
- Cloudfront is fine, but if you doing 100 tb/month probably Cloudflare would be best! - Lambda is fine but if you doing 100M requests/month EC2/ECS would probably work better (or more cost effective)
Consider 1) what you need to get started and ship 2) what you need to scale up 3) what's the most cost effective solution long term
The requirements will change over time, be ready to adapt!
so compare them to buying AWS VMs you don’t need? one of the main attractions of the cloud is the fact that you don’t have to pay for what you don’t use. Elasticity.
Every time I see one of these is: I bought 100 servers (which i use 40 of), I would’ve saved money if I was renting 100 servers of AWS. Yeah … no shit. If you were renting only 40, and had the ability to go down as usage decreases, there would be savings, mind you there are such things as, RIs, savings plans, etc, etc. meh.