These cloud providers are, by definition, charging you more than it would cost you to run it yourself. What you get in return is a guarantee of expertise and an ecosystem.
That is _not_ a given. They're charging you more than it costs _them_ to run it.
They get:
- much lower hardware prices than you
- lower bandwidth prices than you
- likely lower electricity costs than you
It's my impression that AWS competes with the EC2 instance cost (as it that's what new customers look at), and the bandwidth/storage only becomes apparent when you are locked in.
Really what AWS should be is one or two phases of service maturity: dev infrastructure and experimentation is phase one, phase two would be the "its a couple servers" production, BUT: with a scale plan for phase 3 being not-AWS.
Having a mature/battle tested phase 2 --> phase 3 should be a market advantage in the modern business landscape, but that's also a post-acquisition/exit phase, so none of the presumed target HN crowd care about it.
There are extensive articles and public knowledgebases on various technologies and architectures, but there is effectively nothing on AWS independence out there, even though almost every mature organization will need to face it.
I assumed phase 3 was where AWS started giving you massive discounts because you represent so much business for them, but I've never been at a place big enough to swing that stick at them.
Could it be magical thinking based on marketing hype and autoscaling fantasies?
I think you should have multicloud strategy just to improve your bargaining position. They know when you're locked in.
I understand that there are many cases when all of that you mention is needed but for the vast majority of normal businesses having main and standby servers on Hetzner / OVH with on premises backup would cover all their needs like for ever with stellar performance. All at the fraction of the cost. Many business owners just do not know it as they're not technical and their tech stuff for some inexplicable reasons are all for doing "cool" cloud architectures at owner's expense. And of course that magical "cloud" word sells itself. So many times when I come to do something a new client I hear the owner say proudly - we are on a cloud. It takes some careful efforts to explain to them that they're just wasting money while not upsetting them in a first place.
Realistically, a company will charge slightly less than whatever your alternative is, and they will provide products that restrict the number of alternatives you have.
AWS, GCP, and Azure provide several differentiated products that lock you in to higher prices on everything else. Effectively, these three have an oligopoly on "highly differentiated cloud services." They are only competing with each other on those services, and they are competing with "servers plus ingress/egress costs to our differentiated services" on their commodity products. That is the real reason why AWS egress costs are so high. It prevents you from picking and choosing where to buy each part of your cloud footprint, and locks you into buying AWS. Bandwidth costs are what keep you inside the AWS/GCP/Azure walled garden.
Lower-end providers like Linode, DigitalOcean, Vultr, and Cloudflare have parts of the differentiated offering, but not all of it. These people will have lower prices than AWS/GCP/Azure since their offering is less differentiated, but they will still charge you more than you would pay by renting a server, since they offer more products.
Finally, hardware rental providers like Hetzner and other operators are competing directly with you buying the hardware and paying for datacenter space and bandwidth. Datacenters typically charge a premium for power, even when they are in areas with low-cost power.
Notice that none of these companies are competing with large-scale server buyers who have the same hardware/bandwidth/power costs that they do. As such, they do not need to pass the savings they get onto you. That is where they get their profit.
That idea was not expressed in the comment that you replied to. The comment you replied to was in response to a strong claim that ignored the existence of economies of scale, and therefore concluded that it's so obvious that cloud providers must always be more expensive than self-hosting.
AWS's economies of scale have as much to do with its pricing as the phases of the moon, and citing them as a reason why AWS could be cheaper than self-hosting is pretty ignorant in itself.
Whether economies of scale, or the efficiency of managed services at scale, or use-based loss leaders, there are many ways in which AWS services could be, and often are, cheaper than self-hosting. Of course, not always.
At my last company, providing SaaS for the education market, moving from a datacenter to AWS saved almost 70% year-over-year. In the datacenter, we ran machines to cover our peak load, which only happened a few times a year. In AWS, we scaled way down and auto-scaled up during those peak weeks.
Is the entire education market "exceptionally bursty?" I suppose it could be considered so. Bar exams and midterms and finals certainly don't happen every week.
I'll wait for people to tell me how AWS was the wrong solution, and how we did everything wrong before that, but the bottom line is we saved a lot of money, accelerated our development schedule by building all new functionality using the so-called "serverless" stack, and succeeded so well during the pandemic that another company acquired us... which is why it's my former company.
I don't know how big your peak was compared to your average load, but if it was anything like that, serverless was a great call. Buying enough hardware for the bar exam and idling it 99.5% of the time sounds incredibly wasteful. However, this is an exceptionally bursty workload.
Most services do not have this level of traffic variability in such a way that a CDN can't handle it for you. 10x peak to trough variability (after your CDN) is fairly common, but still considered bursty, and in that case, AWS serverless still doesn't look great compared to DO droplets. Many services have daily or weekly cycles (without events like the bar exam), and run analytics workloads in their off hours.
We frequently see links at HN that fall over and die after they're linked. On the one hand, that's usually because they've misconfigured a personal blog and lack caching. On the other hand, some services just weren't ready for HN traffic and failed to scale up. Which.... they wouldn't, were they running on a cloud provider like AWS.
It's possible for any service to be surprisingly bursty, but I suspect more than you think are bursty on a regular basis.
P.S. DO Droplets are Digital Ocean's response to AWS Lambda, and are still "cloud."
It is not obvious that growing your own wheat will always be cheaper than buying bread.
It's obvious that baking your own bread is a bad idea if you're okay with what you can buy from the competitive market. It's equally obvious that baking specialty, artisanal bread (and buying the commodity ingredients, such as flour, salt, and yeast) is cheaper than going around to several grocery stores to try to find the exact kind of loaf you want.
AWS is in an oligopoly, so it should be obvious that its efficiencies (in this case, its economies of scale) are not passed on to the customer. The strong, clearly expressed claim was fine.
Dell can bulk order parts and has a strong negotiating stance, which makes the hardware __CHEAPER FOR THEM__.
It says nothing about how much the market can bear for Dell. And in fact, companies use Dell because of the _guarantees_, not the cost. At any point in time, Dell will replace hardware with no questions asked. They can do this because of the margins between what it costs them and what it costs you.
There are benefits to cloud, cost aint one of them.
edit:
Also, this shit?
> That's just not how this works.
It never makes you sound right.
Your parent comment wasn't saying that AWS is necessarily cheaper than running hardware yourself. It was saying that AWS is not necessarily not cheaper.
You're responding to someone who said "can," not "must."
What they said was "there's no requirement that cloud costs more than traditional hosting. Their scale allows them lower fundamental costs. They could outcompete traditional hosting on cost if they wanted to."
There's no misunderstanding of how comapnies work in what parent poster said; in reality they are 100% correct, and this is the path that CloudFlare is beginning to follow with services like workers and R2.
It's a CDN, not a filestore. CDNs mean hundreds of locations.
Cloudflare is a CDN, which, like all of its competitors, is a lot cheaper than doing it yourself because CDNs are a competitive market. There are at least 10 (if not more) CDN providers out there that have footprints in several thousand datacenters, and the product is completely undifferentiated. In a competitive market, you have to charge related to your costs, because your competitors will cut prices to their costs.
In other words, in a competitive market, economies of scale get passed on to consumers. In monopoly and oligopoly markets, that is not the case.
Object stores are becoming a competitive market, with R2, B3, and Wasabi entering the game, but buying your own machines and using Ceph is still cheaper. You still pay for "value adds" like not having to manage the machines yourself.
A CDN is a thing where you can put things, and request them by web, and have them geo-distributed.
R2 is actually a CDN. So is S3. Yes, I'm aware, the vending companies also sell different products called CDNs. Nonetheless, R2 and S3 (unlike B2) are CDNs. You can make distributed hits to it across the planet to addresses you don't control. You can hit it directly from the web, and it will serve to you from a local node you never knew anything about. Its goals are speed and locality. You can look up programmers debating whether to use CloudFront or S3 as their CDN based on time to invalidate vs actual speed and cost control.
Attempting to describe it as "an object store instead of a CDN" is kind of missing the point. It's both. It's also a webserver. It's also a backup system. It's many things.
You cannot replicate R2 on a single machine. R2 delivers locality and redundancy. Arguing over its title is irrelevant; the reason to point out that it is a CDN is to establish requirements for replacing it with a homebrew solution.
The lowest practical cost for reliable machines is arguably VPSes at around two dollars a month. Those VPSes tend to give you 100-250 megabits of traffic monthly with around 1gig of storage. Assuming two for redundancy, you'll get maybe 500 megabits for about $4 a month. (I personally would not feel safe with two, but we're talking about cost cutting.)
That same 1g of storage on R2 falls in the free forever tier.
To eat up the $4 a month in class B actions, you will first need to consume the 10 million free events, then spend 36 cents per further million. This is a further 11.1 million, or 21.1 million requests per month.
Most low-tier VPSes will struggle around 10 requests per second using NGINX, because their disks and memory are terribly over-burdened. There are 86,400 seconds in a day, suggesting you'll get around 860,000 requests per day if you don't consider time shaping. Considering time shaping - you don't have full requests coming in at every time zone - you'll most likely have closer to 350,000 realistic. This means that your two VPSes will, just barely, be able to close the same amount of traffic.
Very, very slowly. You'll be lucky to get 200ms responses locally, and to get VPSes that cheap, you'll need one in the Netherlands in the burden corridor, and one in the United States, most likely in Texas or Georgia.
Most of your customers will now be getting 300ms round trips.
Why? Because you wanted to save against four dollars a month. This isn't even enough money to biggie-size a value meal.
All so that you could make a two-computer CDN on VPSes.
In the meantime, I've worked at two unicorns and neither of them had traffic anywhere near this large.
You can also do the math with an unmetered 1u. Ten megabit unmetered with 95% availability typically goes for about $19 a month, or $30 a month with a cheap 1u attached. That's about 3.2t of traffic a month.
As soon as you try to put actual specific numbers to it, and calculate actual specific costs, this just falls apart.
.
> There are at least 10 (if not more) CDN providers out there that have footprints in several thousand datacenters
There's more than two thousand of them from the United States alone. Akamai's CDN has over 400,000 nodes at over 50,000 POPs.
I can think of more than 40 in-house just from the US of that size (eg Netflix, Steam.)
What's your point?
.
> R2, B3, and Wasabi entering the game
Wasabi has been around since 2015. B2 isn't really the same kind of thing as R2 or S3.
.
> because CDNs are a competitive market ... Object stores are becoming a competitive market
Object stores have been around decades longer than CDNs.
.
> but buying your own machines and using Ceph is still cheaper.
Not really, no. I just did the numbers, and they did not pan out the way you claim.
Buying your own machines? You're lucky to find a 1u for less than $1500, and colo for a 1u is typically at least $40/mo. That covers 30 million writes and 300 million reads a month for six months at R2. (For scale, the Washington Post gets 86 million uniques a month, so if WaPo is doing four hits per load, which seems high if you sprite your images, then you're talking about larger than WaPo traffic. Spriting your images is a lot less work than building a CDN from scratch.)
The R2 cost for Washington Post is about $230 a month. You ... you really want to fight a cost like that, at a scale like that?
At that price, you're saving about $190 a month on bandwidth, so disregarding the cost of upkeep, it takes about eight months of WaPo tier traffic per slab to break even on each $1,500 slab.
In order to compare against R2 you need multiple locations (redundancy is critical to safety,) so for this to make sense, you need to be able to deliver multiple Washington Posts of traffic. If you can do that, well, I can make $230 a month not a big deal to you. Let's talk.
If you're trying to save money against four dollars a month, which I think was the actual realistic price here, I'd warrant it would be a good idea to look at your salary, convert it to hourly, and figure out how many years you'd have to run that savings to recoup your first hour of thinking about it. (Heck, I would even say that about $230 a month - after all, if you make $125k that's about an hour of your time, and I doubt even a very talented engineer could set up a distributed R2 replacement, including buying and siting machines, that fast.)
The vertical step cost is larger than the total cost in almost any practical system.
In order for the self-hosting costs to pan out against a distributed system like R2, you need to be looking at weekly terabytes of traffic and hundreds of nodes.
Respectfully, no, doing this yourself isn't actually cheaper. Not even close. You'll need a cost outlay to convince me, and it'll have to explain why the one I just did is wrong.
A realistic "buy your own hardware" CDN^H^H^Hobject store doesn't have barebones 1us with tiny drives. A realistic object store has a couple attached JBODs. (But you know, this is some individual's personal file server, which has been lionized into being a "competitor to R2" by ignoring almost everything that R2 actually does.)
And of course, the second you actually pull that cost outlay off, I'm just going to start talking about how much cheaper it would be to build your own datacenter, and then your own internet. Because as long as we're not focusing on there being no business need to justify the outlay, the larger you build, the better.
But it's "object hosting," not an R2 competitor, and you want to self-build. So maybe we just knock this down to two nodes. And then it's only $4,200 up front and $1,700 of traffic a month to break even, plus engineer costs for building, maintaining, and debugging it.
(And honestly, if you're just using it for storage, why not just set up FTP or RCP?)
So I googled "how much traffic should your website get." I got hubspot. https://blog.hubspot.com/blog/tabid/6307/bid/5092/how-many-v...
That article - and I'm not entirely certain that authoritative even exists here if someone defines things, which I haven't - but that article is probably better to listen to than me, at least.
They claim they surveyed about 400 "traffic analysts." There's a common sense line here of "they work for people who can afford entire staff for a job like that," so I guess I assume by default that these folks mostly work for very large sites, and the remainder merely for large sites.
Like I don't think any private pages with 200 hits a month have a "traffic analyst," you know?
Sure enough, even in that survey, 46% of them were in the 1k-15k a month bucket, and less than half a percent were in the 10m+ bucket. And remember, we needed 300 million per server-month to break even.
My strongly held, evidence based opinion is that the scale at which CDNs are cost-competable is fundamentally an irrelevant scale to all but the very largest of sites, in the way that a local single-location restaurant should not be looking into making its own flatware or furniture to control overhead.
And none of this counts the engineer costs.
My opinion is that in order to cost compete against CDNs, you need to have hundreds of large websites as customers.
My opinion is that viewing R2 and S3 as "just a place to store files" is akin to viewing a car as a place to listen to music - not even the primary use case. S3 has had http access from day one.
And yes, you can make a far cheaper radio than a car. But in general, if you try to sell the result to car purchasers, what I believe you will find is that you've misunderstood what the market is attempting to purchase.
My experience is that 90% of the people I know using S3 are using it for web fronting directly, or to back CloudFront.
.
Null hypotheses are important. So is actually having the ability to use what you build.
Imagine how cheap I could make the CDN if you just keep adding zeroes to the userbase all day, right? But it would be a hollow demonstration, because I can't actually deliver the customers to justify it.
The reason I've never heard of Ceph is that nobody I know uses it. Nobody I know uses it because nobody I know would invest this much effort into getting rid of a couple dollar a month datastore.
I know thousands of people who are making SAASes and other sites.
I am of the opinion that recreating object stores and CDNs is one of the smallest levers that you can think about turning, unless you're some starkly uncommon kind of site like a video host.
> R2 is actually a CDN. So is S3. Yes, I'm aware, the vending companies also sell different products called CDNs. Nonetheless, R2 and S3 (unlike B2) are CDNs. You can make distributed hits to it across the planet to addresses you don't control. You can hit it directly from the web, and it will serve to you from a local node you never knew anything about. Its goals are speed and locality. You can look up programmers debating whether to use CloudFront or S3 as their CDN based on time to invalidate vs actual speed and cost control.
S3 does not serve you from a local node (by my understanding of 'local'), it always goes to the source AWS region. It can take advantage of availability zones but it is still regional. It is missing the "D" part of "CDN". Perhaps this argument is just semantics, but the conventionally held understanding of "CDN" is that it's much much more widely geo-distributed. Otherwise you're right that they both provide a storage function.
> Wasabi has been around since 2015. B2 isn't really the same kind of thing as R2 or S3.
B2 added an S3-compatible API, so they're now directly comparable.
Other recent entrants in this market are IDrive e2 (4 USD/mo/TB) and Storj (4 USD/mo/TB). It's definitely a competitive market aside from the anti-competitive pressures that keep people using S3 anyway, such as zero-rated transfer within an AWS region to your other AWS resources.
Storj is the closest competitive object-store to a CDN in that it is possible to get data from the nearest geolocated node, although you'll (A) have to use the non-S3-compatible version of their API to do it, and (B) tolerate a certain degree of crypto-adjacency,
R2 is not a CDN. S3 is not a CDN. Both are object stores. They store objects with some redundancy in a local area. CDNs are globally distributed. Object stores usually aren't. You can use an object store for backups and for holding webpages, but that doesn't make them backup services or webservers either. Using only S3 (without deduplication or compression) for backups is ridiculously expensive, and using it as a webserver limits you to a static site.
In terms of the costs, if you are small, the free tiers (and cloud platforms in general) are great. If you are at the point where you are paying $5-10k/month, you are almost certainly overpaying by using cloud services.
Also, you don't even need an object store for a small app. You can use a server with a hard drive if you have a lot of data to store (plus backups and a redundant server in another place for HA). Put a CDN in front of it - a real CDN - and don't worry about it. Pay the monthly cost for the CDN. Also pay for your backups.
It's fine if you want to pay AWS so that you can focus on things other than efficiency. Most SaaS companies make that trade. But you are paying for it.
The core point, that you appear to have missed in the actual math, is that you will never personally be at that point.
AWS may be able to charge a surcharge for being the biggest and best-known cloud offering that most people have expertise with, raising the de facto cost of switching.
That's totally non-obvious and certainly false in most cases. Cloud providers have huge economies of scale, which certainly makes it less expensive to use cloud if you factor in all your costs related to running it yourself.
This is like saying buying furniture is more expensive than making your own furniture because they have a profit margin. Even if you don't account for your own time, it will certainly be false as you'll pay your raw materials much more.
AWS would pass that difference to Amazon shareholders than passing them to you.
But, now that the emphasis seems to have shifted from growth-at-all-costs to where-can-we-cut-costs, I think there may be more organizations with a large enough server load to realize that they can do it cheaper (or even just get a different cloud provider to do it cheaper).
You see a lot of posters commenting that it isn't obvious that it's more expensive, but that's a large part of why I said I'm flabbergasted at how many people don't find it obvious.
It is obvious, someone may still choose to pay it because there's a natural tradeoff, but it's absolutely obvious.
I suspect most developers don't find it obvious because they've never dealt with actual servers themselves.
It's akin to arguing changing the oil on your vehicle is more expensive than paying someone to do it. No, it's more convenient, and that may make it worth the price.
And also of course, it allows you to go faster, which can have a huge value
I mean, it is fungible enough that they can learn, but they'll probably make a lot of mistakes along the way and be less efficient.
- Disable password-based auth for ssh (require key-based auth) - Enable fail2ban or similar to slow down brute-force login attempts - Configure firewall - Install monitoring tools, antivirus, possibly backup daemons, etc - Setup a sane swapfile for your use case, and configure monitoring tools to alert when memory pressure gets too high - Setup disk mounts, configure monitoring tools to alert when disk space is low, and consider a cron job to automatically clean up tempfiles - Either set up automated updates (typically excluding kernel upgrades), or have a standard schedule for manually applying updates
...and probably other things that I'm forgetting because I'm a developer, and it has been years since I've been a sysadmin.
A couple of others spring to mind:
- if you're not regularly testing your backups, you don't have backups
- monitor SSL certs for expiry. It's amazing how many outages I've seen because this was missed.
- are you allowing direct connection to your server from the internet, or do you want to go through a bastion host. Do you need 2FA on this, say because you need to meet ISO27001.
- do all the above through CM (like Ansible or Puppet) for obvious reasons.
You don't hire a dedicated sys admin because they know some voodoo magic developers don't, you hire a sys admin because they have the _TIME_ to watch for security updates. And you _PAY_ a company like RedHat to ease that burden.
A dedicated sysadmin will typically do a better job than a non-dedicated developer, but if that developer cannot do a passable job, that's a gap in their skillset.
I may not be a network engineer, but I damned well understand DNS, MPLS, networking, gateways, and the like. Any developer who doesn't has gaps.
Yes I exaggerate a bit, but I do so to make a point stand out.
Focusing just on what you're working on is great.
I was commenting on the expense rather than why you would (or wouldn't) want to use cloud.
That's pretty obvious but years ago you would have been downvoted into oblivion for writing that.