S3 Is the Future, S3 Is the Past
btrblocks.com
btrblocks.com
2006-03-14 $0.150/GB-month
2010-11-01 $0.140/GB-month
2012-02-01 $0.125/GB-month
2012-12-01 $0.095/GB-month
2014-02-01 $0.085/GB-month
2014-04-01 $0.030/GB-month
2016-12-01 $0.023/GB-month
God why am I defending Amazon?
For reference transit cost has dropped over the years and is now about at $0.000247/GB or free if there is a peering agreement with the network the data is sent to.
This isn’t just AWS BTW; all tier 1 cloud providers recoup their costs this way.
Yes, you're legally correct, but the point is AWS are clearly overcharging for egress.
As you say:
> Egress pricing pays for the AWS network infrastructure that is incredibly reliable (hardware failures happen regularly yet almost no one notices) and allows it to operate at scale sufficient to absorb even the largest DDoS attacks.
Cloudflare is also pretty good at absorbing DDoS attacks, yet charge no egress costs.
Maybe the real question is, are the tier-1 providers colluding in overcharging for egress?
Egress costs are fairly clearly a vendor lock in strategy. If I want to transfer 1TB to run on cheaper compute at another provider, that costs me $90 (edit) on AWS as opposed to $0.24 at wholesale prices. It's unlikely to be cost effective so I choose AWS compute.
EDIT: thanks to someonebaggy for pointing out my decimal point error.
Ninety dollars to transfer a terabyte is clearly ridiculous.
Many people avoid T1 clouds because they're so expensive, but they seem to have found an extremely profitable market segmentation consisting of the remainder.
And kindly refrain from accusing others of Stockholm Syndrome here. It’s incredibly rude.
Finding any excuse to justify a 100x profit margin is a form of Stockholm syndrome.
GiB is unambiguously the big one, power of two. You want S3 to be priced in dollars per GiB. "shrinkflation power of ten GiB" is getting it backwards.
Amazon S3 storage usage is calculated in binary gigabytes (GB), where 1 GB is 2^30 bytes. This unit of measurement is also known as a gibibyte (GiB), defined by the International Electrotechnical Commission (IEC). Similarly, 1 TB is 2^40 bytes, i.e. 1024 GBs.
I guess they could just change that overnight in the future. Still, a lot better than charging for Storage Units or similarly arbitrary unit.
There's only a small little mention, if we are lucky, on the couple drives that have it (expensive enterprise flagships). It should be a regular sticking point, whether it's there or not. Without pressure it's not going to get regularly available, it feels like.
FDP is so simple. Declare a number for what pool of data you want to write into. Data of the same pool gets written to the same storage such that you can wipe it latter together. It has huge wins though against write amplification! Massive wins. For so close to free.
Some day I want to own a FDP drive. And then I can finally start using the tokio/io-uring support that I contributed! https://github.com/tokio-rs/io-uring/issues/380
The NVMe-KV is more radical. Still worth putting some pressure on, but your drive as KV, as object store, feels harder. Side note, really enjoyed this ceph nvme-kv offload post thing, my favorite tech write up in a while! https://ceph.io/en/news/blog/2026/for-whom-the-door-bell-tol...
FDP is a very minimal addition of control. But yes it is another box to ticket, is another place to upcharge. Yet still, we're only just seeing mainstream products emerge. Kioxia's CM10 for example. https://www.techpowerup.com/351218/kioxia-introduces-first-p...
I'm hoping that the need is great enough to break the industry control. I think for a while the market felt relatively well enough served such that it was unclear whether clear wins would actually result in customers. With AI need for speed, I can definitely imagine incredibly crazy CXL controllers or what not, that allow low latency acces to many many open channel flash systems. Wouldn't that be a thing.
Again though, FDP is such a ridiculously tiny add, and it helps SSDs so much for so many use cases. I really hope it becomes an expectation, not a feature, sooner rather than latter.
Edit: happy to see a new group has shown up asking for open channel flash, Open Flash Platform. https://openflashplatform.org/
It turned out that all of our peers supported blob storage better than SFTP, which has some “show stopper” problems like forced outages caused by mandatory host key rotations.
The standards say SFTP. Most everyone ignores that part and have been using S3 buckets by bilateral consensus instead for years.
But also while it should be faster than it is, how would an S3 designed around SSDs look differently to an API user? I would think the API is basically the same.
- Directory entries are no longer returned in sorted order in ListObjectsV2
- There's an AppendObject API
- There's a RenameObject API
I suspect it's more to do with the fact that with One Zone is a clean rewrite of large parts of the application stack that makes up S3.
S3 is made up of hundreds of microservices[0], there probably isn't anyone at Amazon that actually understands the whole system. Refactoring it to support these features probably requires coordination between a lot of different teams. They might have petabytes of metadata, making a change to how metadata is persisted probably requires a massive risky data migration.
[0]: "All in, S3 today is composed of hundreds of microservices" - https://www.allthingsdistributed.com/2023/07/building-and-op...
If you do a clever bit of caching work in your app like we have with our apps Slyp and SlypBusiness, you could even make it insanely cheap.
We have a file manager that is ultimately responsible for images, files, videos and whatnot. It loads the images/files from the downloaded local cache when requested by a feature in the app.
A standard feature uploads and downloads using signed urls obtained from the backend service. The manager increments the file's version in the backend on each successful upload. For download, the manager compares the version against the locally cached version. If there is an update, the new file is downloaded in the background and overwrites the local cache and notifies all features using the file inside the app.
And you're going to have to pay $15 of storage costs for your terabyte.
Cloudflare can shut you down and charge an arbitrarily high price if they don't like you and feel you're abusing their low prices.
Maybe maybe bringing their own IP would have solved the problem, but Cloudflare was obstructing everything and then did a cutoff without proper warning. It was really bad on Cloudflare's part.
They really hated that particular customer.
Any technology that lowers latency, memory, or cost must be appreciated and recommended.
I will not be trusting Cloudflare with anything production critical.
There are a lot of very competitive alternatives around today are largely unknown. For example B2 Overdrive by Backblaze boasting up to 1 Tbps, Wasabi offering byte-compatible S3 storage with no egress fees for reasonable use and Cloudflare R2 also with no egress, all priced comfortably under S3 before egress (which the the real kicker...how do you even put a cost on lock-in?).
Not shilling any of these. We use R2 and Wasabi as our storage backends...so we obviously like them, but are not paid to say good things about them.
(1) https://www.backblaze.com/cloud-storage/b2-overdrive (2) https://wasabi.com/cloud-object-storage/wasabi-fire (3) https://carolinacloud.substack.com/p/pushing-r2-to-its-limit...
It's not "cheap". 6 months of S3 is at around price of outright buying 4TB SSD at retail price. That before you do any IOPS to it
S3 is terrible deal on any front. it's just easy
(Don't bother with Ceph Object Gateway unless you really really need it - since you're programming your own application, access the storage pool directly with librados)
For one, this has no redundancy and doesn't factor the cost of the machine providing the storage. But also it's an upfront cost for storing 4TB. It would be implied you wouldn't literally store 4TB on the SSD because now you're out of capacity for more data. If you only have 2GB of data in a bucket then S3 is still magnitudes cheaper, faster, and more redundant than anything you could put together yourself because the cost is spread across all the users of the service.
Reality is there are so many times that S3 makes the most sense that it makes people short-circuit and always pick S3 despite the huge hidden IOPS and bandwidth costs everyone rightfully tries to point out.
Of course, that's list prices, you can get savings plans and RIs and discounts.
The cost of RAM per hour in AWS isn't cheap.
What is your datacenter costs for multiple different buildings (assuming colocation/office space)? How much are you going to pay for the electricity? How much are you paying for networking? How much is rent for each U of server? How much do you spend on payroll to have knowledgeable people to manage it 24 hours a day, 7 days a week? Suddenly its not just a one-time $50 payment.
Maybe you don't actually need all of that, just some slices of some of that whole stack of things. Probably true in many situations. But just looking at it as a couple of sticks of RAM is not a like-for-like comparison.
I'm not arguing the cloud is cheap. You can definitely go cheaper and choose your own cost optimizations when managing the hardware yourself, deciding what is actually a necessary cost and what isn't. I totally agree AWS overchages for a lot of what they do; they price their stuff at a premium because they know businesses will pay it. But hosting 2GB of stuff in RAM reliably and indefinitely to anywhere near the guarantees of S3 is going to be a lot more than just a single $50 payment.
Things always seem free when you've already spent the money. It doesn't change the fact you spent quite a bit of money, you're just failing to account for it properly.
The standard here was I've just got 2GB of data I need to reliably store someplace. Its an assumption I've got multiple servers in multiple physically separate buildings already lying around.
This is badly misunderstanding the problem: S3 is a highly available service with geographic redundancy and a huge range of integrated features. Your comparison would need to be updated to include multiple running servers in addition to redundant storage, and the software stack implementing things like immutability, not to mention all of the security features.
That’s not to say that you can’t build equivalents for the parts you use but you either need massive scale or giving up features to do that. For example, if you can tolerate bitrot or long access times if hardware fails, you can definitely get a lower cost per terabyte.
The reason why most people don’t do that, even when they have scale, is that it adds cost and risk everywhere else if you need engineering/ops people working on storage. If you’re, say, the internet archive that might make sense—it’s quite literally why your organization exists—but most other places are going to see all of the integrated features that they don’t have to build and operate paying for the difference between S3 and physical media pricing. For example, if I want to process files as soon as they’re uploaded or have immutability, I can just turn that on rather than having to build more services.
You just plug it in and basically it runs. Thats pretty much it.
Having to explain to an HN audience how installing an SSD is trivial is weird.
If you don't pay the premium for redundancy, your S3 is the same sitting duck as the single SSD in your office, albeit an expensive one. Thinking AWS and therefore integrity and security is magical thinking.
Ask those who lost their data in Bahrain AWS.
But you are entitled to your expensive opinion. Some of us just don't buy it is all.
No, that was literally the first paragraph of my original comment.
> If you don't pay the premium for redundancy, your S3 is the same sitting duck as the single SSD in your office, albeit an expensive one.
Again, this is misunderstanding what the service provides relative to raw storage. I suggest reading the documentation about reliability and thinking about how that SSD in your office fails to provide either geographic redundancy (which all but the one-zone flavor of S3 has) or bitrot protection. Similarly, while it is true that some security concerns still require you to configure things correctly, you should think about the infrastructure you would need to build to match the core service: none of that is something you can’t match but building and operating that has costs.
> Ask those who lost their data in Bahrain AWS.
Let’s think about this slightly longer: how is the SSD in your Bahrain office doing? Iran has to hit multiple AWS data centers to lose your S3 data so to match that you need to build a system which syncs changes at multiple separate campuses in the region, which is significantly more expensive, and when you realize that you probably want a copy somewhere far away, the work to build that will be more than checking a box — and don’t forget that you have to regularly test all of this to make sure you can trust it to work in a disaster.
Again, as readers of the original comment know, I’m not saying just you can’t do this but that it’s a category error to think the comparison is the raw hardware and not the combination of redundant hardware, software, and staffing you need to match capabilities. You can beat that with enough scale but that’s a very different equation.
You shifted the conversation to redundancy and how AWS provides bells and whistles. You are consistent in your position, just not with me and not with PunchyHamster because we both are comparing costs by self hosting.
And for those of you who are willing to pay the premium and don't care about costs, like you, would obviously find AWS attractive.
For a startup like ours and many others, cost is everything. And spending a day or two for bitrot protection and having a couple of SSDs for redundancy and scalability is a no brainer.
You don't need big buildings for resilience. I suggest building things that require real scaling and resilience and you would know that it is about being distributed, having low cost, and owning your systems.
Your argument that it is expensive and tedious to maintain compared to AWS S3 is false. No it isn't.
Fear mongering only works against those who haven't built reliable systems. Building an S3 equivalent with cheap providers for redundancy or having one's own small resilient distributed hardware with staffing most often dwarfs the horribly insane premium paid for AWS S3 services.
But this is only if you want the redundancy which is a different topic altogether.
If all you want is a single replica, which was what PunchyHamster originally commented on, the base price of what AWS S3 provides, which does not include redundancy, makes even lesser sense compared to having your own SSD.
The biggest reason people pay for S3 is because of data transfer cost to other cloud services they are using.
Fundamentally you’re playing a commodity game and that means you need a consistent edge to be worth the fixed costs. If your fully-loaded advantage is less than, say, 30% you’re better off negotiating with your AWS rep and investing your engineering resources in something your customers want.
Only someone who had never done this would say this.
Really? No, it's not a few percent.
This is very different from the idea that the savings are a few percent. In almost all cases the savings are nothing or a large fraction.
Of course, if you had paid extra for the backups and redundancy, your data would survive.
So to address your original point, unless you are paying extra for redundancy, it is cheaper if you use a 4TB SSD at retail price as the original commenter discussed.
The parent commentator is under the illusion that AWS automatically means security and scale and reliability.
I was merely pointing out to him that Iranian attacks must serve as a wake up call for him.
I'd bet the majority of projects, but perhaps not the majority of traffic, can run adequately on a single server and tolerate a few hour maintenance window one weekend every month. Don't waste money on overkill.
First off, for geo-redundancy you need to pay extra, you only get region, and with recent strait crisis amazon already showed that can be hit and be unrecoverable.
And you're badly misunderstanding the point; that it is fucking expensive from every angle.
You can get geo-redundancy cheaper (their competitions prove it all the time); you can get IOPS cheaper; and with massive difference in end of the month bill.
Especially if you need a tons of bandwidth, S3 is very, very, very expensive way to get there.
It might be acceptable if you just run a website, but once you start running data-heavy operations on it, a big SSD attached to server can be order of magnitude cheaper than running it on S3 (and you can still use it to backup the data in case something fails)
Yes you can make the argument for that specific use case.
For everyday use case, using s3 as your primary storage is costly and is not at all ideal.
I’d also like a “shitter” storage service. I.e no redundancy, no multi-AZ. Just really cheap storage for stuff I don’t really care about losing, so that I can store a bunch of stuff in bulk that can easily be recovered.
What you need to understand is that a modern hard drives are severely limited in the amount of IOPS they can perform. Disk size has grown, but read and write speeds have not kept up with it.
In 1990 you could read a whole hard drive in 2 minutes, in 2026 as sizes have grown it takes the entire day.
As a system designer you want the maximum utilisation of your hardware, so you want to be close to saturating the IO that your drives are capable of.
What that means in practice is that only a tiny portion of the drive can be allocated to files that are read often.
That leaves the rest of the drive for files which are not used very often.