Hacker News Reader

Comment by fxtentacle | Hacker News Reader

I would say the main story here is that AWS overcharges on traffic to such a ridiculous degree. My SaaS regularly consumes 50+ TB in traffic, but I pay €150 in monthly fees for that.

For someone coming from my perspective, that would be a huge and unpleasant surprise to be billed $2600 for a service that I know costs $180 elsewhere.

Edit: especially so because people keep repeating the mantra that one should use the cloud to save money by not paying for what you don't use. Obviously, my 3 Xeon servers with 64 GB RAM each are way overpowered for sending out some GB of static files, but I wanted to have a bit of redundancy. But with my setup, there should be plenty of obvious inefficiencies for "the cloud" to eliminate

=> It feels like cloud should be cheaper than my dedicated servers. But it's not, and that is the unpleasant surprise.

View on HN

IMO, it's not so much that they overcharge as that the service was created and priced for a different purpose. It's possible to serve huge files to tens of thousands of people, but it's not really what it's meant for. Naturally, the pricing isn't really optimized for that, and a service that's designed for that can be an order of magnitude cheaper.

Seems to me S3 is more designed for highly reliable data storage. If you wanna store a bunch of data in a way that's highly reliable and resilient to hardware failures and also not have to worry about managing RAIDs and clusters of servers for the data size, S3 is just the thing, and probably priced pretty fairly.

It's also a capable and flexible service. It's possible to use it for things it isn't really designed for and have it behave pretty well. Well enough that you can mostly ignore that the other thing isn't really what it's designed around. The "mostly" is important, though. If you start hitting any extremes while using it in a different way, then you certainly can run into some pathological cases in the pricing structure and pay way too much for something that would have been cheaper in a service dedicated to that.

S3 is much cheaper than the egress traffic. As such, this isn't so much about S3 because you'd be paying the same price for that traffic if you were using EC2 instead of S3.

Looking purely at the traffic price, I believe the word choice "overcharging" is warranted if you can buy the exact same product elsewhere for 90% less.

They over charge.

Their cost is less than .01 cents per gb and look at their published pricing for customers.

And I’m including networking equipment as well as head count in my cost, not just the cost of pipes.

They charge so much for it as a disincentive to use it for CDN purposes. It is not architected to act as a CDN for people, and if they don't overcharge for egress, people will use it as a CDN because it is easy.

In that case, why isn’t EC2 egresa pricing more competitive with other services?

Same reason?

It's dubious that they don't want you using S3 for one of the advertised use-cases but even if we accept that, the logic wouldn't apply to EC2 since there's even less of a chance of that impacting other users. Building your own CDN on EC2 isn't going to produce a different workload than many other activities which are actively encouraged.

Probably not. They have a tutorial on how to host static data with S3, so it appears to be an intended use of the service: https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosti...

If serving static files to thousands and millions of users is not cloudfront is for then I am not sure what else would a CDN be used for .

You don't need to manage any storage to get your bill to drop, S3 can be fronted by your cached proxy (minio or just plain nginx works easily) it is trivial to setup on OVH or hetzner or any other VPS provider.

You get all the benefits of S3 without the egress bill of a major cloud

> I am not sure what else would a CDN be used for

He used a CDN, just didn't read the fineprint:

“Things I should’ve known but didn’t.” Did you know that “The maximum file size Cloudflare’s CDN caches is 512MB for Free, Pro, and Business customers and 5GB for Enterprise customers.” That’s right, Cloudflare saw requests for a 13.7 GB file and sent them straight to origin every time BY DESIGN. Ouch!

It's another case of misunderstanding and misusing a tool/service in my opinion. Cloudflare (and Cloudfront) are intended to be website CDNs. Their use case is _not_ serving 13.7GB objects - and so they didn't. (Whether "passing those requests straight back to the origin every time" is a better/more customer friendly thing than " blocking those requests since they're not being caches due to an object size rule" is a good question...)

If you want "inexpensive large file hosting/serving", you don't use a combinations of "a service with 11 nines of durability" and "a web-infrastructure and website-security company, providing content-delivery-network". As you rightly point out, duct taping this together on VPS providers competing on low-cost bandwidth is the "right thing" for this use case. The gold-plating of 11 nines of durability and a CDN with over 200 globally distributed POPs almost (perhaps "should have"?) cost this guy $2.5k for "taking the easy way out without thinking through the consequences".

In my head, it's like he had a Tesla Model S in the driveway, and needed to transport 6 tons of lead bricks across town, and thought "I know, I'll just load them up on the back seat and drive 50 miles!" and then getting surprised the car needed expensive repairs afterwards.

Yup that's exactly what I was saying. Cloudfare is designed for website resources, mostly smallish images and bits of JS and CSS. They would probably regard 5MB as a rather large resource. I'm not gonna blame them for having no idea what to do when somebody tries to run a 13.7GB download through them.

Couldn’t they have blocked it after the HEAD request , s3 sends contentLength in the head request . Many servers for static files do.

It is not the size of file really the problem if he had consumption of 30TB he will still be charged that much .

The b/w costs are very high on the cloud , 5-6x higher than it is outside, I don’t think there any value they provide that justifies it for vast majority of users .

I am pretty skeptical on anyone at all in the world needing 11 nines of durability. You are 100’s of times more likely to have a meteor strike .

All the major service providers have gone down recently. Cloudflare went down just this month . At best they are delivering 4-5 nines practically speaking (loss of one PoP or region is still down )

CDN is exactly the service which needs to be significantly cheaper at scale, if you really need 200 PoPs then you have at minimum 10,000s of users at which and you will have consumption

If the industry’s position is they only want/can service enterprise customers for whom current pricing is within the budget, then their online pricing models/ marketing are incredibly misleading on who they are targeting .

To me it looks like their business models depends on predatory pricing prosumers y hooking them on with the ease of use.

This is very much like credit card industry. If everyone paid on time , CC providers will never make money, predatory loan pricing at 36% or higher is their real revenue source.

Paying on time / reading the ToS only lets them off the hook legally , morally they are both praying on vulnerable users .

If you're using it properly you can even boot VMs off of it, in the same region (S3 data transfer in region to EC2 is free) like this article is talking about. Just don't enable public read on your bucket to avoid a surprise bill.

Always thought that a great statup idea would be to interact with dedicated server provider and offer on top of those cheap servers AWS like services.

The most common services like S3, Kafka, RDS (pg and mysql), Redis should be enough to cover most use cases.

With k8s and a dedicated smart team, it would be a possible adventure.

Moreover it can work great also on-premises. Several old medium business have their own physical infrastructure and they are not yet ready to move to the cloud.

i don't know about the US, but in europe a ton of these smaller, local parties exist. (usually called something like managed infrastructure or thereabouts).

> Always thought that a great statup idea would be to interact with dedicated server provider and offer on top of those cheap servers AWS like services.

Which is what AWS is anyway...? Also, I don't think AWS' data-centers exist in isolation: https://www.datacenterdynamics.com/en/news/wikileaks-publish... Definitely not the Cloudfront PoPs which ought to be leased, surely?

Also see: https://www.tatacommunications.com/solutions/cloud/infra-dc/

And: https://www.vapor.io/vapor-io-partners-with-cloudflare-on-na...

There's a actually a lot of companies doing that. Google 'aws managed services'

Some of those companies are even more eyewateringly expensive than the story in the article.

I worked on a project a few years back where the client were paying $10k/month as a "managed service" fee, on top of about $4k/month worth of platform at full on-demand prices. I showed the client how they could have it all running on reserved instances for Prod and spot instances for dev/staging for under $2k/month - but no, somebody had signed up for $14k+ per month just to have someone to blame/shout at 24x7 if something went wrong. (And that company was ~85% likely to call me and blame it on the app before they even bothered looking to see if the platform was working...)

In addition to my own products, I offer consulting to medium sized companies and my sales pitch is usually:

How much additional profit would it generate if I could cut your cloud costs by 20%?

Usually, the savings are even higher, but 20% is low enough to be easily believable, yet large enough for them to invite me to learn more.

The reason I do it manually is that many companies are afraid of the workflow interruptions usually associated with moving off the cloud. That's why I first analyze their deployment to calculate possible cost savings and then I offer them to hire me to take care of the migration. I'm pretty sure those companies would not feel comfortable switching to a different cloud by themselves.

The problem is the business model; managing all these common services is a big deal and takes serious engineering effort to keep online. You then fairly quickly should enter into relatively high prices to make this cost-efficient.

The cloud providers, however, have a trick: they can eat these costs and make serious $$$ on the actual compute, storage and bandwidth.

If you’re trying to compete with AWS et al based on price for bandwidth etc “but with common services such as S3”, you’re gonna have a really hard time convincing a customer your expensive managed Kafka is reasonably priced. And then you’ll just find yourself competing on just VPS servers...

For S3, it's not what most of the customers think they're buying, but what Amazon is actually selling is a storage service with 11 nines of durability:

https://aws.amazon.com/s3/faqs/#Durability_.26_Data_Protecti...

It's "Simple Storage Service", and it does that one thing (storing practically unlimited amounts of objects) spectacularly well. Like 11 nines well. It doesn't do "inexpensive egress charges" well. Amazon never claimed that though.

I think a lot of people don't care about those 11 nines, and would love to buy something with perhaps only 5 or 4 nines of durability ("Maybe I'll need to upload it again once every few years, that's cool!") but which optimises for cheap bandwidth costs instead. Right now, I think that thing is an inexpensive VPS with an unlimited traffic (Hetzner were offering "unlimited traffic at 1GB/sec for the first 20TB, then throttled to 100MB/sec" for under 10EUR/month last time I went looking. And they're well above "the bottom of the barrel" pricing in cheap VPS land.)

This is part of AWS's business strategy. Overcharge to an absurd extent (90% margins) and then if a surprise bill happens you can just refund it for 10% of the cost. This cost is a rounding error if it allows you to maintain your margins. It's worth it even if only a single person is willing to pay the regular rate.

Out of interest who are your servers with? That seems a good price and might be something I'm interested in...

Not the OP, but it could be Hetzner. They're in that price range, with that kind of hardware. I myself have been a happy customer there for almost 20 years now.

OP here, yes I use the hetzner marketplace for used dedicated servers. I have found that using hardware that's 6+ months old actually decreased failure rate. Plus it's cheaper and they waive the setup fee.

The marketplace has some really good deals on it! Thanks for sharing.

Of of the good aspects of Hetzner that they also offer desktop CPUs which are often more powerful than the so-called enterprise CPUs (see https://jan.rychter.com/enblog/cloud-server-cpu-performance-...).

I've gotten servers from Hivelocity in the past, they like(d) to offer servers with the "enterprise" equivalent of these CPUs, I'm not sure if Intel still sells them, but there used to be a "xeon i3" basically, usually 4 core CPUs clocked to the moon, that supported ECC. They were really good!

Of course the cost of data egress for AWS does not add up to nearly what they charge for it.

But it's totally unfair to say they're "overcharging". The pricing is set up to encourage using the service properly. For example, did you know you get unlimited free (very fast) transfer between s3 <-> ec2?

You are free to use S3 however you want, but the pricing is set up such that people use a proper cdn backed by s3, do as much as they can between ec2 <-> s3, etc. instead of making s3 the backbone of their public site.

If you want to use it in a way it's not intended it will cost you more $$ which is how it should be.

No one prices to "encourage proper use." They price for a number of variables. And I do agree that arguing they're overcharging is unfair; however, SU-pricing standards include triggers for excessive use as part of the customer guarantee. If Amazon has no problem automating the product, they can automate some billing features, like notifications and bandwidth restrictions when a customer is nearing excessive use based on the population use of a particular service.

Amazon's systems should have notified at least twice and required direct confirmation from the customer for their continued use.

We don't just make business decisions like this based off our self-interest. In this case, a failure to be resourceful on Amazon's part gave the customer one of two options: track everything manually minute-by-minute, or be surprised by an excessive bill.

AWS does have budget notifications that you can set up with a few clicks. It doesn't come by default because AWS doesn't know what's the right amount for those notifications. I already get enough emails from AWS for every account that I make.

I can actually think of possible/real workloads that would burn 2700 usd out of blue in two days and those customers would not be happy if AWS blocks their account because AWS thinks they did something wrong.

AWS Support quite often happily refunds these amounts and you don't need to post to Twitter. I've seen them refund a lot more.

Why haven't they done something to prevent this mistakes? They already refund these things, it's just those refunds is a drop in bucket compared to services that enterprise customers are paying for, and AWS engineers are busy building for them.

AWS gives you a lot of power compared to what you can do on your average hosting provider, but sadly there's also a lot of room to shoot yourself in the foot if you don't know what you are doing.

This is just making excuses for a business model built around taking advantage of small mistakes. If they actually cared they'd allow you to set billing circuit breakers on setup.

It's not impossible to script this functionality if needed.

It's difficult enough and error-prone. What customers want is a bulletproof way not to exceed a defined spending limit. We've been asking for it for many years, and the answer has been "Yes, we hear you, we are working on it", and then nothing ever happens, because this is a strategic business decision that would affect their bottom line. Why should they do it if they don't have to?

Billing notifications aren't much use if you can run up a $1000 bill between going to bed and waking up.

Or a week's vacation

> No one prices to "encourage proper use."

Citation needed.

Network throughput is a finite resource. And spikes from noisy neighbors could absolutely degrade throughput performance for everyone else.

So it makes perfect sense for me that part of Amazon's pricing calculus would be based not just on the cost to deliver the feature but the amount of utilization of the overall network they expect for the use cases their pricing supports.

But...their pricing punishes a spiky neighbor exactly as much as somebody dripping a tiny amount of content steadily for a month.

>But it's totally unfair to say they're "overcharging". The pricing is set up to encourage using the service properly.

That's a very charitable take. A more cynical take is that they do it to encourage lock-in. If you already have your data stored on s3, they can get away with overcharging on compute because they know that if you switched to GCP, you'll end up paying more because of the egress costs.

>but the pricing is set up such that people use a proper cdn backed by s3, do as much as they can between ec2 <-> s3, etc. instead of making s3 the backbone of their public site.

I'm not sure what your point here is. Cloudfront costs 8.5 cents per GB of egress, meanwhile ec2 costs 9 cents per GB of egress. half a cent doesn't make the egress charges any less outrageous. That's not even including the per-request charges that cloudfront has, which probably can make it cost more than 9c/GB depending on your usage profile.

> The pricing is set up to encourage using the service properly.

Do you believe that little gem applies to all products and services, or do you reserve it to downplay AWS's notorious price gouging?

Who is your provider?

Reply on news.ycombinator.com