Hacker News Reader

Comment by danols | Hacker News Reader

I don't understand how it is possible in a functioning market economy to have 8000% markup on a utility product like bandwidth. Seems more like an oligopoly where a few big actors with a big moat has agreed to keep prices inflated. How can a sustained 8000% markup for a product like bandwidth not be considered price fixing?

source: https://blog.cloudflare.com/aws-egregious-egress/

View on HN

Because AWS doesn’t actually want to collect the egress fees. They want you to avoid egress entirely for high-value services, which has all kinds of excellent implications for AWS:

Third party SaaS offerings that move large amounts of data are effectively forced to host in AWS.

Want a small number of high-value servers (e.g. big GPUs, etc) in your own data center or colo to use for non-availability-critical purposes integrated with the rest of your AWS stack? You’d better price in egress!

Want to gradually transition to a competing cloud? Good luck, egress will bankrupt you before you finish the transition.

amluto is spot on. AWS don't want to make money off of egress, they want to make it non-viable to move anything out of AWS that isn't being served to an end consumer.

It is because amulo has a point is why Cloudflare stands a good chance to disrupt AWS in at least few market segments, if not all.

More: https://stratechery.com/2021/cloudflares-disruption/

Time to move hosting back in house.

I wonder, if you made a video streaming app on top of AWS you could negotiate a significant reduction of fees, since you're not ruining their business by transferring out valuable data to competitors. And if that's the case, I wonder if that constitutes a breach of net neutrality in practice, since in practice, a major part of bandwidth costs will depend on what kind of information you are transferring.

My suspicion is that not even Netflix gets the egress out of AWS at commodity market rate. The vast majority of traffic will be handled by their own CDN

Netflix doesn't pay anything for egress of video because it doesn't serve any video from AWS.

As for everything else, AWS offers tiered pricing for everyone and it's the same for everyone. They just don't publish the tiers at that level, but yes, Netflix pays less than you probably do because they're in the higher tier -- but they pay the same as all the other companies at that tier.

Thanks! I had been curious about that for quite some time now.

You don't need to speculate. https://openconnect.netflix.com/en/

Your link has no information about their egress pricing with AWS which is the only thing I was speculating about.

> Want to gradually transition to a competing cloud? Good luck, egress will bankrupt you before you finish the transition.

They do offer the Snowball to lower transfer out cost, they also offer Direct Connect which is more convenient and can be cheaper.

For sure it won't be cheap, but being able to transfer 100 TB for less than 3k isn't too bad. Storing that on S3 would cost you nearly that per month... if you can't spare a month of cost to migrate, you may want to rethink your pricing.

For Direct Connect, they still charge for egress.

> For Direct Connect, they still charge for egress.

They do for both Snowball and Direct Connect, but it's 2 cents per GB, which is what I used to give the 3k estimate.

That still sounds like price fixing, but for a different purpose.

The Cloudflare blog post really only looks at wholesale cost of bandwidth and compares it to the price AWS charges. But I think it's missing a huge component of all of the magic that happens inside AWS between those two things.

I've seen some of the inner workings of the big cloud providers' networking stacks. The networking infrastructure, the software that runs it, the software that exposes it to customers, the thousands of engineers working at any given moment in AWS/GCP/Azure's NOCs to maintain uptime are truly some of the most impressive technical marvels I have ever seen. They aren't as sexy to discuss on HN as something like the managed containers services, functions as a service, EC2 etc, but the networking stacks like the VPC, NAT gateways, subnet routing, privatelinks, security groups, ENIs, nitro cards, etc are pure magic as far as I'm concerned and are so so so much more complicated than a standard data center's networking stack, or even Cloudflare's stack.

To use Cloudflare's "bucket of water" metaphor, AWS isn't even close to just being a dumb bucket of water that you fill with water and then get charged to take out the water. There is so much that happens inside of that bucket to segment your water into different pipes, routing your water in all kinds of customer-customizable ways for many different use cases, mixing/heating/cooling your water as you need, all while guaranteeing things like making sure your water arrives exactly where it is supposed to arrive and doesn't get contaminated or leaked along the way.

Does AWS make a big markup on bandwidth? Yea, surely they do. But is it as simple as Cloudflare says it is? Not even close.

Yes, their network stacks are definitely complex and cost a lot to maintain, I'm sure. But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity. In fact it makes the offering less attractive if the complexity isn't sufficiently abstracted away and distract from product work or if their abstractions are leaky.

Recently I've been working with https://fly.io/ for a new app and it's a breath of fresh air compared to working with the big cloud providers. They offer simple but robust networking primitives built on top of ipv6 and WireGuard and provide a ton of value add on top like global distribution & load balancing, service discovery, TLS termination, all of which just work exactly like I'd expect it to, out of the box without any configuration on my side.

EDIT: Almost forgot to mention: their egress costs are also much more reasonable: https://fly.io/docs/about/pricing/#outbound-data-transfer

I'm watching fly.io with interest, I want to see how they handle the first major incidents - response time, lessons learnt, transparency before I trust them with a production site though. Most SRE skills related to your own operations are all learnt on the battlefield and not via some cliche must-read book from Google engineers afaic.

If its Linode style - delayed status page updates - sometimes as much as 15minutes, zero detail post-mortems - this problem has been fixed by our engineers thank you yada yada, and same issues repeat six months down the line then I will be understandably disappointed.

I've only been with them through one major incident so far, and I recall them handling it reasonably well.

You can see them responding to customers and providing updates in real time here: https://community.fly.io/t/there-seems-to-be-an-outage-with-...

And a detailed postmortem here: https://community.fly.io/t/major-outage-portmortem-2021-10-1...

They also update their status page pretty diligently whenever something goes wrong even for things that don't necessarily impact all customers (the only recent item on there that affected my app directly was the Oct 13 one from what I can remember): https://status.flyio.net/history

> But that doesn't necessarily make it a good deal if the customer isn't able to derive enough additional value from all that complexity.

It’s simply obvious that it’s not a good deal if you’re not their target customer with a use case they cater to. However, it could be a good deal if you have a relevant use case. Unless it’s being suggested that AWS caters to everyone in all cases then it adds nothing to the conversation to point it out.

https://www.hetzner.com/cloud gives you 20TB bandwidth for €3.49/mo VMs, which I've essentially regarded as Hetzner gives unlimited free bandwidth for all servers.

Being lynched for egregious egress fees is only something I've experienced when using mega corp's clouds, where economies of scale suggests their vastly larger size should allow them to provide even better value.

But that's in a normal market, not the artificial lock-in mega cloud corps enjoy where they're able to distort customer behavior from artificially high pricing.

I'm a Hetzner home user and a huge fan, but let's not compare the quality of networking you get for free from them with the networking you get from AWS.

I don't think I've seen a latency spike on AWS in 10 years. Hetzner, it's often possible to observe latency and drops over 10 minutes (and the situation hasn't changed much in about 10 years)

In all the years I've used Hetzner I've never observed these random 10 minute latency drops you speak of. They've always had much faster internet access then I've ever been able to get from my home broadband so I'll even SSH into & use them for network intensive dev tasks like iterating on a new Docker container since it's able to download & build the image packages in a fraction of the time.

The primary issue I have with them is latency access to their DE/FI data centers from the US, if their US DC offered dedicated servers I would be migrating to over to use them instead.

They launched Cloud in the US this month, very likely dedicated will be offered soon enough. The bang for buck on Hetzner is insane, really love them, but have and would rip them out of any business environment I come across, largely due to network quality and attitude to support.

If you haven't experienced Google translating insistently German responses from one of their DC techs you probably haven't been using them for long enough ;)

As for networking, would encourage installing something like Smokeping

I've needed to access their tech support 1 time when my HDD failed and a couple of times for new SSL certs before LetsEncrypt, who were always responsive and supportive. Don't see how derogatory characterizations of their DC techs is in anyway necessary.

But I don't really access AWS support either, when something doesn't work I've just killed the VM and started a new one. It's less disposable with bare metal servers, I can physically restart the server from their control panel or if issues are not fixable, reset the server with a new Linux OS image, which granted would be a lot more time consuming.

I will add that whilst I'm not in the business of dictating which cloud services business customers would use, I'd agree that I would recommend AWS over Hetzner who are a) paying for & would have to administer it themselves and b) is going to have access to all the managed services they would ever need in future.

I would still recommend they consider Hetzner for any high-resources intensive workloads where their raw compute is vastly less expensive. I'll also chose the cheaper reoccurring cost over convenience when I'm able to self-service it myself.

Hetzner support when I have needed it, has always been faster and of better quality than AWS or Azure. All emails and talking was in English

Hetzner has its network hiccups sometimes, but AWS quality may be a joke if you really care about latency tails and even median under any significant load. I didn't analyze the networking itself, but - you run in a VM and share host machine with other clients VMs - you just can't get stable latencies this way. It's night and day when you migrate to baremetal Hetzner and observe how latencies change. (Again - it's about dedicated baremetal - I know nothing about Hetzner's cloud)

It's not really that much magic. It's just a variation of EVPN-VXLAN plus smart NICs that segments and directs the traffic. Then they have normal VM hosts or nowadays devices with ASICs that handle the GW and NAT functionality.

Custom ASICs (Nitro chips) aren't magic? Maybe so, but they cost money to develop.

All of the other networking stuff ( Security Groups, NACLs, flow logs, VPCs, subnets, etc.) you don't directly pay for, isn't magic either, but also cost money.

Nitro is just a fancy converged host adapter with Smart NIC functionality. It's unclear to the industry how much of Nitro is custom, and how much of it is existing IP that is cobbled together (e.g. Graviton and the ARM Neoverse cores).

The ASICs are on the fabric doing the routing and NAT for all the traffic in the AZ. These ASIC are unlikely to be custom. Hyperscale operators typically use open networking hardware with merchant silicon. You can get open networking hardware to do all sorts of packet manipulation, and these devices are a cheaper than traditional manufacturers, but more powerful as they expose more low-level interfaces.

All those features you talk about are implemented from features that are provided by these hardware platforms.

AWS is just putting an managed service together from them, no different to how they take postgres, do some tweaks and rebrand it as an AWS service.

It's weird to me how people think contrasting a raw pipe billed on 95th percentile to a service like S3 or Cloudflare is in any way a fair comparison.

S3 has its own data retrieval costs, as do several of their managed services.

Those are separate charges from the EC2 costs the Cloudflare blog post discussed.

Egress does not mean S3 or Cloudflare. Egress is the raw pipe billing from AWS to the wider internet. Other services are priced differently.

That’s where folks are revealing how clueless they are.

Raw pipe isn’t priced in GB it’s in capacity. To serve GB on Black Friday you need to provision far far more capacity

Yeah but people can do that math. Peak bandwidth is not 200x higher than average bandwidth.

Cloudflare used a 5x multiplier. How high do you think it needs to be? Does total AWS bandwidth even go up that much on black friday?

The thing about a competitive market is that it requires that cost of switching is low. The higher the cost of switching, the more leverage the company has over you.

SaaS and cloud providers in particular have a lot of leverage over their customers due to the typically high cost of switching. E.g. what does it cost to move your whole infrastructure to a new cloud? They can basically gouge you up to that cost threshold.

These cost of switching actually leads to a market that is not competitive in the traditional sense. Yes, competition on the surface, but monopoly within.

It's true that gouging can't be as egregious as a true monopoly environment, but certainly much higher than a low cost of switching environment.

The big tell is margins. Anytime a company can sustain excessively high margins, it's usually a tell that theres a lack of a competitive market. A lot of software companies get high margins either by being first to market (by many years), or leveraging high cost of switching.

Despite software being cheap to deliver, it's also cheap for your competitors to deliver, so margins should be low in a highly competitive market, regardless of marginal cost of production.

I do believe new regulation is needed to handle these business models such that we can continue to foster a competitive environment. It would have to be very carefully crafted to prevent unintended second order effects of course.

Because bandwidth isn't the only AWS product, but you can't buy bandwidth for cheap from XYZ Inc. and connect that to your AWS account. That's not technically possible, you have to buy it from AWS.

So instead of using the cheapest vendor, companies use AWS for a ton of reasons (broad support, everyone knows how to work with it so it makes recruiting easier, hundreds of features, etc...) and their high bandwidth bill is worth it.

AWS is a luxury product and the outbound charge is the delivery fee. You will also find that the delivery fee for a Louis Vuitton bag is more than for a pizza.

AWS spent the first decade of existence advertising to executives as a way to significantly decrease their cost of IT, not as the Louis Vuitton of IT.

Even a Rolls Royce is cheaper than building your own equivalent car.

If you need one car, but probably not if you need tens or hundreds of thousands.

No, it's a very valuable tool for corporate IT.

The egress is a way to snare a lot of extra margin.

It's 100% clear that orgs. wanting to host large public platforms will have 'major concerns' over this pricing issue but that's not their target market.

I'll bet most HNeers are thinking in terms of 'Hosting my App' there vs. AWS Bread and Butter is mostly hosting corporate IT services, which is a different thing.

So it is:

https://www.nytimes.com/2020/10/25/technology/apple-google-s...

I mean the politicians are in on it, and turn against them when their ability to control elections for their own purposes comes into question.

Look at the “testimony” of Robinhood guy. It’s a scripted TV show; you lose.

How much margin do you think there is in coffee or soft drinks?

Apparently nowhere near the 8000%. After https://www.profitableventure.com/income-soda-retailers-marg...:

> Restaurants and bars have around a 70% profit margin on a bottle of soda and soft drink, while retailers typically have between 30–50%.

First - that is 'huge margin' far beyond their other offerings, but on fountain soda it's much greater.

Theatres make 0% margin on the Ticket and probably 90% on the soda and popocorn.

The analogy is reasonable: 'The Food' , 'The Film' or 'The EC2's are the primary ticket item and it's ballpark competitive on a cost basis.

AWS is not competing to be the cheap, low-margin host for YourFreeImageSite.Com and so their prices are going to be different on that point.

Coke and Pepsi have that kind of mark-up on their aluminum can sugar water products.

What about a $3 Coke in a vending machine?

At a grocery store: 24 pack of Coke being $7 but one 20oz bottle being 1.99?

Or that $6 coke at the movies, ballpark, or Disneyland?

$7 for 24 cans of Coke? I live in a wrong country.

This is like going to a supermarket and complaining that the plastic bags for you to store your shopping in cost 10 cents which is a 8000% markup, as if plastic bags are why you're in a supermarket

This is a horrible example that doesn't not apply cleanly.

To fix it, let's change it from buy per bag to per gram you want to leave the store.

So your canned soup costs at least 3 times more to leave the supermarket then in cost to buy it than to eat it in the supermarket.

This is not even an exaggeration. s3 is 0.024/GB and egress bandwidth is 0.09/GB

TOC 'Total Cost of Ownership'.

Nobody is paying for egress, they are paying for everything else.

AWS is focused on corporate IT so egress was always a secondary thing for them.

It might actually be more profitable for them to have uber-markups on that egress then to serve the web space that requires cheap bandwidth.

Its not 8000% on a markup on raw metered bandwidth, they're serving data from a service they operate. Comparing acting like they're just "marking up" bandwidth is an unfair comparison.

What do you mean, how is it possible? That's how markets work. You open a booth in the market, and you can ask any price you want. You can ask for a nickle or a million bucks. There's no rule about how much money you can ask for.

As for your assertion that there's a price fixing conspiracy, who do you imagine they're colluding with? What other company do you find charging these prices?

The parent comment misunderstands that most of Amazon's customers are very clearly not extremely concerned about the bandwidth costs (even if they'd all happily receive a lower bandwidth bill from AWS). They're not there for the cheap bandwidth. That's the central flaw in the parent's bafflement about markets, they did a exceptionally flat appraisal of the AWS value proposition, as though AWS isn't a gigantic business of many dozens of service offerings which act as a customer magnet and retention mechanism.

And many customers simply are not doing petabytes per month.

AWS wasn't meant to support Netflix type loads, those guys will build their own CDN's. For everyone else who wants security groups (totally for free) when some other firewall vendors would charge a small fortune to provision 1Gbps capacity for these types of services... they are fine with the price.

The other issue cloudflare doesn't understand is the bandwidth pricing they quote is for capacity (ie, 10Gbps). AWS has to have enough capacity to serve the peak, but the customer only has to pay for data used. I'm sure at low points the data is free, but maintaining the black friday / superbowl capacity is expensive, so you are paying a premium for that too.

Urm, Netflix is BUILT ON AWS (or at least was 5yr ago):

> Netflix uses Amazon Web Services (AWS) for nearly all its computing and storage needs, including databases, analytics, recommendation engines, video transcoding, and more—hundreds of functions that in total use more than 100,000 server instances on AWS.

https://aws.amazon.com/solutions/case-studies/netflix-kinesi...

You're both right. Netflix is built on AWS - for the sort of workloads you quoted.

It also has it's own entirely separate content delivery system, Open Connect. https://blog.apnic.net/2018/06/20/netflix-content-distributi...

>Netflix Open Connect is our purpose-built Content Delivery Network (CDN) responsible for serving 100% of our video traffic.

TIL, cheers

1) $50/TB is not insane pricing relatively. I used to pay something like $3/GB (way back) which would be $3,000/TB. There are still plenty of folks charging high rates.

2) $50/TB may not be a big part of some customers bills. If it is on cloudfront you drop to 0.02/GB or $20/TB for larger volumes.

3) AWS doesn't charge separately for some services that wrap around networking. Their nitro instances have a pretty good networking story.

4) Especially with this new free tier on cloudfront, a fair number of users will never hit 1TB per month (free) data.

It’s absolutely insane compared with what bare metal providers charge.

But that's apples and oranges... With AWS you get VPCs, Security Groups, NACLs, flow logs, DDoS protection (limited), public IPv4s, custom hardware to make sure you can actually use the 10/40/100Gbps pipe on your server, etc. etc. for "free" ( you only pay for those via bandwidth). The best most bare metal providers have is a firewall(security groups), and that's it.

I can get all the things you stated, including "unlimited" bandwidth for a few bucks a month if I just rent a VPS. I never understood the flocking to AWS. Does nobody like running a server anymore?

> I never understood the flocking to AWS.

Nobody ever got fired for using AWS.

BTW you don't need to "run your own server" per se on the bare metal or VPS providers. You can auto-provision with Terraform, run Kubernetes or Nomad/Consul, etc. You have to do some work to set up your templates and the environment you will use but once it's created you can stamp out copies of it endlessly.

There are cases where AWS et. al. make sense. The bottom line is that you need to do your own spreadsheets modeling your own workload and compare costs. Include extra labor for managing your own stuff and compare it to the added costs of AWS.

What you'll often find is that AWS and such are cheaper at a small to medium scale and DIY becomes cheaper (sometimes radically so) at larger scale.

I don't like being bound to a particular machine (including indirectly via a VM) and having to manually intervene if something goes wrong with that machine. AWS auto scaling, and the equivalent feature from the other big cloud providers, frees me from that.

Most VPS providers don't have VPCs ( private network for all your VPS), or flow logs, and charge you for extra public IPs.

what do bare metal providers charge

Just as an example: Hetzner's dedicated servers, which start around US$30/month, come with unlimited 1gbit connections (really unlimited, not "unlimited until we decide to throttle you").

If you max out that connection non-stop, you can push about 330 TB/month. The same amount of bandwidth from EC2 would cost roughly US$20,000.

Dedicated servers from Hetzner and OVH is what I use. Very sweet deals they have.

Are you actually able to max that out to all network peers? Have you measured this? That sounds fantastic.

No direct experience with Hetzner but for DataPacket, OVH, and FDCservers the answer is yes.

These companies are still pretty profitable.

If you colocate your own hardware you can go even lower than the bare metal hosts.

https://www.hetzner.com/cloud gives you 20TB free bandwidth a month for all its servers, including its €3.49/mo VMs.

They also offer comparatively much cheaper hardware, which is why I've been using them for all my heavy workloads for 8+ years.

Most are unlimited. But the ones where you have an actual bandwidth cap, it comes out at less than $5/TB (if we're including the machine itself), the bandwidth itself, probably less than $1/TB

Some bare metal providers have allocation based on server provisioning rather than specific egress limits. But CloudFront is cheaper with commit pricing because CF commit pricing is very very cheap and you have to factor in the cost of said reverse proxy nodes and only so much can be delivered per node. As you factor in management of the nodes it becomes even more favorable to the hyper scaler like CloudFront.

I have seen large scale deals where CloudFront comes in cheaper than what the smaller CDNs built on bare mantel can reasonably offer even with sizable commits.

Go to datapacket.com or hetzner.com for examples.

VPS providers like Vultr and Digital Ocean are a bit more but still a lot cheaper than the big three.

Bandwidth prices at the big clouds are ludicrously high.

$500/mo unlimited egress

These are not real numbers. I've tried the "free" and "unlimited" providers. They will drop you if you take them up on their offers.

Damn that is wild when you just look at the absolute figures. Really? Where? Hetzner or some other bare metal provider?

Reply on news.ycombinator.com