AWS's Egregious Egress | Hacker News Reader

AWS's Egregious Egress

blog.cloudflare.com

742 pointsjgrahamc229 comments

On the Hotel California Effect

If it's expensive to move data out of AWS, it's not just about making it hard for customers to leave AWS. It means that any third-party service that wants to sell to AWS customers must also use AWS.

For example, Snowflake can't really run its own data centers. They need to rent from AWS. If you're a company on AWS, you won't want to spend $0.08/GB sending things to Snowflake.

The high egress cost gives Amazon a lot of power to keep the third-party ecosystem within AWS and without as much negotiating power on things like EC2 costs. Snowflake can't say to Amazon, "if you don't give us a great discount, we're going to move our operations to DigitalOcean" because Amazon would just say, "no you're not. Your customers aren't going to pay $80/TB to load things into Snowflake when you're charging them $23-40/TB for storage. They'd be paying 2-3.5x your storage costs just to load the data into your system!" Yes, I'm sure that Snowflake does get a discount from AWS, but having less negotiating power can make discounts smaller.

There are other reasons to want a third party to use AWS if you're on AWS. Still, the egress pricing seems to make it very hard for third-party tech providers not to use AWS if their customers are using it.

Amazon is really good at leveraging its semi monopolistic presence in different markets. Once you are big enough you only need to tweak a few parameters to dominate the market.

It's interesting how AWS uses egress to this effect. Your comment is very insightful.

They employ teams of economists to model price elasticity of demand. There are likely many orders of magnitude more than a few parameters. And they look at how a change in one product feature or pricing model cannibalizes or compliments other features and prices.

Thank you for this. I’m surprised that CloudFlare didn’t mention it in an otherwise thorough post. After all, the reason they’re trying to pressure AWS to change is that they’d like to offer services to AWS customers without bandwidth markup. Essentially competing with existing AWS services on an even footing.

This to me makes it unlikely that AWS will change their bandwidth pricing. If it was just about this revenue source disappearing, they might bite the bullet and hope that the lower prices would attract more customers in the long term. But bandwidth pricing is the moat around their castle. If it didn’t exist, Snowflake and others would leave. But more than that, new AWS services benefit from the captive audience choosing them by default. Without the moat, each service has to compete on its merits. AWS Lambda, for example would need to be better than CloudFlare Workers or fly.io containers. It won’t win customers like it does today just because it’s “free” bandwidth.

Removing these fees is a risk AWS is currently choosing not to take. And they won’t change their pricing regardless of the number of blog posts that competitors write. The only thing that would force a change is an exodus of customers leaving AWS citing this as a reason.

And I always thought they will drop the price of their EC2 some day to balance things out. But nope.

I am wondering if Amazon could introduce a bandwidth cost rebate, where the bandwidth cost would be subtracted from your final bill at a maximum of 30% of your total VM bill.

But it is like Apple, once you have a monopoly like market they just dont need to. Amazon is buying as many TSMC capacity as they could as demand for Graviton 2 is way above their projected expectation. It seems they are still growing at such a pace despite their size.

The rebate you mention sort of already exists, just restricted to public sector research customers. Think the cap is that egress is waived as long as it is less than 15% of the bill.

AWS looks like a ripe target for Biden's new emphasis on antitrust and competition in business

Except unlike google play for android there are a lot of competitive alternatives.

I've always referred to this as a roach motel model.

Most proprietary vendors do various tactics under this to retain customers via lockin, versus actually good products.

Software file formats serve the same for proprietary software.

"Cloud" vendors provide easy API services that have no analog to on-prem, so that migrations are painful. And the Great Egress is a painful problem with AWS.

Roach motel indeed. They check in, and they don't check out.

> "Cloud" vendors provide easy API services that have no analog to on-prem, so that migrations are painful.

This is true. But just as true as "cloud vendors provide easy API services which provide value over on-prem options that companies pay for". Whether you see it as positive or negative depends on how much value you get from the service.

And features like AWS PrivateLink play right into this strategy. If you're a SaaS provider built on top of AWS, the egress costs heavily incentivize you and your customer to connect over PrivateLink vs over the internet.

This keeps both parties locked into the AWS ecosystem.

These costs are not only on egress traffic to the internet, but also on other ways to get data out of AWS. For example, the snowball appliance that is basically a pile of hard drives that can be shipped to AWS has free ingress, but costs a few cents per GB for egress [0].

[0] https://aws.amazon.com/snowball/pricing/

Just FYI, the generic word for this medium of data transfer is “sneakernet” (https://en.wikipedia.org/wiki/Sneakernet).

There is an option of peering at 2c/g but yeah it’s not particularly cheap either when you compare it to transit costs in the real dc

AWS, may be, subsidizes the costs of its loss-leaders by (what seems like) charging disproportionately for egress. I reckon, as Cloudflare gears up to take on incumbent cloud leaders, it modus operandi will be to commoditize its competitor's advantages.

Apart from CloudFront and Lambda@Edge, Cloudflare has offerings that compete with Lambda (Workers Unlimited), Web Application Firewall, AWS Elemental (Stream), and (app/network) load balancers (Argo, Spectrum, Magic Transit). Check out Cloudflare's bandwidth / usage pricing for some of those, and it isn't much different than AWS.

Cloudflare, I'd predict, will eventually arrive at a similar pricing model for its IaaS outside of the Bandwidth Alliance (which, ironically, is Hotel California with select though multiple providers at play).

> Apart from CloudFront and Lambda@Edge, Cloudflare has offerings that compete with Lambda (Workers Unlimited)

Oh yeah, and it comes with a KV store capable of storing data across requests. It's awesome.

Just like all the AWS resellers: cheap to launch, expensive to succeed. :)

I don’t think so. None of AWS services seem abnormally cheap. Do you have an example where it seems like they are selling a service at a loss?

I don't know for sure if they sell services at a loss, but they give away some glue infrastructure at no cost to the customer. AWS Batch, AWS Lake Formation, EC2 Auto-Scaling, Data Migration Service (free for 6 months, no-limits), AWS CodeStar, AWS BeanStalk as few examples. I believe, in almost all cases (save traffic over transits like cross-account VPCs, load-balancers, internet-gateways, NATs, etc), intra-region transfers are also free.

https://duckduckgo.com/?q=%22There+is+no+additional+charge+f...

And they do the same with some of their VPN/firewall products which are totally over priced except compared with self hosted products

This is a great point I wouldn't even have thought of, and seems like something anti-trust regulators should be interested in.

i could see this move eventually going sour for amazon down the line, there's always that one false play that sinks a giant. it probably won't be this one or one anytime soon, but definitely seems like that strategy may eventually be ruled against. but who knows, i guess if enough people are willing to accept it then they will stay that way, i honestly was hosting on amazon for a bit, nothing major but the service for me did not justify the cost. but i am somewhat old school.

Working in realtime games that require high bandwidth usage, AWS and basically every other public cloud is fundamentally unusable for us because of this exact problem. Our bare metal infrastructure for the Hypixel Minecraft network uses 3-4PB per month, so we simply lease a 100gbps transit link billed at 95th percentile.

Last I checked, AWS wanted ~$200k per MONTH, for just bandwidth (no compute, memory, storage, or anything else). We'd love to be able to use the cloud, but not at the cost of increasing our monthly expenses by an entire order of magnitude, so we just stick to bare metal colocation.

I honestly believe that if you have the technical skill in-house, and are spending more than, say, $30k/mo on public cloud hosting, that you should seriously evaluate whether bare metal could significantly decrease your costs.

At $JOB, we have a similar issue with static content. We serve over 1PB a month and the price AWS would charge to use CloudFront is an order of magnitude larger than even a Cloudflare Enterprise plan (which comes with some nice bells and whistles that Cloudfront doesn't offer).

Even with discounts from AWS it just doesn't make sense to use AWS to serve up the assets to users.

We do use S3 as our static asset backend and the combination works really well. I would love to see Cloudflare release a S3 compatible storage service though. I think we'd jump onto that in a heartbeat.

Backblaze has S3-compatible API and they have free data transfer to Cloudflare

The last time I looked at BB S3 API they didn't offer the lifecycle controls that AWS over. Mainly the ability to remove old versions of files after a period of time and we didn't want to roll our own expiry solution. Might be worth looking again though.

Yep, B2 does offer this capability now. They've been working hard to add S3-compatible/similar features.

"Lifecycle rules instruct the B2 service to automatically hide and/or delete old files. You can set up rules to do things like delete old versions of files 30 days after a newer version was uploaded."

https://www.backblaze.com/b2/docs/lifecycle_rules.html

Art from Backblaze here...Thanks for the feedback! As treesknees points out, we currently support lifecycle rules with our B2 Native API and our web application - those are available right now if you need to do it ASAP. https://www.backblaze.com/b2/docs/lifecycle_rules.html

CloudFront is also a pretty inferior product in the CDN marketplace. Why do you think Amazon's retail business signed a huge contract with Fastly to be their primary CDN for all mission critical retail images? CloudFront sucks. Not competitive on price, features, performance or anything other than "we already pay AWS so might as well use it".

What is the best CDN offering right now in your opinion?

If you want raw speed/features/reach, then Fastly or Akamai.

CloudFlare's CDN is competitive on features and price but it isn't a market leader in terms of raw speed.

Raw speed in what way?

Cloudflare. Their performance, pricing, and Workers platform means you can build whatever you need.

One advantage of CloudFront is support for uploading large files (5GB+) to the origin server. CDNs tend to enforce a low size limit for uploads. It’s probably an uncommon use case, but the reduced client-side latency is nice for customers who are far away from the origin.

What if your business has fluctuating loads?

I can see how running game servers—with a (somewhat, maybe?) predictable load of players—can be done efficiently on baremetal and with colocating.

For another business that has huge peaks of demand, such as analytics with dynamic queries, I fail to see how baremetal can compare to spinning up hundreds of instances on-demand.

Perhaps it comes down to what services you offer your clients and how you implement them?

Just compare the spot/preemptible instance price or committed-use price to the on-demand price to see how it can be cheaper.

The scale part of the cloud is often not as exciting as it seems on paper. Yes a cloud can auto-scale 100x. But can everything else support that? Like is your DB setup to handle 100x increase in load? (Sure, use dynamo DB, but that has other restrictions).

If the cloud to bare metal price difference is 10x.. you could easily just buy bare metal = 2x your peak load, and still come out ahead..

What you state makes sense. For me, I haven't worked in an environment (yet) that would need to handle fluctuating loads at scale—so my comment is my own speculation based on my experience working for smaller businesses with MUCH less data and bandwidth usage compared to those mentioned here.

And that's why I come to HN, lobsters, etc :) so that I can read and learn from others' experiences...

I've worked with dozens of different companies, and it's pretty rare to truly have such crazy variance in load.

also, the moment it does happen you usually have non technical systems which are even harder to scale. (customer support for instance, or your sales/billing pipeline).

Honestly, a lot of people overengineer the scaling thing.

Google really does need a service that can scale to 100M users.

I doubt someone starting a SaaS to make garbage truck software needs to. Or at a minimum, at the time they need that kind of scale, they will have plenty of money to throw at hardware. It's amazing how many users a single maxed DB can handle.

If you go for horizontal scaling like that, indeed everything needs to be adjusted. If you're asking "is your DB setup to handle 100x increase in load" then likely the answer is no. "How do you shard your writes" is likely a better question then... Aggressive autoscaling requires different thinking.

Depends on how many times you've scaled 100x already. 1 user to 100 users is 100x, 100 users to 10k users is 100x, 10k users to 1 million users is 100x. Aurora serverless autoscaling can handle a LOT, depending on the architecture and use case.

The promise of cloud scalability never really materialized in the way that people talk about. The value in cloud today is really in hosted services. Why have loads of exports for different supporting infra if you can just lease them from AWS through a hosted service.

If you look at what it would cost to run bare metal with top shelf companies, it's often as cheap or cheaper than spot/preemptible instances at the cloud shops.

This... modern architectures for mid sized companies are a combination of hosted services. I still have bad dreams when I look back a on prem Swift stack object store project took years to stand up & integrate all to have a much higher TCO and self manage a bunch of out of date s3 translation libraries.

It was a system that had fixed scale, high licencing costs and did not bridge well into software that increasingly leveraging the AWS environment. Egress was least of our worries relative to integration complexity in services ecosystem.

You have to be a very very large scale before self managed services make sense; as you suffer at integration layers for every developer that is even an inch off the well beaten path.

We always ran most load in our own data centers with the ability to transparently shed load into AWS for large peaks. It saved a lot of money.

Yes, different situations require different solutions. I think most people learn this by the time they reach puberty.

If you're trying to reduce costs on a complicated usage pattern, then you'll probably want to have someone on staff or consultant who knows how to deal with those things.

I don't know your experience, but bare metal has at least an order of magnitude greater efficiency at one or two magnitudes lower cost. If you want to pay $1MM/mo for something that could also be done for (generously) $20K/mo (plus a couplefew salaries that add up to much less than $1MM/mo), that's your business, but I don't see how the use-case you describe requires cloud/AWS.

The thing that people don't get (or have forgotten, or...) is that you don't have to spin up hundreds of instances in the way you do on AWS, you just handle them on a machine or three that already has enough headroom to handle them.

(all hypothetical blather shaped to support my perspective and disprove the utility of yours, of course ;)

> Yes, different situations require different solutions. I think most people learn this by the time they reach puberty.

Not so sure about most people learning this...though, I will say I'm probably a late bloomer compared to most on HN :)

What you state makes sense. I think my tone was dismissive of the idea that running your own machines is worthwhile. To add to that, I haven't (yet) worked on large-scale systems so my comment is purely speculation.

> We'd love to be able to use the cloud

why?

because you are not limited by your fleet.

For pricing reference you can get a 100GbE transit link (from a top-20 sized carrier by CAIDA ASrank size) at major IX points now for well under $6000 a month.

And if you are present with your own bare metal infrastructure at such a place you almost certainly also have the opportunity to connect to a serious IX for settlement-free open peering, and to run PNIs to other major sources or sinks of your traffic. So by no means will all of your traffic be going through transit.

How much would 1k 100GbE links cost from one of those providers? How much would 10k of those links cost? Does the price increase linearly or exponentially?

If you need multiple 100GbE transit connections from ISPs larger than yourself, in multiple locations, you most likely also are a fair sized ISP, so the situation is very different because you'll also be purchasing a variety of transport (point to point circuits) between cities at 100/200/400GbE capacity, various DWDM circuits, lighting your own dark fiber, etc. It's a whole other ball game.

It will be linear until you reach a percentage of the capacity of the provider themselves. Not every company has multiple terabits into each datacenter.

At that point you’re probably buying dark fiber to get where you need to go.

This is super true, yet sadly much of the Japanese mobile gaming industry is paying through the nose for cloud hosting. There just is not a culture of optimizing the cost structure.

Isn't this because they're basically printing money with slightly disguised gambling?

Are they printing money with gacha games? Yes.

Doesn't mean they couldn't do some optimization! But to your point, yeah, I imagine there's probably a little more pressure to develop new features/characters/etc. rather than spend months rigging up in-house network infrastructure.

Are you talking about Amazon?

No, I'm talking about Japanese mobile game companies and their gacha games.

I think a big driver for adoption of AWS and the cloud in general is tech salaries in the US. You'll often hear people on HN talking about how useless it is to try to save $Xk because the engineer cost will be even higher than that.

This is true until you get to a certain scale and then you start looking at your margins and realize a ton of it is your wasteful use of AWS.

I spent a big chunk of my last 2 jobs dealing with that issue.

That's true, and on the opposite side you have companies that manage so much hardware that it starts being profitable for them to become a cloud operator.

also, the math for some cloud providers doesn't work that well if you are in regions which are not EU-west or the US.

even in europe, AWS has "meh" regions compared to azure or many of the colo/dc locations.

Do you mean compared to the cost of living in those regions?

No, ping to users.

Is there any cacheable content? I'm working on high-performance compute on CDN software you can run yourself.

Not when we don't control the game client. We're the largest Minecraft server in the world (220k peak concurrent users) yet we operate entirely as a third party with no involvement from Microsoft.

There's the opportunity cost to throwing away your organization's knowledge of AWS, though. Everyone will have to learn to do things the bespoke way on your bare metal. So developer productivity could decline.

This is where Kubernetes I think helps a ton. I work with bare metal customers all the time that stand up OpenShift on their BM and can migrate k8s apps very easily. Depending on the apps you may need to throw an object storage solution in there too (such as OpenShift Container Storage). It does require a certain scale before this makes sense, but it's not nearly as high of a scale as most people think.

You go on to AWS for their managed services like Fargate, DynamoDB, ECS, S3 etc. Have used OpenShift in the past and had endless problems with cluster stability (especially in 3.x), and weird inconsistencies.

With AWS I could just spin up 10 Kubernetes clusters with pretty much unlimited resources, can't do that in OpenShift because you'd hit a resource quota or limit.

I would say the exact opposite in some ways for teams who are not using AWS yet. My team for example has been operating bare metal for 8 years now, and we know how to do that. Transitioning our team to the "bespoke way" that AWS does things has a huge opportunity cost, too.

This is the correct answer.

People are swimming in the koolaid and don't think a different environment exist.

These kids. When I was a kid......

Imo using cloud vendor apis is more bespoke than using kube/nomad or even old school andible/salt on baremetal cluster. With the exception of s3 all your knowledge will tabled if ever need to switch the vendors

or improve, if you're not opening tickets to raise service limits all the time, and can afford to deploy 50% extra capacity 6 months earlier.

Is *that* why Hytale hasn't been released yet?

This is exactly why we are using DigitalOcean to host our gameservers... Our egress costs are basically 1/10th of what we would pay on the big three.

DigitalOcean Egress - $0.01/GB

How do you orchestrate or manage your servers currently? And do you employ a in-house solution for it as well?

I’ve found that are generally three stages in a company’s life. If you are extremely technical you can make bare metal beat cloud in the beginning. As the environment grows and IT roles diversify cloud starts to make more sense financially then bare metal so you do a cloud migration. If you are lucky then eventually you get to the point that it’s cheaper to run your own hardware then rent someone else’s and you end up migrating out of the cloud again.

If there is a fourth stage then it’s getting to the point that it’s more profitable to run a side business renting out your unused resources to others and you become a cloud.

I’m surprised there isn’t a Walmart cloud or an Exxon cloud by now.

> If there is a fourth stage then it’s getting to the point that it’s more profitable to run a side business renting out your unused resources to others and you become a cloud.

This is a common belief about AWS, but I find the counterpoint persuasive - if AWS involved Amazon renting out its unused resources, it would have to shut down in December. But obviously AWS does not shut down in December. They're not renting out their unused resources; they're renting out their experience running this kind of system.

The fourth stage doesn't even involve owning your own infrastructure.

We pay a 3rd party for our AWS (and Azure) usage to get better rates than AWS would give us (and without a commitment to increased spend like AWS reps have tried to push for Amazon's available programs).

I wouldn't be surprised if they just used volume discounts / negotiated pricing and reservations at their end to make it massively profitable for them.

I think SPOT (pre-emptable) instances are an under appreciated resource in AWS. Technically they can get pulled back at any time but in practice it’s only happened to me once, for a single instance. I’ve lost far more instances to hardware failures in the same period. If you can get your head around that then you can save more money with SPOT plans then any of the regular compute plans. AWS also does an auction for SPOT instances which can push the price really low for ARM and previous generation instances whereas other clouds just give you a set discount when using them.

If you've only had a spot instance pre-empted from you once then you haven't used them very much. We have a very elastic workload that can scale up to a few hundred spot instances and it's not uncommon to see tens of them yanked out from under us at once.

This is our exact journey so far.

1. On prem hosting managed by 1 person who works out of broom closet in some budget colo. Zero compliance or official procedures involved.

2. Cloud hosting managed by whoever needs a thing taken care of. (We are here)

3. On prem managed by dedicated in-house team in multi-colo setup. (We really really want to be here)

I think we will skip the 4th stage. If we are forced to build our own datacenters, something went very badly.

The 30k/m figure posted elsewhere in this thread is a fairly reasonable line in the sand for jumping from 2 to 3. We also have unique regulatory requirements that could move that threshold lower.

3. On prem managed by dedicated in-house team in multi-colo setup. (We really really want to be here)

another major advantage of this kind of setup is that it is possible to give far better (technical support) to your customers, especially if you provide infrastructure services to your customers.

having a competent technical team which can do proper indepth analysis of issues customers are having has a massive positive impact on customer relations. Compared to most technical escalations i have seen at most of the cloud providers, which comes down too "it was an internal problem, but is is fixed now, don't know if this will happen again".

> it is possible to give far better (technical support) to your customers, especially if you provide infrastructure services to your customers.

This is exactly why we want to do it. Being able to offer our customers a 100% vertical solution stack that we completely control & understand is a massive selling point in our industry (and hopefully others).

oh there is definitely a market for this, and cusotmers willing to pay for it.

at $job, me and another team member basically did technical deep dives into issues that occured for $customer. Usually they needed deep root cause analysis as of why something happened. (Mainly for insurance/legal/compensation reasons).

It was a ton of fun and we even got to a point where customers where willing to pay for this service when required. It costs us a lot of hours to do the troubleshooting, and not all customers have such requirements.

Some fun root causes i have found over the year:

- a bug in a firewall which resulted in kernel crash of the packet processing plane (and thus packet loss). We worked extensively with the vendor to fix this. This bug only occured because $company was using a custom TCP stack, and was sending ACK's incorrectly every X packets.

- tons of issues regarding time. Time drift on routers, time drift between processes inside a router. Time drift between hosts.(this happens a lot, and it makes me appriacate the fact that some routers have external clocks as a source option).

- quite a lof of issues with IPsec tunnels. For being a standard, i have never seen suchs a lot of edge cases with non-interoperability.

I think it depends very much on what your product does. E.g. in your case (games) bandwidth is a big deal. The last three companies I've worked at have all had AWS bills of more than $10M/yr and data egress costs were never more than 2-8% of the bill. A mixture of B2C and B2B SaaS companies.

In that case, the economics of bare metal aren't as clear-cut.

This is such a great conversation, and I agree with you - that while the egress is egregious; it's rarely a large portion of the cost.

I run a team that manages our public and private (DC) cloud for a successful tech company. - I need more folks to help us do things better. Please contact me Mr. Pugs or others if you are open to thinking about looking at another company.

I run servers for one of the biggest open source multiplayer games on github. (tgstation/tgstation)

Everybody asks why we don't do cloud, and time and time again i have to point out that the fastest per-thread cpu on ec2 costs more per month then our rented bare metal server, that benchmarks as faster per thread, and has 3x the bandwidth we need, included in the price!.

Even on a cpu level, the obsession with intel """enterprise""" cpus is leaving money on the table.

"but what about web stuff, surely that could go on the cloud!" the best cpus to run (mostly) single threaded game servers right now come with 8 to 16 cores, and we aren't running that many game servers in each region we target, so what else are we gonna do with those spare cores but load up ansible managed docker swarms to handle the webstuff?

Notice how I haven't even needed to get to the egress cost to point out how senseless going to the cloud can be.

I get very happy seeing people run Docker Swarm in production. we need a community!

I run Docker Swarm on my homelab, but I honestly wouldn't touch it for production because it's lacking features that I find critical and doesn't show any signs of developing them soon.

For example, Swarm secrets can only be exposed as files, not as environment variables. You can argue this is more secure because file permissions are more granular than env vars, but IMO that's a silly argument in a container context because containers are almost always single-user to begin with. Moreover, the vast majority of containerized applications expect their secrets in env vars, so you have to resort to fragile entry point wrappers if you want to make use of Swarm secrets.

honestly the biggest failure of swarm secrets is the inability to change them while things are using them.

automating secret usage across containers that are deployed via ansible scripts gets annoying in that situation.

You, sir, are smart, don't believe hype, and know how to math.

You are not AWS's target market.

In the past I worked at a company in the top 20 bandwidth consumer in the world.

95th billing is the way to go. Due to the nature of the company's content (adult) nobody wanted to do peering agreements due to the public outcry if anyone found out but if that had been possible I figure the plan would have been to follow Netflix's open connect journey.

How to run a top 20th largest bandwidth consumer in the world for < $1M/month. https://openconnect.netflix.com/en/

Disclaimer: These are only my own personal opinions

AWS Egress - $0.09/GB

Azure Egress - $0.0875/GB

GCP Egress - $0.11/GB

Alibaba Egress - $0.123

Interesting that no mention is made of the other cloud providers pricing here when AWS isn't even the most expensive egress...Oh! I wonder if it is because Cloudflare has data transfer agreements with GCP, Azure, and Alibaba! But that would mean this is a shoddy hit piece to strong arm AWS into an agreement with them. Nonsense, I'm sure Cloudflare would never do that...

Edit: I've removed claims about Cloudfront's pricing since it isn't actually clear what they charge for egress and seems to vary on a number of factors. Thanks Dylan16807 for pointing that out.

Oracle cloud egress goes from $0.0085 USD to $0.025 USD (depending on the zone) and the first 10 TB are free.

It's quite good; too bad their cloud services are quite bad.

Bad in what sense? Curious because I just signed up for OCI a few hours ago.

My takeaway as a daily user of OCI and an AWS / GCP customer:

Oracle cloud has proven to be refreshingly nice and easy to work with and develop + manage services on.

AWS is fine but can be costly and has technical baggage.

The cluster mess that is GCP is by far the most frustrating. For starters, the UI is indefensibly terrible.

Curious to know what specific complaints or grievances about OCI folks have.

Disclaimer: I currently work at Oracle, but don't speak on behalf of the company in any capacity. This is all my personal opinion and honest experience. YMMV and that's okay!

p.s. The OCI "free-forever" VMs are pretty generous-

2x instances with 1GB ram, much higher network and compute performance compared to GCP free-tier.

Honestly, looking at Oracle Cloud, I think the best thing they could do is spin off and just completely remove any trace of the name Oracle. You have some absolutely fantastic products that are being criminally neglected because people won't go anywhere near the name, and can you blame them?

I dont even think they need to Spin it off. They could reform it as subsidiary and have a different company name.

GCP has the worst UI, except for all the others

Agreed... GCP isn't perfect by any means, but I find it much better than AWS, and miles better than Azure.

Aws ui is old school and has clunky ux but is very snappy even on underpowered hw. GCP is a bloated mess. Takes 30-60s to load every page on my personal mac air with a 1gpbs link. And doesn’t work well in safari

It's fixed now, but for a while I had to open the gcp pubsub part of the console in chrome rather than Firefox because it would just completely fail to load.

The cynical part of me was thinking that this is a good way to increase browser market share, but more likely an honest mistake.

Overall I find the gcp console and gcloud cli very user-friendly compared to aws though

The last time I tried Google Cloud was to test some Serverless code, this was the outcome:

---

Just spent 2 hours trying to make some basic Google Cloud Services examples of Serverless to work, directly from the Serverless page to no avail. GCS was returning all kind of weird errors, including HTTP 500 ...

After that decided to move to use the AWS examples, and the first one I tried worked like a charm! Google has a long way to go to be at the level of AWS, right now the infrastructure seems crappy, if you cannot even run the most simple examples of a simple HTTP serverless function.

EDIT: Also, I would never use Google Cloud, fearing that if for some reason they decide to "block" my account, I would have no recourse. Their support is terrible as well.

Weird, I use GCP (GCS extensively) pretty much exclusively for numerous different clients and can recall having 0 issues. I had to use AWS EKS recently and found the UX to be quite sub-par in the console compared to GCP. EKS is a nightmare compared to GKE as well.

Yes. EKS is a horrible piece of crap. I've tried it and ended up using ECS.

In a past company where I was, we were approached by OCI sales team wanting to move us from AWS (we had a partnership with Oracle in other products).

After doing some due diligence, I couldn't find a reason to move anything. I think Oracle has to find something that just excels in their cloud. Shit, even giving free outbound traffic forever would be a huge plus. Microsoft has their AI stack, Google has their Spanner and other propiertary services, AWS has "everything else". But if Oracle wants to really take clients from others, they ought to do something good. I don't think they can compete with the DigitalOcean/Linoe/Vultr class... So unless they have a "killer app" it will be complicated to find customers that want to go through all the hassle of migrating from an existing cloud solution.

> Microsoft has their AI stack

TBH this is better on Google as well. MS has better corporate integration with IT teams that use MS internally.

One advantage is that GraalVM Enterprise Edition is free to use on OCI infrastructure. I have yet to try it, but could be useful for a nice performance boost.

It's rough all around the edges:

- Want to mount HD Volumes? You've got to copy/paste a bunch of commands as an additional step after attaching it. This, in addition to the mkfs.xyz and all the normal linux stuff

- You thought AWS pricing was opaque? OCI implemented something called OCPU each of which gives 2 cores, but you've got to go to their 'cost simulator' page to make sense, and even then, there are some of their machine types that are not in the cost calculator

- They offer ARM cpus with their proprietary Oracle Linux... but once in, if you try to install the "oracle arm development toolchain" using their own tutorials, commands and sources, it doesn't work.

- To get into a "pay as you go" plan after your trial finishes you have to dance around in 4 or 5 video/calls with one of their "representatives" so that they move you to "PayPal" payment (for some reason they don't accept cards directly but only Paypal)

- Once I was "moved" to the pay-as-you-go, my Billing chart UI just broke, and suddenly I don't have a way to see how much I am spending (empty charts).

- The "performance levels" of block storage is also weird... and just now they released "ultra high performance", but it cannot be used by any machine, and you also need to do some additional crap to make it work.

- The login is strange.. at some point trying to login, I ended up in some strange Oracle screen that had nothing to do with OCI.

- The "Wordpress appliance" (whatever they call it) that they place at the very front of the screen after you login doesn't work: The admim user/password you enter when setting it up does not work. This one made me laugh... so bad.

And so on, and so forth. There's a lot of other papercuts that we've had while trying it.

If you want to create a couple of instances it is OK, but the brokeness becomes old when you are trying to do something real.

The link you have submitted is about Workers however, which is a Cloudflare's function-as-a-service solution. I could not find egress pricing for general bandwidth that does not go through Workers last time I tried. Could you please point me to a document that says CF general egress pricing is $0.045 now?

The original version of my post called out that you cannot actually see Cloudflare's pricing without an account, that link was the best I could find. Funny to be calling out their competitors for pricing when they won't even publish theirs...

https://developers.cloudflare.com/workers/platform/pricing

Edit: I was quoting a 100MB per request limit but that's actually ingress, egress is 512MB/unlimited so if you calculate that out it would be an even smaller fraction of a penny per terabyte for bundled workers.

Cloudflare doesn't do pricing for bandwidth, as far as I know.

https://www.quora.com/How-can-Cloudflare-not-charge-for-band...

In this case is because they actually does not charge bandwidth. I have few services on CF and never paid a dime for bandwidth. Even on free plan I push few terabytes.

What are you on about? A big chunk of the article is about the bandwidth alliance, and your number for Cloudflare is not egress.

Edit: And in particular, if you are using normal cloudflare workers, not the "workers unbound" that are "intended for applications that need long execution times", then the price is $0.50 per million requests and $0 for bandwidth.

Thanks for pointing this out, I've gone ahead and removed mentions of Cloudflare's pricing from my comment since I can't find an authoritative source for their egress pricing in their docs. The point general point still stands, all cloud provider egress costs suck and Cloudflare is just trying to strongarm AWS into entering a bandwidth agreement with them.

Honestly I'd love it even if AWS just joined into the bandwidth alliance. Then I could store backups in glacier but have several options to retrieve it at a reasonable price.

No it doesn't. Since the article mentions that Amazon is charging egress pricing for egress data that doesn't go though the public internet, which is much cheaper and isn't egress in the first place.

Except at Amazon/AWS ofc.

/thread

Thanks for this.

I was trying to find a simple comparison of pricing on the article but only found an overly simple water analogy.

I don't know about GCP but Azure's egress costs go down as you use more and more dropping as low as .05/GB, and if you are using Cloudflare on top of it the costs go down even more. Then don't forget ingress fees, Azure charges nothing, while AWS charges ~.015/GB in. Since a lot of protocols rely on heavy ingress as well that starts to add up. AWS is notoriously expensive in bandwidth costs, that's nothing new.

I took the article as pointing out that they specifically do not charge AWS for data transfer to CloudFlare and yet AWS charges full-price for bandwidth while other clouds do not. That's not a shoddy hit-piece, it's exposing some predatory business practices. Of course it'll benefit CloudFlare, but that doesn't mean it's a disingenuous article.

If you're doing a lot of egress look into AWS Direct Connect (I think GCP has an equivalent too.) That gets you down to $0.02/GB

Digital Ocean: 0.01/GB

Netcup: basically free (40TB is baked into the VPS price)

I think it’s safe to say that all of the major providers egregiously overcharge for bandwidth.

I’ve never heard of Netcup. How are they compared to say DO?

The article is about egress data that doesn't go over the public internet, which is much cheaper.

Other cloud companies don't use the same price for that, while Amazon does. With an markup...

Cloudflare does multiple petabytes of traffic on their enterprise plan for ~$10k/m.

Disclosure: $NET shareholder

Yes, most people think about Azure/GCP/AWS/Alibaba only. But there are many others, smaller and bigger providers. For example:

Scaleway (small, in Europe) is like DO: Egress cost is part of the instance price

Oracle cloud: Egress cost depends on the monthly usage and and the destination country (free of charge up into $0.050 USD per GB per month) [1]

Egress shouldn't be as expensive as it is in major clouds.

[1] https://www.oracle.com/cloud/networking/networking-pricing.h...

On that basis, I am saving $90,000 a year by having a dedicated server from Hetzner (running a Tor relay). If I could afford that, I would pay off my mortgage in a couple of years.

This blog post is pretty disappointing. While true that AWS's egress can be costly, it is interesting that this article makes no mention of the fact that "non-partnership egress" on GCP is actually more expensive than AWS and that Azure is roughly on par. This article misses the opportunity to talk about cloud providers in a broad context and instead attempts to publicly shame AWS for not cutting a partner discount with them on egress for customers. Don't get me wrong, I'd love an egress discount, but this is "partner channel propaganda" masked as an AWS call-out.

This take is repeated a few times in this thread, but I'd counterpoint that AWS is the market leader, and Azure/GCP would be under much more competitive pressure to lower their bandwidth pricing if AWS brought theirs into line.

This article isn’t about reducing the cost of egress though, its about giving a discount when you peer with other partners. Its a very self serving argument.

I speculate that the reason why Amazon charges so much for egress (other than to nickel-and-dime its customers) is that it inhibits hybrid-cloud or multi-cloud architectures. It promotes lock-in.

In literally every business domain, Amazon succeeds not because of "customer obsession", but because of their willingness to engage in anti-competitive or outright monopolistic practices.

Interesting to see them grow so big then - they used to be the minnow to the Walmarts / Targets etc.

At least in my experience (retail product ordering) their service flat out IS head and shoulders above much of the compeitition.

I order off a Facebook ad - I get a drop shipment from china that takes forever, is effectively not returnable etc etc.

I order from Amazon - delivery is insanely reliable, when I have an issue refunds are handled incredibly well. Even special cases (I had a weird issue with an iphone order that required a manual override and refund despite what looked like a failure to return - I got a real person and a refund promptly - they would have been relatively justified NOT to refund but I thought I'd ask).

Even on AWS my support contract get's real responses. When I had a billing / customer service issue - a real person (sounded like native English speaking US call center) handled it very well.

My brokers is constantly experiencing "unexpected call volume" (hour long waits). The IRS when I call has "unexpected call volume" and hour long waits. Calling southwest they have "unexpected call volume" and hour long waits. A ton of places make it virtually impossible to actually call and talk to anyone (much smaller than Amazon BTW) - you can search website all day long. Or you are speaking to call centers that can't do anything at all.

I contrast AWS to the treatment we got as a paying google suite customer (endless different sales people calling us, but you don't have good paths in to trouble shoot weird state issues etc beyond basic customer service). And AWS devices support employees work calendars (via gsuite) while google devices DO NOT!

> I contrast AWS to the treatment we got as a paying google suite customer

Did you have an additional paid support contract with that Google Suite/Workspace account? What would your customer service have been with AWS if you didn't have that additional support contract? If you're paying for premium support on one service it seems a little unfair to compare that to the included base tier support of the other.

I’ve had very poor experiences with gsuite support. Once your issue gets past the front line support it disappears into an opaque void where nobody knows anything or can talk to anyone. Once I was asked to write a business case clarifying why answering my support question benefits Google. (Found a bug in gmail css handling, wanted a workaround).

With AWS support I can get an answer to my toughest “found a bug” issues in hours or a couple days at worst.

I’m pretty sure if I called AWS support on Thanksgiving and asked for help cooking a turkey they would send me a recipe. Their support is really amazing.

> In literally every business domain, Amazon succeeds not because of "customer obsession", but because of their willingness to engage in anti-competitive or outright monopolistic practices.

You're obviously casting a blanket that is far too wide. That's the hate Amazon train running out of control.

They legitimately do a great job for retail customers, re customer obsession. Their retail customer service and willingness to make things right, is excellent.

I've been buying from Amazon.com for ~23 years or so now. They've never failed to make something right. I've had maybe three bad purchases in those 23 years out of hundreds of orders (meaning something arrived that I didn't order, or it arrived damaged), all were made right with little effort on my part. Amazon almost never protests if something goes wrong with an order.

Their retail customers by and large adore them for that reason.

Their kindle ebook solution won because it was an excellent experience overall with a vast selection that was priced very much in the favor of consumers (and still is). They produced a great ebook at a great price. It's very easy to use and their pricing was so tremendous publishers sued them to try to force book prices higher on consumers.

Their Prime membership program has been no more anti-competitive than Costco's membership program.

Your scenario damns Amazon every direction no matter what they do, which reveals the plot. If they raise prices, they're a brutal monopolist taking advantage to gouge consumers. If they give consumers cheap prices, they're an anti-competitive monopolist undercutting on prices. And if by chance they perfectly align with everyone else on pricing, well they're obviously colluding to set prices, a clear anti-competitive practice.

> Their retail customer service and willingness to make things right, is excellent.

Excellent, right. I could tell you a story about how one of my German business accounts got closed after they allowed me to: a) open it, b) place an order with a line of credit payment type - only option possible because I couldn't even link a SEPA bank account to it c) they have shipped the order d) the line of credit was repaid, outstanding balance: €0.

A week later, account closed. They have sent me an automated email asking for "company documents". To my questions regarding which documents do they want, which I explicitly listed out what I can provide, they have sent me a blanket email asking for "company documents". Managed to get in touch with a human, the woman said "documents", any further communication would trigger the blanket email with request for "company documents". What documents, never figured out, it was outrageous. Stopped buying with Amazon (via 3 businesses and 1 private account, 4 accounts in total). Wife dumped Amazon too. Now giving money to German online shops or buy directly from manufacturer / supplier / distributor / merchant online store, even if less convenient than Amazon.

What I found out by that, many items offered on Amazon are about 40% more expensive than buying directly from manufacturer / supplier / merchant store. People are getting ripped off left and right and even pay for Prime delivery. Amazing. The best example was a t the beginning of the covid mess. Wanted to buy 4 bottles of hand sanitizer for the office. They wanted €180 and given me a delivery estimate of 2 weeks. So I listed the shops where the article comes from. Went to the first one, found the article in stock, paid €68.99 and had it delivered two days later. From the same shop that Amazon told me it would take TWO WEEKS to deliver.

Or another story, happened before the first one: as self employed, opened a business account next to the regular one. After they have changed their tax locality to Luxemburg, they lost all my purchase history on the business account. They couldn't figure out where it went. The only way to get invoices for past purchases for my own accounting, was to dig out all the purchase confirmation emails which had direct links to invoices. They've lost my history. If I did not keep the confirmation emails, I could have been deep in trouble because I had these expenses on business books but no invoices, and they couldn't be bothered. Nuts.

Never going back to Amazon for reason other than listing merchant shops to buy directly from.

Other than the timeframe (since they've opened), i can say the same thing about AliExpress. Same day support. Out of the 2 issues I had in the many years and hundreds of orders, all were resolved with a quick refund. Amazon isn't special. They just have such a high price tag, on our expense and not to the manufacturer's benefit, that they can afford to throw money at human support.

Amazon.com is full of drop-shipped no-name almost literal polished plastic turds from China. A few useless stock-photo level product images. No technical specifications, just the same generic terms repeated as `description`. A completely unknown brand and company name. Of course only available for a few months, so bad reviews with this product name and brand don't accumulate all over the Internet. Rinse, repeat.

It's great that they "make things right" somehow. But their marketplace is crap, they are the embodiment of a cancerous flea market that grew to enormous proportions.

The problem is, Amazon shouldn't be able to set prices so freely. Market forces don't apply when you can just dump all your profit into a new niche, obliterate the competition, capture the audience, then set prices to maximize profit, and move to the next market segment. (Just as Google's money distorts the browser market.)

(Books happen to be the segment where most of this doesn't apply, because books, like any content are not substitutable, yet copies are perfectly fungible. At best there's a very minimal concern about the condition of used books. And it shows that Amazon's whole model is built on this. One completely unknown PowerBank/USB-hub/gadget is just as good as an other.)

I wonder if net neutrality rules, combined with the absolute dominance of public cloud infrastructure in the US, could force providers to have more reasonable egress pricing.

"In other words, ingress (data sent to AWS) doesn’t cost them any more or less than egress (data sent from AWS). And yet, they charge customers more to take data out than put it in. It’s a head scratcher.

We’ve tried to be charitable in trying to understand why AWS would charge this way. Disappointingly, there just doesn't seem to be an innocent explanation. "

One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. At the very least, this could cause a customer service headache. Perhaps AWS decided they'd rather not deal with that?

> One consideration could be that you can't directly control data ingress.

Are you thinking UDP? Or for a DDOS? Connection setup overhead could be accounted for. I highly doubt this is the reason.

If there's an IP exposed on the internet, you can just send it tcp payloads. The end destination will silently drop them, but it doesn't mean people can't send you gigs of useless data.

I think you mean UDP. TCP requires a 3-way handshake first.

Intermediate routers don't care about that; they only forward the IP packets; four target host/firewall will drop them (because they don't belong to a valid connection) but they will be still accounted for as ingress traffic.

Correct, but you can't get much bandwidth through until the 3-way handshake is completed. Sending a bunch of unanswered SYN packets isn't really a great way to instigate a DDoS, compared to sending avalanches of 64KiB UDP packets.

As long as there is no connection tracking you can send whatever crap you want, including replayed packets from the middle of a connection, perhaps even huge packets with a syn flag ... As long as the accounting happens before a firewall performs basic TCP sanity checks you're going to pay for it

in some setups, you could send non IP packets (especially in MPLS fabric if accounting happens by LSP).

At a routing and peering level. Once you have an announcement for your netblock out there, traffic will start to head towards it. A lot of this is due to the BGP Path Selection Algorithm.

You can try and influence how traffic arrives, by doing things like, AS prepends, but you are still going to get traffic.

The main reason for this is that the other side that is egressing to you has their own egress policy that also follows path selection. Things like localpref and weight will force my traffic to leave via a path before it considers how a network has AS padded.

As an example: Lets say I want to egress (company A) to a downstream company (company B). If I learn routes to Company B via multiple ways: peering fabric (low cost), paid peering (medium cost), transit1 (high cost, variable quality), transit2 (low cost variable quality), I can choose which way my traffic goes, via localpref, weight etc.

Only when I view the paths equally (equal localpref, weight etc.) will I evaluate the shortest AS Path (which the receiving company has influence on).

The only way to completely not get inbound traffic via a specific link, is to remove your BGP advertisement for your netblock from that link. (some providers also let you do this selectively via BGP communities).

There are also some other tips/tricks - such as adding a more specific prefix to a certain link, to attract traffic, but care needs to be made to have a fallback route in case things go wonky.

Coming from someone who believes that the AWS egress fees are highway robbery, I'd throw this out there. In order to maintain settlement free peering, you need to keep egress and ingress somewhat even with your peers.

Amazon proper probably has 3x egress vs ingress between prime and twitch. AWS is an easy avenue to balance it out.

All that being said: I would guess there's almost 0 chance that's what's happening here. Amazon is just using it as a form of lock-in IMO.

I agree that it is just lock-in. The price of egress is the same even when using devices like the Snowball where the data is loaded onto a server and physically shipped to you.

Most request/response models are very asymmetric. I can request a relatively large asset from S3 with a small request and maliciously generate fees as an attacker.

AWS egress is vendor lock-in 100%.

It's also just a way to segment customers. The people doing tons of egress, on average, probably have bigger budgets for cloud computing. There's no particular reason AWS costs have to mirror how they charge.

The only service AWS hasn't consistently lowered pricing on (unlike almost everything else) is their egress cost. Everyone complains about vendor "lock in" for anything AWS related, and I've never thought so since they've always reduced prices across the board. But egress is the exception. They're keeping pricing high for this exact reason to prevent folks from moving their data elsewhere.

I've got about 25TB of stuff I'd love to keep all in Glacier Deep Archive, but the restore costs are just too insane to justify due to the egress pricing still at .09c after all these years. Too bad.

I view network egress as the primary lock-in mechanism for all of AWS.

If you want to migrate to another provider, network egress costs mean that you'll spend multiples of your normal monthly operating costs to do so. That stifles competition.

But well before that point, most services in AWS have already been built around avoiding network egress wherever possible. You're always going to prefer AWS APIs and services, even if they aren't the best for your use case, because services outside AWS have a network egress tariff placed on them. So you aren't always buying best-of-breed, you're buying the best of what AWS chooses to offer (or the selection of vendors that choose to build in AWS to remain competitive).

And if that isn't good enough, your only other option is to migrate out (and pay those egress costs!).

> Everyone complains about vendor "lock in" for anything AWS related, and I've never thought so since they've always reduced prices across the board.

How would keeping prices high for these services keep people locked in? The whole point of vendor lock in is there is some other indirect cost that you bear by switching, which is why you don't switch.

For AWS, this indirect cost is bandwidth. It is the lock in mechanism.

> How would keeping prices high for these services keep people locked in?

By doing the opposite and raising prices across the board randomly because they know folks can't spend the time and money to migrate without a large support burden or time/money sunk cost.

The only service not going down in price is egress and the only thing I've seen which has been like this, to directly discourage folks from migrating their data to say GCP when their offering(s) look more attractive.

If you have larger data needs you might look at snowcone -> snowmobile family of solutions. Generally 0.03/GB and scales to 50 petabyte + transfer volumes.

I've actually looked at that before if I was wanting to a large restore if my home nas just totally crashed. .03c is still going to cost like $1000 which still is pretty steep. Mostly because you know these devices aren't being plugged in and the data extracted from s3 over the WAN, they are likely plugged in directly in the datacenter over some insane internal high-speed link.

Yeah, it depends on use case. For business use cases in the event of a big issue / local data loss - $1,000 range might be a rounding error - that's def who they are targeting.

For 40 TB range stuff I've benchmarked some of the free / unlimited options and I'm not sure you could really get data back out and local that efficiently? It's still a week with AWS though as well.

One note - ingress to AWS is free - so if your job model is mostly writing into AWS, with a once every 5 years extract - the overall cost per GB transferred may not be that bad.

rsync.net is 2 cents per GB/month.

AWS glacier and deep glacier are .4 and .1 cents per GB/month. Free ingress. 40TB on AWS + ingress = $40/month.

A lot of the "outrage" on HN around AWS pricing is not necessarily looking at total costs of solutions AWS offers.

The other things - AWS offers fully transparent pricing.

Even folks like cloudflare - folks say it's an unlimited CDN for $200/month. No it's not.

"Use of the Service for the storage or caching of video (unless purchased separately as a Paid Service) or a disproportionate percentage of pictures, audio files, or other non-HTML content, is prohibited."

For business - they like the clarity / simplicity of AWS. For bigger data you can backup locally and have your offsite be AWS - hopefully never to use it or pay the ship out expense.

If you get DDoSed you also pay through the roof in fees (for whatever is in your VPC - e.g. ELB which are billed per throughput) unless you buy their anti-DDoS services (AWS Shield Advanced) for 3k$/mo - without that AWS won't waive the fees.

> One consideration could be that you can't directly control data ingress. Charging for ingress opens an avenue for attackers to attempt to bankrupt you by sending unrequested packets to your network. At the very least, this could cause a customer service headache. Perhaps AWS decided they'd rather not deal with that?

An equally likely reason is by making ingress free it helps balance out ingress and egress traffic flows which are usually desirable for settlement-free peering agreements.

The simple explanation is that ingress is far, far lower than egress so the cost isn't relevant since links tend to be symmetrical. So the cost to them is determined based on egress, their ingress provisioning happens at no additional cost.

Great post. Minor rant: However while Cloudflare does not charge for edge-egress, their pricing story is terrible, awful and dare I say terrible again.

It looks like they took their internal evolving developer docs and turned it into pricing for customers.

There is a tree of docs and pricing you have to get to the punchline which is that Cloudflare for the most part is several orders of magnitude cheaper than AWS or any other cloud provider. And each 'team' seems to have their own little pricing page with arcane details - exactly the definition of 'ship your org chart as products'. Just check out Workers pricing for instance. The product is great but the pricing story is silly.

But it's almost like they want to drive customers directly to AWS with Amazon's simple pricing calculator and tools.

In what sense is it order of magnitudes cheaper? Egress traffic?

If you are serving static assets then essentially it's pretty cheap. You can use KV / Workers and then use the caching to minimize cost which measures requests not bandwidth.

Something is very off. The post claims an 8000% markup but others have pointed out that AWS is in line with other cloud providers and the same math shows cloudflare with a ~3800% markup. From u/andrewguenther:

AWS Egress - $0.09/GB

Azure Egress - $0.0875/GB

GCP Egress - $0.11/GB

Alibaba Egress - $0.123

Cloudflare - $0.09/GB pre-April 2021 $0.045 now

1. If AWS is charging me $6 per Mbps for something that costs them 8¢, why is Cloudflare charging me $3 for the same 8¢ good?

2. What is going on with the price of the underlying technologies that make up network capacity. Is it just fiber and switches? Are these costs going down?

Buying 1 Mibps means you can use up to 1Mibps. Paying $/GiB means you can use 100s of Gibps for a split second and only pay the $/GiB fee.

In the past i did a TCO comparison for a large spikey workload where the $/GiB cost even at list was cheaper than allocating enough bandwidth for peak events.

At small scale the $/Mibps likely comes out cheaper, but for any large company sitting on 100s of Gibps of excess capacity for an event that occurs once or twice a year, or a few hours per day is very costly and might eat up savings from p99/p95th billing.

> Something is very off. The post claims an 8000% markup but others have pointed out that AWS is in line with other cloud providers

None of the biggest ones are trying to compete on egress. Especially because, why make it easy to run some of your services outside their cloud? Please move it all in to the same place and dedicate your budget to them.

> Cloudflare - $0.09/GB pre-April 2021 $0.045 now

That's not general egress, that's one random service optimized for using more CPU time. If you use the less-CPU-time version, bandwidth is free/negligible.

> What is going on with the price of the underlying technologies that make up network capacity. Is it just fiber and switches? Are these costs going down?

Better tech allows vastly increasing amounts of data to flow through the same wires using the same amount of equipment. The price per unit goes up, but the price per byte goes down.

It looks like Cloudflare cut egress pricing for workers unbound to $.0045 per GB, but that’s still half of AWS or “40x markup” for me in Canada.

The original workers model of pay per execution is incredible. IIRC I figured one run is the equivalent of 6KB of egress at AWS. Is there a point where Cloudflare will start charging egress on those if I transfer too much? I want to use it for something, but don’t like the “free bandwidth until you hit the invisible limit” side of things Cloudflare has going.

1. https://blog.cloudflare.com/workers-unbound-ga/

There's a typo, it's $0.045 per GB (half of AWS, as mentioned). I got excited for a second!

Damn. I'm stupid today. I don't think I can edit anymore either :-(

Cloudflare argo pricing (basically stay on the CF network) is almost identical to GCP Premium Tier and AWS default tier. They claim it does some cool optimizations to traffic path, no different to BGP optimizers - it is simply ridiculous to presume GCP/AWS doesnt optimize their routes too on a continous basis - yet apples and oranges doesn't apply CF says. I applaud CF nudging AWS to reduce bandwidth fees and things like bandwidth alliance but they sure do a hold my beer on hypocrisy.

and what doesn't make sense about that, is in some respects argo reduces load on their network, because your origin makes a single outgoing connection to cf, that is then used to route traffic to you, reducing the effect of 3-6 packet tcp/tls connection start.

That's technically Argo Tunnel tbf - their naming is worse than Google's, CF pushes pure Argo as better path from CF ingestion point to origin server because it now goes CF ingestion -> CF network -> Nearest CF network pop to origin -> origin, they claim non Argo is usually CF ingestion -> transit all the way -> origin which is probably more expensive for them too.

> The original workers model of pay per execution is incredible.

It's still there!

> The only rationale we can reasonably come up with for AWS’s egress pricing: locking customers into their cloud, and making it prohibitively expensive to get customer data back out. So much for being customer-first.

Presuming the average AWS customer is running an API SaaS, said customer’s API response load (= egress bandwidth) will tend to be a good measure of how much “useful work” said SaaS is doing for its own clients — either in the form of API responses, requests to third parties, emails sent, notifications pushed, uploads to other providers, etc.

As such, egress bandwidth tends to be a good measure of having product-market fit, and thereby of willingness-to-pay.

Things that trigger mostly ingress load — dropping attacks at a firewall, accepting uploads, receiving emails/notifications, etc. aren’t as emblematic of product-market fit (they aren’t things you would traditionally charge customers for “by volume”), but rather are usually activities done as loss-leaders; and so do not correlate to willingness-to-pay.

It's very much by design. They want ingress not egress to create "gravity." The end game is for virtually all compute to happen in the cloud and be accessed only by thin client. This pricing drives things in this direction by making companies that would use the cloud to just service local compute less profitable than companies that host everything in the cloud and provide only web or other thin client access.

There are providers that don't do this, but they are much less batteries-included. The best bandwidth deals are via bare metal hosting which is basically just box-in-a-rack rental. You can get bandwidth there by capacity rather than transfer quota. It's many many orders of magnitude cheaper.

In AWS's defense, their egress pricing covers basically 100% of the cost of networking for their whole platform. VPC, ingress, and many other features and services roll up under that bill. On that basis I have to disagree with the analysis in this post.

However, their main point is well-made, its a lopsided agreement that traps customer's data where in some cases its just not affordable to leave. Its not well known to customers until they face the bill... there was a story of a giant data project at NASA where they were planning to store hundreds of petabytes of data on AWS, but neglected to account for the costs of egress which through the entire project into peril: https://www.theregister.com/2020/03/19/nasa_cloud_data_migra...

If you are in Brazil, AWS Egress is so expensive (and the BRL is so undervalued against the USD right now) that hosting in a bare metal with generous free bandwidth tiers (like 20TB you can find pretty much anywhere) is a huge competitive advantage against those that are hosting on AWS.

This is a bit shameful. Publicly ousting AWS because they basically don't want to join Cloudflare's little attempt at market domination named, in 1984 style, Bandwidth Alliance. Any attempt AWS makes or for that matter any company makes to build a moat around a business can be categorized as anti-consumer behavior. Of course, AWS should do their best to make sure they make it harder for their customer to leave on a dime while at the same time doing their best to keep them happy. Why wouldn't any business do that? It's not a charity. The entire cloud infrastructure providers could be seen the same way, you build your infra in the cloud and it'll become significantly more difficult to move away to your own datacenter at some point in the future. This is well know yet, that's not an issue raised here but asymmetric bandwidth cost is the axe Cloudflare chooses to grind?

It seems to be an egregious and opportunistic move by Cloudflare to go after AWS right as their parent CEO steps down after a divorce.

Weird that Digital Ocean is not part of the bandwidth alliance.

If they were, would that mean I may not get charged bandwidth for traffic from my DO VPS to CF proxies?

Oh that is interesting because I remember DO and Linode were part of their partners during the initial announcement. And now its name is gone.

https://www.cloudflare.com/en-gb/bandwidth-alliance/

Thank you cloudflare for calling AWS out on their ridiculous transfer pricing. Anybody working in this space has been frustrated with it for over a decade. We all know it's a ripoff, and we all know it's priced in a way to lock you and others into AWS.

I've used AWS at every job I've ever had. But that bandwidth pricing has been incredibly frustrating.

Also where possible we now host on lightsail, because the bandwidth savings make the hardware completely free. It's not the best hardware, but it works. The web console is also a joy to use by comparison.

Or, it's priced at market -- within 10% of Azure, GCP, and Alibaba, and not the highest priced among them.

Except it's not when it comes to peered bandwidth, like with cloudflare. Which is a big deal especially to cloudflare.

If you're in Alibaba, Azure or Google Cloud you can cut your costs by putting cloudflare in front, even though that adds another service you pay for.

And all of them have ridiculous bandwidth pricing compared to the entire rest of the hosting industry.

Calling the pricing ridiculous is why engineers are underpaid.

Don’t value bottoms up, you undersell yourself. Value top down.

I only use AWS because companies I work for are locked in. When I am spending my own money, I love the low costs at Hetzner and when I need to spin up VPSs I find GCP much more convenient to use. I am also evaluating Oracle right now because they have a generous free tier VPS and their services are easy enough to use.

I pay $$$ for Apple’s products and services, but I don’t enjoy using AWS. I do enjoy Amazon’s sales and delivery platform for buying physical things, and their digital media, so I don’t mind paying $$ for that.

This post is not very fair, and seems to have started with a conclusion "big company screwing their customers," instead of really digging in to learn why things are the way they are for good non-customer-screwing reasons.

The post DOES actually call out that wholesale bandwidth is symmetric, but then jumps to the OPPOSITE of the correct interpretation of that: That AWS's pricing to the end customer should be equally symmetric, and the fact that it isn't is evidence of AWS's customer-screwing behavior.

The actual reason is that AWS's customers, like those of other cloud providers, are running externally-facing services that mostly send data OUT to their users. Given the inherent symmetry in wholesale bandwidth, it means that AWS has to build out based on their total egress bandwidth. As customers egress more data their users (or wherever/whoever), AWS has to not only pay for more wholesale bandwidth, they also have to build out all the additional infrastructure (routers, etc) to support it. It's very expensive.

The ingress bandwidth, being less than the egress bandwidth already noted above, therefore is effectively free to AWS, and they can charge very low rates to their customers for it.

Digital oceans bandwidth charges are about 0.01/GB, which is the cheapest I’ve found. Anybody know of similar or better offerings elsewhere?

Hetzner Cloud is known in the DACH region (Germany, Austria, Switzerland). They charge 1.19€ ($1) per TB and 20TB per Compute/VM is included.

Scaleway - egress free

Azure discount for Cloudflare seems tiny though, but that won’t fit into their recent marketing efforts Cloudflare has been doing.

I didn't see the amount of the discount in the article. Is there a place where one can see the pricing (i.e. 10TB azure -> CF)?

I couldn’t find it when I searched. Some BA participants have fair usage limits too. Ex: I think Wasabi is about 1x your stored data per month.

Reads like CF has a bone to pick with AWS. Are they mad about AWS not in the peering group? Is the AWS price really out of line - the comparison to wholesale cost only seems to miss a lot of infrastructure that gets baked into the final retail prices - it's not simply (Retail-Wholesale=7000% margin).

Feels like a weak "hit piece" to me.

I think it's quite fair. You're not paying anywhere near close to the price of the infra with AWS; you're paying for the convenience, but be mindful that the cost has a break-even point and many companies may be beyond that point.

I don't have the experience to assess whether the total cost of ownership is favorable to most AWS clients or not.

They directly address these costs in the article as "rounding error". Do you have some evidence to dispute that? Did I misunderstand them?

They are in the market for reducing that bandwidth by caching stuff at CloudFlare, so it's not exactly surprising.

Speaking of egress... I've been thinking about hosting stuff on Hetzner and using Backblaze B2 for storage to get a cheap alternative to EC2+S3, but unfortunately they're not partners so I won't get free traffic, or low latency :(.

They (B2) are with Packet (rebranded to Equinix Metal), last I checked.

If tou put the CF in between will not pay the egress. But will be better to Hetzer be part of the alliance.

Does this make sense?

He seems to be comparing peak bandwidth pricing (megabits per second) to average throughput pricing (megabits).

Peak bandwidth pricing is really a lie (if you actually used it all the time you'd be looking for a new host)

Calculated assuming 20% utilization, which is the low end of industry average (20% – 40%).

I wonder if this could be taken as IE on Windows by default. Would this be anti competitive knowing there are industry efforts to benefit customers on other big customers.

Back in the old day, we were charged by the maximum bandwidth. Not quite scalable maybe, but it definitely won’t bankrupt you if you’re not careful.

While it ain't wrong, I'm not sure I want to trade one cartel for another, Cloudflare.

No one will migrate off baremetal if ingress were to be that expensive.

I would point out that Oracle's Egress is $0.008 per GB, based on their estimator.

That makes a hybrid cloud (on-premises/cloud) more than doable -lots more flexibility. The rest of the cost structure is transparent, with industry leading pricing and performance (take a look at HeatWave for an example of this).

As a cloud advisor, I often see a lack of proper due diligence in cloud adoption decisions. The typical cloud decision is made at the CFO or CIO level without full visibility into the cost of complex application delivery.

These are not stupid people. My take is that a deliberate fog is created with many cloud vendors to obfuscate the true cost of business.

Three blindsides hit the executives.

One, performance at production scale require ala carte additions for reasons outside of the workload's demands.

A case in point, a cloud vendor required an 8 vCPU instance to equal the performance of a four core, on-premises box at production workloads. The storage driver ate half of the available CPU and IO traffic was routed over a single vNIC shared with application traffic. This was unexpected by the customer -something hidden in the vendor's implementation.

Two, lack of transparency, leads to billing surprises.

Another of my clients budgeted $45k for a 3 month production cloud deployment test. The CFO had an anti-pleasant meeting with the CIO when the first month's bill hit $40k. The cause was egress charges. The experiment ended and they stayed on-premises. At least they had tried the workload at production scale.

The third is cloud 'divorces' are almost never considered. The cost of 'divorce' is extremely high, cost prohibitive in many cases but not just for egress fees.

For instances a client using a cloud native database found initial performance good for inserts, but after time query performance became unacceptable.

The promised scaling was not there. Fortunately the lesson was learned while egress was 'affordable' for moving 2TB of production data out. The pain came into play with the additional computing resources needed to export (not copy out) the data out for repatriation.

Insult was added by maintaining two production environments for the 10 days to perform the export and required expertise to implement it. That was for one application's analytic database. Now imagine moving full technology stacks for multiple critical application.

How would I avoid the issues:

1. If billing isn't fully transparent, then move to another vendor.

Too often, a small application gets to a $100k annual spend, the cloud vendor give a fair discount at this point. Once the spend hits a $1m, usually to compensation for vendor deficiencies at production scale, there is no additional assistance offered. The vendor has you exactly where they want you and your are locked. Greater discounts for 3 year terms you say, you've got three years of sub par service and get to spend more money to solution yourself out of the hole.

2. Test at production workloads. Does the spend match what you need or do you need ala carte upgrades to get you where your workload's performance needs to be today and out years?

On then other hand are you paying for the architectural deficiency of the vendor compared to your current as built.

3. If you are dissatisfied with the quality of product or more importantly the level of service, can you just leave then and there?

A quality cloud vendor will create a great experience pre and post sale. The will not create artificial barriers to departure.

The golden rule for a valued cloud vendor is, "I have to prove myself, every day, in every way to keep my customer for tomorrow."

If you sense that isn't the demonstrated action of a perspective or current cloud vendor -move along.

I don’t think the traditional waterfall style of infrastructure planning (all planning up-front, implementation on rails) is optimal for the cloud. Even the best plans are likely to run into unexpected difficulties.

Unfortunately suggesting actual DevOps practices (i.e. giving developers some actual control over infrastructure and a more agile approach to infrastructure development) is viewed as heretical and crazy in large enterprises.

The end result is enterprises paying cloud premium prices, but, with none of the flexibility and agility that make those prices worthwhile.

TL;DR: Cloudflare are upset that AWS won’t peer with them in terms of Cloudflare’s choosing, and this is the angry shitpost.

It does seem like Cloudflare are not the only ones upset about how AWS handles traffic

Base on the costs of transferring data into AWS being much cheaper, I wonder if a reverse transfer protocol could be designed with an adaptation of binary search.

You divide your data in blocks of, let's say 100 Mb, the receiver creates a block of random data and sends it to a process running on AWS. The process has to answer, higher, lower or exact (and then both parties move to the next block).

Probably it will be very slow, but it will lower a lot the bandwidth usage for egression.

Getting out more data than bandwidth allows is a information theoretical impossibility. Your proposal has the flaw that it needs one pass per bit in the block on average.

We consistently hear from our customers that egress and transit costs are one of the most difficult things to track from a cost perspective as it relates to AWS.

If anyone is struggling to see egress as it relates to S3 specifically, we put together this guide here which may be relevant for this conversation as well: https://www.vantage.sh/blog/how-to-see-s3-bucket-request-met...

I've assumed this was well-known... AWS has never been a good deal on the billing side. You're paying a huge premium for API access and the strength of the AWS brand.

I was a bit confused as to why Cloudflare was blogging about it until halfway through the article, when The Bandwidth Alliance came in. Makes sense now: they just want to shame Amazon into offering free egress bandwidth to Cloudflare. This is basically a hit piece.

Also, another aside: for hosting providers (AWS included), ingress bandwidth is typically free because it's also basically free for the provider. Outbound dominates so much that the inbound traffic is a rounding error if it's billed by their transit providers at all. But I agree that the high egress prices are a vendor lock-in strategy.

Perhaps I misunderstood, but the article says Amazon already does not pay for egress to Cloudflare. But Amazon still charges their customers as if they did. Did I misunderstand something?

You sure did. Peering isn’t free, or even cheap. Cloudflare are whining for special treatment for their bandwidth cartel, and AWS are large enough to tell them to get fucked.

The key is realising that Cloudflare have always been about disingenuously leveraging other people’s stuff, since they’re basically just a middleman.

Peering's definitely cheap compared to what Amazon's charging. Amazon even has a peering product for 16.5 cents per Mbps, but it's not usable in this circumstance.

"Please join our free peering group of many companies." is neither a cartel nor a request for special treatment.

What are they "disingenuously leveraging"?

Everyone else’s content. Cloudflare purport to provide edge services. Their free product will actually increase your TTI and reduce your overall availability. Cloudflare’s business model is to lower the marginal cost of their revenue-generating business by increasing their overall volumes, and every new sucker on the free plan helps.

As ever, if you’re not paying for the product, then you are the product.

Having AWS give them something for nothing in that regard is a) part of trying to establish themselves as an equivalent tier player (they’re not) and b) leverage that access to get better deals elsewhere.

AWS are likely not stupid enough to get suckered into increasing Cloudflare’s market power for no return, and the resort to faux-analytical self-aggrandising content marketing is a weak attempt at improving negotiating position.

Amazon will particularly hate it because it’s framed so negatively, which is culturally antithetical. If this is the shit that Cloudflare will wring out of their dirty laundry in public, I shudder to imagine how poisonous their internal tone just be.

The content ring is part of their traffic boosting goal and sure it’s a cartel. It’s a group of natural competitors, setting prices as a club.

> As ever, if you’re not paying for the product, then you are the product.

Okay but I care about this purely as an AWS customer. I've used them and want to use them more, and have never used cloudflare.

> Having AWS give them something for nothing

> AWS are likely not stupid enough to get suckered into increasing Cloudflare’s market power for no return

By this logic, should they never drop egress prices at all? Should they actually raise them?

And Amazon doesn't get "no return" for dropping prices. It brings them more users.

Some of the other big clouds also have huge egress fees, but there are lots of providers that don't, and many users move to those specifically for that reason.

The biggest reason to keep egress fees up is to keep a high incentive for users to stay inside the walled garden of a single cloud, and that's very anti-competition and therefore anti-consumer.

> The content ring is part of their traffic boosting goal and sure it’s a cartel. It’s a group of natural competitors, setting prices as a club.

Peering should be extremely cheap though. And that only helps customers.

"Hey everyone, let's collaborate to reduce our profit margin on this product to merely high levels." is not a bad thing, and it doesn't fit the definition of a cartel either.

I recommend reading the Peering Playback. It will open your eyes to the reality of the situation. It sounds like you want lower prices, that’s fine, but don’t latch onto the swill that cloudflare are peddling. It’s a super distorted, self-interested view.

In fact, given the scale disparity, peering with Cloudflare likely increases AWS’s marginal costs.

Ultimately, Cloudflare are a rent-seeking middleman, and antithetical to the end-to-end architecture to boot.

> In fact, given the scale disparity, peering with Cloudflare likely increases AWS’s marginal costs.

Look, I don't care how it's done at a technical level. If they'd like to just stop charging so much, with no routing changes, that's fine too. The problem is that I can get full transit service for a tiny fraction of what AWS charges for egress. I want a competitive price. And I don't care how selfish cloudflare is being while they highlight a real problem with AWS that's been obvious for many years. Their selfishness doesn't make what they're saying 'swill' either.

And it's pretty hard to distort "this is how much egress costs to supply, rounding up. this is how much amazon charges.". Is their number egregiously wrong? It's reasonably in line with what I've researched in the past.

> I can get full transit service for a tiny fraction of what AWS charges for egress.

That is rather the point: all you get is a full transit service, and the false equivalence is palpable.

I'm very much reminded of the equally fictitious comparisons made between EC2 and Hetzner, or racking up white-box servers.

> I don't care how it's done at a technical level

That is not something I would personally recommend bragging about on this forum, especially when it leads to sleepwalking into accepting a snake-oil salesman's contrived misrepresentations.

> That is rather the point: all you get is a full transit service, and the false equivalence is palpable.

False equivalence how? AWS's offering isn't better, it's just expensive.

> That is not something I would personally recommend bragging about on this forum, especially when it leads to sleepwalking into accepting a snake-oil salesman's contrived misrepresentations.

1. What a rude way to interpret what I said. I don't have a preference between different options to deliver data that all work fine. I'm interested to know what happens, but it doesn't affect my purchasing.

2. Nothing you've said backs up the idea of this being "snake oil". They're not making any promises about amazing products that can't be delivered. They're just asking for a price change on an existing product. That doesn't even resemble snake oil.

You’re not buying transit from AWS.

This pig-headed insistence on comparing things that are not alike, and then complaining when called out for disregarding the details, doesn’t warrant further comment.

You haven't named a single difference. You insist on vaguely insulting me and ignoring me when I ask why they should be treated differently.

They're both delivering packets from a datacenter in a specific spot to anywhere on the planet. They're charged based on different metrics, but it's easy to make a rough translation between those metrics.

I ask if the numbers are wrong, you insult me.

I ask why it's a false equivalence, you insult me.

And you think I'm the one not contributing to this conversation?

Reply on news.ycombinator.com