Comment by api | Hacker News Reader

Comment by api | Hacker News Reader

Parent Full thread

Based on the pricing you see at VPS providers, bare metal providers, and colocation, I have long called BS on large cloud provider bandwidth pricing. It's completely insane, like more than 10X higher than VPSes and more than 100-1000X higher than wholesale (bare metal and colo).

Take this for example: https://www.fdcservers.com/

That company is profitable, which means they have to be paying less than this for bandwidth or at least selling bandwidth at cost and paying less for other things. I'm sure they are overselling but even then this model couldn't work if bandwidth were anywhere even close to what cloud charges for it. (We run a few things there for high bandwidth simple services and it works great and seems to be as advertised. There are many other companies with insanely low pricing on bandwidth, but they're one of the cheapest.)

The key to understanding why I really think AWS/GCP/etc. price bandwidth the way they do is that ingress is free and egress is absurdly expensive. I call this "roach motel pricing" after the old roach motel ad: "roaches go in, but they don't come out." The obvious objective is to get data to flow effortlessly in but make it costly to either relay data for edge or endpoint first computation or move data out. The whole pricing structure is to encourage cloud providers to suck in the entire computing ecosystem and host it internally within large data centers, which is their business model. It works very well.

The instant I saw this pricing structure when cloud first started to be introduced I immediately understood what was happening here.

Searching for terms like "wholesale bandwidth transit" might be a better reference for comparison of actual cost. The providers you mention are depending somewhat on customers that statistically underutilize what they have and using obscure TOS clauses to chase away heavy users.

That's the list price. You don't pay the list price if you purchase at high volume. Bare metal and VPS providers are surely very high volume customers.

This is perhaps an argument against pure DIY colo, but most small scale colo isn't connected "naked" to the Internet but gets racked in a data center that offers a bandwidth package and does peering for you. The only people who peer directly tend to be clouds, bare metal hosts, CDNs, and ISPs. Even if you have your own IP space and an ASN you are usually behind someone who is bulk purchasing transit and handling peering.

To be fair very large customers can negotiate costs down at cloud providers too, but you have to be spending tens of thousands a month to even have a chance at getting their attention for any kind of special deal.

Company I used to work for had an AWS bill at 7 figures and sometimes 8 (a very time based offering)

AFAICT we had a blanket 20% off the bill, nothing special about egress.

It's probably the closest proxy for which you can get meaningful numbers.

One complicating factor of egress is that it's not all created equal. Some providers have really good peering agreements. Others (even some big players...) have bad ones. Some cheaper-than-AWS options aren't really selling the same thing.

It's really hard to evaluate that before you hit issues in production, too.

> Take this for example: https://www.fdcservers.com/

You took "as example" literally the cheapest provider that exists in this space. (and lowest network quality/connections)

Okay, then here's more:

Hetzner, Datapacket, Hivelocity, Reliablesite, and OVH.

I did pick them as an extreme example, but even the more expensive bare metal and colo providers are like 100X cheaper than cloud. VPS providers like Digital Ocean and Vultr are ~10X cheaper.

Cloud sells egress bandwidth at an enormous markup. Meanwhile ingress is free. If this were a technical limitation ingress would be billed too because there's nothing asymmetric about the Internet. The asymmetry you usually see on home connections is mostly a DOCSIS (cable modem) limitation or a way of segmenting personal vs. business accounts.

BTW we run a few things at FDCServers (among others) and monitor them and they seem fine. We do use multiple providers in case one of them has a huge issue.

Edit: these providers can't be overselling too much because the fact that they have such cheap bandwidth makes them a magnet for bandwidth-heavy work loads: game servers, seed boxes, DIY CDNs, TURN servers, etc.

> Cloud sells egress bandwidth at an enormous markup. Meanwhile ingress is free. If this were a technical limitation ingress would be billed too because there's nothing asymmetric about the Internet.

Symmetry is the entire problem.

1) to maintain settlement-free peering, you need traffic-symmetry. Amazon — on their own - transmits far more data than they receive. They benefit far more on this account from ingress traffic than more egress. Hence, the economic signal that ingress is free.

2) If you have unbalanced network traffic, your scaling factor becomes the transmit direction that is more heavily utilized. A 100gbps circuit can generally send and receive 100gbps simultaneously. But it can’t be used to send 150gbps and receive 50gbps. So if you have a bunch of people all wanting to transmit from your circuit far more than they receive, you have to upgrade your circuits and routers earlier than you otherwise would for a given aggregate amount of traffic. This is expensive, but adding more ingress traffic is free.

Assuming that's all true, it still doesn't explain the magnitude of the markup. VPS providers can do $0.01/gig outbound almost across the board. This includes some very large ones, and because bandwidth is cheap there people tend to host a lot of high bandwidth things there.

GCP, AWS, etc. are roughly 10X more expensive than this. Are they 10X better?

The cost is very high. I can't stress this enough. A terabyte outbound at AWS is roughly $80. For around $300/month I can get 1gbps unmetered at a highly regarded high quality bare metal provider and transmit well over 200TiB in a month. I know because I've done it continuously for years. It works fine. That means AWS is marking up bandwidth by at least 50X at a minimum.

(The above are rough numbers, but they capture the magnitude of the difference.)

If what you say is true and the markup is reasonable then all these companies should be out of business. Either that or they're overselling bandwidth by a giant margin, but that doesn't check out because they specifically advertise as places to put your extremely high bandwidth work loads. If your business model depends on people not doing X, don't advertise as a place for people to do X.

Note: at no point in my comment did I say I thought the specific dollar amount of the bandwidth markup was reasonable or unreasonable. I pointed out reasons why differential bandwidth pricing for ingress/egress exists at the major cloud providers, when you claimed there were no technical reasons.

Also note: VPS providers generally do not exist at the scale where they could or would pursue settlement-free peering. They buy transit from one (or, hopefully, more) upstream providers who have to worry about the first point I made. The VPS providers still have to worry about point #2, but they can do that by courting different kinds of customers, without the baggage (in the context of running a network) of a big legacy content distribution business that both Amazon and Google have.

Reply on news.ycombinator.com