Hacker News Reader

Comment by datavirtue | Hacker News Reader

Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service.

I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.

View on HN

There are, literally, dozens of hosting providers that bake unlimited egress into their offerings - still. Bandwidth is cheap and has only gotten cheaper. Remember that Amazon doesn't always need to pay for Internet access. It's very likely they participate in Internet exchanges tied to other large providers. Given Amazon has a massive footprint they likely participate in these all over the globe to reduce congestion on their Internet connectivity.

They very much have massive margin on egress. Given some of the cost comparisons floated today comparing R2 to S3 egress AWS is likely hitting 1000s of times (likely more) their return on the actual bandwidth cost they pay for month over month.

CPU isn't cheap, typical PHP+MySQl pages are ridiculously heavy, that's why Cloudflare got themselves into business.

Usually DoS attack doesn't exhaust the bandwidth, come on..

> Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests?

Come on, Amazon is not serving traffic through a Comcast business connection, they're peering directly with other large operators for free or for next to nothing.

I mean that's another vector for savings... Maybe Cloudflare bought their own cable modem instead of paying the $20/month to rent one. :)

In the few times I've priced blended connections at most Colo locations unmetered 10G is ~500/month to ~1000/month.

5Gbps*1 month is ~1.5 PB. AWS is about 0.02/gb or ~30k/1.5pb.

Approximately 30x the cost.

These are old numbers for specific use cases so I'm not sure how much that has changed.

AWS bandwidth is definitely over priced however the comparison to an unmetered colo isn't quite fair. What happens when that unmetered colo gets hit by a 100G ddos attack? Everything you have on it goes down?

In fairness, Amazon also charges for more than basic DDoS protection:

https://aws.amazon.com/shield/pricing/

but regardless, the EGRESS charges are what are absurd. The only logical reason for charging so much for data exiting is to make sure that you can't practically leave the AWS ecosystem.

If the discussion is about what it costs amazon, that's not really relevant. They avoid DDOS simply by being enormous.

We have a customer on Azure running a bunch of medium-sized websites. Their mean outbound throughput is just 25 Mbps, but they're paying on the order of $3K/month all up for that Internet egress. (Not just bandwidth but some additional overpriced services on top like App Gateway.)

I have had to. I pay $400/month for the following:

- A 42U cabinet - A 15A 120V circuit - An unmetered 1Gbps IP transit link

All at a proper datacenter, namely Hurricane Electric's fmt2 facility. Includes a /29 of IPv4 and a /48 of IPv6, allows me to announce the /24 and /36 I own, and there's a free internet exchange onsite which I have a 1Gbps port at.

1800 Watts for a full rack? That’s bordering on useless. You can run maybe two real servers on 1800 W.

You need to provision enough power for simultaneous startup after a power outage, unless you have some really smart PDUs and automation. We have a 10 year old DC with 30A@208V per rack and we have to leave racks half full because modern servers are so power-dense.

Gatekeeping server power!? We run 1U on 250-350w (230v). So yes, odd power allocation but servers are real way before 900w. We're running dual Xeon Gold low core count machines (because of Microsoft licensing) and they're pretty decent.

What’s the startup peak draw of those severs? Have you measured it? Remember everything turns on at once after an outage.

In any case, even at your optimistic 250W per rack unit, a full rack of servers with two switches is more than 10kW!

Absolutely, I'm with you. I just meant that 900w isn't where "real servers" start. If you're at the point where you're experiencing issues with scalable power on you could disable that feature and script ipmitool to power the servers on once ipmi is reachable but in a queue. Could help you eek out a bit more density.

You could possibly run this fictional daemon on your management switch :)

My 900W per server number comes from experience with “hyper-converged” infrastructure where each 2U node has two filled CPU sockets, gobs of RDIMMs, and is stuffed with flash-based storage.

I think this is the most common “enterprise” datacenter server type in 2021, mostly due to licensing constraints from VMware/Microsoft/Red Hat/Oracle/etc. Such servers give the most “bang” per dollar when licensing costs are included.

We're running VMware with vSAN on 1u nodes with 2x Xeon Gold 8 core cpus @ 3.6ghz base, something about the Microsoft SPLA licensing as we're mainly virtualizing Windows machines. 384 GB RAM per node, and we're just pulling about 400.

https://i.imgur.com/Zr5rq4X.png a graph if you're curious :)

Fair enough, but for that price it's hard to beat. Sadly, getting another rack is cheaper than another power circuit though.

Nope, plenty of server hosts provide bandwidth for free, or near enough.

Just one example is Hetzner, who include 20TB of bandwidth, with anything over that charged at just €1/TB.

Meanwhile, AWS is gouging at €80/TB.

You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/

To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.

The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent.

The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe.

If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps.

(Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)

Yet CloudFlare is able to do it for free up to a point, and at a lower-than-AWS fee after that. It might not be the exact figure but there is most definitely a huge markup there.

And your "business internet connection that is offered by ISPs" is a joke compared to the backbone AWS uses.

Remember those ISPs need to make margin, that's why they charge what they do.

AWS _could_ make it's margin on services alone. They just choose not to.

Reply on news.ycombinator.com