Comment by mikiem | Hacker News Reader

Comment by mikiem | Hacker News Reader

As a provider of IaaS Cloud and of dedicated servers and colo, I hear this argument all the time. No one ever seems to include the Network Engineers, monitoring systems, the routers (better have more than 1!), the switches (distribution and access layers), the maintenance, software licenses (where applicable), customer support, cost of IP addresses, Account Payable, ARIN membership, RADB membership, cross-connects, optics, spares and/or support contracts, etc... and finally, you do not use a 1Mbps at 100% for 24hrs per day, so while 1Mbps for a month is ~320GB, in reality, the way most people transfer data, 320GB would look more like 3Mbps at 95th percentile (the way burstable bandwidth is billed)

A basic 1Gbps commit on a 10Gbps port in a data center might cost you from $0.50/Mbps (something like Cogent) to maybe $1.50/Mbps (let's say Level 3), other providers could be $4+/Mbps. By the time you factor in all of the above overhead costs, the true cost of the bandwidth is much much higher on a per Mbps basis.

Don't forget to significantly over-build your stuff, or you might get knocked off-line for anomalies or DoS attacks.

Admittedly, the scale of Google, AWS, Azure makes the cost per Mbps much much lower, but when as others have pointed out, AWS, Google, Azure don't need to charge less than they do.

Straw man argument, you're looking at around $300/month for 20A, 10+ rack units, 100mbit commit to play with in a lot of decent US facilities. That's a few pathetically provisioned VMs on most clouds. I have such a colo for my personal use. This doesn't exclude me from using other providers for DNS and MX slaving, or things like object storage (i.e. I use Tarsnap for DR) and there isn't really any overlap or lost efficiency in mixing and matching those higher level services. My facility's house bandwidth blend provides free DoS mitigation as a nice bonus.

Most facilities and providers don't require you to have your own ARIN registration and block. In fact it's quite difficult to get even the smallest blocks right now. Instead, an ISP will lease you IPs usually at around $1/month.

A pair of x86-64 boxes running OpenBSD can easily virtual switch, route and firewall well above 1gbps. They can announce BGP if needed, if you have a delegation or are muli-homing. A pair of high quality 10g switches may push a few grand on the secondary market but you can ~get away~ with Ubiquiti gear.

This kind of setup can scale up pretty high by dropping the OpenBSD boxes from the forwarding plane and just using them as route reflectors into higher end switches.

There is a slight labor disadvantage that amortizes quickly. i.e. setting up switches and routers isn't hard, and if you aren't sysadmin skilled enough to do that in a few days you probably shouldn't be running VM infrastructure either and retreat to something like Heroku, Lambda or GAE where there is much less room for footshots.

So TL;DR $300/mo and 5k CapEx will run a company, double or triple that for DR of a serious startup. Reap significant saving as a larger company by partnering or outsourcing remote hands and basic ops with a company like yours for the day to day stuff.

This is bull. I've used colo and been paying 95th percentile billing for a decade. We've run our own hardware. Much of your list didn't even come in to it. E.g. you don't need your own router, you get awesome ddos mitigation from upstream providers like NTT super cheap. It's been a major competitive advantage to get bandwidth this way.

Cost of IPs? Memberships? Wtf are you talking about? Cross connects? This stuff is all free, not needed or cheap and a one time fee.

Really depends on your scale. If you just run half a rack of equipment a lot of these costs are factored in by your housing company. But you come to a certain point where you need to invest in a RIPE membership, where you need a router that costs 100k€. Plus a spare one. Where you need someone to watch the monitoring 24h sitting in the DC - just in case. Even if a lot of this stuff is free, it needs to be setup up and managed all the time.

Let alone the cost for DDoS mitigation. Thats an easy 300k€ worth of equipment plus lets say 2x40 GBit/s links for lets say 40k€ each per link and month. Over the course of two years thats easy 1 million euros, just to handle DDoS. Even if you just need 10 GBit/s capacity, you might need 10x that capacity in a DDoS situtation.

Here's the reason no one thinks about those things: no one makes their numbers public.

Or at least I've never stumbled across a blog post with anything like a line item cost range for everything that goes into DC or cloud networking.

And in the absence of transparency, yeah, people are going to assume they're getting screwed.

(I understand why this is. The networking free market seems to do a decent job at fulfilling needs, but it turns all those things into secret sauce that shouldn't be shared.)

Yup. I took a swag at a generalized breakdown of "data transfer" costs https://news.ycombinator.com/item?id=7479030 four years ago. The values might be slightly different today, the logic isnt.

I suppose part of the issue is marketing. The customer reads the bill as "bandwidth." Would consumers understand better if there was a separate "network infrastructure" line item? Would they accept differntial pricing for network quality or class? Dont see why we'd find out as long as theres no downward pricing pressure.

How could I forget electricity: A/B power for a pair of Cisco ASR 9006 routers costs us over $1,000/mo!

The article compares cloud costs to putting a little x86 server on a colo-provided 100 Mbps connection. I think you are not talking about the same kind of service, and also this reference price does include amortized cost of network engineering.

A lot of the things you mention are presumably covered in the cloud provider's service prices for things other than bandwidth.

Another possible explanation is that the ratio of available (or build-out-able) compute power to available bandwidth they have is such that they just can't give you cheap bandwidth and their pricing reflects that.

Cloud data transfer is very overpriced, no doubt about it.

I'm currently paying 0,017 USD for high quality transfer, with a managed network. I could get even lower pricing by switching to 95th-percentile. Maintenance on server hardware is damn cheap with remote hands and hardware warranty.

It would cost me over five times as much to use GCP with 50 TB traffic/month, all things considered.

Cloud providers are cool, but bandwidth is still very overpriced.

Also, your transit prices are way higher that reality, at least for Europe.

Actually, we just bought bandwidth for a roll out at Equinix in Munich. $0.50 for Cogent (when added to several other 1G commit on 10G ports in our account. A single 1G commit on a single 10G port would cost more) and we were quoted $1.43 for Level 3 after rejecting a $1.70 quote. Both Cogent and Level 3 were 2yr terms, and in an "on net" location. We are going with another provider besides Level 3 there, but I used these as examples in the parent. You thought it was not "reality", and I refute that.

Reply on news.ycombinator.com