Hacker News Reader

Comment by kuon | Hacker News Reader

You can have a 100Gb uplink on a dedicated fibre for less than 1000$/month now. Which is insanely less than cloud bandwidth. Of course there are tons of other costs, but that alone can suffice to justify moving out of the cloud for bandwidth intensive app.

View on HN

We went to cloud because 1) we only need 3 infra guys to run our entire platform and 2) we can trivially scale up or down as needed. The first saves us hundreds of thousands in skilled labor and the second lets us take on new customers with thousands of agents in a matter of days without having to provision in advance.

1) You may more than pay for that labor in cloud costs, but you can also pretty easily operate rented dedicated hardware with a 3-man team if they know how to do it, the tools to scale are there they're just different.

2) I don't know what your setup looks like, but renting a dedicated server off of Hetzner takes a few minutes, maybe hours at most.

My personal opinion is that most workloads that have a load balancer anyways would be best suited to a mix of dedicated/owned infrastructure for baseline operation and dynamic scaling to a cloud for burst. The downsides to that approach are it requires all of skillset A (systems administration, devops) and some amount of skillset B (public cloud), and the networking constraints can be challenging depending on how state is managed.

With 3 people it’s basically impossible to build a ha storage solution that can scale to a certain amount - it’s also impossible to keep that maintained.

Can you give a ballpark figure of what scale you have in mind?

Some distributed databases and file systems are notoriously finicky to operate; Ceph comes to mind in particular. Choice of technology and architecture matters here a lot. Content addressed storage using something like Minio with erasure codes should scale pretty easily and could be maintained by a small ops team. I personally know a couple of people that were effectively solo operations for 100PB Elasticsearch clusters, but I'd say they're more than a bit above average skill level and they actively despised Elasticsearch (and Java) coming out of it.

Just to clarify, AWS lets you provision bare-metal too if your goal is to just rent hardware someone else is maintaining. And things like trivially distributing load and storage across multiple datacenters/regions is another big bonus for us.

Hetzner has all of that. And cloud. Its just that their dedicated server offerings are SO attractive that people keep mentioning that. Otherwise its not like their cloud offering is also very attractive.

Correct, but most of your cost in public clouds is in bandwidth, not server rental. To my knowledge, AWS also charges a hefty premium for their dedicated servers compared to competitors.

Just curious where do you get 100Gb with Internet transit and dedicated fiber for 1000$/month? I'm in a small town in eastern Germany and looked for a simple Gigabit fiber access for our office without any bandwidth guarantees and it's 1000€/month for 1Gb here with the most budget provider but with some nebulous bandwidth guarantees. I'm not talking about residential fiber that also very expensive after a certain threshold. I know there is init7 in Switzerland but it's the exception to the rule in Europe it seems. Getting a fast fiber and good transit is still expensive?

I'm in Switzerland, so maybe I am biased, I have 10Gbit/s dedicated on a 100Gbit/s link for about 600$/month. In practice I have 25Gbit/s with most datacenters in europe, 100Gbit/s with some that are close (OVH, Hetzner), and 10Gbit/s with the rest of the world.

Is this a home connection ?

No, dedicated business fiber from sunrise.

Sorry, I should have been more specific: does this offer stands if I need 200G ? Is the bandwith garanteed (as-in: I can use it all day long, as would a datacenter do) or is it burst-only (so its targets are homes and offices) ?

I expect the latter

I have 10Gbit/s guaranteed with 100Gbit/s burst.

Running a service takes more than a fat pipe. You need to handle power outages, need redundant internet connections, ect, ect.

Yes, but for example a 10Gbit/s pipe is about 3PB of transfer capacity per month which is about 150 000$/month in S3 traffic. A 40kW UPS which can handle about 2 racks (2x42U) of high density servers, with a generator cost about 50k$. A redundant link with your own AS so you can BGP should cost about 5k$ per month (at least here in switzerland).

Of course it really depends on the application, but if you host something like a streaming video service where bandwidth is the main factor, you can quickly reach a point where self hosting is cheaper.

10Gbps is one "teen with a stolen credit card" DDoS event away from being unusable. If you're running a big service that someone may dislike, that's really not enough.

That's why you put your services behind a CDN, even if it's not cacheable traffic. Then you can rate limit what's coming to you.

With the cloud, that DDoS can bankrupt you by causing you unconstrained bills instead.

Oh definitely. I would've been more clear - I meant: you still can't stop there and you'll need a third-party to take the traffic with either solution.

As you’ve already alluded to elsewhere though - you host it behind a cdn or something. A single ec2 instance is just as vulnerable to a teen with a stolen credit card attack.

Yea, I call BS on 100Gb uplink for $1000. I have racked a lot of servers at different data centers. No way.

I am in Switzerland where you have 25Gbit/s for about 70$/month so I understand that this might be an exception. But even if it is 10 000$/month, it is still widely cheaper than cloud bandwidth.

https://www.init7.net/en/

Business pricelist as PDF: https://www.init7.net/de/angebot/business-internet/business_...

It's interesting as it seperately lists price for 24x7 support.

CHF 111 ≈ $129

If you don't mind me asking, to where? That is, what uplink do you see to your nearest AWS or gcloud? In the US, advertised residential speeds don't nessarly translate to realized gains. Just pushes the bottle neck to the ISP.

Agreed that either way it is way cheaper than cloud bandwidth.

Reply on news.ycombinator.com