Hacker News Reader

Comment by lpcvoid | Hacker News Reader

Can we at some point acknowledge that constant cloud disruptions are too costly, and can we then finally move all of our hosting back on-prem?

View on HN

It's the old IBM thing. If your website goes down along with everyone else's because of Cloudflare, you shrug and say "nothing we could do, we were following the industry standard". If your website goes down because of on-prem then it's very much your problem and maybe you get to look forward to an exciting debrief with your manager's manager.

That's lazy engineering and I don't think we as technical, rational people should make that our way of working. I know the saying, but I disagree with it. My fuckups, my problem, but at least I can avoid fuckups actively if I am in charge.

How do you mitigate large scale DDoS?

I don't, since my stuff is reachable only within the company network/VPN. If I needed to though, I would consult the BSI list of official DDOS mitigation services [0] and evaluate each one before deciding. I would not auto-pick Cloudflare.

[0] (German) https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Cyber-Si...

When the solution you pick inevitably has downtime too you’re in the same boat.

DDoS mitigation is one of those areas that an on-prem solution just isn’t well suited to solve.

Yeah, but people aren't using Cloudflare just for DDOS Mitigation. Some are running pretty much everything over it, from DNS to edge caching to load balancing and even hosting. That's what I oppose mainly.

Unless you are really big, onprem stuff would be 90% internal anyway. For everything public you'd host your hardware in a datacenter with better high speed connectivity. And pretty much every single datacenter I interacted with in the last 5 years does have a DDOS protection solution that you can order for your network.

The problem is the people that sign our checks usually aren't technical, rational people.

The system isn't designed for technical, rational decision making.

That's fair, yeah, and I agree it's not always feasible - but if you have any influence over technical direction at your org, I encourage what I wrote above. Otherwise yeah, let the pea counters in the C-Levels dig their own grave.

Funnily and ironically enough, I was trying to check out a few things on Ansible Galaxy and... I ended up here trying to submit the link for the CF ongoing incident

I would only consider doing stuff on-prem because of services like Cloudflare. You can have some of the global features like edge-caching while also getting the (cost) benefits of on-prem.

can you define "constant"

Well, between AWS US EAST 1 killing half the internet, and this incident, not even a month passed. Meanwhile, my physical servers don't care and happily serve many people at a cheaper cost than any cloud offer.

We had an Azure outage in between those 2 as well.

You realize these are two different companies right? If you’re saying “I’m an AWS customer with cloudflare in front” I think you’ve failed to realize that two 99.9% available services in series have a combined availability of ~99.8% - that’s just math.

Your physical servers should have similar issues if you put a CDN in front unless the physical server is able to achieve a 100% uptime (100% * 3 9s = 3 9s). Or you don’t have a CDN but can be trivially knocked offline by the tiniest botnet (or even hitting hacker news front page)

I do. But I put both into the "cloud offering off-prem for very much money" shoebox. I setup a CDN once using VPS from different hosting providers for under 100 USD a month, which I would vastly prefer over trusting anything cloud.

And yes, I know that there's sites that need the scale of an operation like Cloudflare or AWS. But 99.9(...)% of pages don't, and people should start realizing that.

People who don't need that, also don't care much for an hour or two of service disruption. Most users will have far worse disruptions with the alternatives.

How do you back up?

We have a few colocated servers offsite, each in a different region, each with a zpool of mirrored spinning rust. We use rsync across those at different times.

never build on us-east-1, everyone knows that ;)

Reply on news.ycombinator.com