Hacker News Reader

Comment by mehrdadn | Hacker News Reader

Does anybody else feel like there have been a lot of outages in recent months? And I don't mean Google -- I mean lots of others too (I seem to recall CloudFlare, Facebook, etc.)... are they really increasing or are we just hearing more about them? Seems a bit odd.

View on HN

Now that you mention it, I just realized why. The current few months are the intern season!

That's more or less inevitable. As complexity increases (which it does naturally, if there's no effort to decrease it) at some point it begins to outstrip the limits of human understanding.

I've been saying this repeatedly (and downvoted for it repeatedly): if you want truly reliable systems, use simple, boring technology, and don't fuck with it after it's set up, and run it yourself. 99.99% of all these outages are due to screwing up something that already works, something that if it was in your own rack you could just leave alone and not touch at all.

My horribly out of date system works, therefore I should never strive to improve it or god forbid update it (since that involves “fucking with it” in ways that can break it from version to version)? That gets you technical debt and that’s not fun.

I'll tell you more. Much of the world is run by "horribly out of date" systems that nobody has touched in years _because they work_. And it all works fine. No "cloud", no Rust or Go rewrites, no Haskell, no fancy javascript frameworks or anything like that. Just boring ol' files, boring relational DBs with boring schemas, constraints and stored procedures, boring old languages, boring old hardware, boring old operating systems underneath it all. Don't screw with it and it will work for a decade. Start screwing with it and it will be busted every month, like Google Cloud.

You can't create "technical debt" if you don't change anything in the first place.

I got an email yesterday that told me the boring old HPUX server (which was racked before my intern was even born) barfed all over its boring old 50-pin SCSI drive and ops went scrambling to find one in storage so the boring old Oracle DB that was responsible for production lines running could be recovered. Took us around an hour, cost us a boring 5 figures. Luckily our sysadmin knows how to hide “unused parts” for days like that or we’d have been really in trouble.

> You can’t create “technical debt” if you don’t change anything in the first place.

Rubbish. The bits really do rot, and if you don’t do _something_ on occasion you end up with an entire data center no one wants to touch because the dust in the servers might be structural at this point.

I’m not saying go rewrite your apps against the Kafka instance your junior devs are fucking with, but you have to do something to fight the entropy.

The first part of your post sounds like a success story to me. You got many years of use out of that server. 5 figures is cheap comparing to the cost (including the collateral damage cost) of a hotshot SWE or devops guy who insists on using the most resume-worthy, most bleeding-edge technology available.

I wish I could upvote you twice - I just finished a multi-week effort to unwind (defuse?) some of the resume-driven architectures that were left behind when resume-driven development was successful.

There's boring and there's legacy. Legacy is when the hardware is unsupported and doesn't get software/security updates. You don't want to let it become legacy.

The counter-story to yours is running that database on MongoDB in the cloud on a cluster. Instead you'd be having crazy MongoDB issues, data inconsistencies, connectivity issues when the cloud is down, etc etc.

The solution is somewhere in the middle. You can have modern, supported hardware running a LTS Linux and that counts as boring.

I think you are right. Where I’ve seen success in “the boring middle” is when an appropriate amount of tension exists in the engineering organization: you want some teams and groups pushing to try new things, but they need to push against a boundary - preferably something, not someone - and the boundary should define your organization’s best practices and standards. This way a team doesn’t get to sneak clustered Mongo into the cloud and make your ordering systems talk to it.

But over time boring IT turns into legacy, and without some tension to the system pushing it forward your standards end up locking you into legacy forever.

Sounds like you were bitten by the technical debt that was created by placing an important database behind a single point of failure - a single physical server. Of course something would break sooner or later, PA-RISC and Itanium servers were great, but still not magic.

Important things go onto clusters, or at least have a (hot or cold) standby server.

> you have to do something to fight the entropy.

Stuff breaks. So you fix it. Boring old stuff needs fixing too sometimes. Problem is, old stuff gets obsolete, can't get replacement parts, because of progress. (or something). It's the same story since the first looms were made centuries ago.

What you can't fix you can't really depend on. Our time scales are just compressed to ridiculousness because the pace of change is off the charts these days. So basically, you can't really depend on anything working more than a few months before falling over. Sucks.

That's not entirely true. you don't have to try to add features in order for the operating environment of a legacy system to change. More users, transaction count fields overflowing, timestamp fuels hard coated without the century or with 32 bit time_t values...

Or it may simply not meet the needs of users anymore.

I would hardly hold the air traffic control system up as a model to aspire to, for example. The only reason we run the old one is that the upgrade attempts all failed.

Nothing ever is "entirely" true.

Of course - the point this person was making is that this is substantially false in the way they described.

> You can't create "technical debt" if you don't change anything in the first place.

Tell that to your security team.

Curious how you would imagine handling something like GDPR or SOX compliance in this alternative world you’re proposing. You can’t magically foresee new requirements and new implied complexity for all future time.

To many European companies GDPR did not really change the operational requirements - only the penalties for not meeting them.

That’s too clever by half. Avoiding substantial financial penalties for not meeting an operational requirement is an operational requirement.

> Don't screw with it and it will work for a decade.

Don't screw with it and it will have security issues after a few months?

COBOL is a big example of this. It's still ubiquitous in many industries where reliability is near the top of priorities. And I imagine in 50 years we'll still very likely still have many critical systems operating on COBOL. I wonder how many will be running 'The Next Big Thing' language/api then...?

Technical debt is about the increasing difficulty to add features to a system.... if you aren't adding features, technical debt is not really an issue.

> 99.99% of all these outages are due to screwing up something that already works

Fiber optic cables are a great technology, but they don't react well to being cut in half by a backhoe. Is the solution you are recommending that we stop using fiber optic cables, or that we stop using backhoes?

Stopping depending so much on remote datacenters unnecessarily would be a good start.

I'm a remote employee of a distributed company. Where do you suggest we deploy our code/services?

I feel a sense of karmic balance on backhoe fade myself, because after running networks for decades, I now own a backhoe. So far I've managed to dig up a power cable I forgot I laid to an outbuilding, and a coax satellite dish feeder.

So a vulnerability is identified in a version of software you're running within your stack and doing nothing means you will most likely lose important and sensitive customer information if you do nothing about it.

Do you:

1) Don't fuck with it?

2) Make a mitigating code change. Patch / fix it (fuck with it)?

Vulnerabilities don't always matter. If it's some godforsaken internal-only backend that never sees external traffic, study whether there's risk, and if there is none, let it be.

If you must fix it, the correct solution is to replace the affected software with the same (or almost the same) version of the software with the fix. No API changes, no other fixes.

Sorry, but that's bullshit.

Once an attacker is in your organization he will look for exactly that kind of internal-only backend were exploits are already available and the attack vector is known.

There is no such thing as a internal-only backend regarding security.

Let's assume the attacker used social engineering to get credentials from an unprivileged user and uses these to log in to a remote desktop. (I know there are ways to prevent that but I think there are many examples shown that public facing remote desktop is not two unrealistic) Once he is inside your company he can reach the "internal-only" backend and uses the privilege escalation bug you thought is not worth fixing to get root.

Cloud should be a backup, a failover, but people build their entire business on other people's hardware because they can sell the cost per hour easier than the price of a new server which is cheaper in the long run. At this point, with so many outages showing the need for self-hosting, not allowing customers to do so shows how little you care about them.

The challenge of self hosting is then you don't get the sophisticated load balancing the CSPs like my employer, Microsoft and Amazon offer. You also don't get the dedicated networks.

But it's true, it's much cheaper if you can find a way to replicate those or do without.

> At this point, with so many outages showing the need for self-hosting,

Are they really showing that? None of the major cloud providers, even constrained to a single region (or even AZ) seems on average less reliable than the on prem datacenters I've seen, and there's

> not allowing customers to do so shows how little you care about them.

While the solutions may not be as complete for all use cases as public-cloud-only ones, are any of the major cloud providers not working to enable and selling their capacity to support hybrid-cloud deployments?

Have you seen Jonathan's Blow talk that touched on this? I enjoyed it. I think his fundamental point is that as we build on complexity, future generations lose track of the underpinnings and things start failing for unexpected reasons and we may eventually lose our capability entirely. But he does meander a lot.

I've definitely seen this where I work - the "old guard" setup the system that put the company in a prime market position, the newer people are just doing API calls and scratching their heads if it doesn't work.

Here's a reddit link because YouTube is blocked here.

https://www.reddit.com/r/programming/comments/bq1dt6/jonatha...

As more businesses move their compute to the cloud, one might predict that more people will be impacted by outages in the large cloud providers. This in turn means that the affected people will start up-voting these threads. Expect these to be more common.

And unfortunately that is making the web more centralized.

I don't see how this is something that's specific to the last few months though.

It's just global warming again. The weather in the clouds gets increasingly unpredictable :)

I came here to say this - it's like the cloud as a whole is imploding lately.

Seems like if it continues to be a problem that more multi-cloud solutions will present themselves (Terraform does that sort of thing, right?).

Terraform gives you a single management stack to a number of services and endpoints, but it doesn’t magically make your solution multi-cloud...you still need to understand the architecture you are deploying and the idiosyncrasies of each provider and the services used (not a bad thing imo).

microclouds!

What do people do to mitigate DNS services from going down? Is it possible to have multiple services for that? And CDN's too as per our recent CloudFlare issues.

DNS caching combined with multiple servers makes it one of the most reliable services by design.

For small hobby projects I simply use a 3rd party 2ndary DNS service.

Terraform does not handle data locality. Since compute generally sits next to data for latency and cost reasons one should first think about how to ensure that their (perhaps considerable) data set is stored and synchronised elsewhere before worrying about which infrastructure manifest tool to use.

Tinfoil hat: Maybe someone practicing for an attack?

It's almost as if we had made an overly complicated system with too much "efficiency" and thus not enough redundancy, centralizing on too few pieces of what used to be a quite widely dispersed system.

The more "the cloud" replaces many, many servers at lots of different places, the more the outages (which once happened all the time, but to many different organizations at different times) will become big enough to notice.

So, yeah, not just your imagination.

> It's almost as if we had made an overly complicated system with too much "efficiency" and thus not enough redundancy, centralizing on too few pieces of what used to be a quite widely dispersed system. The more "the cloud" replaces many, many servers at lots of different places, the more the outages (which once happened all the time, but to many different organizations at different times) will become big enough to notice.

This is just for the last few months...?

Reply on news.ycombinator.com