Hacker News Reader

> It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things.

> I think in 2022 we're back in a similar situation again. Virtualisation, containers, cloud services and whatnot have regressed development reliability.

Realistically you probably want to find the middle ground between these: not copying PHP code through SFTP or even working on it directly, but also not running an enterprise Kubernetes cluster for your simple CRUD app.

Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes).

> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."

That sort of happened with Roblox, they had an outage that went on for 3 days due to Nomad rolling over and dying in their infra and nobody really knew what the problem was or how to fix it for quite a while: https://roblox.fandom.com/wiki/2021_Roblox_outage

From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.

> Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes).

This is essentially why I’m now in the midst of a career change. I realized I want no part in what is more of a cargo cult than an engineering discipline.

I am curious, if not too personal, would you mind sharing what do you plan on working on?

I recently enrolled in a newly launched BSc program at my local university titled Sustainable Urban Development, blending administrative and social sciences with engineering in an attempt to tackle current and future issues related to urbanization, urban planning and sustainable development.

https://www.tuni.fi/en/study-with-us/technology-sustainable-...

Good luck!

Thanks :)

> From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.

I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done.

A team is left behind to operate that infrastructure, but they may not necessarily understand it to the depth required to react properly when it fails or when it needs updating, etc..

There's a 20+ year old book called The Ingenuity Gap that addresses this exact challenge with complex systems that we aren't equipped to do much more than "keep the lights on"..

It definitely feels like we are running up against similar situations in the software world, particularly with the huge amount of job churn (and corporate brain drain) that the pandemic produced.

You see this at all different levels of complexity.

I'm on the board of a very small non-profit and periodically I see someone come in with grand plans for updating the website and other infrastructure. And you pretty much know that, given free rein, they'll come up with something that will be way beyond what the person coming in after them will be in a position to maintain.

As a volunteer at a small non-profit attempting to keep a similar-sounding setup working long enough to replace it, this is the nightmare I am attempting to unwind. The problem when moving to off-the-shelf software for things is everyone wanting their particular favorite features. Said features were all custom-coded by someone without formal development training, of course.

> I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done.

Most companies do not need complex infrastructure either. It is just cargo culting ( or Google culting )

> and then leave once the interesting/challenging work is done.

Don't underestimate constructive dismissal.

The courts have begun to take a somewhat dim view of executives being completely ignorant of Too Good to be True scenarios, which has been a viable form of plausible deniability forever and a day. I'm not treating the workers badly, George is treating the workers badly. George, who I brag about all the time and have promoted three times in five years.

There's this sort of notion of 'what have you done for me lately?' that pushes us to underappreciate people who set up systems that usually run on autopilot. Bosses can't absorb the idea that just because nothing has broken doesn't mean it won't break, and these may be the easiest plates in the world to spin but they are in fact spinning, and so this person who "doesn't look busy" is doing so on purpose. It's what some clever person called "on-site on-call" to differentiate doing nothing from actively doing nothing. It's very reminiscent of the experience some people have buying someone a gift (a subscription, a car, etc) that has an ongoing cost and then they hear from the person how disappointed they are in the new/other gifts they got. "Last year they bought me a car and this year all I got was a new iPad." No, this year you got an iPad and your rich uncle is still paying for your car, you little shit.

Steve may be maintaining systems that save us 4 headcount, but he looks like a cost center to the business. He's lucky we don't lay him off, so don't talk to me about giving him a 10% raise for boosting that to 4.5 headcount.

"I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it.""

That day is today: https://confluence.status.atlassian.com/incidents/hf1xxft08n...

6 days and counting

Insanity at its best!

Atlassian annual revenue for 2021 was $2.089B, a 29.42% increase from 2020.

Atlassian annual revenue for 2020 was $1.614B, a 33.39% increase from 2019.

Atlassian annual revenue for 2019 was $1.21B, a 37.36% increase from 2018.

Nah, Atlassian has not given up, and isn’t going to. @nonrandomstring wasn’t referring to incidents that take more than a minute to resolve, they were predicting an incident of such magnitude that a large company shuts down over it. This is definitely not that.

I read it differently: the company will not shut down, but abandon attempts to resolve the issue (whether for a product or a set of customers) due to its complexity. Using major in GP's description was to highlight that the company has necessary resources and still can't figure out what's going on.

Fair enough, there is a valid more optimistic interpretation than mine. Still, this is not even the case you’re suggesting. Atlassian isn’t abandoning any features or customers here, this case isn’t indicative of anything other than a run of the mill outage that’s taking longer than a day to fix.

My Atlassian products have been fine for the last few days, and indeed last few years. I think there was a 1 hour upgrade window a couple of months ago.

But then we self-host.

Excellent comment and I can't speak directly for the author, but that's what I read too. Incidental Complexity is a term related to this. Incidental Complexity is any complexity that is not part of solving the core domain problem and has been introduced by the tooling or approach you chose instead.

The most common example I see is using enterprise tooling for small teams and projects, and then those projects getting snowed under by the weight of the hidden and incidental complexity they had built without noticing.

Essential vs incidental complexity is one of those concepts everybody who wants to call themselves a software engineer should be intimately familiar with. As they say, anyone can design a bridge, but it takes an engineer to just barely design a bridge.

Incidental Complexity pushes me towards serverless offerings. Of course, serverless brings with it its own test/deploy quagmire, but I'd argue for personal projects, it doesn't matter as much.

> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."

On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity, and they will have had hundreds of service outages. This is in stark contrast to the single-user free service we’re commenting on, where the owner was already treading water and perhaps just didn’t know it, and where the tradeoffs never had financial upside.

On the other hand, you’re not just right about your prediction, but it has already happened in the broader sense, just maybe not over a single incident. Complexity has already killed many software companies, when the software gets too big to manage and can’t be easily refactored, runs on old languages and tools that stop getting updates, relies on too many services that slowly go out of both fashion and support contracts. The list of companies that got too invested in a single codebase and stuck themselves with the inability to adapt quickly enough due to the weight and complexity of what they built is pretty long and growing.

> "On one hand a single incident is unlikely to be the end of a large company because companies exist to make money, and also because by the time a company gets to a thousand engineers, it’s engineering function is already centered on managing complexity."

Absolutely the companies themselves can be very robust. Some friends and colleagues who work for <big defence> tell me they don't fear for their jobs if a project is crashing. Internally such firms are structured to run dozens of projects, all well compartmentalised, such that if one fails they'll just get moved to another. The company itself is survivable via diversity (plus there's .gov money close at hand to bail out in an emergency).

What concerns me about civil resilience is where we've got unexamined mono-cultures like Google Docs or GitHub that become de-facto pillars for countless other businesses. When Trump ordered Adobe to cut-off Venezuela that should have been a resilience wake-up call for everybody (and it prefigures the present Russian situation of course).

There could be many reasons (more likely an unfortunate confluence of unrelated upstream reasons) that topple a vital part of public infrastructure. That's why big-tech monopoly, which erodes hybrid resilience, bothers the hell out of me.

True but there is also a difference in priority with different package maintainers which isn't always helpful.

Why were old features removed from the upstream packages? Probably because upstream either wanted to ditch old/insecure protocols or the usage was too low to warrant the investment in keeping it alive. That is completely reasonable for them.

Then look at something like SSLPing, it wants to support old protocols because that is part of the feature set for older servers which have no realistic need to update.

It's something I find a bit troubling about the very vocal "security" crowd for whom only the latest and greatest ciphers/protocols/short-lived ssl certs/stronger DH key pairs are acceptable and everything else is garbage.

Sadly, it seems that realistically, only paying/charging will help this because then it is reasonable for someone to spend 40 hours of their week keeping up with the changes/building their own libraries/etc. to keep their project running.

Yeah; I was going to write that this shutdown notice is a great microcosm for what's wrong with the field.

You focus on containers, but this project had dependencies fail at each level. The author also cites init breaking, the crypto library losing support for old protocols, node breaking compatibility, and I think some frontend junk I haven't heard of also fell out of support.

The project sounds like it could have been a well-scoped command line option to SSL or something, but, because it was built with current best practices, keeping the dependencies up to date and maintaining it would probably take multiple full time engineers / swe's at a FAANG.

On top of that, there's no sustainable business models for good ?aaS tools (small, or large), just for platforms and monopolistic stuff.

> because it was built with current best practices, keeping the dependencies up to date and maintaining it would probably take multiple full time engineers / swe's at a FAANG.

Surely significantly less than one full time? But like 10% of an extra full time job that costs you money isn't a very good proposition

Each dependency gets to choose when they break your service (they can have a zero day exploit that's only fixed in a compatibility breaking release, for example).

You need two people if you want one to be able to go on vacation while you maintain a 24 hour SLA for security patches. You need a third because this sort of thing generates employee burn out once it is feature complete.

You could try having it be 10% of one employee, but that employee would need to be a full stack expert from low level Linux stuff to JavaScript GUI junk. Good luck finding two people like that, then telling them they own ten feature complete services like this.

The people that fit such a bill and have the personaility type to be happy in such a job are expensive and rare!

> 24 hour SLA for security patches

I think that's a significant step up from your earlier guess at "maintaining" the software. There's a big spectrum between "project has been shut down" and "round the clock SLA".

"When we reach a situation where some unknown person pushes an upstream update, which causes a cascade of problems that even someone with deep technical knowledge and daily connection to the project cannot unpick, that's a breakdown in modularity/dependency."

Agreed. I believe that the "UNIX philosophy" is a good strategy for avoiding these scenarios and I believe it is underappreciated - even by architects and designers that should know better.

Forgive me for injecting this word but this is why I recoil so violently from systemd and all of its accoutrements. I can sense how this is going to fail even if I can't clearly elucidate it just yet ...

Nonsense. It was a side project that cost a lot of money. He had to take care of his actual job, family, hobbies, income producing side projects, etc. He had no single person on the planet to help him maintain it or update it. So the dependencies went out of date over the years.

Projects with resources can be maintained and updated. This one has no resources and there is no reason to update it because it's a drag on that person's life.

> we reach a situation where some unknown person

We reach an unnecessary situation. The vast majority of these dependencies don't even add any real value - not enough to justify the danger of having your app just stop working inexplicably.

https://status.atlassian.com/

Atlassian has had a major incident ongoing for about 6 days now, so we may be at that point sooner than you were hoping for.

Why do we need Docker? Just so that it’s easy to ship to server? How about rsync + taking the risk of dependencies conflicting?

Sorry not expert at all the infra stuff. But I hope we can question things.