Comment by KronisLV | Hacker News Reader

Comment by KronisLV | Hacker News Reader

Parent Full thread

> It's what we faced in the 1990s and from it evolved package managers, version control, dependency management and all the good things.

> I think in 2022 we're back in a similar situation again. Virtualisation, containers, cloud services and whatnot have regressed development reliability.

Realistically you probably want to find the middle ground between these: not copying PHP code through SFTP or even working on it directly, but also not running an enterprise Kubernetes cluster for your simple CRUD app.

Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes).

> I predict the day when a really major service provider comes out and says; "Sorry everyone. It's been 10 days downtime. We've put our 1000 top engineers on the problem and nobody knows what's wrong. We just can't fix it."

That sort of happened with Roblox, they had an outage that went on for 3 days due to Nomad rolling over and dying in their infra and nobody really knew what the problem was or how to fix it for quite a while: https://roblox.fandom.com/wiki/2021_Roblox_outage

From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.

> From here on out, we'll see more and more outages like this in large companies with complex infrastructures, at least that's my prediction.

I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done.

A team is left behind to operate that infrastructure, but they may not necessarily understand it to the depth required to react properly when it fails or when it needs updating, etc..

There's a 20+ year old book called The Ingenuity Gap that addresses this exact challenge with complex systems that we aren't equipped to do much more than "keep the lights on"..

It definitely feels like we are running up against similar situations in the software world, particularly with the huge amount of job churn (and corporate brain drain) that the pandemic produced.

You see this at all different levels of complexity.

I'm on the board of a very small non-profit and periodically I see someone come in with grand plans for updating the website and other infrastructure. And you pretty much know that, given free rein, they'll come up with something that will be way beyond what the person coming in after them will be in a position to maintain.

As a volunteer at a small non-profit attempting to keep a similar-sounding setup working long enough to replace it, this is the nightmare I am attempting to unwind. The problem when moving to off-the-shelf software for things is everyone wanting their particular favorite features. Said features were all custom-coded by someone without formal development training, of course.

> I think part of this problem is that people set up complex fancy infrastructure at a company, and then leave once the interesting/challenging work is done.

Most companies do not need complex infrastructure either. It is just cargo culting ( or Google culting )

> and then leave once the interesting/challenging work is done.

Don't underestimate constructive dismissal.

The courts have begun to take a somewhat dim view of executives being completely ignorant of Too Good to be True scenarios, which has been a viable form of plausible deniability forever and a day. I'm not treating the workers badly, George is treating the workers badly. George, who I brag about all the time and have promoted three times in five years.

There's this sort of notion of 'what have you done for me lately?' that pushes us to underappreciate people who set up systems that usually run on autopilot. Bosses can't absorb the idea that just because nothing has broken doesn't mean it won't break, and these may be the easiest plates in the world to spin but they are in fact spinning, and so this person who "doesn't look busy" is doing so on purpose. It's what some clever person called "on-site on-call" to differentiate doing nothing from actively doing nothing. It's very reminiscent of the experience some people have buying someone a gift (a subscription, a car, etc) that has an ongoing cost and then they hear from the person how disappointed they are in the new/other gifts they got. "Last year they bought me a car and this year all I got was a new iPad." No, this year you got an iPad and your rich uncle is still paying for your car, you little shit.

Steve may be maintaining systems that save us 4 headcount, but he looks like a cost center to the business. He's lucky we don't lay him off, so don't talk to me about giving him a 10% raise for boosting that to 4.5 headcount.

> Doing that can be pretty tough, though, especially with peer pressure to use hot tech so you aren't left behind in the industry due to stagnating skills (e.g. needing to learn GraphQL and gRPC when REST is still good enough, for example; same for containers and Kubernetes).

This is essentially why I’m now in the midst of a career change. I realized I want no part in what is more of a cargo cult than an engineering discipline.

I am curious, if not too personal, would you mind sharing what do you plan on working on?

I recently enrolled in a newly launched BSc program at my local university titled Sustainable Urban Development, blending administrative and social sciences with engineering in an attempt to tackle current and future issues related to urbanization, urban planning and sustainable development.

https://www.tuni.fi/en/study-with-us/technology-sustainable-...

Good luck!

Thanks :)

Reply on news.ycombinator.com