Regardless, I hope you enjoy your retirement when it comes. You have earned it.
It used to be that you can't get people fresh out of school without basic network knowledge, and over the past decade or so, you can. Heck, it's more likely that new candidates don't know how things work, because they have been trained to pass hiring barriers instead of actually having to know how things work.
This makes sense in other areas, especially those that have existed for a long time (i.e. accounting, metal work) and aren't layered in abstractions covering 10 different disciplines. But IT is relatively young, yet still works on the same foundations it always did. Even things like ethernet that's used by mainframes, minicomputers, microcomputers but also clouds, IoT and workforce, is simply ignored because it's supported to magically work and abstracted away by 10 sys calls, 100 node packages and 20 layers of XHR promises. When it breaks, you suddenly don't know how to fix it, and because you also don't know what those 100 node packages are doing (which would be unrealistic anyway) but also don't even know what a syscall is, you will neither know how to find out if that is the problem, and even if you did, you won't know how to fix it.
Some basic understanding of the ecosystem in which IT exists wouldn't be too much to ask, but because of the high demand we're just accepting overspecialised people for jobs that don't exist (no role in a company is "react-redux integration specialist"), because it used to work out in the past (but in the past you had to have the same foundational knowledge).
This is the equivalent of throwing spaghetti at the wall and seeing what sticks, because it's cheaper to hire 10 new people and hoping one of them lucks out and accidentally fixes something than hiring or training one person that can actually consistently diagnose and resolve issues.
And before someone assumes this means some people are "bad" because they don't know "everything" (not talking about knowing everything): this is mainly a disconnect between what is needed and what is applied for. And that in turn is because it's hard to describe what is needed, and in return is hard to prevent how as a candidate you would fulfil that need. (i.e. job postings with 15 years of senior Kubernetes experience come to mind)
If you know a bit about tasks, queues, control loops, you can pretty much fill a job for legacy systems like servers running bare metal linux or windows, but also containers, container orchestrators and orchestration tooling for infrastructure as code, because at the end of the day they all are implementations of the same principles, but slightly specialised for the problem at hand. They all do the same thing: you specify what you want, and it is up to the system to make it real and report how that's going. Reading the output of invoke-rc.d, service, systemctl, kubectl, docker and nerdctl is specialised on resource names, but identical in terms of action flow. They all have a concept of a resource, a concept of something that needs to happen and a concept of telling you how it went/how it's going. But teaching people that you need to understand the principles and apply them on different implementations and abstraction levels is hard, so we just teach people the contents of a MAN page instead of actually telling them that the man command exists and you can just read it whenever you like...
Most of the feedback loop is if you can get jobs of ever higher compensation. If that loop stops considering old school sysadmin foundation skills, then people will stop learning those skills because they can get jobs without them.
There's been lots of progress on making things convenient for developers/IT, so that's also why: you don't need those old skills to use those, or at least your entire team doesn't.
If you write in Go, Rust, Java, C#.NET, Python or TypeScript, all of those foundations stand and are practically unchanged. Some layers might have been added, i.e. when you describe how your service is supposed to run in a Docker manifest, or in a Nomad configuration, or just a systemd unit file. But those are just formatting issues with plenty of manuals to go around. Yet still, when something goes wrong (i.e. application exits with some status integer), an engineer used to be able to understand that:
- Your application or its management system detected a problem
- The OS provided an indication that your application has an issue
- If you don't know all codes, you know that they are perhaps called "exit code" or "return code", and you at least know how to look those up
Say it turns out that your application is throwing a SEGFAULT, as an engineer, aren't you supposed to know that a segmentation fault happened, and you can suspect some shared library to be the first thing to check before going deep with some debugger? Maybe run a command to find out what libraries are needed, check if they are accessible and contained within the context of your application (i.e. if not in a complete OS, but a container or jail or chroot).While in theory only a part of an entire team might need to know this, having mostly teams where 0% of the team members know this might be a problem, don't you agree?
Same goes for "I am calling this HTTP API but it doesn't work", that's just a lack of information to begin with. This question is asked plenty of times, and usually the first thing we need to know is "what is the error" because for some reason the developer doesn't understand that it could be a myriad of things and asking for help because "the thing had a booboo" doesn't mean much. Then onwards to:
- Is the URL valid
- Is the FQDN valid
- Are you even using an FQDN or did you think everything in the world runs on your DHCP-supplied search domain (common in windows land)
- Can you resolve the name via DNS
- Can you connect to the IP on the port (commence lesson about netcat and similar tools, every single time, again and again)
- Did you ensure the security groups or legacy firewalls are actually configured for your desired traffic
- Is the service on the other end actually up and does it even exist
None of this is deep systems understanding for greybeards, it's just basic client-server model programming. But still, there are more team members that don't know how to verify connectivity than teams that have at least 1 who does.Every time someone needs to reach out to that special someone they think magically knows everything in the world, that's a waste of time for two people, and a break from concentration. Advanced issues like "my packet payload get truncated and I don't know why, they are just TLS and below the MTU size", sure, can't expect everyone to understand how to check that stuff out. But we're talking about the basics here.
We're running low on greybeards and we're not getting new ones because "they don't need to know" (developers, ops), and this is going to be a problem that gets bigger, not smaller.
For these people needless complexity was a benefit. I saw complex neural networks poorly built for the most laughably simple problem, but arguing that it could be solved more simply got you a talking to, and anyone who actually did solve the problem was quickly let go.
Part of the consequences of being in a tech bubble that has lasted so long is that reality stops being important, so technical skills don't matter and at the extreme threaten more important illusions.
In a company where you're expected to make more money than you spend, simplicity and technical competency can be hugely valuable. In a world were 'profit' is a foreign concept, trying to save time and resources is seen as a threat to the complexity theater other people are creating to boost the importance of their work/team.
Don't get me wrong, there was plenty of bs in the pre-bubble world, but there was a time when technical teams had greybeards you would consult and learn from. People like this are one of the reasons I started a career in tech to begin with.
Humans are a diverse population. Apathy is one response to facing overwhelming pressure of legacy systems, monolithic organizations. Perseverance is another.
Look at George Hotz and https://comma.ai/ - I am skeptical he and his organization would "no one gives a fuck" about broken shit.
- https://www.youtube.com/watch?v=FOuCKQfybz8 personality snippet