Hacker News Reader

Comment by j-pb | Hacker News Reader

I know that it's parody, but I think the whole sysadmin culture is incredibly toxic and glorifies toxicity.

I've never met more socially inept yet condescending people in computer science. The whole "loser", "not my job", "automating your job away thing" totally clashes with the real-world requirements that the profession entails.

Unless you work at google in a datacenter in the arctic you'll have to help people, and you'll have to do it over and over and over again, because frankly most sysadmins don't make the money, the people they support do. Yet, they look down on the people they run the systems for. To them they are just cogs in a machine doing non intellectual work. They see it as an insult that their "losers" won't learn and stop to appreciate and marvel at their beautiful systems, ignoring the fact that most ops positions at a non IT company are a lot less stressful, yet much better paid than all other jobs.

/rant

View on HN

Like much dark humour, it's inappropriate and often viewed in poor taste outside of those communities.

I'm not sure how else to describe BOFH except as revenge-fantasy porn. It's a vent/release for people who are often stuck between a rock and a hard place.

In most organisations, IT is seen a as a cost-centre, rather than a business enabler. So on one hand, you have management often wanting to cut costs to the bone - on the other, you have the very same people complaining that you're implementing certain policies and being a hard-ass about equipment/software/etc. Then when things go down and it's due to the business cutting costs - well, it's still IT's fault.

Then there's the often poorly thought through and inane requests, where people refuse to listen to reason.

And then you have a lot of people who have no awareness of anything outside of themselves - Their issues are always the most important, they shouldn't be required to learn basic IT skills, and anything that goes wrong it's IT's fault, even if they've been provided ample training and education opportunities.

I see similarly dark humour and in-jokes in other high-stress professions.

See also: medical slang

http://messybeast.com/dragonqueen/medical-acronyms.htm

>> In most organisations, IT is seen a as a cost-centre, rather than a business enabler. So on one hand, you have management often wanting to cut costs to the bone - on the other, you have the very same people complaining that you're implementing certain policies and being a hard-ass about equipment/software/etc. Then when things go down and it's due to the business cutting costs - well, it's still IT's fault.

IT leadership tends to be passive and disempowered. I'm not completely sure why this is, but the behaviours are often the opposite of what you'll often see from leadership in other departments.

What I have observed, and it may stem from the sysadmin mythos, is that you can work behind the scenes staying away from complicated interpersonal relationships, but you won't be successful. This leads to IT never really being embedded fully in projects and seeking out ownership of projects with other parts of the organisation. This transactional relationship puts IT in a bad spot and they fail to do things to improve the organisation.

This evolves into things like R&D in pharma companies having a parallel IT department because the main IT department isn't trusted. The folks on the ground in regular IT would be interested in helping the scientists and delivering tools and services for them, but IT leadership didn't really take much interest in the needs and differences from the mainstream user population (sales, marketing, etc.) so trust broke down.

I once got to participate in a leadership program at a global org. They selected up and coming peers from other departments. My observation was that the folks from the IT would not be the ones I'd choose for key leadership positions, but I was glad to see they were getting to see where other folks were in their development.

So, the dark humour of BOFH is a release for individuals, but it's also an indictment of a culture which IT won't easily escape from.

> I'm not completely sure why this is, but the behaviours are often the opposite of what you'll often see from leadership in other departments.

I think part of the problem is that the vast majority of people who are in CTO/CIO type roles have zero IT knowledge. In the case where it's someone who "came up from the trenches", they were often in a hands-on role for only a relatively short period of time, a long time ago, and were never particularly skilled.

In both cases, they tend to trust their own intuition/gut instinct over that of the advice they get from their team.

Getting good managers who trust their team to give them good advice, and are able to fight (and win) for those things to make things better are few and far between.

Then again, having a good team to begin with can be a problem, too - there's a lot of people in the industry who really simply shouldn't be, and make a lot of poor decisions that just cause pain for others.

So what if IT folks are not the ones you would choose for “key leadership positions”? Maybe they understand something that you, and most other engineers don’t. At half the places I worked at as a sysadm or IT engineer I had access to pay bands, vendor contracts, etc. I was never impressed.

All that I saw was a lot of people fighting very hard over a very marginal and pathetic amount of money, equity, and status, while we wrote yet another multi-million dollar check to some shitty consulting firm in Eastern Europe.

You may think that we’re out of the game, but the truth is we understand it in a way you could hardly imagine.

I never met a sysadmin that was anything like the bofh.

Look, it is a funny story, nothing more.

Just like there is dilbert and other stuff.

Do I want to become a bofh? NO

Do I like a complete Fantasy story in which a sysadmin runs amok?

Yes, yes I do.

I would never compare that to reality, just as I wouldn't expect that lord of the rings is a accurate history documentation and that star wars a prophecy.

To your other points:

I rarely met toxic people and when I did it was one of the following reasons:

* As soon as a system was down(Of course, no money for HA or preemptive maintenance) users called and screamed(Yes screamed)

at the it people till one of them had enough and screamed back.

* The Sysadmin was responsible for all the systems 24/7 but got paid 40 hours.

* The sysadmin had to repeateadly explain to a user how to change his or her password and how to login to computer.

* Sysadmin got the blame for not backing up a system becaus some enduser deployed in on their smuggled in nas

You will see that most people that become toxic, suprise suprise, are in a toxic environment.

Just to iterate over your last point:

The Job of a sysadmin is to build and deply systems, keep them running, document and plan new things.

It is NOT the job of a sysadmin to help people learn how to use software, which they should know about because they knew they had to use it, when they applied for the job. That IS helpdesk work.

E.g. "The software writes I can't connect to the server" is a valid question.

"I don't know how to make a paragraph bold in word" is not valid.

Because that would mean that you just hired a person who has no idea on how do to his job. How to you expect a sysadmin to know every programm inside out?

Heck I know how to deploy photoshop, butI have no clue how do to anything in it and I frankly don't care.

> The Job of a sysadmin is to build and deply systems, keep them running, document and plan new things. It is NOT the job of a sysadmin to help people learn how to use software, which they should know about because they knew they had to use it, when they applied for the job. That IS helpdesk work.

Unless your company is too small to have a helpdesk, then that's precisely what they're often hired to do, hate doing it, and are shitty and toxic about it.

Long time sysadmin here:

I'll happily admit that sometimes, especially once, becoming everyones favorite sysadmin took way less effort than it should take.

That said: either I've been extremely lucky to work with smart and nice sysadmins or you have been extremely unlucky or you are posting hyperbole.

Edit: In reality it also goes both ways: I had one particular user going to my boss because "we sysadmins were totally unreasonable" or something. So my boss told me (since I was both the group leader for user support and also the defacto user advocate) to do whatever it took to make him happy. As far as I understood from the IT manager it was some program that wasn't in our official repository that he needed or something.

Turns out it was a "music program". This left me wondering since that particular user was in management. Then he gives me the name of said "music program" and it is limewire and the reason he need it is for his kids.

At that point I became "unreasonable" towards him as well ;-)

I agree. I mean I’ve got my war stories from when I was a sysadm, and I laugh about them, but at the end of the day I knew that computers were (are) really arcane, and the people I was hired to support had jobs to do that didn’t include the words “computer science” (but did include words like “histopathologist”)

BOFH to me was always the epitome of those few people who liked to lord it over others who they considered to be less smart than them - “lusers”. I knew a few admins and programmers like this and they weren’t ever half as smart as they thought they were.

BOFH always felt to me like a celebration of stupidity, which is never something I find entertaining, and I was never quite convinced that it was the victims who were stupid.

I have had my own share of problems with admins and corporate IT. But in most companies it really is a thankless job where you can’t win. When things work you have to justify why you are sitting around while things “just” work anyway. When things don’t work you get yelled at by some exec, probably the same who previously denied budget requests for better equipment. Add to that the fact that some users are just idiots and most companies see you as pure cost that best gets offshore. In my view definitely a job to be avoided.

"you'll have to help people, and you'll have to do it over and over and over again"

We do, again and again, and from some people we get nothing but condescension in return, and treated like unskilled janitors who only ever stand in the way.

If I had a penny for every stuck up sales/marketing person or exec who though everyone in IT was lower than pond scum...

This condescension even comes from developers, who you'd think would have some sympathy for other tech roles, but the attitude is often that devs are gods compared to the lowly ops guys.

"most sysadmins don't make the money, the people they support do"

This attitude is part of the problem. You don't appreciate that it's a team effort.

It got a lot worse since 'devops' came around, in my experience. As an example: people started getting increasingly pissed off when proprietary software (unpackaged, but usually with a shell installation script) couldn't get installed on a network wide setup of ~200 nodes fast enough to their liking. Since at home they could do that in 15m following the vendor's instructions (which involved running the installer script as root).

It fell to deaf ears that a packaged approach was the right way to go, given said software installed its own jungle all around the filesystem hierarchy, numerous times conflicting with software from other vendors with the same kind of recklessness.

Quite clear that any of these people lacked the understanding that having a longterm stable environment where everyone uses the same version of everything, is uncomparable with doing unsafe installations following the first forum-post you find online.

Devops learned them that cutting corners to empower yourself is OK, they could run their own systems quicker. Needless to say all the debugging (the search-engine based kind) that needed to be done after the facts was suddenly part of their job description, and filled up most of their time...

I recently got called in to support one of my folks because some devops team was absolutely incensed they couldn't just pull in some rando docker container they'd found on the net. From a .cn domain. That failed the static analysis suite. For a production financial application. Fail fast indeed.

I get we need to adapt to support devops. I'm investing big resources to support that. It would be so much easier if so many devs didn't think devops == no oversight.

So what actually prevented them from just doing this?

The dev folks have an 'approved repository' workflow; when this team tried to deploy to QA the rouge container got flagged. Plus it failed the static analysis step in the pipeline. I assume a sufficiently clever dev could find a way around this, but it worked here. Maybe we just got lucky this time, but we did an asset audit when we found this one and didn't find anything that had slipped through.

We're looking at deploying some tools (e.g. Dome9) that we think will close some of the loopholes and provide more automation. However, due to the complexity of these environments, I'm not sure there's a perfect solution.

For some reason it become socially acceptable to bash folks who do system administration for a living.

Seems that you are really frustrated because you had a bad experience while working with a sysadmin and I don't think it is fair to despise people who do system administration just because of that.

This is toxic, and is _not_ ok.

Sysadmins are busy people, with their own reponsibilities, tasks, etc. They are always willing to help, assuming that you know how to communicate nicelly, tell about what you want to achieve, being a humble, honest human being.

Assuming that the sysadmin profession entails helping people all the time is plain wrong. Don't abuse.

Though, even if you ask for help several times about a specific subject is very likelly you will receive useful links, docs etc to get you "unblocked".

You still have to read and try to help yourself with the new piece of knowledge that you received.

_Demanding_ to help all the time is _not_ ok.

Peace!

Absolutely!

Back in college, I used to laugh along with BOFH style comedy. I still do find it funny, but now that I've been on the receiving end of this brand of toxic behavior in my own life, it's all a little too real for me.

Interestingly, at my current job (research assistant at a university) I've found an almost identical culture among lab support staff. I'd chock it up to the same factors of long hours, a thankless job, and the occasional user that is in fact an idiot. It's also a field that tends to accumulate people without a lot of social awareness, but where the job involves a lot of human interaction. They're also likely the ones getting blamed for problems they don't have a ton of power to solve.

I understand why they feel the way they do, but I wish we could do without all of the abusive behavior. In the end it's real people's careers that they're taking their anger out on.

I've had research that my career depends on delayed for at least a sum total of months at this point (in only ~2.5 years of time), because of situations like this from both toxic IT and research staff dragging their feet and being hostile to work.

I find a lot of undesirable behaviour and condesention between groups is based on insecurity. And not just between engineers and other engineers.

I once worked somewhere with kind of old-school sysadmin guys. I had to spend so much time asking and badgering one to add disk space/allocate me more disk space on a computing cluster to load a model. The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk. In a cloud environment I could do this myself instantly or not even have to do it at all.

Especially as the cloud becomes more popular, and devops/modern tooling begins to be adopted in house even for companies that aren’t “on the cloud”, I see the role of sysadmins who aren’t also software engineers going away.

What was the iops you eventually got or needed?

Since you mentioned model and cluster I’m curious as to wether or not others were also on said cluster.

The way this computing cluster worked, each person had some default disk/memory space they were allocated out of a large total. I don’t know exactly how it was set up. People were running things like Hadoop jobs, Spark, Jupyter notebooks. I think the default disk space allowed was very small, like 5GB, and I had some large NLP model that was 8GB (probably the entire library and the part I needed was smaller).

I needed to save the NLP library to my user account’s disk so I would only have to download it once. I think I could redownload it every job, but that would take a pretty long time, and had other issues.

So it wasn’t even IOPS, it was just the disk space I was allocated on my user account. Probably a one line command by the sysadmin guy, unless he also needed to go get another hard drive and plug it in, in which case it would be 10 minutes and then a one line command.

"The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk."

So what do you do when that disk dies and you lose all your data?

Oh, you want backups? Well, maybe you could do them yourself.

Oh, you don't want to do the backups yourself but want the sysadmins to do your backups for you, do you?

Or what happens when the network that's hosting the disk becomes inaccessible? You want to get to your data anyway, so that means adding network redundancy.

And you also probably don't want downtime if the server hosting the disk goes down, or the server is up but the disk dies, so that means RAID or maybe a distributed filesystem.

And what about making sure the data is secure and the servers hosting it meet compliance requirements?

And when something goes wrong you want someone in ops to troubleshoot it, right? Maybe you want six 9's of availability, and you want us to wear a pager so we can work on fixing it at any time of the day or night, right?

All of this will take planning and time and more hardware, configuration and monitoring, perhaps a bigger headcount since sysadmin teams are often already running short-staffed and don't have the time to dedicate to supporting even more infrastructure than they already do.

You see, it's not always so simple as "just add another disk".

Or I just use the cloud and configure all that myself without an arbitrary gatekeeper

I didn’t even care about any of that stuff. I just needed to be able to work on my own model using some other model that I could always redownload if there was an issue. If I had been empowered to just do it all myself it would have been a 0-10 minute task.

> I know that it's parody, but I think the whole sysadmin culture is incredibly toxic and glorifies toxicity.

It has been quite a long time since the days of grumpy greybeards ruled the roost.

Sysadmins, much like developers, are labour force multipliers. One big difference is, sysadmins and IT are often treated by businesses as an absolute minimal-possible-cost departments. They're frequently underfunded, short staffed, and rarely have any management buy in for hardware or software necessary to do the job reliably. That leaves sysadmins scrambling around trying to pull one miracle after another after another out of their butts, making hardware and software perform to a degree that it just isn't capable of doing. When you push software and hardware beyond its limits, that means lots of emergencies, lots of firefighting. It's really not rare to have to argue for having any kind of redundancy in hardware or software ("we can't justify the expense of a warm spare, or even a cold spare", "But this is the mission critical software our entire business relies on. Every second it's down is tens of thousands of dollars down the drain", "yeah, but we just don't have the budget for it, we need that for the CEO's golf trips".)

Throw on top of that business groups that keep buying software and hardware without engaging the sysadmins to evaluate suitability; software that often involves working further miracles to make it work for the company because fuck me so much of that shit is written by people that have no clue about a safe operating environment, but hey the CEO's nephew produced it and we've already paid them hundreds of thousands of dollars for it that he's already spent on a sports car and we can't get a refund (I truly wish I was exaggerating there, that's a for real situation a friend found themselves in).

This is the continual story of sysadmin life. It's one reason why I eventually moved on to systems engineering / cloud platforms ops. It's just sheer constant stress courtesy of management that just don't give a shit while things are working, and then lay a crap load of blame on you when things fail.

Want to make any guesses now why sysadmins are frequently grumpy? It's not because they set out to be, or want to be. Very few do (most of those obstructionist greybeards are long gone). It's because they literally don't have time or energy left after being continually burned out from performing one miracle after another. For what it's worth, I see the exact same kinds of grumpiness from developers and other staff in businesses that don't give a shit about how much they're burning out their staff. It just seems like it's more visible in sysadmins because they end up having to deal with a far wider range of staff across a company.

A well funded IT stack makes it as friction-less as possible for employees in a company to get their job done. Technical problems employees have get dealt with quickly so they can be back up and working at peak productivity, because there will be sufficient helpdesk employees with sufficient training to keep the quality bar high (trying to get funding for training is another nightmare, especially for helpdesk). Platforms and technology they rely on are reliable. Mistakes made by employees are easily recovered from (files restored from backups, ransomware is isolated and easily remedied). The list goes on and on.

> When you push software and hardware beyond its limits, that means lots of emergencies, lots of firefighting. It's really not rare to have to argue for having any kind of redundancy in hardware or software ("we can't justify the expense of a warm spare, or even a cold spare", "But this is the mission critical software our entire business relies on. Every second it's down is tens of thousands of dollars down the drain", "yeah, but we just don't have the budget for it, we need that for the CEO's golf trips".)

> This is the continual story of sysadmin life. .... It's just sheer constant stress courtesy of management that just don't give a shit while things are working, and then lay a crap load of blame on you when things fail.

yes, this, a thousand times; my job security lies in my shit eating grin and capacity to ignore exactly this structural idiocy.

Isn't this why those jobs also have huge churn? The people who can get out, do. Leaving only those without the ability to find another job.

I know as a developer I have churned quickly when it turned out that "occasional overtime" meant constantly babysitting systems that were never finished, just patched together enough to run for a day without obvious disasters. That was the explicit metric in one place, they literally would not address a bug unless it caused the system to fail before the daily restart. "daily" meaning some developer had to wake up at 2am and remote in to manually restart a bunch of things in exactly the right order, then debug and fix any newly revealed problems (because this was also when new versions were started in production for the first time).

My record so far is less than an hour in a new job - I turned up for my first day and one boss was berating the other boss over the phone. As soon as they saw me they handed me the phone and ran out of the room. I got an earful before I managed to say "Hi, I'm psychlist and I was supposed to start working here today". I did actually let them talk me through fixing the immediate problem, then I hung up and walked out.

Don't work in those places. Just... don't.

> frankly most sysadmins don't make the money, the people they support do

> the fact that most ops positions at a non IT company are a lot less stressful, yet much better paid than all other jobs.

Which is it?

Make money for the company vs. make money for themselves.

Reply on news.ycombinator.com