The Bastard Operator From Hell (1999) | Hacker News Reader

The Bastard Operator From Hell (1999)

bofh.bjash.com

196 pointsiudqnolq94 comments

There's a typical amount of seething involved in any field where someone has a minor input or touch-point with your job and they can do something you absolutely cannot or would not do.

Typically it's regular folks against the office IT people, but for the Hacker News crowd the bogeymen have to be in either Sales or Infrastructure.

Nowadays it's not as fashionable to bash the sales people, because the famous tech entrepreneurs and VCs have beaten into our heads how important sales folks are and you don't piss off the income stream. So instead "we enlightened few" take it out on the other one. They're a cost center anyway, right?

And as much as most of us think we can productionize some distributed data service serving mega-high volume traffic, we either just can't, can't afford to spend the time to, or would rather not. And we definitely don't want to be paged about it at 3 in the morning on a Sunday either.thank.you.very.much.

I can say this confidently because I work on both sides of this fence.

So rather than shitting on one of the only "shared jokes" that the people doing that thankless sysadmin job get to enjoy, maybe have a little empathy towards them instead.

> There's a typical amount of seething involved in any field where someone has a minor input or touch-point with your job and they can do something you absolutely cannot or would not do.

I'm not sure I buy into this theory. I think most people have no idea what goes on behind the scenes, and so ascribe slowdowns, missed SLA, or other failing metrics to amorphous "others" rather than individuals. Under such a system they certainly don't work to identify frictions along touch-points.

just read all of the sysadmin-hate in this thread.

> I think most people have no idea what goes on behind the scenes,

I agree with this.

> and so ascribe slowdowns, missed SLA, or other failing metrics to amorphous "others" rather than individuals.

The amorphous "other" is generally "IT" when it comes to anything computer related.

IT are the 'big evil' and the cause of most people being unable to do work, if you follow the common narrative.

For example, it's IT's fault that this machine is so slow...Not finance's for restricting budgets and using the cheapest RAM/drive configuration they can get away with.

It's IT's fault that I can't install this super important Outlook/Chrome/whatever extension that I saw on a blog somewhere. It's also IT's fault if Outlook/Chrome/whatever crashes or is slow because I've installed all these extensions.

It's IT's fault that I can't just use wordpress and put custom themes and unvetted PHP code on our main domain so I can do some promotion. It's also IT's fault if that Wordpress install gets hijacked and is used to phish/whatever customers/staff/etc.

It's IT's fault that I can't send 500MB attachments and cc 250 people at other companies. It's also IT's fault that the network (and my computer) is so slow when I'm trying to open an email.

It's IT's fault that the printers are locked down to only print in B&W, even though they have that colour option - and not because finance is sick of paying for me to print 500 full page colour birthday/soccer/wedding invitations every other month.

It's IT's fault that I got phished, even though I ignored the IT security warnings, and slept/skipped through the mandatory online training. It's also IT's fault for letting phishing emails come through, even though it was detected and put in my spam folder with a header clearly saying that it was a suspicious email.

It's IT's fault that I can't use an unencrypted USB drive to move files between computers, even thouh it's because we have had too many colleagues leave USB drives full of customer data in taxis or lost in the mail.

Don't you understand? IT just makes everyone's life more difficult.

I'm playing this up a bit, but these things happen, and people do blame IT for these things.

Thank you for explaining it in better terms than I could. I was just trying to point out that developers are also super susceptible to this kind of thinking.

The BOFH is largely a funny and positive character. In the end the character is pro-sysadmin, not anti-sysadmin.

As a physicist I also would not be insulted by the characters in the "Big Bang Theory", but strangely enough some are.

I guess it comes down to having a sense of humor.

Hmm, not sure I agree, the jokes tend to be at the expense of characters with social development issues but even if you put that aside - watch it without a laugh track, it just isn't funny.

As an aside I've never thought Friends was funny nor deserved the accolades it got.

I wouldn't say I lack a sense of humour either, I'm not overly offended by either of these shows, I just find them genuinely unamusing.

> So rather than shitting on one of the only "shared jokes" that the people doing that thankless sysadmin job get to enjoy, maybe have a little empathy towards them instead.

That's the worst BOFH-comments made by a layer 8 problem I have ever read. We sysops don't need pity, we demand absolute obedience by our l(ocal)users or we'll get all AIX on you! ;)

Maybe I'm in the minority here but I don't think this is clever at all. BOFH isn't some Dr. House character who's insufferable personality is the price you pay for genius. BOFH isn't doing some sort of vigilantism, correcting injustices in the world. He acts like all the people he screws deserve it for being dumb, but the "dumb" decision being punished is not expecting to be sabotaged for no reason. That's not dumb of them, that's a necessary assumption to make civilization work. He phrases these stories as if he's some clever trickster that hasn't found a worthy opponent, but where's the cleverness in "HAHA deleted your thesis using my admin priv"? I can't even respect that in the technically talented hacker sense. He didn't hack or do anything intelligent as part of his cruelness. He just abused his normal work credentials. Don't let the tone fool you, as far as sophistication goes this stuff is on the level of a shirtless parking lot fist fight. This isn't even funny in the harmless prank sense because it flat out isn't harmless. BOFH is nothing more than an asshole who is actively counterproductive at his job. Why does anyone here aspire or relate to BOFH? Why are so many finding this to be a funny and agreeable part of "hacker culture".

People enjoy watching a fictitious, narcissistic psychopath from afar. I think you're reading too much into it.

It reminds me of Always Sunny in Philadelphia.

See I like Always Sunny but the joke in Always Sunny isn't about narcissistic psychopaths from afar. There's many layers of joke in that show. Sometimes the joke is that the apparent voice of reason has just as poor reasoning as the rest and they only got it right by coincidence. Sometimes the joke is that every time you think there's a genuine moment the rug gets pulled out and its revealed to be a scam or setup. Sometimes the joke is in the way they don't react to any tragedy going on around them. Almost always the joke is to subvert the tropes of the "20-somethings urbanite sitcom" genre (cheers/friends/HIMYM).

Sounds like Seinfeld, which I hate.

I enjoy Watching House, but wouldn't want him as my doctor and would probably fire him if he was in my staff. Same with BOFH.

I’d be ok with House as my Dr I think, not so much as my colleague or staff.

He's a parody, not a role model. People like watching Tom and Jerry but that doesn't mean they think Jerry is some kind of heroic character.

This is like Tom and Jerry if instead of constantly matching wits and ending in stalemate, Tom put a plain old mouse trap in front of the hole and killed Jerry within the first 5 seconds of the show and then Tom patted himself before the episode abruptly ended.

You're misunderstanding the power dynamic. The BOFH is the mouse.

You are describing the derranged Tom amd Jerry boys in the 90s wished existed.

But Itchy and Scratchy does exist!

This is a psychopath fantasy. It's the IT version of You (the novel / TV show), Dexter etc.

Nobody wants to be a psychopath unless they are, but there's a certain feeling of liberty that comes with imagining doing things without caring about the consequences.

There's only one joke here, which is repeated over and over and over: it's the callous carelessness by which the protagonist breaks every single social rule imaginable. Nearly every single sentence contains this joke, and there are no other jokes.

It's like Dave Chappelle's "so I kicked her in the pussy" skit, but without the buildup. And ofc way less well done because this isn't Dave Chappelle.

I don't know how often I can watch someone godmode noclip allammo pwn the opposing team and still find it novel, entertaining or engaging.

He just wants to fuck with people. It'd be a different story if he had a different self-serving material goal every time and used surprising, morally despicable means to attain it, that still caught you off guard after the 10th episode.

This is what BOFH really is. It's that sick revenge fantasy you have where you brutally beat that reckless driver asshole who almost hit you while playing indy 500 on the highway, before your morning coffee, after running him off the road. In reality you sure as hell aren't running people off the road and beating them to within an inch of death with a tire iron. It's just a fantasy to release anger or deal with it.

BOFH is the psycho fantasy you have after dealing with the same idiot who yet again forgot their password and locked themselves out. Or that snippy idiot in the office who keeps insisting there is something wrong with their computer and its not as fast as the one they have at home or the "better" computer the person in the adjacent cubicle who also coincidentally can't stand them. But hey, it's still your problem to deal with. So screw em, why not delete their shittly work or their account for that matter? That'll show em.

I thought that was the joke.

You can laugh at a character without respecting or agreeing with the character. I personally find the absurdity of the stories funny (maybe because it caught me off guard).

The name of the character is "Bastard Operator from Hell." He is supposed to be terrible. He's like Cartman.

On the other hand, most of the other characters are also terrible, or incompetent or attempting to be abusive, and that echoes with the experience of many IT people dealing with users. Seeing those fuckers get even a fictional comeuppance is somewhat cathartic.

Bofh is a little further out than, say, thedailywtf, but largely in the same vein. There are so many shitshows and people "faking it until they make it" horror stories floating around.

> BOFH is nothing more than an asshole who is actively counterproductive at his job.

I think you're right, but if, as a sysadmin, I chuckle at these stories and it helps me remember why I should make an effort for my high-maintenance users (instead of deleting their $HOME), is that ok?

Something along similar lines that I really enjoyed was the "Website is Down" series of videos, starting with "Sales Guy vs Web Dude":

https://www.youtube.com/watch?v=uRGljemfwUE

Wow, what a flashback. I had forgotten about this classic.

just like alt.sysadmin.recovery

Big fan of BOFH but I’d never heard this, very funny, thanks for sharing

I know it's supposed to be a parody, but it hits very close to home.

thank you for this masterpiece.

I know that it's parody, but I think the whole sysadmin culture is incredibly toxic and glorifies toxicity.

I've never met more socially inept yet condescending people in computer science. The whole "loser", "not my job", "automating your job away thing" totally clashes with the real-world requirements that the profession entails.

Unless you work at google in a datacenter in the arctic you'll have to help people, and you'll have to do it over and over and over again, because frankly most sysadmins don't make the money, the people they support do. Yet, they look down on the people they run the systems for. To them they are just cogs in a machine doing non intellectual work. They see it as an insult that their "losers" won't learn and stop to appreciate and marvel at their beautiful systems, ignoring the fact that most ops positions at a non IT company are a lot less stressful, yet much better paid than all other jobs.

/rant

I agree. I mean I’ve got my war stories from when I was a sysadm, and I laugh about them, but at the end of the day I knew that computers were (are) really arcane, and the people I was hired to support had jobs to do that didn’t include the words “computer science” (but did include words like “histopathologist”)

BOFH to me was always the epitome of those few people who liked to lord it over others who they considered to be less smart than them - “lusers”. I knew a few admins and programmers like this and they weren’t ever half as smart as they thought they were.

BOFH always felt to me like a celebration of stupidity, which is never something I find entertaining, and I was never quite convinced that it was the victims who were stupid.

Like much dark humour, it's inappropriate and often viewed in poor taste outside of those communities.

I'm not sure how else to describe BOFH except as revenge-fantasy porn. It's a vent/release for people who are often stuck between a rock and a hard place.

In most organisations, IT is seen a as a cost-centre, rather than a business enabler. So on one hand, you have management often wanting to cut costs to the bone - on the other, you have the very same people complaining that you're implementing certain policies and being a hard-ass about equipment/software/etc. Then when things go down and it's due to the business cutting costs - well, it's still IT's fault.

Then there's the often poorly thought through and inane requests, where people refuse to listen to reason.

And then you have a lot of people who have no awareness of anything outside of themselves - Their issues are always the most important, they shouldn't be required to learn basic IT skills, and anything that goes wrong it's IT's fault, even if they've been provided ample training and education opportunities.

I see similarly dark humour and in-jokes in other high-stress professions.

See also: medical slang

http://messybeast.com/dragonqueen/medical-acronyms.htm

>> In most organisations, IT is seen a as a cost-centre, rather than a business enabler. So on one hand, you have management often wanting to cut costs to the bone - on the other, you have the very same people complaining that you're implementing certain policies and being a hard-ass about equipment/software/etc. Then when things go down and it's due to the business cutting costs - well, it's still IT's fault.

IT leadership tends to be passive and disempowered. I'm not completely sure why this is, but the behaviours are often the opposite of what you'll often see from leadership in other departments.

What I have observed, and it may stem from the sysadmin mythos, is that you can work behind the scenes staying away from complicated interpersonal relationships, but you won't be successful. This leads to IT never really being embedded fully in projects and seeking out ownership of projects with other parts of the organisation. This transactional relationship puts IT in a bad spot and they fail to do things to improve the organisation.

This evolves into things like R&D in pharma companies having a parallel IT department because the main IT department isn't trusted. The folks on the ground in regular IT would be interested in helping the scientists and delivering tools and services for them, but IT leadership didn't really take much interest in the needs and differences from the mainstream user population (sales, marketing, etc.) so trust broke down.

I once got to participate in a leadership program at a global org. They selected up and coming peers from other departments. My observation was that the folks from the IT would not be the ones I'd choose for key leadership positions, but I was glad to see they were getting to see where other folks were in their development.

So, the dark humour of BOFH is a release for individuals, but it's also an indictment of a culture which IT won't easily escape from.

> I'm not completely sure why this is, but the behaviours are often the opposite of what you'll often see from leadership in other departments.

I think part of the problem is that the vast majority of people who are in CTO/CIO type roles have zero IT knowledge. In the case where it's someone who "came up from the trenches", they were often in a hands-on role for only a relatively short period of time, a long time ago, and were never particularly skilled.

In both cases, they tend to trust their own intuition/gut instinct over that of the advice they get from their team.

Getting good managers who trust their team to give them good advice, and are able to fight (and win) for those things to make things better are few and far between.

Then again, having a good team to begin with can be a problem, too - there's a lot of people in the industry who really simply shouldn't be, and make a lot of poor decisions that just cause pain for others.

So what if IT folks are not the ones you would choose for “key leadership positions”? Maybe they understand something that you, and most other engineers don’t. At half the places I worked at as a sysadm or IT engineer I had access to pay bands, vendor contracts, etc. I was never impressed.

All that I saw was a lot of people fighting very hard over a very marginal and pathetic amount of money, equity, and status, while we wrote yet another multi-million dollar check to some shitty consulting firm in Eastern Europe.

You may think that we’re out of the game, but the truth is we understand it in a way you could hardly imagine.

Long time sysadmin here:

I'll happily admit that sometimes, especially once, becoming everyones favorite sysadmin took way less effort than it should take.

That said: either I've been extremely lucky to work with smart and nice sysadmins or you have been extremely unlucky or you are posting hyperbole.

Edit: In reality it also goes both ways: I had one particular user going to my boss because "we sysadmins were totally unreasonable" or something. So my boss told me (since I was both the group leader for user support and also the defacto user advocate) to do whatever it took to make him happy. As far as I understood from the IT manager it was some program that wasn't in our official repository that he needed or something.

Turns out it was a "music program". This left me wondering since that particular user was in management. Then he gives me the name of said "music program" and it is limewire and the reason he need it is for his kids.

At that point I became "unreasonable" towards him as well ;-)

I find a lot of undesirable behaviour and condesention between groups is based on insecurity. And not just between engineers and other engineers.

I never met a sysadmin that was anything like the bofh.

Look, it is a funny story, nothing more.

Just like there is dilbert and other stuff.

Do I want to become a bofh? NO

Do I like a complete Fantasy story in which a sysadmin runs amok?

Yes, yes I do.

I would never compare that to reality, just as I wouldn't expect that lord of the rings is a accurate history documentation and that star wars a prophecy.

To your other points:

I rarely met toxic people and when I did it was one of the following reasons:

* As soon as a system was down(Of course, no money for HA or preemptive maintenance) users called and screamed(Yes screamed)

at the it people till one of them had enough and screamed back.

* The Sysadmin was responsible for all the systems 24/7 but got paid 40 hours.

* The sysadmin had to repeateadly explain to a user how to change his or her password and how to login to computer.

* Sysadmin got the blame for not backing up a system becaus some enduser deployed in on their smuggled in nas

You will see that most people that become toxic, suprise suprise, are in a toxic environment.

Just to iterate over your last point:

The Job of a sysadmin is to build and deply systems, keep them running, document and plan new things.

It is NOT the job of a sysadmin to help people learn how to use software, which they should know about because they knew they had to use it, when they applied for the job. That IS helpdesk work.

E.g. "The software writes I can't connect to the server" is a valid question.

"I don't know how to make a paragraph bold in word" is not valid.

Because that would mean that you just hired a person who has no idea on how do to his job. How to you expect a sysadmin to know every programm inside out?

Heck I know how to deploy photoshop, butI have no clue how do to anything in it and I frankly don't care.

> The Job of a sysadmin is to build and deply systems, keep them running, document and plan new things. It is NOT the job of a sysadmin to help people learn how to use software, which they should know about because they knew they had to use it, when they applied for the job. That IS helpdesk work.

Unless your company is too small to have a helpdesk, then that's precisely what they're often hired to do, hate doing it, and are shitty and toxic about it.

I once worked somewhere with kind of old-school sysadmin guys. I had to spend so much time asking and badgering one to add disk space/allocate me more disk space on a computing cluster to load a model. The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk. In a cloud environment I could do this myself instantly or not even have to do it at all.

Especially as the cloud becomes more popular, and devops/modern tooling begins to be adopted in house even for companies that aren’t “on the cloud”, I see the role of sysadmins who aren’t also software engineers going away.

What was the iops you eventually got or needed?

Since you mentioned model and cluster I’m curious as to wether or not others were also on said cluster.

The way this computing cluster worked, each person had some default disk/memory space they were allocated out of a large total. I don’t know exactly how it was set up. People were running things like Hadoop jobs, Spark, Jupyter notebooks. I think the default disk space allowed was very small, like 5GB, and I had some large NLP model that was 8GB (probably the entire library and the part I needed was smaller).

I needed to save the NLP library to my user account’s disk so I would only have to download it once. I think I could redownload it every job, but that would take a pretty long time, and had other issues.

So it wasn’t even IOPS, it was just the disk space I was allocated on my user account. Probably a one line command by the sysadmin guy, unless he also needed to go get another hard drive and plug it in, in which case it would be 10 minutes and then a one line command.

"The cost of the excess space was probably around $1-5, averaged out of the cost of a large disk."

So what do you do when that disk dies and you lose all your data?

Oh, you want backups? Well, maybe you could do them yourself.

Oh, you don't want to do the backups yourself but want the sysadmins to do your backups for you, do you?

Or what happens when the network that's hosting the disk becomes inaccessible? You want to get to your data anyway, so that means adding network redundancy.

And you also probably don't want downtime if the server hosting the disk goes down, or the server is up but the disk dies, so that means RAID or maybe a distributed filesystem.

And what about making sure the data is secure and the servers hosting it meet compliance requirements?

And when something goes wrong you want someone in ops to troubleshoot it, right? Maybe you want six 9's of availability, and you want us to wear a pager so we can work on fixing it at any time of the day or night, right?

All of this will take planning and time and more hardware, configuration and monitoring, perhaps a bigger headcount since sysadmin teams are often already running short-staffed and don't have the time to dedicate to supporting even more infrastructure than they already do.

You see, it's not always so simple as "just add another disk".

Or I just use the cloud and configure all that myself without an arbitrary gatekeeper

I didn’t even care about any of that stuff. I just needed to be able to work on my own model using some other model that I could always redownload if there was an issue. If I had been empowered to just do it all myself it would have been a 0-10 minute task.

I have had my own share of problems with admins and corporate IT. But in most companies it really is a thankless job where you can’t win. When things work you have to justify why you are sitting around while things “just” work anyway. When things don’t work you get yelled at by some exec, probably the same who previously denied budget requests for better equipment. Add to that the fact that some users are just idiots and most companies see you as pure cost that best gets offshore. In my view definitely a job to be avoided.

> frankly most sysadmins don't make the money, the people they support do

> the fact that most ops positions at a non IT company are a lot less stressful, yet much better paid than all other jobs.

Which is it?

Make money for the company vs. make money for themselves.

> I know that it's parody, but I think the whole sysadmin culture is incredibly toxic and glorifies toxicity.

It has been quite a long time since the days of grumpy greybeards ruled the roost.

Sysadmins, much like developers, are labour force multipliers. One big difference is, sysadmins and IT are often treated by businesses as an absolute minimal-possible-cost departments. They're frequently underfunded, short staffed, and rarely have any management buy in for hardware or software necessary to do the job reliably. That leaves sysadmins scrambling around trying to pull one miracle after another after another out of their butts, making hardware and software perform to a degree that it just isn't capable of doing. When you push software and hardware beyond its limits, that means lots of emergencies, lots of firefighting. It's really not rare to have to argue for having any kind of redundancy in hardware or software ("we can't justify the expense of a warm spare, or even a cold spare", "But this is the mission critical software our entire business relies on. Every second it's down is tens of thousands of dollars down the drain", "yeah, but we just don't have the budget for it, we need that for the CEO's golf trips".)

Throw on top of that business groups that keep buying software and hardware without engaging the sysadmins to evaluate suitability; software that often involves working further miracles to make it work for the company because fuck me so much of that shit is written by people that have no clue about a safe operating environment, but hey the CEO's nephew produced it and we've already paid them hundreds of thousands of dollars for it that he's already spent on a sports car and we can't get a refund (I truly wish I was exaggerating there, that's a for real situation a friend found themselves in).

This is the continual story of sysadmin life. It's one reason why I eventually moved on to systems engineering / cloud platforms ops. It's just sheer constant stress courtesy of management that just don't give a shit while things are working, and then lay a crap load of blame on you when things fail.

Want to make any guesses now why sysadmins are frequently grumpy? It's not because they set out to be, or want to be. Very few do (most of those obstructionist greybeards are long gone). It's because they literally don't have time or energy left after being continually burned out from performing one miracle after another. For what it's worth, I see the exact same kinds of grumpiness from developers and other staff in businesses that don't give a shit about how much they're burning out their staff. It just seems like it's more visible in sysadmins because they end up having to deal with a far wider range of staff across a company.

A well funded IT stack makes it as friction-less as possible for employees in a company to get their job done. Technical problems employees have get dealt with quickly so they can be back up and working at peak productivity, because there will be sufficient helpdesk employees with sufficient training to keep the quality bar high (trying to get funding for training is another nightmare, especially for helpdesk). Platforms and technology they rely on are reliable. Mistakes made by employees are easily recovered from (files restored from backups, ransomware is isolated and easily remedied). The list goes on and on.

> When you push software and hardware beyond its limits, that means lots of emergencies, lots of firefighting. It's really not rare to have to argue for having any kind of redundancy in hardware or software ("we can't justify the expense of a warm spare, or even a cold spare", "But this is the mission critical software our entire business relies on. Every second it's down is tens of thousands of dollars down the drain", "yeah, but we just don't have the budget for it, we need that for the CEO's golf trips".)

> This is the continual story of sysadmin life. .... It's just sheer constant stress courtesy of management that just don't give a shit while things are working, and then lay a crap load of blame on you when things fail.

yes, this, a thousand times; my job security lies in my shit eating grin and capacity to ignore exactly this structural idiocy.

Isn't this why those jobs also have huge churn? The people who can get out, do. Leaving only those without the ability to find another job.

I know as a developer I have churned quickly when it turned out that "occasional overtime" meant constantly babysitting systems that were never finished, just patched together enough to run for a day without obvious disasters. That was the explicit metric in one place, they literally would not address a bug unless it caused the system to fail before the daily restart. "daily" meaning some developer had to wake up at 2am and remote in to manually restart a bunch of things in exactly the right order, then debug and fix any newly revealed problems (because this was also when new versions were started in production for the first time).

My record so far is less than an hour in a new job - I turned up for my first day and one boss was berating the other boss over the phone. As soon as they saw me they handed me the phone and ran out of the room. I got an earful before I managed to say "Hi, I'm psychlist and I was supposed to start working here today". I did actually let them talk me through fixing the immediate problem, then I hung up and walked out.

Don't work in those places. Just... don't.

"you'll have to help people, and you'll have to do it over and over and over again"

We do, again and again, and from some people we get nothing but condescension in return, and treated like unskilled janitors who only ever stand in the way.

If I had a penny for every stuck up sales/marketing person or exec who though everyone in IT was lower than pond scum...

This condescension even comes from developers, who you'd think would have some sympathy for other tech roles, but the attitude is often that devs are gods compared to the lowly ops guys.

"most sysadmins don't make the money, the people they support do"

This attitude is part of the problem. You don't appreciate that it's a team effort.

It got a lot worse since 'devops' came around, in my experience. As an example: people started getting increasingly pissed off when proprietary software (unpackaged, but usually with a shell installation script) couldn't get installed on a network wide setup of ~200 nodes fast enough to their liking. Since at home they could do that in 15m following the vendor's instructions (which involved running the installer script as root).

It fell to deaf ears that a packaged approach was the right way to go, given said software installed its own jungle all around the filesystem hierarchy, numerous times conflicting with software from other vendors with the same kind of recklessness.

Quite clear that any of these people lacked the understanding that having a longterm stable environment where everyone uses the same version of everything, is uncomparable with doing unsafe installations following the first forum-post you find online.

Devops learned them that cutting corners to empower yourself is OK, they could run their own systems quicker. Needless to say all the debugging (the search-engine based kind) that needed to be done after the facts was suddenly part of their job description, and filled up most of their time...

I recently got called in to support one of my folks because some devops team was absolutely incensed they couldn't just pull in some rando docker container they'd found on the net. From a .cn domain. That failed the static analysis suite. For a production financial application. Fail fast indeed.

I get we need to adapt to support devops. I'm investing big resources to support that. It would be so much easier if so many devs didn't think devops == no oversight.

So what actually prevented them from just doing this?

The dev folks have an 'approved repository' workflow; when this team tried to deploy to QA the rouge container got flagged. Plus it failed the static analysis step in the pipeline. I assume a sufficiently clever dev could find a way around this, but it worked here. Maybe we just got lucky this time, but we did an asset audit when we found this one and didn't find anything that had slipped through.

We're looking at deploying some tools (e.g. Dome9) that we think will close some of the loopholes and provide more automation. However, due to the complexity of these environments, I'm not sure there's a perfect solution.

For some reason it become socially acceptable to bash folks who do system administration for a living.

Seems that you are really frustrated because you had a bad experience while working with a sysadmin and I don't think it is fair to despise people who do system administration just because of that.

This is toxic, and is _not_ ok.

Sysadmins are busy people, with their own reponsibilities, tasks, etc. They are always willing to help, assuming that you know how to communicate nicelly, tell about what you want to achieve, being a humble, honest human being.

Assuming that the sysadmin profession entails helping people all the time is plain wrong. Don't abuse.

Though, even if you ask for help several times about a specific subject is very likelly you will receive useful links, docs etc to get you "unblocked".

You still have to read and try to help yourself with the new piece of knowledge that you received.

_Demanding_ to help all the time is _not_ ok.

Peace!

Absolutely!

Back in college, I used to laugh along with BOFH style comedy. I still do find it funny, but now that I've been on the receiving end of this brand of toxic behavior in my own life, it's all a little too real for me.

Interestingly, at my current job (research assistant at a university) I've found an almost identical culture among lab support staff. I'd chock it up to the same factors of long hours, a thankless job, and the occasional user that is in fact an idiot. It's also a field that tends to accumulate people without a lot of social awareness, but where the job involves a lot of human interaction. They're also likely the ones getting blamed for problems they don't have a ton of power to solve.

I understand why they feel the way they do, but I wish we could do without all of the abusive behavior. In the end it's real people's careers that they're taking their anger out on.

I've had research that my career depends on delayed for at least a sum total of months at this point (in only ~2.5 years of time), because of situations like this from both toxic IT and research staff dragging their feet and being hostile to work.

My friends and I still make jokes about suggesting people get to the bottom of their problems with a "power test"; that is, flicking the main power switch on and off as fast as possible for about 30 seconds.

Great. Makes for an excellent bit of early Internet humor study.

I like BOFH. I and my co workers used to look forward to them.

I just learned today that he's published some recent updates in the Register. Latest is from four days ago!

https://www.theregister.co.uk/data_centre/bofh/

Sadly, I don't think they're as good as the older ones.

Well cool. Pity they might not be as good. But, I will read anyway.

Get a little nostalgia fix.

Thanks!

A new one comes out on El Reg once a week or so but, yeah, there's less death and destruction than there was back in the good ol' days.

I have compiled an index of the new stories arranged by year of publication on my website, which is a bit easier to navigate than scrolling through the (incomplete) list on the Register website: https://bearbin.net/bofh

Links to the official archive and author bio are dead.

Thanks for letting me know, I've replaced them with archived versions.

The BOFH is a response to the kinds of things that still happen, for a nice little modern-day horror story take a look at https://www.reddit.com/r/sysadmin/comments/eaphr8/a_dropbox_...

The meat buried in there: [...]"it wasn't just the database it was the entire app and website. The app was actually just a server instance in Heroku that was spun up whenever there was an update and would make crazy api calls to the drop box account read information from hardcoded database files. He immediately called drop box support to figure out what in god's name was going on and to his horror after several escalations gained access to the account and found that the account had 497 TB of 500 TB space used up and the team was on the verge of running out."

also

"This single drop box account was also their version control."

Ahh, for the life of me I can't rememeber another site that was similar but not a parody. It was tales from a Tech Support worker, which were pretty funny but he wasn't being malicious.

I've search my pinboard for it and I can't find it anywhere! If anyone knows of it I'd be grateful.

Are you thinking of 'Chronicles of George' (https://www.chroniclesofgeorge.com/)?

That's exactly it! Thanks so much.

I’ve been reading the BOFH since the BBS days. He’s still at it, for The Register.

I gave up following the ref about 15 years ago - originally body was every Friday if I recall, but it dropped to irregularly a few times a year. The plots also went from short circuiting machines and deleting files to mass murder for no reason.

The first BOFH was there right at the time I learned Unix in an apprenticeship school. It got me into the right vibe.

Damn programming and architecture stole me from system administration :)

Yip, earliest I recall seeing it was mid 80's, unsure if it was in the industry magazine "Computer Weekly - https://en.wikipedia.org/wiki/Computer_Weekly" (the one you got for free and was impossible to cancel) or UNIX World. But certainly fun times.

EDIT - wrong magazine

For me it was end 90s. It was "classic" at that time already. When I remember right, I learned the concept of quotas at the same time and the practical usage from BOFH.

The problem with these stories, and pretty much any that use shock as there source of humor, is that they just aren't that funny. "It's just a joke" only works if the content actually develops a punchline. But instead, it's the same story, over and over again. The only change is that the BOFH goes from fighting with incompetent vendors foisted on him by management to just being the same, exact kind of dick to the people he's supposed to be supporting, making him exactly the same, frustratingly useless "help" he complains about with the vendors.

Even with a reading that these are projections of what he sees as how other people see him (he doesn't actually want to watch cartoons all day and delete peoples files intentionally, that's just what people think he does), the joke runs tired after the 3rd iteration, and then continues ad nauseum.

This is a hacker culture classic.

for both good and bad

I never encountered a BOFH in my career but can certainly understand their frustration with users. Maybe I haven’t worked for the “wrong” companies but the first BOFH moment would be a sysadmins last where I’ve worked. If it became a pattern then the IT manager and the sysadmin would have been fired. Then again, I’ve worked in Microsoft shops and never had to deal with a crusty, silver pony tailed Unix sysadmins so maybe the stack plays a part.

Indeed!

  $ curl  http://bofh.bjash.com/ -I
  HTTP/1.1 200 OK
  Content-Length: 22100
  Content-Type: text/html
  Last-Modified: Mon, 09 Sep 2019 18:36:39 GMT
  Accept-Ranges: bytes
  ETag: "916293843d67d51:0"
  Server: Microsoft-IIS/10.0
  X-Powered-By: ASP.NET
  Date: Mon, 30 Dec 2019 18:53:47 GMT

Could add 1999 to the title.

Ok, added. Is that the correct year?

I had a hard time figuring it out but my sense was these posts were happening over time starting in 1999.

A small but good thread from 2018: https://news.ycombinator.com/item?id=17160902

I used to read it initially and some of it was funny because it was harmless but overtime it became more and more illegal, mean, vindictive just to hurt. I could not read it anymore.

that is in needs of an update: something like 'the cloud setup from beyond hell'

Here's the real world version: https://freethoughtblogs.com/stderr/2019/12/28/its-still-abo...

clickety clickety

I think people are missing the whole context of BOFH.

In the '90s there was no prestige in working in IT maybe as a developer but not as a Sysadmin.

Sysadmin and Support (the guy who fixes your e-mail or set up Netscape) usually were the same guy.

The prestige positions were in Marketing, Law and executive/finance.

The assistant on HR?Finance? They usually earned more than the Sysadmin.

People used to tell me:

"People who work on computers will never grow in the company. They stay there working and nobody notices them then. If you want to grow and climb de ladder go to Finance or Marketing."

So then who were the people who worked on computers?

The ones that really liked it and the ones socially inept. Sometimes Both. It was the "misfits" people dept. Data Centers? Basement or an ugly off-site. Location of Dept? The worst possible.

(Check the IT CROWD series).

But it was very rare someone that Did not like computers and did not have a hacker mentality to work on the field. Because of low pay and prestige.

Then BOFH represents the "Dr. Evil" that makes all right. Punishes the people that earn 2-3x times more and still can't do anything in IT. It is dark humor like "Nick burns, your computer guy" from SNL.

The big difference is the people who started at 00 and later got a Very different market and with other people went after looking for prestige.

Take Bezos, Gates, Stallman, Linus and any other big tech guy before 00. They were "NERDS". Nerds only became cool after bringing billions to the table.

What a young, socially inept, smart guy and nerd will do in his 20's?

1) Have a lot of girlfriends? Party like crazy?

2) Fail miserably in #1 and code?

I remember on dates saying that I work with "IT" and the face of frustration in my dates.

One of my date father saying to her "He will not be able to support you... If he was a Lawyer yes.."

There was no "diversity" need because the requiremnt was: be smart and crazy to work in IT.

Last comment:

Once in a HR meeting they asked what traits they need to fill an position without thinking too much I said:

1. Low self-esteem;

2. Smart;

3. Desesperate.

My coworker was there agreeing. The HR said that we can't put it. So we change the words a little.

I agree with you about IT careers. Back in the late 80s when I was starting college, it was exactly the way you describe. It is really interesting how things changed in the late 90's regarding this. I remember it falling back for a while after the dot com boom, only to pick back up in late 2000. A lot of people really don't realize that the prestige that comes from IT is only a recent thing, I think because having grown up around it, it's all that they know.

From the site:

> Warning: This site contains adult themes, violence, profanities, and other such fun stuff. If it bothers you, just don't read it.

Reply on news.ycombinator.com