Hacker News Reader

Comment by uvesten | Hacker News Reader

From the article: ”Often they have not even bothered to do basic troubleshooting, things like read the documentation on what the error message is attempting to tell you.”

This has been the bane of my work happiness for a while now. I keep having to tell junior devs to actually _read_ the fine error message, just in case it actually _contains information about the error_, you know. Not that it seems to help much, it’s like they can’t get the concept into their heads.

This is 100% a problem with younger, bootcamp-”educated” devs, in my experience. I know the common wisdom on social media is ”no one reads text anymore”, but if that includes aspiring developers, it might be tough to replace the current workforce when that day comes…

View on HN

It was not about 27 years ago that the senior sysadmin at my first real job told me: "Most young people don't have any hacker spirit at all, they just give up the first time they see an error message. The world is not going to survive that."

I was doubtful that this was a universal truth then, and I think it's the same now: there are a lot of people who do mediocre work, and they are and were supported by a smaller group of people who do really good work. And the world keeps turning.

One of the joys of the Internet and of open source is the increased ease of sharing ideas and solutions.

> they are and were supported by a smaller group of people who do really good work

I think the only solution for this is when "support" includes education. In the simplest form, we can give support by helping that colleague to find the issue himself, rather than giving the solution directly. In a more advanced form, you're making structural changes to your company. Like in how you share knowledge with the team.

Reality check: they run off and nag someone else...very discreetly...until someone caves and fixes it for them. If they get enough pushback they will just quit.

I think that education ought to be available to everyone, but spending effort trying to teach someone who doesn't want to learn is a waste of everyone's time.

We've all heard the stories where someone bright joins a new workplace, is assigned drudgework and after a bit automates it until they only work two hours a week? That's someone who is willing to learn, and everyone else in the office was willing to experience boredom in order to avoid learning.

For all I know, some of those people would have perked right up if the subject matter were milling, millinery or masonry moving millier-weights of stone. Not everyone is interested in the same things, and even when their job is all about it, sometimes people aren't invested in it.

If it works, everyone involved wins. Sometimes it doesn't work.

This is where leadership comes in, to make clear division of responsibilities.

You're doing everyone a disfavour giving out full answers or not demanding some homework first, and often, it's really an ego-issue.

Salient point and makes a ton of sense. I think this is accurate, totally coincides with my entire career/work experience.

> This is 100% a problem with younger, bootcamp-”educated” devs, in my experience.

In my experience it’s also common with older and college-educated ones; contractors trying to avoid extra hours; senior architects; and especially anyone who thinks ops is someone else’s job. It’s definitely not specific to age or training mode.

There are a few contributing factors I see: tunnel-vision focused on the particular detail they think they’re working on, causing them to ignore anything they “know” isn’t related; shoddy tools like much of the Java ecosystem where poor culture around logging trains every user that it’s normal to have huge amounts of log spew; etc. but the biggest problem I have seen is ego — either unwillingness to believe that the product of their staggering intellect could be less than perfect or that the mundane task of getting their grand vision to actually work is for the little people.

I’m thinking of a “senior architect” who was quite surprised to learn that networks are neither perfect nor instantaneous, and that his app might have some issues due to needing thousands of XHR calls to load the UI. It was so much easier to ignore the error messages and say the problem was Chrome. He had a CS degree – the problem was the wrong mindset and having been enabled to avoid good troubleshooting skills.

It could be argued that there is a technological Maya that has to be overcome before you mature as a developer. Most people grow up spoon fed the "it just works" ideal, even old school computer people never really had to doubt that their floppy drive would read a disk or that their PC BIOS would accurately boot their computer.

It's only after the technological illusion of Maya breaks that you realize floppies have read heads, hard drives have moving parts, CPUs have conductive traces and all of these are vulnerable to breakdown, entropy exists in the system and cannot be expelled, that the previous "ideal" state of your system was temporary, an illusion, that nothing always works the way it is supposed to and that your options boil down to "burn it to the ground and start over" or "leap into Hades both feet first to rescue the soul of what you love".

Most people go the first route. Buy a new one. Replace what is broken with something else. That way the illusions are never broken. The technology didn't fail, only its current & easily replaceable avatar.

As the Son of God once did, after its death it will rise again, immortally replaceable.

However, it is only after you have faced that 2nd trial by fire and returned with your elixir that you as a changed being can peer through the veil. The meme about "CPUs being rocks we filled with lightning & tricked into thinking" rings differently to you now.

You're touched the bones of the God and found that they crumble. There is no God here, only a beautiful shambling nightmare that has eaten the minds and souls of millions, built by mad scientists and engineers in a vain attempt to create the God whose physical absence they find themselves longing for the same way a neglected child longs for the embrace of their mother.

https://psychology.wikia.org/wiki/Maya_(illusion)

I think people don't read error messages if fixing the issue is beyond their immediate capabilities.

Computers are scary things that fail in counterintuitive ways. When the handle of your tea cup breaks, the issue is intuitive and most people will be able to understand why it is happening and how to work around it(handle it carefully from the top end end enjoy your tea?).

But when it comes to computers, often you need deep understanding of its inner workings to make sense of your observations of problems. Why Xcode would say that it failed to compile my project because usefulExtensions.swift already exists? What it is supposed to mean, I see only one file with that name? That information gives intuitive idea about the issue only if you know how the compiling process works.

Why would I know why the package couldn't be found? Unless of course I know how that package manager works. Then I can check if the package manager is configured to look at the correct places.

Most error messages are like that. Instantly makes intuitive sense if you know how everything is glued together and makes no sense and needs study if it's outside of you domain of expertise. No one reads error messages unless they can recognise the pattern instantly and there's a data(like the name of the variable) guiding you to the fix.

> Computers are scary things that fail in counterintuitive ways.

Not being scared of the tool and believing one's inherent supremacy over it must be the most basic criterion for practicing this craft, but these days this fear is nursed, at times encouraged, at times even exalted (corollary of the failure fetish) especially by those who publicly place themselves as ambassadors.

Any introduction to computers must start with the statement that they are all heaps of plastic and sand and the only things they are able to do are because some mortal sat down and spent time figuring it out.

People starting out now are at a disadvantage because their first encounters happen mostly through extremely polished looking apps and it is hard to see at the outset how one could go from weird incantations in a text editor to that.

I think it's harder now than it used to be. When I started programming, any error message you saw would almost certainly originate from the beige box on the desk in front of you, which had only a single CPU core and often didn't really do multi-tasking. Over time, our computers, operating systems, applications and networks have become vastly more complex, with the effect that you can't easily build an intuition about which component is responsible for a failure and why.

Personally, I love debugging things. I have a very good "theory of mind" for dealing with computer failures, and figuring out why the computer isn't doing what I might naively expect it to do is a lot of fun. However, it's only fun because I've been able to stay on top of the curve as the systems I work with have become more complex. Starting from zero today sounds a lot more daunting.

True, the stacks are more complex today, but the resources are greater. Back in the days when the error was in the beige box, if the books/CDs/DVDs you had on your shelf didn't address the exact problem, you had to roll up your sleeves or you were SOL.

Nowadays, research skills are more important, but I see a lot of devs who just don't have them. Can't find the answer on the first page of your first (poorly formed) search? Run get the senior dev. To me it reads like incuriousity and laziness, or lack of training.

I don't mind doing some coaching, but if you're a dev, and you can't even be bothered to read the error message, what does that say about your effectiveness?

/rant

It gets a lot less fun when there's a customer visible outage and you need to know why things are failing NOW because dollars are on the line.

> Any introduction to computers must start with the statement that they are all heaps of plastic

This scales to everything IMHO, everything is simple once you understand it. Levels of abstractions is what makes it scary and complex. I.e. electricity or fire is also not scary once you know how to handle it.

One should not have an unreasonable fear of fire, but "unreasonable" is a quite key word there. One should have a healthy respect for it and feel some fear if those around you don't.

This is the correct comment.

First of all I agree with you. But I would like to note one aspect of this that has always been true. Many times the error message itself is terrible. They say that an error happened and then they give you absolutely no context for the error. My favorite example is when an application happily bubbles up the error from the operating system when a File read/write operation fails. The OS will tell what kind of operation failed. It won't tell you which file you were trying to read or write. It won't tell you what location you were trying to read or write. Basically none of the context you need to really understand what went wrong. You'll have to go read the code in order to mentally reconstruct what the context is. It's silly. If you are writing a file and don't at a minimum catch and then wrap the error with an error of your own that adds the necessary context then you are contributing to the problem here. And that's just one common example. There are many more.

I'm an SRE, so my programs are generally Python scripts < 1K LOC; maybe this isn't scalable, but I write verbose log statements (if it's launched with --verbose, of course). It's not that much effort to change `except OSError as e: log.error(e)` to `except OSError as e: log.error(f"Error accessing {file} - {e}")`

If I know typical causes of errors (forgot to connect to the VPN, etc.), I'll include them in the log message as well as things to check.

That might at least in part be because you are an SRE and at some point you hit your limit of inscrutable error messages happening in production.

This is a large and ongoing part of becoming a developer. It also happens again every time you try to learn a sufficiently new or alien technology. You know you start to make progress when the error messages begin to make sense.

Often you need to know a lot of context before you're even able to determine what the error message is! One error message can lead to a cascade of other error messages, or it's something breaking down as a result of multiple layers of indirection, requiring the developer to careful track the trail of what went wrong and led to another thing failing, which broke down the next thing and ultimately, decided to stop the program and mention only the very last thing falling apart to the user. There might be a directly sensible connection with the original error, but often it's quite unrelated. An experienced developer often immediately recognizes: this is not the actual error message, that other thing is! But for a junior it's all equally incomprehensible.

It is detective work with many false leads, and being very new at something it can be so overwhelming you don't know where to begin and immediately assume you will not succeed finding out 'whodunnit', asking your senior co-worker for help.

> I think people don't read error messages if fixing the issue is beyond their immediate capabilities.

How would they obtain those abilities though if not while spending time on the issues brought up and learning how to learn.

I think sometimes people are just bored and can't be bothered to find the cause and solution to their issues, and over a long period of time that mentality sticks and becomes second nature resulting in phrases like "this software sucks, I need to read the docs to use it".

Sure, if they study it then they learn it.

The problem is, learning is taxing and many times you encounter these errors when you have more important things to do.

When you want to develop your game and the IDE is complaining about something about locating some files, do you think that it is good idea to learn how that IDE organises dependencies?

Sometimes you suck it up and learn it and you know next time. However, your first instinct would be to look for ways to make the error go away so that you can immediately start working on the task that you are supposed to work on. That's why we have abstractions and when things work fine we don't know how things work.

It shouldn't be expected of you having complete knowledge of all computer systems, tools and frameworks before you can make a ball image bounce on the screen.

Taxing as hell. I have personal projects well underway that go unfinished because of tooling complexity or some other issue causing me to completely derail and spend days figuring out some type of in issue that has absolutely nothing to do with what I'm trying to accomplish. Granted, since I have gotten away from visual studio it is much better but I'm t still happens. If it isn't the IDE or package upgrades it's AWS or Azure issues.

> But when it comes to computers, often you need deep understanding of its inner workings to make sense of your observations of problems.

They're supposed to have that knowledge, or at least not be afraid to dive in and get that knowledge.

There's only one way to build an intuition of what kind of problem probably causes some error (most famously, if the error is completely incomprehensible, you missed a closing thingy on the previous line), and that's by doing the work a lot.

Unfortunately, the way we’ve organized this industry, a junior developer stuck on an Agile(TM) hamster wheel has no time to dive in and figure it out.

Very very true. When something causes a 1 point story to take three days; bring on the hacks and compromises and ignore anything that doesn't need dealt with to get it out the door.

> They're supposed to have that knowledge

I don't think so. We can do so many amazing things with the computers precisely because we don't have to know how things work. Computers are so many levels of abstractions over printed metal on melted sand.

People who know what they are doing will understand the errors of their own creations and will learn the workings of the tools they use to some degrees and will be able to understand the failing modes of these tools with experience over time. No one starts with complete knowledge before start building things.

> or at least not be afraid to dive in and get that knowledge.

Of course they should have the drive but people's first instinct would be to make the error go away so that they can do their actual work. People have limited time and energy, you can't expect a JS developer, for example, to study inner workings of a Linux box to understand all errors. It's cool when they do and gives them superpowers but it also makes them less productive as JS developers. Sometimes you simply need to implement that button to render on the server without studying the server.

Improving compiler errors can go a long way.

When even line numbers are missing, simple syntax errors can generate new errors and mind numbing troubleshooting.

My own perspective on this was that it felt like my brain would turn to goop when I got an error message, my eyes would cross and I would start frantically googling literally anything, skimming stack overflow, and getting nothing done. In order to progress I had to learn to slow down and start reading error messages and learning what they meant. Sometimes this meant I had to look up a bunch of words, one after another, to understand something incredibly dense.

So yeah, I think I largely agree with your assessment, and would only go on to state that the path forward is slowing down to learn vocabulary and think critically. You really speed up after that.

“I think people don't read error messages if fixing the issue is beyond their immediate capabilities.”

At least capture them so somebody who knows that area can make sense of them.

But the moment you get these error messages they have to become your domain of study.

I have experienced this same issue many times. But from my experience, I cannot simply pinpoint it to young developers or their education. I have seen this behavior with several older people, university educated, with many years of professional experience. So not just an age/generational thing, in my opinion.

Maybe they have seen so many red herrings that they don't even trust that the error message could contain something useful and relevant? Or maybe they just learned to skim through everything, and don't actually read stuff.

I have the same experience.

I also don't understand why, when they ask for help, they can never be bothered to say what they're trying to do, what error message they got, etc. It feels like they're doing me a favor when I try to help them fix something.

Then there’s the StackOverflow effect: where you ask a reasonable question X and get a bunch of upvoted condemnations along with directions to do Y instead.

Or, why would you ever consider doing X at all? Ok, thanks for being unhelpful.

Right. When I was learning to code a more seasoned dev told me "The code doesn't do what you want it to do, only what you _tell_ it to do."

Really helped me gain the mindset that not only was it my mistake that resulted in code not running, but that it was fixable. Like a game of ping pong. You hit the ball, sometimes the compiler hits it back.

I think it comes down to passion and curiosity. Those are two things can happen at any age and education. It is also something that ebs and flows based on energy levels.

I think the same. It's so typical everyone here tries to pin characteristics to exact professional groups. The amount of anecdotal evidence here is too damn high.

Keep at it: some crusty guy telling me to just read the sodding error 12 years ago opened my eyes.

I think there's something about certain kinds of tools that are cryptic, unpredictable, and frustrating that can teach you to be helpless - to just Google and hope. It's fixable though.

I don’t know most of the times error messages on computers are garbage. People build up some type of fear for the error message format.

I have an example:

I built a logistics and invoicing tool a few years back with error messages that where human readable with clear proper messages that told the user exactly what they did wrong and it even proposed how they might fix the problem.

I don’t know how many times I had to go to the users workstations read the text out loud for them like they where a 5 year old and ask them what they thought it meant. They always knew what it meant but I had to read it for them it was embarrassing.

And these where university educated accountants that where using the software.

After a lifetime of garbage error messages like “error code 4513” people just zone out.

For real. How difficult would it be to have the computer tell us what the error code means instead of a single sentence.

I get codes back in the day when storage for a whole book was costly, but that isn't the case anymore. Just tell us the error, show us the pointers, and then tell us what typical fixes are instead of expecting us to go to the internet for a solution.

My take is that cryptic error messages have always been a cross between 'protecting company secrets' and 'never really admit to a mistake'. Sure, MS or Apple could just throw up a dialog that said "Sorry, we trashed the file you were working on, here are the last 1024 chars" but people would actually be more angry then, instead of just "Error 1234 occurred" (or at least they'd have to go look it up to be mad).

As developers, we also have to be used to a lot of completely unhelpful errors. Yeah, couldn't connect to the DB, sure... oh, but actually because my code ate all of memory, why didn't you say that in the first place?

I suspect it grew out of electrical engineering. Machines/parts could only return errors as integers, so you would expect people to open up the documents and read that error code 1021 meant that the unit had caught fire.

Software just kept the tradition of error codes, since that meant you could also sell that juicy documentation (localized into whatever language you wanted) to the user as well. I suspect it also a localization issue because OracleDB would never return the table/column in the error message, so as to be easier to translate.

Why would a user-friendly system need error messages in the first place?

The modal dialog breaks UX spectacularly.

Even logs should have messages that actually are intuitive to follow.

From my mentioned example one reason was cause the user loaded a csv file with incompatible data.

Or cause the user had filled in part two of a task but not part one and then tried to continue with parts of the task that where dependent on filling in part one.

Or the user tried to synchronize orders from the erp system but the erp system would not return any orders.

Still there are better and worse ways to identify and present failure. A modal often being worse for instance.

I always preach to people that they have to make it easy for others to help them. Don’t just say “it doesn’t work” and expect them to analyze the issue and take care of things. Instead provide information what you did, send log files, screenshots and whatever other information you may have.

I think the people most likely to fall into the “it doesn’t work” category are people who don’t have much experience troubleshooting difficult problems.

In the end it’s about compassion and understanding of each other. Unfortunately in a lot of companies the only direction people are getting is “get it done on time”. It’s rare that management asks people to have empathy for each other.

> I keep having to tell junior devs to actually _read_ the fine error message

I wonder if it's also to do with the environment in which they learn. When I was learning to program, like probably others here, I didn't have anyone around me who knew anything about computers so was generally on my own until my first job and had to dig through stack traces and read error messages and had to try and figure out what was wrong. Kind of a blessing and a curse as I imagine my rate would have been a quicker and I wouldn't have hit so many brick walls but I learned to debug independently.

> This is 100% a problem with younger, bootcamp-"educated" devs, in my experience.

You'll get a lot of pushback here but it's definitely true.

That doesn't mean it doesn't happen with CS grads as well, but it's quite rampant among bootcamp devs. I think the reason for that is that, since the bootcamps are so short, they "stay on rails" and mostly work on simple projects (that will give out something they can push to a github repo and use as a portfolio).

It's the same with git. Every bootcamp will use git and claim to teach it to their grads, but then watch them do anything on a repo with multiple users. A lot of them just rote memorized commands to pull and push to main and that's it. Branching? Rebase? Using the commit history? Never heard of.

For new hires from serious Engineering or CS Degree, they should have had at least a few classes dedicated to projects where they built something non-trivial. On top of theoretical classes teaching the fundamentals.

I do have one anecdote/counterpoint. At my company we do DevOps, but we have a central group that creates the templates we are supposed to use for various AWS resources, build plans, and deploy plans. It makes it extremely frustrating to troubleshoot issues specific to the templates or how you configured them because they usually aren't something you can google and the documentation for them is extremely basic. Sometimes you have to ask the people who created them since they have the deep understanding and experience.

I've been contracting at a company for a few years and things progressed from manually setting up servers (before I was involved) to using Ansible to provision multiple servers and now it's transitioning to a Kubernetes cluster and infrastructure as code from day 1 along with tomes of documentation to go with it.

It really is worth it to go the extra mile and write comprehensive docs, even going as far as writing them in a conversational tone as if it's a blog post or a book. I'm really happy I found a company who treats documentation and workflows as first class resources.

For a small team where only 1 person is working on this it helps eliminate the bus factor and it also makes it easier to have non-hardcore ops folks do code reviews on your IaC. Having them be able to get the gist of it with a little bit of background knowledge is so much better than nothing. All of this results in higher reliability of the services your company offers.

I share this pain, but believe it or not, it’s actually much better than everyone doing their own thing all willy nilly.

I agree that it has to be done to have consistency. I'm just pointing out that when using IaC designed by another team where the internal workings are largely hidden, us primarily dev guys are going to need some help with issues that appear to be ops related. (And yeah, occasionally root cause will be that I'm just an idiot)

”We’ll just rewrite that microservice from scratch and the error probably goes away. Preferably with the newest framework du jour”

Not to totally discount your point, but consider that rewriting is a form of study.

2021, State of the Practice.

well except for using Rust, and a bunch of dependencies from the web, version determined when downloaded at compile time.

To be fair, Cargo defaults to using lockfiles, so you'd have to go out of your way to hit both your points at the same time.

very true, some hyperbole to make a point.

"lack of newness" is a characteristic many will expend untold hours to extinguish. to my perspective, the "rewrite it in Rust crowd" is the peak; all non-Rust code is soiled, and worthy of replacement.

(it is very possible that the "rewrite in Rust" movement is just a guerrilla marketing project)

I don't really care for Rust, but all non-memory safe code could benefit from being replaced. This does, for some people, mean rewriting it in Rust because C and C++ make it harder to achieve the goal of memory safety.

this feels like one of those things that needs to be tackled from both ends

teach people to read error messages and simultaneously improve the readability (and utility) or error messages

i don't know why we put up with such bad error messages anymore. i imagine it's a function of stockholm syndrome and the difficulty in getting messages changed

I sort of blame exceptions here. They make it really easy to just let the error bubble up to the top. But often times the place where the error gets thrown doesn't have all the necessary context to have a really good error message. If you want a good readable error message you have to trap the exception at the appropriate place and then wrap it with the appropriate amount of context in the message. But the easy path is to pretend there is no error and let the very top layer surface anything that went wrong to the user.

Exceptions are appropriate if you create unique application errors or kill your app with it.

Force them to write heavily templates C++ code, they either learn how to read pages of comprehensible error message to diagnose one little typo, or they can’t do their job and are forced to look for a new one.

It’s usually not hard to figure out what’s wrong from the messages, but man do they look scary and hard to understand when they appear. Yes I’ve been writing C++17 lately using some very template heavy libraries.

“Force them to write heavily templates C++ code, they either learn how to read pages of comprehensible error message to diagnose one little typo, or they can’t do their job and are forced to look for a new one.”

When I did C++ we sometimes made little competitions for the smallest change that can produce the craziest error messages. On the other hand I always found it extremely satisfying to make one little change that removed thousands of errors and warnings.

I've avoided C++ for most of my career, but one idiotic mistake I do remember making was accidentally leaving an open curlybrace at the end of one of my source files. The C++ compiler ran and reported 1000s of compilation errors all through every single other file -- in my code, throughout all the library code I included.

Easily diagnosed if you're working incrementally, one small change at a time, and making checkpoints with version control: `git diff`, carefully review the diff of what you changed since the last checkpoint where things were more or less working. I must have not been disciplined enough to work like that at the time.

Troubleshooting systems integration failures is also character building for getting better at diagnosis from errors. Sure, it's failing, but let's try to figure out the immediate layer of failure from the logs, error messages, symptoms: name resolution? tcp? tls? http proxy? authentication? authorisation? api spec misalignment? error in our application code or the system we're directly talking to? unexpected data? error in some other system that we depend upon transitively? each time you hit a new novel failure mode, or fail at one level deeper, you're making progress!

I had a nice one today where it complained about some thing not being invokable deep in some std code somewhere. Lots of crazy template instantiation errors. It turned out I forgot to pass the variant parameter to std::visit.

I don't think it's just because of bootcamp education, I think people are growing up in a world where error messages are either never displayed, or displayed in the from of "The program has did a sad :( try again later."

They're not used to reading error messages because they've been brought up seeing nothing but completely useless error messages.

We “process” maybe 100 times more text than we used - but to do that we scan for patterns and ignore without reading most of it. (Partly due to the prevalence of advertising as well, of which we prefer to read 0%)

> This is 100% a problem with younger, bootcamp-”educated” devs, in my experience.

I resent this. Not because I'm a bootcamp-"educated" dev. I'm not. But it suggests somehow that devs with CS degrees are somehow better in this aspect. If anything, they're arguably worse (obligatory, not everyone disclaimer).

I have never met these kinds of people and it boggles my mind that they exist. Actually reading error messages is the most basic thing I'd expect from a developer considering that you're going to encounter them regularly during the development process.

Some seniors remind me of street preachers shouting "Hell and damnation awaits you!", and they get much the same response from people. Not saying this is you, but I'd be interested to know what ways you've tried and what's worked or not.

It’s like script kiddies showing you all of the innocuous stuff they found with Google, they are illiterate and so their imaginations run wild.

The script kiddies of yore, however, did have a desire to get stuff working.

This lead to a lot of them learning common failure modes of the software they ran.

Ironically the people who were script kiddies in their teens have been some of the best troubleshooters I know.

Selective memory, nobody cares about Igor who never got further than being a script kiddie.

I have the opposite experience, boomer devs that see red or yellow in their terminal and send me an email or jira ticket before trying to parse it. The younger generation at least can make judgement calls on warnings.

Why are software devs responsible for first level support? Software devs are expensive, support staff are not. Seems like bad business to me.

Big fat "it depends" on that. It might be superficially correct (dependong on the scope/skills of "support" staff). Even if there is a meaningful financial difference between the day rates, it doesn't necessarily follow when the process is viewed end to end:

1. If ops staff have limited expertise/authority, it's less likely they can resolve problems. They might acknowledge (so maintaining some aspect of client SLA), or have a limited set of pre-defined remedial actions (reset button). Anything beyond that, though, and it needs the dev team. So it's arguable whether the ops staff provide much value in the equation.

2. As a dev, there's nothing quite like the prospect of being paged at 2am on a Sunday to incentive more robust code.

End to end dev accountability isn't a panacea either - but the problem is more nuanced than just pay rates.

One of the best arguments for shared responsibility is that it avoids “not my job” thinking. I’ve seen large organizations burn resources and downtime because developers and ops are in an adversarial relationship where nobody has a stronger sense of responsibility for an application working than they do for shifting the blame to their counterparts. If everyone is getting things escalated to them it tends to cut through that cycle.

also, devs seem to really underestimate the pay support staff makes, especially those capable of troubleshooting deep, low-level and complex issues. this might be less visible on the Dev side, but troubleshooting infrastructure is not an easy to find skill, especially if dozens of moving parts are involved.

More power to them. The rest of us 'Ops' guys are slowly transitioning to DevOps Engineers and SRE's and are gladly taking handfuls of cash because I both know how to read hex dumps, and are also smart enough to know your IP address isn't going to change because I replaced your ethernet cable.

L1 support are users of the software, just like their customers. If the application doesn't provide them with enough information to do their job, then that's a failure of the application.

And by "application" I mean the combined software, release, documentation, runbooks, etc. Not just the latest git tag pushed.

Often the root cause is in dev code without proper logging and documentation = dev problem. The only way for Ops to troubleshoot might be to sniff the network and make educated guesses, if it's recurring. All that extra work costs much more and halts more urgent production work.

You broke it, you fix it.

Reply on news.ycombinator.com