Hacker News Reader

Comment by lordnacho | Hacker News Reader

> This was a very fun problem to debug.

I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention.

Something this deep is especially frustrating. Nobody suspects the standard library or the compiler. Devs have been taught from a young age that it's always you, not the tools you were given, and that's generally true.

One time, I actually did find a standard library bug. I ended up taking apart absolutely everything on my side, because of course the last hypothesis you test is that the pieces you have from the SDK are broken. So a huge amount of time is spent chasing the wrong lead when it actually is a fundamental problem.

On top of this, the thing is a race condition, so you can't even reliably reproduce it. You think it's gone like they did initially, and then it's back. Like cancer.

View on HN

It feels like this comment was almost a purely additive anecdote of your own experience with a similar kind of issue, but you've spoiled it by deciding to tell the author that they're incorrect about how they felt during the process?

Maybe different people find different things fun.

Not saying he's wrong, sometimes the word "fun" connotes something slightly different what what it literally means. "Satisfying" is something I'd use for the end state. Maybe "challenging" for the intermediate state. But while you're in a high-pressure situation that you don't understand, that is rarely "fun" in the literal sense.

You wouldn't pay to be given compiler race condition bugs, right?

Maybe stop digging here and just let it be fun for the author?

> Not saying he's wrong

https://heinen.dev/ - I’m Thea “Teddy” Heinen (she/her or they/them)!

I like these bugs. They’re intricate, technical puzzles, that can take weeks to figure out. You need a proper strategy to figure them out, cannot rely on simple tactics, and when you finally understand what’s going on, it’s immensely satisfying.

This, and now there’s pernosco which makes everything much easier.

Now, under pressure, this is going to be a nightmare unless you have a high tolerance to stress.

I wouldn't pay to be given any kind of work, but there are some aspects of my job that I find more or less 'fun'.

Hunting bugs that people have given up on or have no ideas on how to tackle is near the top of that list.

Some people are perverse individuals and actually enjoy debugging very esoteric things. What might be frustrating to you might be the very thing that gets someone else very excited.

Probably just meant satisfying instead of fun. I found a bug in sscanf for the gcc arm toolchain that ships with Ubuntu (and Debian), and it wasn't fun since I had deadlines to deal with. Workaround was to use the official ARM one. But after 2 days, it was satisfying to nail the exact problem and write a regression test.

> I'm sure it was a relief to find a thorough solution that addressed the root cause. But it doesn't seem plausible that it was fun while it was unexplained. When I have this kind of bug it eats my whole attention.

Yeah, and that's fun for me. Some of my most fun bugs to debug have been compiler, or even CPU issues.

It becomes fun when you narrow down to the solution. Before that it's hell.

I don't think I'd be allowed spend weeks to debug something like this. Credit to Cloudflare's PMs.

Apparently they have a "unexplained crashes must have an explanation determined" policy ever since there was a trend of uninvestigated unexplained crashes that were canaries in the mine for a security issue.

https://blog.cloudflare.com/however-improbable-the-story-of-...

> But [the Cloudbleed sensitive information disclosure security incident] wasn’t the only consequence of the bug. Sometimes it could lead to an invalid memory read, causing the NGINX process to crash, and we had metrics showing these crashes in the weeks leading up to the discovery of Cloudbleed. So one of the measures we took to prevent such a problem happening again was to require that every crash be investigated in detail.

Since then, they have a "no crashes go uninvestigated" policy, which for the scale Cloudflare operates at, seems pretty impressive.

Yes, and we set up all the tooling for that and I would look at the output every single day and keep an eye on what was happening. Any team that didn't fix a crash quickly got a personal message from me. That responsibility has been taken over by others now.

The people who find the fun are often good at identifying when it is the standard library or the compiler.

I find this sort of thing to be tremendously fun. It can be frustrating as well, but overall it’s my favorite part of my job. I don’t see why this would be implausible. Different people enjoy different things.

> Devs have been taught from a young age that it's always you, not the tools you were given, and that's generally true.

That's not been my experience at all FWIW. Tools get things wrong all the time.

Simply that more mature projects with heavy use like eg; gcc or clang/llvm generally tend to have had major bugs stamped out by this point. They do still happen though.

More nascent language and compiler ecosystems are more likely to run into issues. Especially languages with runtimes.

Hey; it could've been type-3 fun.

Segfaults with no use of “Unsafe” equivalents in managed languages can give immediate indication it’s not a code problem.

They explicitly mention there was usage of unsafe, and they weren't sure that's not the cause.

Right, in a 3rd-party library.

Although I’m good enough at it, like you I hate this kind of debugging experience, and try hard to avoid putting myself in a position where I have to do it. It’s not fun for me at all.

I also don’t like many puzzle games, like Sudoku, because to me they feel like this kind of work. Many colleagues of mine have expressed bafflement that I don’t find such puzzles fun and give me all kinds of grief about how I ought to enjoy them, since they do.

It’s the same thing here, just flipped around: this person seems to enjoy the debugging experience; just let them be. Or recruit them, because that temperament is valuable.

Reply on news.ycombinator.com