Hacker News Reader

Comment by staticassertion | Hacker News Reader

> There are bugs where you know exactly how to reproduce it, but it takes a long time (like 20 minutes or way longer) to reproduce the bug. This is hard because it’s hard to maintain your focus: maybe you can only try 1 experiment per day!

Ugh I hate these. It's almost worse than the "only happens sometimes". I'd rather walk down a longer path fast than a shorter path miserably slow.

> there’s no output at all

Hangs!! I hate hangs. I hate them. Crashes give information. Hangs could be so many things, but it is the absence of information.

> 4. one of your assumptions is wrong

#1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". So may "impossible" bugs I run into are in fact impossible... in the codebase I'm looking at. And it turns out what's deployed is some other code where it is very much possible.

View on HN

At one point, I encountered a low level bug that would only manifest if the computer was physically located in a particular room. It ended up being an undocumented cabling revision, but I revisited a lot of assumptions about the nature of software and reality before getting to that point.

I have come to love the feeling that the universe is broken when bug hunting. It means the solution is close at hand. If there are still lots of plausible explanations left then it means there is lots of work left to do. Things always seem most insane just before figuring it out.

And then you find it and you remember the sentence: “Everything that can fail, will fail”.

Thank you for this positivity! I'll try to remember this next time I am frustrated.

Sounds like the 500 mile email bug

https://web.mit.edu/jemorris/humor/500-miles

The thing I like about this story is that all the facts fit.

The user may be explaining something that is crazy to you initially, but the first step is to gather the facts, THEN try to figure it out.

So often, the part of troubleshooting is "How can I get more information about X?"

In this case, the seemingly unrelated step of telnet into SMTP gathered a few facts together which helped to explain it all.

I’d love to see a gallery of these kinds of bugs and the actual solutions. This was an awesome story— thanks for linking to it.

Also I didn’t know about the units cli program!

http://beza1e1.tuxen.de/lore/index.html

Units is a wonder. I use it pretty much daily. It's a shame units_cur is so fragile though.

"but I revisited a lot of assumptions about the nature of software and reality before getting to that point."

I encountered some bugs that felt like this (software only though), where I did not found the all explaining logical solution and ended up rewriting that part.

But I saved the whole state of the project and data in a zipped file and keep them locked away - so one day, when I feel like madness (and maybe have better tools avaiable) I will jump into it, either to proof my own stupidity - or the existence of dark magic, poltergeister and demons.

>#1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at".

Once I got so far as stepping through some malfunctioning code at ASM level on a production machine, only to discover that the ASM had a jump if equal instruction while the source code prescribed exactly the opposite. I compared to the executable binary that was supposed to be there, and it was different from what was actually there. There had been a bit flip somewhere along the line.

I now spend more time than I probably should comparing hashes of any executable binary that's acting strangely enough.

We had a fun one at my first job where the same Java webapp was behaving differently on a couple of servers (this is back when servers were administered manually by sysadmins). We did a recursive md5sum of the homedir on each server and got the same results.

Turns out that by default tomcat will use filesystem order when loading jars from its library directory, and some of the servers used different filesystems, so even though each one had the same set of jars in the library directory, they would be loaded in a different order which lead to different behaviour.

I worked at Microsoft back when they were looking at the Itanium as a Win64 platform. We were making a .Net console application and one day I was in the lab while our daily test automation was running, and the Itanium machines had a bunch of crazy random chars and colors on the console windows. It wasn't causing tests to fail, but I rolled up my sleeves and debugged the issue, thinking it was probably just some buggy test thing.

It turned out we were doing something nobody else was doing in .Net at the time, which was to return a structure by value while we were populating the console buffer. Apparently the JIT compiler lost track of which registers it was using to store the struct and overwrote them. In the disassembly it looked like "move some stuff into a register, then move more stuff into the same register". Thought I was losing it for a second.

I reckon the moral is don't get into the mindset that the test is always broken. Sometimes it's actually the product.

I've been there a couple of times.

Relevant: https://xkcd.com/1316/

The rise of reversible debugging for C/C++ made debugging much more enjoyable for me. For those tedious to reproduce bugs you can reproduce and record it once and replay it many times with the bonus of reverse-* operations. With rr the execution isn't even slowed that much.

Indeed. I actually quite like hung programs :) Attach a debugger, and now lessee what it is doing...

If anyone is complaining about a few hours here and there, relax. It took 7 years for me to solve an intermittent one once.

The issue turned out to be an immeasurably small thread synchronisation issue that went away when we bought new CPUs which were fast enough for it to not happen.

In the end this task was rewritten in a single threaded process as the CPUs are now fast enough to complete the work on one thread.

This is all well and good but for people like me, I generally need to fix a bug in a week before people start looking at me weird.

This hurts me to read

That's less than 1% of the pain I experienced there.

> #1 assumption I always make sure to check now is "is the code that's running actually from the source code I'm looking at". So may "impossible" bugs I run into are in fact impossible... in the codebase I'm looking at. And it turns out what's deployed is some other code where it is very much possible.

cough I spent two days doing this when testing different application servers for a Java servlet I was consulting on. Two days of banging my head and wondering why it works on that server and not on that server.

I was deploying the new .war to the wrong place and the server autodeploy was picking up the old version every time.

I've been doing this for over two decades.

> Hangs could be so many things, but it is the absence of information.

start adding information.

you need to know what the code is getting to and/or what its not getting to and progressively narrow it down. spamming debug print statements can be useful if you have no better ideas to get yourself moving.

or if you can manage to hook up an interrupt handler to be able to get it to dump a stack trace that'll immediately tell you.

or attach a debugger while its spinning in order to get a stack trace.

or attach something like strace to dump out system calls or ltrace to dump out libc calls. that can't catch processing spinning in "math" though, but that absence of information is a hint that eliminates lots of possibilities.

you may need to add a feature to the software in order to do this kind of debugging, so you write your own USR1 interrupt handler or something. it will be worth it.

i think the kids these days use stuff like systemtap?

One of the best pieces of programming advice I've ever received was from my undergrad advisor. "You'll never win a staring contest with the code." If you're in the middle of debugging and you find yourself staring at the code hoping that it will suddenly reveal answers, that's not a productive position to be in. In those cases, better to figure out what diagnostics can help, and to get more information.

>> there’s no output at all

> Hangs!! I hate hangs. I hate them. Crashes give information. Hangs could be so many things, but it is the absence of information.

variants that made me scream include:

- misleading information, the kind you can't dismiss as irrelevant, and you go down a thousand rabbit holes that seemingly lead nowhere, but it somehow all ties together into understanding the issue, only after a thousand turns and while keeping the holistic view of it all

- useless information. kind of like the one above but every piece of information you get truly leads nowhere

Once I had an actual crash that somehow happens before jumping into the program's code (like, before main), but changing your code does toggle the trigger... but when it crashes you have no core and can't attach a debugger, so you're left with debugging on the happy path and constantly think about what could possibly go wrong, and you don't even know what you're looking for. understanding why crt0/ld trips over is not fun.

The ones I really hate are the ones that disappear when run under a debugger, or with higher logging levels. Usually race conditions.

Ah, the dreaded heisenbug

The solution is simple, though: ship with the debugger.

…with the JVM debug port open, so you can unlock your customers when they are stuck.

Even works for software embedded on rockets - After all, network latency to the moon is only 2s, right?

Yep, GDB also regularly splits up TCP network packets, so beware of changing stream packet boundaries

> 20 minutes or way longer

One of my coworkers was working on a bug that only manifested itself after ~2 weeks of putting the system under continuous stress. Bad times.

Stress tests often seem to unearth strange bugs. Once when we used to sell boxes running a single-threaded real-time OS, the OS vendor gave us a separate machine that would sample the registers and memory of the box 10000 times a second (using jumper cables on the motherboard), in the hope that we would catch the exact state of the system as it crashed after days of stress testing. The sampler machine was a crazy overpowered beast of a machine with really high clock CPUs and oodles of RAM so it could dump as many samples as possible into a ring buffer in memory.

> the code that's running actually from the source code I'm looking at

Super common when working with Clojure(script)! Bites me in the ass all the time.

Speaking of clojurescript and bad assumptions:

Clojure and clojurescript are both dynamically typed, but clojure is strongly typed (type errors throw exceptions), and clojurescript is weakly typed (type errors are technically valid code). Too many clojure enthusiasts act like they're both the same language, but that is an absolutely massive difference.

Try the following code out in each: (+ "1" 1). In clojure you get an exception, in clojurescript you get "11" (maybe a warning at compile time which won't show it's face at runtime when the data is fed to you from an API call).

That bug silently corrupted analytics data for 3 months for a service I worked on. That alone was the reason I stopped using dynamically typed languages.

> That alone was the reason I stopped using dynamically typed languages.

Nitpick, but weakly typed not dynamically typed is the problem here. Python would throw an exception (though I know mostly prefer JS with TypeScript to Python nowadays.)

Yes, in this particular instance that would be correct. But tracking down that bug made me realize something. A well tested program, fed improperly typed inputs, will fail in different places depending on it's type discipline.

With the weakly typed language, it will fail anywhere within your codebase and won't tell you that it is doing so.

With the strong/dynamic types, it will still fail anywhere, and while it will fail loudly, it still leaves you with the work of tracking down why. You won't know that it is due to improperly typed IO until you've traced that data flow all the way back from the point of the failure. For large apps, this can be a nightmare.

With a static/strong typing discipline, type errors can only occur in one place: exactly at the point where objects are constructed during IO.

And let's say that that upstream API provider (that you have no control over) silently switched to bignum strings instead of floats, your strong type system caught the problem immediately, and now you have to accommodate. You change the object, and get a stream of type errors in your IDE, and you go around fixing them all until they disappear. Almost like magic, your program works again. No new tests, no running multiple times until you track down all the type errors. With a dynamically typed language you'll have no such luck.

Frustratingly many statically typed languages are not very strongly typed since they don't force you to check for null values. Kotlin/Swift/Rust really are a huge improvement over Java/Go/C#.

At it's best TypeScript feels very similar to programming in one of these OCaml like languages, though you're still vulnerable to errors in untyped dependencies or people on your project abusing the `any` escape hatch.

(+ 1 “1”) is not a type error for CLJS, because it’s not a type error for JS. Clojure is designed as a hosted language and it shares all the basic types and operations with the host platform.

That's perfectly fine. In fact, it would probably have too much overhead to try to build a dynamic/strong runtime on top of javascript, so adopting the platform's type discipline was probably a wise choice.

However, the type discipline is a fundamental aspect of the definition of a programming language, and you can't exactly call it the same language if it has a completely different type discipline on different platforms. And it would be nice if clojure advocates stopped lying to people about how it is the same language. Be transparent about how different it really is, even if it looks the same on the surface.

That’s quite bad. It basically means that Clojure and clojurescript are different languages. So you can’t blindly share client/server code.

It took me some time but I have a good workflow for debugging hangs/deadlocks/etc. If you know what part of the code produces it, you iterate over it indefinitely in a debugger until it hangs, then once you notice the iteration has stopped you “step in” to the debugger. Then you run another script that dumps the current trace back for each existing thread. That should be enough to detect the lock normally.

The big problem is if you don’t know how to reproduce the issue in a debugger for some reason.

Last time I had a "silent crash" heisenbug that produced no output, I solved it by system-level tracing (ETW, Event Tracing for Windows). It turned out not to be a crash at all! We had a script in our test harness that cleaned up stray processes, and that script sometimes managed to run long enough that it caught the start of the next test, and killed the tested executables before they managed to output anything.

Your last point on the correct code running, ugh. I haven't gotten better at it even thought I'm aware of it more now. Had an incredibly persistent linting config issue just last week, turns out I had the wrong config open for editing in vim.

I also hate hangs. If you are working with the JVM you can send it a signal kill -3 [pid] which can give you a thread dump. That's a QUIT signal if you are curious. I once found a stuck HTTP request when I did this, so I fixed the issue by adding a timeout to the HTTP connection object.

When you are designing a system or a program you should have a way to get diagnostic information to make it easier to fix issues that can arise.

Always design in timeouts. And always make them configurable.

Yes, I agree.

JVM also has jstack and other tools like jprofd to introspect running vms.

First day of electrical engineering classes we were taught, before troubleshooting a circuit - make sure you're plugged in.

Assumptions are evil :)

Reply on news.ycombinator.com