Hacker News Reader

Comment by rdtsc | Hacker News Reader

Other coders' nightmares:

* Security : customers acccounts leaked, identities stolen, bank account depleted

* Undetected data corruption : customers' data is corrupted and it has seeped into the backup stream, even the off-site one, and that is corrupted as well now.

* Rarely occurring simingly random concurrency related bugs that manifest themselves once a month or longer.

* Time : making assumptions about it, dealing with it distributed systems etc. Nobody thinks about it until it stops working or systems' time gets snapped back somehow.

View on HN

* Works on my machine: You can't repro a customer bug in-house. Something about your internal setup is making a difference. Even worse: You work for a big-corp that is a platform for said customers, and so god knows what another team has running in your environment in the name of "dogfood all the things"

Years ago, my wife had a bug that only happened in a non-debug build and only on one specific system. It was a recursive tree traversal algorithm, written in C, with an OB1 error in some pointer arithmetic. If memory was allocated in a particular order, the function would read off the end of level X's data and find level X+1's data, which was structured exactly as expected, so it would end up processing certain nodes twice. In the debug build there was a debug variable being initialized to null, which was in between the two blocks of data and therefore got interpreted as a terminator.

What a horrible coincidence. I'm surprised they found it.

I saw one once where the cell carrier actually made the difference. As in I had to fly to a foreign country before I was able to reproduce the bug.

Something like this happened a few months ago, only this time I was the user and I drove the person writing the system up the wall because initially he couldn't reproduce it.

After writing a test script and letting it sit in a loop the bug became reproducible: 1152 times got a replication.

https://github.com/spf13/hugo/issues/1293

What amazes me about CS - and this almost literally leaves me speechless - is how little interest there is in classifying and avoiding bugs, and working towards a General Theory of Why Stuff Might Not Work. Even a set of basic widely accepted heuristics would be good.

Instead of real observation, there's a lot of leaning on type theory, OOP, immutability, and other ideas that scratch some sort of neatness itch and/or advance the careers of opinionated language creators, but have little/no empirical support to prove they actually produce more reliable code.

Meanwhile formal techniques and languages that do have empirical support are sidelined and ignored.

Bugs like yours are not uncommon. If anything, because of Darwinian debugging - the obvious ones die first - you'd expect a good proportion of hard-to-reproduce bugs in any code base.

So they shouldn't be considered weird aberrations. I'm not sure why CS theory doesn't have a better handle on them.

Yes, this amazes me as well. Shallow bugs are many and are fixed quickly and easily so in the end the ones that remain are nasty and hard to fix.

Every anti-bug measure tends to work like yet another brush aimed at a wall. But most brushes overlap partially or even wholly with areas of the wall already painted with previous brushes. The only strategy that would really work would cover all of the wall in a very systematic fashion.

Time : making assumptions about it, dealing with it distributed systems etc. Nobody thinks about it until it stops working or systems' time gets snapped back somehow.

I once wrote an industrial control program, and used a time function that rolled over at midnight. To make matters worse, we never bothered to set the clocks on the computers, so it seemed like a completely random glitch, and was a devil to debug.

The one I'm dealing with currently. Sometimes half an array in perl is missing its data, but the program should terminate early if that data is missing. So it gets past one check and then disappears sometimes. Never locally in dev. Only in prod. And not always reproducible. Customer data is getting messed up sometimes, and the code paths don't support this possibility. frustration.

Kinda sounds like you've got a race. Are you using threads at all?

AnyEvent for concurrency; no threads aside from the fork that causes child workers to pick up a message. The validation is in the parent, the child uses the data. The race would have to be something mutating the array, but I can only find one place where mutation happens. That part is a tight for each loop that appends source data to the target array. I will likely have to add logging, but the issue is so infrequent that the extra logging will be a mess due to the volume of data this service handles (around a billion requests per day, and maybe between zero and 200 requests will face this issue). Something will work out eventually; the bug must be exterminated ;)

Oh man, I used to work on a large service that had to aggregate data from a bunch of downstream services and submit it upstream. At the end of the month, billing would run off the data and send out invoices -- tracking down subtle data corruption was nightmarish and cost me my youth.

I have one of those random concurrency bugs in my code right now - it takes a month to show up and so far not in the debug build. The code doesn't crash, the program just stops running as though manually paused.

Is this on a unix machine?

If so crash the program with a kill -9 set up to write the core to disk, then run gdb on the executable and reload that core file. Then run 'where'. This will tell you where you're hanging and how you got there as well as all the parameters to functions on the way down.

A ps -aux beforehand can give you another hint, look at the status letter.

Good luck!

I have a Linux version, but unfortunately the version that stops is the Windows version. At this point if I solve this it is going to be by chance.

Ai, in that case I have no bag-of-tricks for you, my last windows programming was about a decade ago and I don't miss it in the least. Much good luck, if you ever do figure out what caused it you should definitely do a write-up, there is probably a more general lesson to be learned from nailing that one down.

ps aux (the dash is for sysv syntax: ps -eif)

Aargh. Sorry about that. It's so easy to mess that one up. Gets me with tar all the time as well. Thanks for the correction.

This reminds me of the buggy pic-C compiler standard library I was using. Some functions calls with particular parameters (maybe the printf family) would halt program for no apparent reason.

Currently dealing with an intermittent concurrency related bug. It's horrible.

Have one that took six months to find.

The database person spent a month trying to figure out why there were entries with odd dates. Like one or two a month out of a few hundred thousand. Eventually figured out it wasn't his fault. His end was getting an occasional packet with a good crc and odd data. Most of the time the parser would kick those out. But a few were well formed enough to go through.

I chased it down eventually to a documentation error in the datasheet for the radio transceiver we were using. The datasheet very clearly stated that when a packet came in with a bad crc... nothing would happen. No end of packet interrupt. So I had to use a timer to detect a bad crc. (Okay whatever, I mean I hate those guys, but I've seen worse)

Turns out when the radio received a packet with a wrong/bad crc the radio silently reset the receiver and started waiting for a new packet. If a the radio received a packet before the timeout hammer came down then the driver would think the previously received packet was 'good'

Opps.

Was interesting because the symptom appeared at the farthest end of the system the tool graphing data for the customer and was caused by something at the other end. The error could have been literally anywhere in our system.

This is why i want to avoid mutable data as much as possible

Anything involving interrupts and timing. Especially when the timing is so critical that it is time to break out the logic analyzer. Bugs like that can keep you busy for weeks.

Let me add

* gcc dies with internal compiler error when you try to compile with debug symbols.

Oh, boy.

By this time I've killed gcc, javac, ghc, and the .Net compiler. Worse, the ghc bug was the only one I could track.

I'm so thankful that I work client-side and not server-side sometimes.

Reply on news.ycombinator.com