Hacker News Reader

Comment by kabdib | Hacker News Reader

One of my favorite questions is: "What's the best bug you've ever found?"

Usually I get a "Huh?"

The really stellar folks will tell you about the bug that took two weeks to find and boiled down to a single missing comma in a vendor's library routine. Or /something/. But "Huh" is a bad sign.

"Huh?" is not a fail, but it usually correlates with people who can't write a function to find the length of a string, and I see a depressing number of those folks, even ones with the tag "Senior" in their title.

Hiring strangers is terrifying.

View on HN

One of my favorite questions is: "What's the best bug you've ever found?"

One of the problems I always have with questions like that is this: I don't keep mental lists of events or other things ordered by potential "interestingness" to myself or other people. I just don't think that way for some reason.

I might fumble to come up with an answer to that question, and then an hour after the interview is over, I'll remember a really good story the interviewer would have liked, but in the meantime I've looked like an inexperienced, bumbling fool.

Not that it really matters, of course--eventually I'll get through an interview without hitting a question that doesn't work for me, get hired, and then actually get to do some good work for somebody. Getting through that process feels fairly random to me.

Same here. To compensate, I load my memory up with "interesting" anecdotes for interviews. Including my most embarrassing moment, which luckily happens to center around a bug. (Failing that, I'd probably grab a recent bug and dress it up to be interesting. I guess you can always riff about techniques which prevent that particular class of bugs.)

But now of course, if I ever go on an interview again, I'd dredge up some funny bug. Just in case. :)

That was my first thought, too. I've had some bugs that actually make good stories, if told right... But I had a lot of other stories that are better. The only halfway interesting bug I can think of right now is actually pretty lame, and the rest of the story that goes with it is halfway interesting.

There's always a danger when picking out certain criteria and claiming they mean something. In this case, 'ability to remember an interesting bug' does not actually select for good programmers. It just happens to overlap somewhat.

Best bug I helped find was a flaw in some FPGA code for a special DAQ board for a muon detector I was building. When the board was in a temperature above about 74F, the FPGA would often (but not always) fail to initiate some ADCs correctly because some delays would cause part of the circuit to go out of sync, and when it was below, it always worked fine. So, testing in the office worked fine, but running the board outside wit a slightly different setup in the summer would mean a failure, but not always.

The horrible part was after all this, I had to wire up a bunch of NIM modules instead with all the special logic (coincidence/anti-coincidence triggers + a bunch of other stuff so I could have separate binned energy counters) and then calibration manually because the engineer for that board was busy with other things to fix the bug.

I would have loved to tell this story at an interview for a job, except I could almost never get past HR drones, probably because my degree said physics and not CS or EE.

heh cool story...

Compiler bugs are always satisfying to find. "It's never the compiler", but sometimes it is: http://llvm.org/bugs/show_bug.cgi?id=12419

But I think my best one was when I found a bug in a third party communication library, that only manifested itself on ARM architecture. Our CPU was PPC, and our simulator was x86. On both of those, unaligned memory reads work fine. But the actual unit we'd talk to had an ARM CPU. On ARM it just reads from the closest(?) aligned address. I found that by looking at network logs and reading the source (which we thankfully had).

Funny - I remember when one of our new Engineers needed a memory-copy method for our ARM embedded solution - he went to Linux source and got some library routine.

It faulted when I used it the 1st time. Fixed the bug (alignment of source), ran again and it faulted again.

So I spent 10 minutes writing a test - move 0-128 bytes from source buffer offset 0-128 to destination buffer offset 0-128. Simple, overkill right?

11 bugs later the damned memory copy thing worked. 11.

The next thing to ask is, What did I learn from that bug? What I learned is, accept NO CODE as bug-free, no matter the source, no matter what authoritative base it came from.

Other learning: why oh why don't CPU designers put a damned memory-copy instruction into the machine? We all need it, all the time, for every project and we all hack something together that works until it doesn't. Sigh.

> why oh why don't CPU designers put a damned memory-copy instruction into the machine?

x86 has had REP MOVSB since forever, complete with a directional flag so you can handle the cases where the source and destination regions overlap. But it went out of favor since for a while from 80386 to early Pentium processors (when Linux was written), REP MOVSx was slower than writing an explicit memcopy loop.

That said, such an instruction would seem to go against RISC philosophies, where you want your operations to be small and atomic and predictable in terms of time and resource consumption.

Right! Foolish programmer! Using REP MOVSB has been broken since about the 2nd issuance of the processor. Dumb folks (read: DOS) used it as a timing loop to calibrate interrupt timers, complained when it got faster and broke their code so Intel 'dumbed it down' til its about the worst way to move memory you could try.

So you say it works again. Cool!

Maybe what we really need is some sort of 'architecture library' that compilers resort to for things like this. Maybe an instruction, maybe a routine, but guaranteed to work for every wrinkle in the architecture.

Because if its not in the compiler, folks will continue to cobble together buggy code of their own, with only a vague idea of the vast architecture landscape they are navigating blindly.

You can't express memcpy in hardware any more efficiently than you can in C because of the way memory controllers work. It'd end up being microcoded, and ARM can't afford that for the same reason it can't afford unaligned access.

I think x86 does have a microcoded memcpy (rep stos) but efficiency varies.

You mention the memory controller; that's probably where the logic belongs, not on the processor. So the microcode would come down to "ask mc to move; wait for completion"

That's not actually any better speed-wise. And the CPU would still have to microcode the copy because of caches (think of what involvement the memory controller has in doing a cache to cache copy)

The real gain in being able to have the memory controller do a memcpy() independent of the CPU would be to let the CPU operate on data out of its caches in parallel to the memcpy() being executed. But that only helps for a very specific class of memcpy() and is highly system dependent (you have to worry about the expense of keeping caches coherent among other things.) Anyway, an integrated GPU or other additional block of hardware behind the memory controller is a better candidate for this sort of thing than a user-level CPU instruction.

Also, use a better libc.

...I was tempted to relay the story of the best bug I ever found, and realized that it literally boils down to exactly what you said-a missing comma in a vendor's library. Wow.

What made it fun was that it was a SAS program (which I knew) with extremely complex, 5-layer deep macros that called VBScript (which I did not know) all over the place. Debugging stuff in a language you don't know under a tight deadline-that's a thrill!

Oh boy.

I seem to remember the

  (member 1 '(1 2 3)) -> t
  (equal '(1 2) '(1 2)) -> t
  (member '(1 2) '('(1 2) '(3 4))) -> nil

bug. It took me just three hours, but boy was it annoying. In the end, it was just me not knowing the language of course.

I can read just enough of that that it confuses me why it doesn't work. Care to explain?

The problem is, that

  (equal 1 1) -> t
  (equal '(1 2) '(1 2)) -> t

  (eq 1 1) -> t
  (eq '(1 2) '(1 2)) -> nil

because eq tests for the same symbol, while equal test for the same list. Member uses eq, not equal.

Ah, I see. Thanks.

If we are talking Common Lisp, the default test is actually EQL. ((eq 1 1) is not necessarily T).

I'd love to answer that question, so I will:

It was a production bug where a customer's birthday was rejected as invalid. This was in Germany, which is relevant because it turned out to depend on the machine's timezone. You see, when Java parses a date, it computes all fields of a Calendar object and then does a sanity check on them. Some smartass had decided in Java 1.4 to have that reject daylight savings time offsets greater than 1 hour. Unfortunately, Berlin and the Soviet-occupied part of Germany had a 2 hour daylight savings time offset in the summer of 1945, because that corresponded to Moscow time.

I don't remember specific bugs in general. I think I try to block out the memories...

Several:

1. debugging an options calculation on a spark workstation for a Merrill Lynch trader who was incognito and on his honeymoon. A certain options valuation table he was using had expired and needed to be updated. Ad interim his position went haywire. I didn't have the exact source, so the debugger was off by one line, then two lines, then three lines. But I found where the table was used and was able to patch and recompile the code.

2. This dates me somewhat (recruiters consider me too old) but I was bitten by the 80286 POPF bug, which caused the interrupt disable mask to be ignored in an interrupt driver I had written for a Motorola 6852 (if I recall). I found the bug but the client announced it was going out of business (I wasn't responsible).

The version I got in an interview was, "What's your favorite algorithm?"

I stalled 30 seconds, pulled Lottery Scheduling out of my ass, and proceeded to do pretty well on the interview, then be told that they wouldn't hire me because they could tell I'd go to graduate school.

If you're ok with someone pulling an interesting story from memory rather than actually computing a total ordering on bugs or algorithms for "best", these questions work very well.

How do you know that those who tell you about such a bug are stellar folk, and those who don't aren't?

I think the question is great to see if somebody is on the same wavelength as oneself and probably a good conversation starter. I'm not sure if it helps with distinguishing between good and not so good programmers.

Reply on news.ycombinator.com