Ask HN: What was the worst bug you've ever solved ?
What is the worst bug (the software kind) that you've encountered in your hacking career ?
The testing worked well on the emulator so I decided to test it over the public network to an actual handset. Only I forgot to advance a recordset through which I was looping, so the code never hit the end of recordset condition. It took me some time to notice there was a problem...
The fact that I crippled a national SMS network for a few hours was bad.
The fact that my company had to pay for each SMS, wiping out out profit for that month was worse.
The fact the handset was mine and on my first date with a girl later that evening (whom I later married) my handset kept beeping with incoming text messages (about 96,000 if I remember) was the ultimate.
The handset didn't have a silent-no vibrate function (either it beeped or it vibrated or it did both) and the SMS inbox filled up after 200 or so messages meant it took days for the inbox to fill up, me to clear it message by message, then fill up again ad nauseam.
Still, I laugh about it now...
Then again $60 is only about 1200 messages.
You must not be in the US..
Long story short: a nasty race condition meant that it was over-ordering duplicate products from the suppliers to the tune of tens of thousands of dollars per day.
On the general theme, my most frightening software experience was when I met a guy who was the star programmer for a company doing controllers for elevators. I got talking to him, and he showed me some code. It took me about 15 seconds to identify an edge case that would engage the motor while the elevator was at the top floor (thereby attempting to pull the elevator into the roof?). It took me a further hour to explain what the edge case was. The bug wasn't that scary, as I'm sure there are hardware failsafes, but the general dimness of the guy writing software to control lifts was scary.
I started taking the stairs after that.
My job was as an assistant to the inventory guy, a 70 year old feisty man with one hand named Al.
I was often bored when Al would tell me to 'go hide somewhere' so I wrote some software to help him manage the inventory system. The power engineers in charge saw this and after a few small programming assignments had me work on updating the newly installed SCADA control system. This was a specialized programming environment that controlled all the power in the town.
We were setting it up to buy power from the local college during the yearly 'peaks' in August, thus reducing our yearly electrical bill by potentially millions of dollars.
After a month of working on it and incrementally adding my changes I screwed up. I knew this when I submitted a change and all the alarms went off at the substation.
Half the towns power was out. I got it back on after an hour, and nobody called with ventilator issues, so I think there was no real harm done.
The engineer in charge of the department laughed it off when he saw my apprehension about the situation. He said in the grand scheme of things they have made far bigger mistakes than that, probably referring to the blown transformer a couple months earlier.
I was given the ticket and found a "SLEEP 10" (causing a 10 second pause for each item) in the 10 year old BASIC code, put in by the original programmer for debugging purposes, and never removed before it was promoted.
I removed the "SLEEP 10" and run time went from 12 hours to 23 seconds.
The users loved me, but my boss was not pleased. He said, "You should have changed it to a SLEEP 5 so we had something else to give them the next time they complained."
I half-seriously suggested adding FPU detection and intentional delay loops if no FPU was present.
I was fixing an account balance on a customer's master database. Any change that happens on the master gets replicated to 30 branches, usually at 10 minute intervals.
I wrote the UPDATE statement, highlighted it, and pressed "Execute". Unfortunately, I didn't select the WHERE clause, so I basically gave all their customers (85000) a 50c credit balance. The IDE I was using also had a bug that caused it to ignore the auto-commit setting (which was turned off), so it basically committed the transaction. First I tried a ROLLBACK, which failed obviously. Realizing I had to act quickly, I disconnected the network cable (I was working on the DBMS server) to stop replication. I extracted the transaction log from the current database into a textfile (a few hundred MB) and restored the database from the most recent backup. Then I basically ran the extracted transaction log as SQL scripts against the database, hoping that it wouldn't fall over. I didn't want to do a normal restore because I was afraid of it showing up in the logs.
Within a very stressful 60 minutes I had everything back to normal. I never told a soul IRL about it.
UPDATE tablename SET xxx='newvalue' WHERE ALLRECORDS;
Or something to that effect.
After a few seconds I thought it was taking too much time only to realise I forgot one AND in the WHERE... I struggled with the backup but managed to get the data back just after the first call.
I first write out a select that will show me the records I'm about to modify, then replace the select + fields with the update. And I try to keep modifying live databases using interactive SQL to an absolute minimum.
Check three times, then hit enter.
Unfortunately, in my case, I ended up having to rebuild all the data from week-old backups and transaction logs.
I started a job recently at an ecommerce company. There was a long-standing bug with the cart display in the upper right of the page always saying that the cart was empty. People would report it all the time, and the quite smart lead programmer said it was something really complicated that he hadn't had time to investigate.
But eventually, after I sort of knew my way around the code, and when I finished up all the tasks on my to-do list, he handed me that as a why-not-investigate sort of thing. He didn't really have any idea, just that the previous long-gone coder had said it was complicated and in the depths of the way the front end code interacted with the order system.
So, I reproduce it, and look at the template code. These two lines, right next to each other:
[% cart_summ = ourdb.cart_summary %] [% IF cart_cumm.qty > 0 %]
Note that the two variables don't match. And this was broken on the site for _FOUR YEARS_. And nobody looked because someone said he had and it was hard, and nobody had time for a hard problem.
facepalm
Looking in the logs we could see that these users were somehow losing their authentication cookie and the application was correctly bouncing them to login. So how were they losing their cookie? Assuming it was a bug in the code we searched and searched to no avail.
Finally I discovered that the hardware load-balancer our CTO/'IT' guy' had insisted on was the culprit. The load balancer would buffer fragmented requests and re-assemble them before sending them on to the server. Unfortunatly the load balancer had a huge bug in its firmare.
If a user was using firefox, on windows, and their request was fragmented such that the first packet contained data up to the end of a header line including the \r but not the \n, so the next packet would start with a \n and then a header name, the load balancer would insert a \n\r between the two packets, thus effectively truncating the HTTP headers, usually before the cookie lines.
When I found this bug I couldn't believe that this was actually happening, I thought I was taking crazy pills, but you could run a sniff on the front and back side of the load balancer and see the request go in well formed and come out busted. We ditched the hardware load balancer and all was well.
I got it working fine on our desktop simulation, but running on the actual hardware it was consistently off. After extensive testing, I realized that it was a bug in the compiler for the target hardware, such that a very particular type of structure (something like, {int, char, float}) was being packed incorrectly, resulting in a 2-byte pad that shouldn't be there. If I reordered the structure elements, it was fine, but that particular grouping and order refused to work correctly.
It was GCC, so we could fix the compiler ourselves, right? Not really, as, for avionics systems the compiler has to be thoroughly qualified for avionics use, and changes equal requalification. I "fixed" it by storing the float as an array of characters, converting it to and from a real float type as we needed to use the data value.
Trivial, perhaps, but I was very excited to resolve the problem, after spending days barking up wrong trees. One usually expects that the problem is not in the compiler... :-)
Anyway, the standard way (to handle protocols) is to parse the thing not making any assumption of the struct layout.
If I am understanding what you are saying, we really couldn't do that, as the server sent and expected to receive binary blobs of data; the only way to know what was what was to have a map of where the data elements were.
As a (former) hardware engineer, I've worked on many projects where bugs have physical effects. This can range anywhere from amusing to seriously dangerous.
One such bug involved a mistake in the assembly diagram and silkscreen for a circuit board. The result was that a tantalum capacitor was installed backwards on a 12V supply rail.
Tantalum capacitors are polarized, and they fail in a spectacular way when reverse-biased. In this case, the supply rail could source upwards of 20A, so the fireworks were loud and impressive. Luckily the cap was easily replaced and the only permanent damage was cosmetic.
Hardest-to-troubleshoot bug:
In my subsequent return to the world of software, I worked on device drivers for network interfaces (among other things).
NICs frequently operate through a circularly-linked list of packet descriptors, which contain pointers to buffers in RAM where the NIC can DMA packet data. The hardware fills the DMA buffers and marks the descriptor as "used," and the driver chases the NIC around the ring, processing the packet data and marking the descriptors as free.
In testing, we discovered that under long periods (hours, usually) of heavy load, the system would occasionally freak out and stop processing packets. Sometime later, various software modules would crash.
Working backwards through the post-mortem data, I saw that the NIC would get "lost" and dump packet data all over system memory. I dumped the descriptor ring (tens-of-thousands of entries) and wrote some scripts to check it for consistency.
To make a very long story short, when the NIC was stormed with lots of 64b packets with no gaps, it would eventually screw up a DMA transfer and corrupt the "next" pointer in the descriptor ring. On the subsequent trip through the ring, the NIC would chase an errant pointer off into system memory and corrupt other system data structures.
Since hardware can DMA anywhere in RAM, the OS is powerless to stop it. The resulting errors can be ridiculously hard to track down and fix.
Here's a video of part of the result: http://www.youtube.com/watch?v=RUhLDtPnSuQ
- 2 holes in drywall
- 1 bent bookshelf
- 1 dent in concrete floor
- 1 frightened Jessica
- http://www.youtube.com/watch?v=qkenIInV9rI
Nice balancing by the way, smooth dampening.
Edit: Ah yes, now I see it, that's a small scope and a lab powersupply in the background, I estimated the size off by a lot.
I figured the whole thing was about a foot tall or so, sorry!
http://www.youtube.com/watch?v=b7i2KkYYulI
Damage: Blown tire, dented rim, looking like fools in front of our peers.
(a) The CAN bus (which reports wheel velocity and position of the steering wheel among other things) micro-controller hardware stopped sending interrupts which caused the main computer to think we were stopped. As far as I can tell this is just a straight up hardware bug with the Philips ARM chip we were using. This caused the accelerator controller to floor it because it was only seeing the last CAN message we ever got which happened to be zero velocity. Same thing with the steering (hence the big swerve).
or
(b) I failed to consider the contingency of not getting any CAN interrupts (either because the of the [very intermittent] hardware bug or because the connector got disconnected--turns out the symptoms are the same) and didn't have any code written to deal with it. Guess what I wrote that night. :-) Luckily I had all day to think about it while a new tire was fetched and by the time I figured out what was going on it took about 10 minutes to code the fix: a watchdog timer that shuts the world down if there are no inputs from the sensors for any reason.
It seems obvious in retrospect but when things are working you sometimes forget about weird failure cases.
That way you can avoid a paralysis of the control software until the vehicle has really come to a halt.
First, using perl a (later-fired) co-worker added a hardcoded check like the following:
if ($client_id = "specific_id") { #email reports }
Needless to say, we emailed reports for all of our clients to a specific client, didn't go over too well considering that many of them were competitors.. It was particularly bad because he had previously been talked to about flipping the constants to avoid the = vs == bug.
Second, possibly abused but not known for sure, was found a few years after initially being put out. Our webapp created a session ID for each user, MD5 hash.
Except it started like: StringBuffer md5HashedBuffer = new StringBuffer(userId);
Which, because the userId was an int, simply creates a string buffer of size userId, not a string buffer initially populated with userId.
The rest of the hash was added afterwards, then the one-time created, with the result that everybody's session id was the same. Changing your user id in the GET or POST would allow you to be logged in as a different user.
Unfortunately, I forgot to return the printer name to a variable when promoting into production. Hilarity ensued.
Guy I knew - awesomely good - hex edited a DOS boot sector on a in house machine to use FUCK.SYS instead of whatever.sys it normally is (I forget). He renamed that file to fuck.sys, rebooted and the machine ran. Cool!
We laughed and reinstalled DOS and two days later the boss come charging in yelling 'I have a client on the phone who says his new machine can't find FUCK.SYS!'
The awesome guy goes 'uh oh'. I laughed.
How about most widespread? Once while trying to debug a CPAN module I figured out that if $condition was false then Perl had a bug causing
my $foo = $bar if $condition;
to leave $foo with whatever value it had on the previous function call. (The exact behavior is more complex than that, but that's a good first approximation.) I then made the mistake of reporting this in a way that made it clear that
my $foo if 0;
was a static variable. Cue years of people like me trying to get the bug fixed and other people trying to keep it around. In the meantime EVERY large Perl codebase that I've looked in has had this idiom somewhere and it has caused hard to notice and reproduce bugs.
How about worst damage to a system? Due to a typo I once caused my employer to send the Bloomberg's ftp system every large file that it had ever sent. Since it sent a large file every day, this crashed their ftp server, meaning that a number of feed providers for Bloomberg didn't have their feeds update that day. I implemented no less than 3 fixes that day, any one of which would keep the same mistake from causing a problem in the future.
How about most (initially) bizarre? Debugging a webpage that a co-worker produced where, depending on the browser window size, you couldn't type into the form. The bug turned out to be a div that had been made invisible but left on top of the page. At the wrong window size it was in front of the form, so you couldn't click on the form elements behind it. (I figured this out by first recreating it with static HTML, then using binary search to figure out what parts of the page I could whack and still reproduce it until I had a minimal example. Then it was trivial to solve.)
Took a while before the phone rang if I wanted to have a look.
It turned out they had a little animated gif in there with the inter-frame interval set to 0, causing a divide by 0 in Explorer.
That gif was pretty much the last suspect on the list.
Divide & conquer until you are simply staring at the solution and still you don't see it...
At the time, Java was a new feature within that platform, so there weren't many apps that mized both languages.
After getting to this point, IBM joined in the fix effort, and we had daily conference calls, on which we always had IBM execs lurking because their 6.0 release of the platform was imminent, and this bug had the potential to wreak major havoc if not fixed before launch.
So I cannot personally claim to have done the actual bugfix - the IBM programmer did that. But it was a great learning experience to work together with IBM to find and fix it.
The worst one I ever caused was when Visa started carrying two amount fields in their credit card records. One was the amount in original currency, the other was converted to the receiving system's local currency. I used the original amount. Our hand-made test data used the same currency for both amounts, so no problem in test. Imagine my surprise when we went live and our system started posting original currency amounts to cardholder accounts, which at the time only supported US dollars. Luckily, we caught it early and senior management and cardholders were all good sports about it. I think those credit card statements with massive amounts became collector's items.
This also meant you couldn't use READ INTO for variable length records (which is equivalent to a READ followed by a MOVE) without taking some care.
As you say: "newbies shake their heads..."
It came down to me to find and fix the problem, and it was subtle. The clue lay in the fact that all of the sites that crashed did so within about a minute of one another.
Turns out that some of the old, old sections of the software had been written by the MD, who, despite referring to himself as 'the emperor of c', was in fact an atrocious programmer.
The actual trigger was the comms system looking at a byte that determined as to whether a message had been received. This byte was set to the character 'A' if a message was received. It just so happened that the first byte of the current value of the number of seconds since 1970 evaluated to 'A', and had been written into that memory location a negative index into an array that hadn't been initialised.
This negative index into an array that shouldn't have been empty caused a section of memory to be overwritten that made the comms system think that it had received a packet. This snowballed quickly, and took down the system within about five seconds of boot.
Took the best part of two days to track down, and, of course, it was everyone else's fault but the emperors.
As it turns out, the Poco library didn't read Steven's UNIX book all that closely, and they closed all of the file descriptors when turning a process into a daemon, instead of reopening them to point to /dev/null. So, standard output was closed, and its file descriptor was reused for the TCP socket. Of course, things like "cout" always assume that standard output is at a particular descriptor, so all the standard output from the program was getting written to the TCP socket.
Boy, that was confusing.
A little race condition in the code allowed one of the smaller solenoids to stay in a duty-cycled state, effectively turning the coil into a small space heater. Given the right play conditions and length of play, the coil could catch fire, and a couple of times it did. Lots of wood and plastic under glass made for a fun little display.
I heard one story about a unit in Paris being dragged out of a cafe and into the street, then put out with axes and buckets of water. Wish I had been there to see that.
Unfortunately, this seemed to happen quite often. Jobs would abort randomly, after about eight hours, sometimes much less and sometimes not at all. Overheating was the obvious first suspect, so that was investigated and ruled out. The hardware was running cool and was in perfect working order. The customer started splitting jobs into shorter batches and combining the results by hand. We wrote code to help them automate this workaround. But batches were still randomly aborting. And I couldn't replicate the bug, despite having identical hardware to the customer. Something in the customer's environment was essential.
Eventually, somebody at the customer figured out the problem. My boss called me up and said, "The customer suspects your code is not time-travel compliant." It was true. My code assumed that time always goes forwards. If time ever went backwards while my code was waiting for hardware, it would immediately time out and abort the batch. And our customer encountered a bug where time did appear to go backwards occasionally. I was too stressed out over other tasks to ask for details of the bug. I just sent them a fix and breathed a sigh of relief when they accepted it.
A Google search now reveals that there was an issue with time going backwards under Xen on dual-core Opterons, which is what the target platform was. They never told us they were using Xen. Maybe that's why they were much nicer to us after the problem was diagnosed!
Jasper L. systems administrator of an early web hosting company calls up one evening, there is a problem with the paging system.
A certain host is being paged as 'offline' but when checking the machine works fine.
So I go there bit by bit we check out the software, everything works fine, but sure enough every 10 minutes or so the machine (called 'chopper', I'll remember that for the rest of my life) gets reported as 'down' again.
But there is absolutely nothing wrong with it.
After ruling out all the software bits we figure it must be hardware somehow. The way the supervisor works is it sends a ping packet to the machine, and if the machine responds it is deemed to be up. But chopper misses one ping out of every 5 or so, and sometimes several in a row.
We swap out the computer, move the scsi drive to another box and boot it. Chopper registers as off-line, but works just fine.
More confusion, finally, out of desperation we start messing with the network. This is all 10Base T, coaxial cables with little T connectors on the machines and a terminator at the end of the line on the last 'T' to make sure the impedance is right.
The terminator was on, so that wasn't it.
We went for a break at that point, we'd been at it for hours. Finally, by elimination we figure the only things we haven't changed yet are the drive and the 'T' connector, but surely that can't be it.
And it was... somehow that T connector was a pretty good filter for some bits in the incoming or outgoing ping packet changing one of the bits, causing the IP checksum to fail. No returned ping... we replaced it back and forth 3 more times just to make sure we weren't seeing things.
And--even though I've strung cables for both kinds, and of course 10Base2 takes far less cable, the maintenance issues and troubleshooting headaches of the bus topology make me so glad we've progressed to more robust network topologies.
An ASP.NET app that didn't consume it's record sets before closing it's database connections, thereby breaking IIS's connection pooling and causing hundreds SQL server login/logouts per second. Lots of interesting side effects on that bug.
An obscure page latching problem on a high volume SQL server (hundreds of transactions & thousands of queries per second). The SQL server would spend all of it's time waiting on page latches, response time would grow exponentially. An eleven hour phone call with MS support finally identified the problem.
Oracle 10.2.0.[1234]
We need to have a word.
Example:
struct FooSort {
bool operator()(Foo const& a, Foo const& b) const {
- return not a < b ;
+ return b < a;
}
}; bool operator()(Foo const& a, Foo const& b) const {
return a.some_float_func() < b.some_float_func();
}
Can you elaborate more on this? Are you implying that two identical calculations will return different results?
double foo(double x) { return x*3.3; }
double bar1(double x) { return foo(x); }
double bar2(double x) { return foo(x); }
void t_bar(double x) { assert(bar1(x) == bar2(x)); }
Obviously there is a precision difference between the two numbers, enough to make a < b and b < a return true in a surprising number of cases. The way I ended up fixing it was by putting the result of the function call in a member variable in every struct, pre-computing all of the results, and comparing based on that value.
But in this case I was just adding some error checks, which weren't even being triggered. Clearly this shouldn't affect the results of our numerical calculations. Since my code shouldn't affect the calculations, I was convinced that our existing numerical code had a subtle memory or timing bug. (I knew that floating-point code was tricky, but clearly I was doing exactly the same operations on exactly the same values.) I spent days staring at code, and then my boss told me to stop working on it since the results were clearly correct in both versions, even if they weren't identical.
A few weeks later I read about how values change when they're copied out of the x87 stack into registers. And I thought, naw, we couldn't possibly be using x87 arithmetic. But we were. Which was horrifying, since floating-point calculations could be a bottleneck under some workloads. But we had been running that way since before I started working on it, so at least it wasn't my fault. I added a compiler option to request sse2 floating point instead of x87 floating point. Voila, predictable floating-point results, plus measurably faster performance on a few tests.
I really, really wish the STL had a mode you could compile it in which boiled down to, "Double check everything no matter how slow it makes things." Maybe it has it and I don't know it. But that was my top irritation with the little bit of C++ I've done.
I have no idea how a non-strict comparator can lead to memory corruption rather than just indeterminate sort order, but it is no fun tracking that down.
-It was a first gen PS2 game that had to get pushed out since it was considered a launch title. The dungeons in the game were seeded and randomly generated - In some instances, a lever to proceed to the next dungeon was made inaccessible due to the random seed that enclosed the lever with 'wall' tiles - the player either had to start the game over and hope the dungeon was seeded correctly, OR play through the game without finishing that particular dungeon. It was not total killer in the sense that the game could still be completed, but it was tough to know that such a glaring bug got through...
-Fix was made for the Korean version :) Basically a check to make sure that wall polys did not enclose any levers prior to generating the dungeon, if the lever was bounded, re-seed, generate and check again...
My incomplete set of unit tests didn't catch it because they were imported by seperate submodules of mine.
I only managed to find it by writing a unit test, going back in version control, searching for the first revision where the problem occurred, and going line by line through the changeset (luckily I commit early and often).
Right after I checked in my code (a completely unrelated feature), this ftp facility broke such that you couldnt ftp an image to any router on any platform. So I was assigned this bug which basically stalled the release on every platform that we supported. (The CTO himself called me.) Now you have to realise this BigCo supplies every cell phone carrier in the nation - Sprint, Verizon etc.
What I found was that the ftp system had a bug such that if the image being ftp-ed was an exact multiple of 8k (or something like that) then it would fail. My checkin made the image file an exact multiple of 8K. (Story of my life!)
I found the bug, emailed the CTO and he assigned someone from the core team to fix it. That guy calls me up and in the end I fixed it myself using vi and kibitz.
Two of the previous employees got into a debate on whether or not you could directly connect to the database through JavaScript and fetch recordsets, etc. To prove that you could, one to the developers did just that...and left it in place...in production...with the full connection string and user credentials in plain site in the javascript code on the page.
Countless sql injection holes were plugged over the years, but not before we got hit with an attack that plugged javascript ad code into 50 or more tables and a few hundred thousand rows.
The initial developers used ".inc" as the file extensions for include files like headers, footers and database access. These files sometimes had html, javascript, asp or all 3. You never knew. The problem was IIS treated these like text files and would gleefully serve them up if accessed them directly - revealing server side sourcecode, more connection strings, etc, etc.
Our accounting "system" was woefully inadequate. Our sales people always said "yes" and we eventually ran into some clients that wanted their invoices formatted and calculated in a way our system couldn't handle. We were always far too busy to take the time needed to properly engineer the new solution, so this one clients bill was done manually - through SQL Management Studio - every month, for close to two years before I left. Reconciliation of the bill, was also done manually. The kicker is, billing was done based on dates that assignments were completed. These dates were not only not locked down once billing had begun, but new assignments could be injected into the billing period long after the invoice was generated, because multiple factors. Needless to say, it took me half a day, once a month to reconcile their invoices and on several occasions, due to the very manual nature, payments were applied to the wrong assignments, transactions, etc. Once to the tune of $200,000. That was fun to fix. :)
I could go on and on and on and on....
Back in 1970, I was at my first programming job, working for a very small consulting company. The contract was to convert a large system from 1401 Autocoder to System/360 Cobol. However, instead of the source code for the current system, we were given system specs, which were almost flawless.
However, in final acceptance testing, one program, for one particular accounting line, kept producing results that differed from the current system. The customer would not accept this, even though my code implemented the specified calculation precisely.
Eventually, after a few rounds of back and forth, I demanded to see the source code of the current system, and after much more back and forth, the source listing was produced. Looking at the source, it seemed to be doing exactly what my code was doing.
There was one fact not taken into account - the present system, rather than running on the 1410 for which it was written, was running in emulation mode on their new 360. And they hadn't bought a license for the compiler. So the small accounting change they had made a few months prior had been made to the object code - the card deck itself. Luckily the programmer who had made the change had written a note to that effect on the listing - in the object code part, where he eventually sheepishly showed it to me. We then got paid.
(I also once had a bug that almost got national distribution via Time magazine. Luckily they had only printed a few thousand copies before someone noticed it. That one would have gotten me fired, but for the fact that the same release also included a halftone compression routine that enabled them to push back their photo deadline by a day. The reason the bug escaped my testing was that it only occurred in pictures where the line count was of the form 4n+3 - I had tested with both even and odd line counts, but never managed to use one that produced the off by one error.)
" #define INTERVAL 10 * 86400 "
read the above 'C' statement. Yeah, there are no brackets surrounding the '10*86400', INTERVAL was being used in some calculations.i wasted a lot of time debugging this crap.
One customer had intermittent communication failures that we couldn't identify for weeks. I finally went on-site and hooked up a serial port logger between the tablet and device. After an hour of testing, I finally captured what caused the communication failure. It was some debug message from the GUI app, somehow coming across the serial port. I called back to the office and had a guy from our group go over the the VB app team and ask which genius was logging stuff to the serial port. Sure enough, some guy had been using VB's remote debugging feature with a null modem cable and left it turned on in the shipped app.
Because there was so much output, I did a `| head` to keep it manageable. Saw what I was doing wrong, fixed it, reran the command, it seemed fine - but the output image looked exactly the same as it had before.
It took me about an hour to realise that once head had exited, the pipeline sent a signal to my program, killing it. The image wasn't getting written until after all the pixels had been processed, so it never got to that stage.
It had to do with a connection being reset in the space of time between the server checking if the connection was good, and actually using it.
The client could reproduce it at will, and after a little bit of code at home, I could, too. Back at the office, no repro.
It took a lot of back and forth (to put it mildly), but the problem at work was that the network was too slow - the server got the reset with plenty of time to "notice" it, and the crash didn't occur.
Once we set up the repro on its own switch at work, everything failed in the expected manner.
Turned out to be a race condition in an interrupt handler, where the OS would say "wake me up when something interesting happens" but something else would sneak in, clobber the wake-up trigger, which meant "... never."
Two weeks to find it, fixed by swapping two instructions in some assembly glue.
The harder a bug is to find, the simpler the fix is.
Change one line of code: $3.00
Find which line of code to change: $3,436.88I did this daemon in Perl, using fork() to search multiple websites in parallel. When a new search was initiated, the daemon broke the search in multiple packages (of 5000 items to be searched). After initiating 5000 searches (keeping the maximum number of active processes at 30), I did a wait for all the children to finish, so that I can mark the package as "done".
The trouble with fork is that opened sockets are sometimes not forked very well. And I forked my database connection (a DBD::Pg). Forking a DBD::Pg connection went well in my tests, but once in a few thousand forked processes, a process would stall. Sometimes it recovered after a few hours, sometimes it didn't.
I tried setting an ALARM that would make the process auto-kill itself after some time. Didn't work. The final workaround was to monitor the child processes from the parent, and kill -9 the ones that hanged. But the whole processing became too slow.
Finally I gave up on the idea of packages and on marking the search as "done". So I just marked the search as "done" after the last child was forked.
It was a difficult bug because the client had no idea it was a bug, and waited patiently for the results (sometimes as much as 2 days). It was also difficult because at first I had no idea why my processes hanged.
I was consulting for a multinational on a commodities trading app. I broke the stuff sending trade data to the Risk Management system that does automatic hedge trades. My manager comes down the hall and tells me I've caused $4 million in exposure in the past two hours. At the end, her comment is: "A million here, a million there, business as usual!"
I told that story to a former coworker, and he pointed out, "Well, your boss was calm about it. I bet those traders were livid!"
I must not state the accuracy of that price...
Needless to say we ended up with more robust load management code and tuned the output of some processes.
After some analysis with firebug, I just figured out that the variable's value wasn't being preserved after a particular point. So I just had to take it's value and assign it back to itself.
It's in this file: http://demos.daylife.com/api_tester/js/daylife.tester.js (lines prefixed with comments "IE Fix")
[Full disclosure: I was not and I'm not a part of Daylife. I only participated in a contest they once conducted and was asked to try solving their problem by one of their staff.]
> r = /^x$/g
/^x$/g
> r.exec("x")
x
> r.exec("x")
null
> r.exec("x")
x
> r.exec("x")
null
Second worst: linking c programs when some of the symbols are duplicated in more than one library yet are defined differently, Nobody mentioning this to the developer made it all the more exciting.
Bugs are a great learning experience.
I used to work on software for a very expensive (started at 70k) DNA/protein analysis hardware solution. This was in the late 90s, and our GUI was a couple million lines of MFC code. I was responsible for an analysis package written in cross platform (linux and windows) C++, but the main UI and all the complex interface was MFC.
My then boss was a unix idiot who hated Windows. Which is fine and all, but a Windows product was paying our salaries. So one day he decided to rewrite my (well tested, though not unit tested) file handling code written using the win32 API and port it to posix. Not to accomplish anything different, mind you, and not to add it to the cross platform bits since it was useless in isolation and, oh yeah, we had multiple millions of lines of MFC so nothing was ever getting ported.
In any case, during this "upgrade", he found a function called DirTouch, which was intended to make sure certain directories existed during the normal course of saving data. Well, this gentleman subtly changed the semantics from "create the directory if it doesn't exist" to "create the directory if it doesn't exist, but if it does already, then silently delete whatever is there". This wasn't the whole bug, but this change of semantics to a destructive function was the root cause.
This change got shipped. One of our customers killed literally more than $1.3MM worth of data, since each machine run might cost $100K given lab time, reagents, prep, etc.
But yeah, I initially took responsibility for the mess, since it was most likely my code that did it. After figuring out precisely what happened, and having to demonstrate to my boss that the old code did not delete stuff out of directories, that responsibility got walked back a bit. Still... I got "laid off" within 5 months of that.
Basically, there was blame to go around: the university should have had a comprehensive backup solution in place, particularly given the expense involved in creating the data; I should have been more careful to not needlessly call this directory touch operation while creating cached bits in the data files (the "data file" that the user thought of was actually a directory because we were running into fat16 and fat32 file size limits), but at the end of the day, my boss took code that used to work and turned it into code that not only didn't work but was broken in the worst way possible for no other reason than to fuck about with posix idiocy. In a giant win32/mfc app.