How to fix bugs in 24 hours or less
dolthub.com
dolthub.com
Saying that your developers will solve every problem in 24 hours seems like a toxic pr move.
"How to fix bugs in 24h", LOL.
Sounds like a hell of a story.
At some point it was discovered that every once in a while, bots that were supposed to just go back and forth the entire game world forever would get stuck. It was immediately obvious that they were getting stuck at machine boundaries (the big game world was split into a grid, and different machines would run the simulation for different parts of the grid). This suggested the bug was in the very non-trivial code that handled entity migration between machines.
This was a nightmare to debug. Distributed logging isn't fun. Bugs in distributed systems have a tendency to be heisenbugs. We could reproduce the bug more or less reliably, but sometimes it took hours of running the simulation until it manifested; worse, not manifesting for a few hours wasn't a clear signal that the bug had been fixed.
My investigations were broad and deep. I looked at the Kryo serialization protocols at the byte level. I scrutinized the Akka code we were using for messaging. I rewrote bits and pieces of the migration code in the hope it would fix the bug. Many other engineers also looked at all this and found nothing. A Principal Engineer became convinced this had to be a bug in Scala's implementation of Map. I was very close to giving up multiple times.
At some point there was a breakthrough -- another engineer discovered a workaround. A violent but effective one: flushing every cache and other bits of internal state except the ground truth would get the entities unstuck. We added a button to the debug world viewer appropriately labelled YOLO RESYNC. We were so desperate about this bug, we seriously discussed triggering a YOLO RESYNC periodically.
But if YOLO RESYNC fixed the issue, it meant that there was some sort of problem with the state of the system. I spent some more days and weeks diffing the state before and after YOLO RESYNC (more difficult than it sounds in a not-entirely-deterministic distributed simulation) and narrowed it down more and more until I finally found a very subtle bug in our pubsub implementation. I don't remember exactly what the issue was, but there was some sort of optimization to prevent a message from being sent to a recipient under certain conditions that would "guarantee" the recipient would have gotten the message in some other way -- and the condition was very subtly buggy. Fixing it was an one- or two-line change.
I still remember the JIRA ticket: ENG-168. It tested my sanity and my resilience for longer and harder that anything else before or after.
[EDIT] I saved this ticket as a PDF as a traumatic memory. It was in January 2015 so I got some details wrong, the main one being that it only took about two weeks from the bug report (Jan 28) and the fix (Feb 10). I swear it felt like 3 months.
Thanks for sharing!
Some times bugs aren't so much a bug as the anthill.
Note that a bug fixed in 24hrs is also a bug that doesn't have to be fixed later. I mean the development work has to be done at some point anyway, and this may even save some time discussing and bouncing around the issue.
Some huge factors are:
- how fast you can test things (e.g. change and re-run test)
- how good runtime (e.g. debugger) and logging visibility you have
- how good (representative) are the test data
In other words, fast ways to "look into" the matter. Usually if you can iterate fast and get a reproduction, you can also solve it fast.
It's much less common that the issue is a big architectural concern that needs total rethinking/refactoring.
I've worked at several companies you know (https://www.linkedin.com/in/macneale) - and this is the least toxic company I have ever worked at. Hands down. We take pride in running a tight ship.
Until we outsourced code production to a country halfway round the globe. Completely changed the dynamics from solving interesting problems in nicely written code, to chasing people around you've never met and never will to fix bugs in 1000+ lines functions, while you know it's an exercise in futility as no doubt there will be lots and lots more where that came from.
"But they're cheaper!"
"Just fix the bugs!"
Etc...
It's. Just. Not. Possible to convince anyone that it's not like hiring a recent immigrant to clean your yard. Sure, their English might not be so good, but they can hold a broom, right? No big deal! Worst case, if they miss a spot, you can just point to it and they'll fix it.
The destruction of code quality is an invisible, insidious thing that can end billion-dollar companies while management whistles all the way to the bank collecting their bonus cheques for yet another awesome quarter.
Assuming that wasn’t the point all along.
On top of all that, if not managed carefully, the 24-hour guarantee could perversely incentivize customers to classify all requests as "bugs" to get faster service, blurring the line with genuine feature requests.
From the article it sounds like a completely different game than what you mention, which I also experienced at times.
It also should come with a culture of treating bugs like you do a production incident: You do retrospectives and figure out what you can change to your process to reduce the likelihood of the same type of bug happening again. You likely also have a more extensive suite of tests to prevent regressions.
But if you make these sorts of promises you have to make sacrifices.
I think it’s refreshing to see this kind of thing though.
I guess it's preference, but I'd rather ship quickly and cover most of my users, and respond to any bugs that do surface (and there's every chance they won't) for obscure issues that probably won't really impact revenue.
I wonder why the rest of the industry continues to produce so many bugs, didn’t they figure this out?
They may also be in a position where they can afford to delay releasing a feature until all bugs are fixed. Most teams on the other hand ship with known bugs, because a feature with bugs today will make more money than a feature without bugs tomorrow.
It looks likes all of their bugs are labeled with "Cirtical" priority. It sounds really weird.