6,991 karma · joined August 12, 2013
Actor / filmmaker: http://gabrielgambetta.biz
Location: Zürich, Switzerland
Remote: Yes
Willing to relocate: No
Technologies: C++, Python, Java, JavaScript, HTML, Django, SQL, PHP
CV: 7 years at Google, 4 years at Improbable, taught Computer Graphics, published a book, founded and ran a gamedev company for 10 years. More details at https://www.linkedin.com/in/gabrielgambetta
Email: gabriel.gambetta@gmail.com
I'm a generalist, full-stack in the original sense of the term. I've done a little bit of everything in the last 30 years. I left Google a few months ago and I'm looking for something to do -- it's one of these "I'll know it when I see it" situations.Ideally looking for something I haven't done before -- I'm curious about AR/VR, robots, drones, fintech. I'm not particularly interested in AI, blockchain or web3. I can write code, I can write for humans, I can lead teams, I can teach and mentor.
If you're working on something interesting, let's have a chat!
[1] https://gabrielgambetta.com/client-server-game-architecture....
I'll take present-day Ubuntu any day. I install it on my desktop, and it just works, GPU and all. I install it on my weird laptop with a touchscreen that swivels 360 degrees and can rotate to a portrait desktop, and it also just works.
Put it another way, both (10.00, 10.01) and (10.00, 20.00) map to (gold, silver), despite being qualitatively very different.
#define STACKTRACE GLOBAL_STACK_FILE[GLOBAL_STACK_IDX] = __FILE__; GLOBAL_STACK_LINE[GLOBAL_STACK_IDX++] = __LINE__; StackTraceCleaner stc;
StackTraceCleaner was a class that didn't do anything but execute GLOBAL_STACK_IDX-- in its destructor.So at any point in time I could inspect GLOBAL_STACK_FILE and GLOBAL_STACK_LINE and have a complete stack trace of the game.
Obviously this only worked because these games weren't performance-critical and because they were essentially single-threaded, but it did the job at the time. We're talking about a time when Visual Studio 6's support for templates was half-broken, and the STL wasn't exactly S, to the point that I had to roll out my own string, smart pointers, containers, etc -- made twice as hard because of the aforementioned broken template support in VS6 :(
I do miss these simpler, more innocent times, though.
I wonder how much of the spaghettiness BASIC is infamous for is due to the language, and how much is about the programmer's mental models -- my recent raytracer for the ZX Spectrum [0] is a pretty straightforward transliteration of my JS ones.
At some point it was discovered that every once in a while, bots that were supposed to just go back and forth the entire game world forever would get stuck. It was immediately obvious that they were getting stuck at machine boundaries (the big game world was split into a grid, and different machines would run the simulation for different parts of the grid). This suggested the bug was in the very non-trivial code that handled entity migration between machines.
This was a nightmare to debug. Distributed logging isn't fun. Bugs in distributed systems have a tendency to be heisenbugs. We could reproduce the bug more or less reliably, but sometimes it took hours of running the simulation until it manifested; worse, not manifesting for a few hours wasn't a clear signal that the bug had been fixed.
My investigations were broad and deep. I looked at the Kryo serialization protocols at the byte level. I scrutinized the Akka code we were using for messaging. I rewrote bits and pieces of the migration code in the hope it would fix the bug. Many other engineers also looked at all this and found nothing. A Principal Engineer became convinced this had to be a bug in Scala's implementation of Map. I was very close to giving up multiple times.
At some point there was a breakthrough -- another engineer discovered a workaround. A violent but effective one: flushing every cache and other bits of internal state except the ground truth would get the entities unstuck. We added a button to the debug world viewer appropriately labelled YOLO RESYNC. We were so desperate about this bug, we seriously discussed triggering a YOLO RESYNC periodically.
But if YOLO RESYNC fixed the issue, it meant that there was some sort of problem with the state of the system. I spent some more days and weeks diffing the state before and after YOLO RESYNC (more difficult than it sounds in a not-entirely-deterministic distributed simulation) and narrowed it down more and more until I finally found a very subtle bug in our pubsub implementation. I don't remember exactly what the issue was, but there was some sort of optimization to prevent a message from being sent to a recipient under certain conditions that would "guarantee" the recipient would have gotten the message in some other way -- and the condition was very subtly buggy. Fixing it was an one- or two-line change.
I still remember the JIRA ticket: ENG-168. It tested my sanity and my resilience for longer and harder that anything else before or after.
[EDIT] I saved this ticket as a PDF as a traumatic memory. It was in January 2015 so I got some details wrong, the main one being that it only took about two weeks from the bug report (Jan 28) and the fix (Feb 10). I swear it felt like 3 months.
"How to fix bugs in 24h", LOL.