One reason it was so hard to pin down is that it was impossible to tell when the bug actually happened -- all of the cases we had were essentially "hey something bad happened in the last ten hours and now my quest is broken" (5/18)
Depending on how early they realized it was limited to companions on the ship, with a specific state (the source is not clear on how early they determined that to be) they could log events that damage a player when in that specific state.
That would have gotten them to fall damage, at which point they might be able to make the leap to ladders. If not, they could then focus their logging around changes in elevation when in that state.
But that's backward reasoning from the already known end state, so it's not fair to apply in hindsight. It's also not clear how much log data they can ingest, whether end-user telemetry was feasible, etc...
Turns out the flaw in the logic was that the NPCs can reach great height on the ship. But the flaw in the logic just as easily could have been in the combat. (Or some other way that the tweeter didn't mention).
In any case, it's really fun to read the "aha! got it" moment.
> The only place in the game when a companion is present but not in the active party is when the player is on their ship
> Eventually we figured out that "undamageable" does not mean "invulnerable" -- they can't take damage from attacks but can still get hurt from other things
> One of those things: falling a great distance
> The problem with that is that there are no spots in the player's ship that are high enough to result in a lethal fall
> So now we had to figure out how companions were mysteriously ending up way above the level
> I looked into tons of theories
This does look like logging would have caught it. They knew exactly what they were looking for: companions reaching strange heights while the player is on their ship.
[1] https://llvm.org/docs/XRay.html#flight-data-recorder-mode
Many bug reporting systems already work this way, such as the one used by Firefox, I think.
Just because something can be done by violating user privacy, that doesn't mean it's the only way, or even that alternatives are difficult.
The problem is that devs reeeeally need a mix of very specific data for debugging, and very aggregated data for prioritization... There's all kinds of cool differential privacy tools for doing things like this, but no way for users to know or trust that they do what's claimed on the tin.
I've been floating the idea of setting up some log shipper plugins for Unity/Unreal to make a turnkey solution you could drop in and self host.
But it's difficult to police good telemetry and easy to change silently. Bad actors have continually given justifiable reasons for data collection, sometimes even with a pinky-swear that they'll not use it for evil. But the terms can be updated by the party collecting the data at any time they desire, with little recourse for the user other than to stop using that product or service.
So no, I'm not worked up over that kind of telemetry, but I'm still going to block it. It doesn't make sense not to.
If it were opt-in, it could be used well however, similar to what the Telltale games had done.
65% of people made choice A. 35% of people made choice B.
They have a sort of proxy for that with Achievements however.