07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
Ouch
Ouch
Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/
Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha.
I would be very interested to hear more about how the break-glass process worked.
Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits).
Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue.
I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)