One specific incident comes to mind. I was working on Heroes of Newerth, a DotA clone. We achieved a bit of popularity in those days, and we reached around 50k concurrent users. So our servers tended to melt.
One day, the servers went south, and although games were being completed, none of the stats were being recorded. (Think of it like, the database was in read only mode.)
I remember it feeling a little surreal how they decided to handle it. They kept cool, looked into the problem, and got the servers back up within a couple hours. Then they went back to their usual jobs and didn’t seem bothered by the ticking time bomb.
I think in my usual 22yo fashion, I said that it was dumb that no one seemed to care that the servers were going down, and that we had to do better for the players.
What I should have done was ask questions. Should we try to figure out the root cause, and make sure this doesn’t happen again? Or was it really not a big deal?
A few hours’ downtime may or may not have been a big deal. We did end up suffering from some pretty serious server malfunctions which limited our ability to scale. But I certainly mishandled how to approach it, and won no friends.
Every team is dysfunctional in their own way. It’s important to adapt to the dysfunction and push things in a positive direction, not merely to be right.