One Rogue User Took Down Our API
betterprogramming.pub
betterprogramming.pub
> Without any end in sight, that team in charge of the Admin service did what they thought was their best option: they revoked our API token.
> It didn’t take long before the user complaints came filing in and the status page was updated alerting users to the situation. When the team realized what our situation had spiraled into, we immediately jumped into disaster recovery mode.
This sounds like a total communication failure to me! They are two teams of the same company, but they behave like complete strangers!
The API team started flooding CDN Admin endpoint, but didn't let them know -- couldn't they pop in into team's channel and let them know? And it sounds like the CDN Admin did not even try to communicate to API team before revoking their token? And even once the token was revoked, they did not let API team know ... instead API team had to look at user complains (why? wasn't there monitoring?)
This is pretty sad story which shows an importance of internal communications in a company.
With that said, it definitely was a breakdown of communication, which is something we highlighted in our post mortem. We do have efficient ways of communicating, but due to the time of day, etc. the communication was very lack luster. They did tell us about removing the API token, but it was after they had already done it and we noticed the errors.
In terms of monitoring/user complaints. Some of this was poetic license. We were engaged with users during the partial outage and they alerted us when they noticed all their requests were failing and it had moved to a global outage. I just described it in a way that was better for story telling and removed some context so the article wasn't 15 mins long.