108 karma · joined July 13, 2016
But everytime I hear this story, I got to wonder who would pick up an object generating heat and not question why it's still hot hours later?
One way we have been trying to improve this is working with PagerDuty reporting and looking at the total amount of interruptions (not just pages but anytime PagerDuty reminds you for an alert/expired snooze/escalation) with the team. It's very easy to forget the oncall as you leave, but having more eyes on the shifts starts to bring awareness and lots of "why is that still broken" questions that are better answered at 10am vs 3am on a Sunday. I came from a large Operation Center so I know the pain of bad alerts, mostly cya stuff where it was put in place just to make sure the last guy can't get blamed. Sort of like adding 100's of random smoke detectors in a build without any fire suppression. The intention is good but the results are poor.
Outside of the meeting with the team, we also have proper handoff meetings with off call and on call, so they can share what's going on verbally instead of tagging the next person with the alerts. Makes it easier to share what's going on, any weird problems, notes. Also we're not using a 24/7 oncall coverage but 12/5 and 48/2 for the weekends, it's a small change but helps so much. The worst I ran was a 7/24 at a major email company and was paged every three hours, for the entire week. After that I knew the team didn't want to change and I needed to do something about it.
Also Notion seems to store code blocks without formatting them, I found that in Evernote, the backtick and single quotes would get formatted even inside a code block.
What we have seen is children will run up and hug the robot and dart around the base, which I would assume doesn't expect a human to be at this level. Another incident was a dog owner walking around the robot, but the dog went the other direction, leaving the leash between them. The robot started to run over the leash and almost drag the dog until the owner backtracked and pulled the leash free.