One hour long component of a (now deprecated) SRE interview loop was for the candidate to SSH into a series of EC2 instances and debug issues which got progressively harder as the interview wore on.
I had wasted a substantial amount of time on the first and easiest problem by really overthinking it, and not trying the simplest of debugging techniques first. By the time I got to the final, hardest problem, I had just over 5 minutes remaining. The interviewer gave me a pass to skip it, but I was having fun, and really wanted to take a crack at it.
The final problem was to try to figure out why logging into a particular machine with SSH was slow. While I sat waiting for a prompt, I had a number of thoughts. Is a reverse DNS lookup timing out? Is there a huge i/o load on the machine? Am I going to have to wire up strace to `sshd` and log in again?
When I finally get to a shell prompt, I instinctively run `w` and it just hangs. I hit ^C, `strace` it, and discover that it's blocking on:
fcntl64(5, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) ...
I look up a bit, and discover that file descriptor 5 is /var/run/utmp. So `w` is trying to get an advisory lock on utmp and failing. Then it hits me, `sshd` is likely also trying to acquire a lock on utmp, failing, and then eventually timing out.A little bit later, I've found and killed the the rogue program that held the lock, and SSH logins were fast again. Solving that last problem so quickly really boosted my spirits, and gave me the energy to push through the harder interviews that came later in the day.
Thanks w!