amessina1

I don't have deep knowledge of details of compute networks, there is a team of TSE who deal with network cases who know more than me. But the whole point of troubleshooting is not knowing what is wrong, but being able to find what is wrong. In order to do that you need good basis, and those you can make by studying how networks and linux systems work (someone here posted some titles) and with experience (I have some grey hair myself). But every time you troubleshoot something you end up touching something you don't know, and that's where you learn something new you might use next time. For example I didn't know about dropwatch, a colleague suggested it to me.

During the interview process at Google we don't expect candidates to be able to get to this level of depth, but we try to hire candidates that could, over time and depending on their skill set, potentially reach a similar level of depth and ability to troubleshoot cases.

Author of the article here. I only thought of the possibility of making a blog post after the case was closed and I started telling my colleagues about it, and realized I would have loved to read about this.

The case was indeed fun to work with, but the main reason why it had such a fast and happy resolution was because the customer was very responsive and very cooperative.

I cannot talk for every Technical Solution Engineer, but I can tell you that I have no particular interest in simply closing a ticket: I want to go down the rabbit hole and solve technical issues, and I know many of my colleagues feel the same.

I am also far from being the most senior or skilled TSE in Google Cloud Support, I just wrote an article about one of most interesting cases I had.

For Google Cloud, 250$ per month per user: https://cloud.google.com/support

I totally agree, but keep in mind that it is an iterative process: you have a case, you apply the runbook/playbook you have, if they are not enough you use your skill/knowledge to solve the case, then you update the playbooks.

Follow the sun has a lot of overhead: imagine having to dump the engineers' thoughts and current hypothesis and load them in the brain of the next oncallers. Also it only really works if also the customer is active 24/7, as often you might need the customer to perform some action on their systems. Once the time pressure is off you might get better results dedicating the engineer who is best suited to work on the case (both from the point of view of the timezone and skill set) and give them the time needed to troubleshoot the issue.

1) you cannot have a runbook for everything, and even if you have a runbook you the best you could have found in this case is that something weird was happening in the VM. The setting had an insanely big value but it was accepted by the kernel, so you would assume it was a valid one. 2) the customer provided a huge amount of details, but it is usually very hard to explain what did you change from the base image. Most customers might not be willing to provide the full spec of their running system, as they might contain information they don't want to disclose. It is easier when the issue starts right after a change has been made, but this was not the case.