I'm not surprised that Twitter has those teams, I'm always surprised that more places don't. About 22 years ago I was the only person at the company I worked for at the time who could analyze Solaris core dumps, and understood enough about the JVM to diagnose deep problems. In the 5 years until the rest of the engineering staff caught up to where they stopped needing me for every incident, I probably saved enough money to pay my salary 100x over.

Never saw a dime of that. The one time someone offered me a spot bonus to come solve a problem, they reneged on it: A manager in charge of a project came to me for help. We're crashing all the time, he says. IBM and the team can't figure it out, he says. If you can solve this, I'll give you a $5000 spot bonus, he says.

I would have done it anyway, because it's my, you know, job? But whatever, I won't turn down free money.

So I wander over to the team that's been looking at this and get the lowdown. They keep getting out of memory errors.

Me: So what does the heapanalyzer output look like? Team: Huh?

Me: You...you've been having out of memory errors and haven't looked at the heap? Team: Buh?

So I get the heapdump and look at it. Immediately it's clear that the system is overflowing with http session objects.

Me: Anything in the log files related to sessions? Team: Just these messages about null pointer exceptions during session cleanup...do you think they're related somehow? Me: <Bangs head on desk>

A little more research reveals that there were two issues at play. The first is that we had a custom HttpSessionListener that was doing some cleanup when sessions were unbound. It would sometimes throw an exception. We were using IBM WAS, and it turned out that when a sessionDestroyed method threw an exception, WAS would abort all session cleanup. So we'd wind up in a cycle: the session cleanup thread would start, process a few sessions, hit one that threw an exception on cleanup, and which would abort cleaning up any other sessions.

We did a quick fix of wrapping all the code in the sessionDestroyed method with a blanket try/catch and logging the exception for later fixing, and IBM later released a patch for WAS that fixed the session cleanup code to continue even if sessionDestroyed threw an exception.

So, I very quickly solved this problem and waited for my $5000 spot bonus. And waited. And waited...

I went back to the manager and asked him about it. Over the next few weeks, he proceeded to tell me the following series of stories:

* It was in the works, and I'd have it soon.

* He had to get approval from his superiors.

* Because so many people had worked on the problem, it was decided that it should be split among the group, and that I'd have to share it with the people that couldn't fix it.

* No bonus.

So even though it was his idea to try to bribe me to fix a problem, they still failed to follow through on it.

Another story: We had an issue once where they finally brought me in after a year of problems. One of our Java systems was failing intermittently, and the development team had given up and couldn't figure out what was wrong. The boss told me it was now my problem, that I was to dedicate myself 100% of the time to solving the problem, and I could rewrite as much as much of the system as needed, basically total freedom (and responsibility). About halfway through the spiel where they were talking about the architecture and implementation, someone mentioned that the system was dumping core. I immediately stopped them right there.

Me: You realize that if it's a coredump, it's not our fault, right? Boss: Huh?

Me: If a Java program coredumps, it's either a bug in a 3rd party JNI library, a bug in the JVM, or a bug in the OS. What did the coredump show? Boss: Wha?

Me: You guys have had this problem for a year and haven't looked at the coredumps? Boss: Blurgh?

So I fire up dbx and take a look at the last few coredumps. Pretty much instantly I can see the problem is in a JDBC type 2 (JNI native code) driver for DB2. We contact IBM, and after a bunch of hemming and hawing they admit there's a problem that's fixed in the latest driver patch. We upgrade the driver and poof! the problem is gone.

We had a year of failures, causing problems for customers, as well as all the wasted man hours trying to fix something in our code that simply could not have been fixed that way, all because the main dev team for this product had no idea how to debug it. I had an answer within 30 minutes of being brought in to the problem, and the solution was deployed within days.