Thank god I checked one last time the environment variables before running into a disaster.
Have you been in a huge mistake involved? what happen?
I'd just taken over as "team lead" of the internal support desk for an interest rate swap trading system at a tier 1 bank. By team lead, I mean that I was nominally in charge of the one contractor who was vastly more experienced than me. She was also famously prickly, particularly with upstart novices such as myself. The lead role was considered a right of passage because of this.
Anyway, one morning she comes to me, ashen faced. She had just deleted the first leg of every single swap that the bank held. At that time, this would have been a relatively high proportion of every swap traded in history. We did have overnight backups but these took about 8 hours to bring online and the prod DB would need to be down for all that time. Such was Sybase back in the day.
Now, it just so happened that one of the operations team leads was a drinking buddy of mine. I called her and asked if there was any way that they could stay out of the system for the whole day. Bless her, she re-arranged schedules, got people to come back in later, pulled in favours from the US ops team etc etc.
In the end we managed the recovery and the day's trading was overlaid without any management escalation happening. Something close to a miracle.
Suffice to say that, from that day on, said contractor was the epitome of sweetness and light to both me and the ops team.
The company sent email alerts to customers whenever data changed. It turned out that a lot of people were not getting emails and they hadn't been for some months. The alerting system was written in Java (not my language), and was written by a part of the company in Argentina so not really anything to do with us.
The consultant was tasked with finding the bug, he stayed home to work on it so he could have no distractions.
I thought well I will look at code changes for the last few months, maybe I will notice something - and it turned out that the consultant tasked with finding the problem had some months ago made a change to the Java code dealing with alerting, which was weird. I went and looked at the before and after - the before was typical verbose Java code and I could sort of follow along and see it was for sending off alerts whenever the data changed ok. The new code seemed to just drop off into nothing halfway through.
So we were having a phone call with the consultant and he said well I am going to talk with 'head programmer in Argentina' in an hour to find out but can't figure out what is wrong.
And I Was like well I was looking through the code...
and you could see the consultant being somewhat annoyed at my chiming in because hey, what is Bryan doing looking at Java code he doesn't know crap about that language. I have to humor him, the big doofus. ugh. politeness sucks.
and I said so a could months ago somebody changed the code in this Java file here and their name was 'consultant name' and maybe that guy could help us figure out what the code is doing, because I don't think it will ever work.
And it turned out I was right and he was wrong and that was why many customers had not gotten their alerts for a couple months.
To pay for it he had to bring everybody pastries the next day.
About a week later I shipped a bug that made a bold text not bold on some types of documents. To pay for it I brought the biggest fanciest cake I could find, but I don't think anybody understood why.
I made a little Sinatra server to get some data from an oracle database and then put it into a Redis Queue to be analyzed by a bayesian classified - 80,000+ documents.
The first version of the code was a little repetitive so I put a main part into a function, checked it worked, went home for the weekend.
Turns out after my code change it was opening a new DB connection for every document. Someone else caught it and turned it off, wasn't that big a mistake anyway because not in production but still embarrassing.
One time working international standardization, I asked one of the people on the committee I was on a question, sent to the mail list of course, I got a long 3 page email in response. I started reading, all they were doing was trash talking one of the people on the committee, who I thought was also one of the more helpful people to be honest. So after skimming through and realizing they never answered my question I answered some nice little thank you thing.
However I had my outlook configured to send stuff automatically to the email list when it had to do with this stuff, and I didn't notice her thing at the bottom of the email "for obvious reasons please don't send this to the mailing list"