Also add what was your lesson learned!
It happened on my third week as a junior developer on a very nice startup company - its kinda my big dream to work in a startup.
It was a friday morning and I was just starting my day at work(I was working remotely) when suddenly one of the cofounders sent out an email that our website is timing out. So I checked out out nagios to see if the website is receiving a large amount of traffic and surprisingly I can even count the number of connections using my fingers. I was a newly hire back then and our lead developer is currently flying on his way back home. My other teammate is not yet online because he is in a different timezone and it is not yet his time to work. So basically I was the only developer available at the time. When my figure out that I have no idea of what is happening, he asked me to just shut down the server so that our customers will not be able to process erroneous transactions. The website is hosted on AWS EC2 and I cannot find our Amazon login credentials(its either I was too dumb at the time or too nervous because later I found out that our lead dev gave it to us a week before) then I decided to shutdown the sever through cli, you know shutdown -h now.
Now the other developer got online and asked me what happened. I told him everything then he decided to power up the server so that he can investigate the issue. He logged in to AWS console but he cannot find the server. It turns out that the server's shutdown behavior was set to terminate. And yes, I just destroyed/deleted the server that the website is using. To cut the story short, our lead developer came online and he rebuilt a new server. But still the timing out issue is still there. He found out that it was coming from a MySQL connection and the root cause was that select statement that is very slow. And guess who wrote that query. Yeah its me. A new release was just deployed the previous day and that query was used in one of the new features. The website became operational the following day and everything came back to normal.
The next day I became emotional and was depressed the following week that I handed down my resignation because I felt like I dont deserve to work for their company. They tried to talked me out on not leaving. The lead dev even said nice things to me(that Im a good coder and even him will write the same kind of select query if it was assigned to him). But my mind was too clouded and made a very poor judgment to pursue my resignation. And here I am now stuck on a corporate job trying to figure things out and getting my shit back together hoping someday I can work in a startup again and not f'up.
What you did right: You accepted responsibility. Good, this is a hard trait to find. The "perfect" engineer mixes equal parts intellegence and humility.
What you did wrong: You didn't trust your team. They were all very supportive of you, but you didn't trust their assesment of you.
Trust your team; trust yourself. Have confidence in your ability to learn from your mistakes.
You'll do fine. Start looking for another start-up job while working at a corp.
The best time to find a new job, is when you already have one.
As someone on the go, I thought it was a good idea to keep the source code for that app on my flash drive (there was no Github back then). For 6 months I worked directly on that flash drive, adding new features to support the large project, and expanding the abilities of the application to gain us even more favor. One day, I plugged in the flash drive and Windows gave the warning that it was corrupt and needed to be formatted. Immediately my heart sank, and the drive was indeed dead. My last backup was about 3 months old, and didn't even include some resources like icons and graphics.
Long story short, I had to sit there for weeks and re-code everything I'd lost, using the latest release as a reference to what was missing. On the plus side, my design was probably better the second time around, but nobody was pleased that any new releases would be delayed a month at least.
I now keep that flash drive, still in its corrupt state, as a permanent fixture on all the desks I've worked at since. It's a constant reminder to not be stupid when it comes to time-expensive intellectual property.
I worked at a data centre which had an IP KVM attached to all of their machines. When you were logged in as 'admin', there was a mode you could toggle that would send all of your keystrokes to every server, but still only displayed the one you were logged into, so there was no (clear?) visual indication that this was going to happen. Coworker hit Ctrl-Alt-Del to reboot a stuck server, and rebooted every non-Windows server in the data centre (and we only had one Windows server).
Every customer got some level of compensation, the noisy ones got a lot of it, and no one ever logged in as admin again other than to relabel servers in the server list.
When I worked for a financial institution my manager gave me a production level username and password to help me get through the mounds of red tape which usually prevented any real work from getting done. We were idealists at the time. Well I ended up typing that password wrong, more than 3 times...shit, I locked the account. Apparently half of production's apps were using this same account to access various parts of the network. Essentially, I brought down half our infrastructure in one afternoon.
Lesson learned:
Don't use the same account for half your production apps. Not really my fault :).
So.. I write a quick script to resize the master images and re-generate around 2,000 thumbnails. Except... I copy/paste the source path to destination - and I mistype 200px width as 20. Now we have a whole site with long thin product images and no originals to recover from! As in the linked story, no backups were in place and all work was done on production. Lost a weeks wages over that, and had to manually re-add everything from a stack of CD's :)
Lesson learned? Don't let pressure force you into making bad decisions. I knew I really shouldn't be doing that but I was young & foolish.
What EMR system do you deal with?
- Doing a port of PS2 -> Gamecube, one guy asks me 'do we need this assert?' I go 'nah it'll be fine'...cue a month later when we have a intermittent soak crash after several hours which I find out would have been caught instantly by the assert I said was ok to remove...took some time to find :/
Funny thing is, it got through a code review my own personal testing and QA testing.
Once the problem came to light it was a very obvious quick fix though.
At first maintaining and expanding functionality was not too hard. As time went by it became harder and harder.
The fix was to stop everything about a couple of years after the product was already shipping and take three months to re-write it in C. After that adding feature requests and improving functionality was an absolute breeze.