Hacker News Reader

Comment by zer00eyz | Hacker News Reader

HA! This is my favorite interview question to ask candidates:

"What is your all time biggest screw up, and how did you come back from it" - I then tell them the story of me loosing several hundred thousand dollars and the funny things that happened around it to set the tone. If you have been in tech for any length of time you have one of these stories (if not a few). I have heard some great ones by simply asking and it gives great insight into a candidate (humor, stress response, the things you have seen).

View on HN

i was a college kid at my first "computer job" doing desktop support for a client in downtown Fort Worth in a fairly nice office suite. Remember those old AT power switches? I wired one backwards, turned it on, and it blew all the insulation off the wires in the case and caught on fire. All in a very nice office, fortunately no sprinkles or fire alarm but I was very embarrassed and nearly cost my boss the client.

EDIT: I once got a call from junior developer who had issued an sql statement on an auto-commit database like "update <table> set x=y; where <some condition>; she fat fingered the semicolon and blew up the whole table. She was crying, I felt pretty bad we were able to get the table restored and back online in a few hours. She still tells me how sorry she was to this day ( about 15 years later ).

You can tell from your grammar you pronounce 'sql' in your head as S-Q-L and not sequel.

Wow. I had to read that statement a couple of time to understand exactly what you had inferred. That’s ... really neat.

It flowed just fine for me because I pronounce it S-Q-L in my head as well. I don't know why saying SEQUAL has always felt unnatural to me. Probably because I came across it independently and learned about it by myself instead of having someone else introduce it to me as SEQUAL before seeing the acronym.

And how does that several hundred thousand dollar story go? (if you have time, and care to elaborate, that is) I'm curious.

You have to apply for a job to find out

Me too. I just want to hear the "funny" part of losing that much money. I think the story would be a bit more somber, something along the lines of "...and that's how I ended up blacklisted from the Financial industry, and living in a van down by the river."

I was swapping tables on a live production server,* and accidentally removed all rows on the production table rather than the auxiliary one. With autocommit on. I immediately confessed to my boss, and luckily, there was a backup.

However, I never did what I saw other people do at another job (more than once), which was to run a delete on a live production table intentionally, but unintentionally leave off the "where" clause. And commit without thinking.

I'm not aware of anything I've screwed up that had a dollar figure attached.

*I am not and never have been a sysadmin per se.

You didn't remove rows, you had an unscheduled test of your restore procedures.

I've done something similar before, forgot to disable autocommit in DataGrip and ran an UPDATE without a where clause - thankfully we take weekly full backups, nightly incrementals and archive log segments so I shut down the database, performed a point-in-time-restore and the damage was undone.

This was actually a good teaching moment, said database is used as a holding area for incoming data from hospitals before we load it into our billing system so our EDI team has limited write permissions on the database - it's much harder for them to run a "oops, time to grab a backup" query but it was a nice anecdote to use when telling them to run everything in a transaction if they're running anything other than SELECT statements.

    [13:45] core-router# debug all

    *** Session Terminated ***

...oops.

You're not a NetEng until you've done this!

I've done it more than once, so I must be great :)

I've done this once, and only once.

What valuable skill or quality does this show?

1. How are you at dealing with stress? 2. How are you at dealing with the instinctual desire to hide your mistakes, and instead immediately take steps to remediate them?

Honesty and the ability to own up to your mistakes. It can also show arrogance depending on their response.

I always ask potential coworkers during an interview for a story like that for two reasons: 1) it'll tell me whether or not they have humility, honesty, etc; 2) and, because I love sharing war stories.

One candidate we interviewed answered my question saying that he had never made a mistake like that. Then he went on to tell me a story about how a bad patch from Oracle which he applied brought down production this one time.

The guy seemed arrogant, and I felt like he was lying or he is overstating his skills. Either way, he ended up getting hired by another team (knowing full well that we didn't like him), and he only stayed for about a month or two.

Go figure.

In addition to what others have said about honesty and ownership of mistakes, it shows you how much they're capable of learning from the inevitable mistakes.

Even more importantly, if they haven't made any major mistakes yet, there's a good chance they'll make it once you hire them, because they haven't been through the whole experience and have no scars to show for it. Or maybe they haven't had yet a job where they were responsible for anything that could result in a major incident. There's no lesson in paranoia as great as wiping the wrong storage array, shutting down services in an order other than the sanctioned one, etc.

I am looking for a few things:

Did you own the mistake. (responsibility)

What steps did you take to resolve it. (accountablity and ownership)

Are you able to laugh about it, or did you learn something from it. I would hope that you can make claim to one or the other. (personality)

Can you tell a story. Because the story is personal, it is topic where you have "mastery" and should demonstrate how you communicate. (communication)

An understanding of your professional background - most of these stories have other "players" in them that help move them along and tell me who you have/had to interact with in the past. (experience)

Remember I give my own example that not only hits these points but probably illicit a laugh our two out of the candidate, and remind them that I am not any different from them. I tend to be able to have more conversational interviews after this question, and get better responses and answers from candidates.

Reply on news.ycombinator.com