My $500M Mars rover mistake

Really well written story.

As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day.

Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had in place was awful and nothing like what we have today—just about every aspect of the process was manual. Anyway, everything was going well. I wrote a few scripts and SQL queries to automate the parts I could. They gave me the production credentials for when I’m ready to deploy. I decided to run what you could call my migration script one last time just to be sure I’m ready. The very moment after I hit the Enter key, I realized I had made a mistake: I had just updated the script with the production credentials just before I made the decision to do another test run. The errors started piling and their service was unresponsive. I was 100% sure I had just wiped their database and I was losing it internally. What saved me was that one of their guys had just a couple hours earlier completed a backup of their database in anticipation of the launch; in the end, they lost a tiny bit of data but most of it was recovered via the backup. Ever since then, “careful” is an extreme understatement when it comes to how I interact with database systems—and production systems in general. Never again.

I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process.

Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE.

I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of the ray of hope. That weight, I've never had a project that so many people depended on.

All heroes in my book.

This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the crucial variables was blank in production. I could have expected this if I had spent more time preparing and reading documentation. Sales died completely, and my heart sank. After a lengthy rollback procedure that resulted in a few hours without sales, a massive surge of angry customers, and a loss of several million euros, I approached the CEO of the company. I still remember catching him in an elevator. I explained that this incident was all my fault and I had failed to properly analyse the environment. I assured him that I was ready to bear full consequences, including being fired. He burst into laughter and said something like this: "Why would I want to get rid of you? You made a mistake that you'll never do again. You are better at your job than you were yesterday!" This experience was formative to me on many levels including true leadership. I successfully completed many high risk projects since than.

I work in TV. During my first job at a small market station 30 years ago, I was training to be a tape operator for a newscast. All the tapes for the show were in a giant stack. There were four playback VTRs. My job was to load each tape and cue it to a 1-second preroll. When a tape played and it was time to eject that tape, it was _very_ easy to lose your place and hit the eject button on the VTR that was currently being played on the air instead of the one that they just finished with. The fella who was training me did something very annoying, but it was effective: every time I went to hit the eject button, he would make a loud cautionary sound by sucking air through his closed teeth as if to tell me I was about to make a terrible mistake. I would hesitate, double check and triple check to make sure it was the right VTR, and then I would eject the tape. He made that sound every single time my finger went for the eject button. It really got on my nerves, but it was a very good way to condition me to be cautious. Our station had a policy: the first time you eject a tape on the air got you a day off without pay; the second time put you on probation; the third time was dismissal. I had several co-workers lose their jobs and wreck the newscast due to their chronic carelessness. Thanks to my annoying trainer, I learned to check, check again, and check again. I never ejected a tape on the air. It certainly would not have been a half-billion dollar mistake if I had, but at that point in my career it would have felt like it to me.

I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move.

However, I don't agree that this is the "real" lesson.

Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them custom test connectors that are incompatible with components that are liable to go up in smoke. That's the lesson. This isn't a little breadboard project they're dealing with, it's a vast project built by countless people in a government agency that has a reputation for formal procedures that are the source of great time, expense, and in some cases ridicule.

The "trust the 28 year old with the $500m robot that can go boom if they slip up" logic seems very peculiar.

I'll add my story here for posterity:

My first job out of university, I was working for a content marketing startup who's tech stack involved PHP and PerconaDB (MySQL). I was relatively inexperienced with PHP but had the false confidence of a new grad (didn't get a job for 6 months after graduating - so I was desperate to impress).

I was tasked with updating some feature flags that would turn on a new feature for all clients, except for those that explicitly wanted it off. These flags were stored in the database as integers (specifically values 4 and 5) in an array (as a string).

I decided to use the PHP function (array_reverse)[https://www.php.net/manual/en/function.array-reverse.php] to achieve the necessary goal. However, what I didn't know (and didn't read up on the documentation) is that, without the 2nd argument, it only reversed the values not the keys. This corrupted the database with the exact opposite of what was needed (somehow this went through QA just fine).

I found out about this hours later (used to commute ~3 hrs each way) and by that time, the senior leadership was involved (small startup). It was an easy fix - just a reverse script - but it highlighted many issues (QA, DB Backups not working etc.)

I distinctly remember (and appreciate) that the lead architect came up to me the next day and told me that it was rite of passage of working with PHP - a mistake that he too had made early in his career.

I ended up being fired (grew as an engineer and was better off for it) but in that moment and weeks after it, it definitely demoralized me.

The story is compellingly written, but I thought it was also confusing.

It sounds as if this team made several mistakes, not just one mistake. It's also not clear if the result of these mistakes was that there might be real damage to the spacecraft, or if the result was just wasted time and hours of confusion about why the spacecraft wouldn't start up.

The first mistake is they didn't realize that the multimeter was not only measuring, but it was also completing the circuit.

That sounds like a real bad idea. But if it was totally necessary to arrange it like that, then that multimeter should never have been touched.

That's not just one guy's error. It's at least two guys at fault, along with whoever is managing them, and whoever is in charge of the system that allows it.

The second mistake is with the break-out-box. They think he misdirected power wrongly into the spacecraft. Then they jump to the conclusion that has generated a power surge which has damaged the spacecraft, because it won't start up.

But they're not sure where the power surge went and what might be damaged. Anyhow they're wrong.

The reason the spacecraft won't start up is just because he took the multimeter out of the circuit before the accident.

I'm still sort of confused about what happened or if they ever really figured out what happened.

He said "Weeks of analyses followed on the RAT-Revolve motor H-bridge channel leading to detailed discussions of possible thin-film demetallization".

Does this mean that they decided that the misdirected power surge might have flowed into the RAT-Revolve motor H-bridge channel and damaged that?

I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover.

No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure involving accelerated fatigue life prediction, or stress corrosion cracking. My rushing of putting parts into production processes that didn't catch something before it went out the door.

It's interesting to read people's failure stories from similar fields but, to me, the ones that people so openly write about and get shared here on HN always come across as... well, workplace induced PTSD is not a competition. It's just therapy bills for some of us more than for others.

Reminds me of the NOAA-N Prime satellite that fell over, because there weren't enough bolts holding it to the test stand.

The roots cause, and someone correct me if this is not accurate, was that the x-ray tested bolts to hold it down were so expensive, that they had been "borrowed" to use on another project, and not returned, so that when the time came to flip the satellite into a horizontal position, it fell to the floor. Repairs cost $135M.

https://en.m.wikipedia.org/wiki/NOAA-19

Interesting to see that the worry could have been avoided if they had lined up their timelines better in the first place. If they'd compared the timestamp on the test readout to the last timestamp from the telemetry system, they'd have seen that the telemetry failed BEFORE the test was executed. Partially caused by using imprecise language "we seem to have lost all spacecraft telemetry just a bit ago" rather than an accurate timestamp.

A cautionary lesson in properly checking how exactly events are connected during an incident. Easy to look at two separate signals and assume they must be causal in a particular direction, when in reality it is the other way around.

It's a interesting story, but the author may be overselling it. It's not a _failure_ story, nor was it a 500M mistake. I get that it was really stressful and the mistake could have cost him the job, but it didn't; it also didn't cost NASA anything other than a few hours of work (which, during testing, I would guess it's expected).

When I'm asked to share failures, I'm usually not thinking about "that one time when I almost screwed up but everything was fine", instead, I'm thinking of when I actually did damage to the business and had to fix it somehow.

One thing about long aerospace missions like this with huge lead times that always gets me - you can spend years of your life working on a mission, only for it all to fail with potentially years until you can try again.

This is a refreshingly humanizing article, but is also one written from the perspective of a survivor. Imagine if the rover were actually lost. I asked the question "what would you do if the mission failed after all of this work? How could you cope?" to the folks at (now bankrupt) Masten Aerospace during a job interview, and maybe it was a bad time to ask such a question, but I didn't get the sense they knew either. "The best thing we can do is learn from failure," one of them told me. An excellent thing to do, but not exactly what I asked. This to me stands out as the defining personal risk of caring about your job and working in aerospace. Get too invested, and you may literally see your life's work go up in flames.

> And I still remember the shock when Project Manager Pete delivered the decision and the follow-on news: ‘These tests will continue. And Chris will continue to lead them as we have paid for his education. He’s the last person on Earth who would make this mistake again.’

I wonder whether Pete had followed this 1989 general aviation/accident analysis story:

> When he returned to the airfield Bob Hoover walked over to the man who had nearly caused his death and, according to the California Fullerton News-Tribune, said: "There isn’t a man alive who hasn’t made a mistake. But I’m positive you’ll never make this mistake again. That’s why I want to make sure that you’re the only one to refuel my plane tomorrow. I won’t let anyone else on the field touch it."

-- https://www.squawkpoint.com/2014/01/criticism/

(The incident above led to the creation and eventual mandated use of a new safety nozzle for refueling, which seems like a better long-term solution than having the people who've nearly killed you nearby to fuel your plane indefinitely: https://en.wikipedia.org/wiki/Bob_Hoover#Hoover_nozzle_and_H...)

If there is the possibility to make a mistake, somebody will certainly make it. You expect all the humans involved to be competent. But relying on that competence is a mistake. The emotional stress of dealing with such enormous responsibilities, the often long work hours and the long list of procedures will make any competent professional to inadvertently slip up at some point.

In case of electrical connectors, the connectors are often grouped together in such a way as to avoid making wrong connections. Connectors with different sizes, keying, gender, etc are chosen to make this happen. This precaution is taken at design time. JPL is extremely experienced in these matters. There is probably something else left unsaid, that led to this mistake being possible.

Meanwhile, motor controllers using H-bridge is something that's never boring. I once saw a motor control fail so spectacularly that we were scratching our heads for days afterwards. As always, a failure is never due to a single cause (due to careful design and redundancies). It's a chain of seemingly innocuous events with a disastrous final outcome. But the chain was so mind-bending that we had to write it down just to remember how it happened. Recently, I was watching the Chernobyl nuclear disaster and I got reminded of this failure. Our failure was nowhere near as disastrous - but the initial mistakes, the control system instability, the human intervention and the ultimate failure propagation were very similar in nature. Needless to say, it sent us back to the drawing board for a complete redesign. The robustness of the final design taught me the same lesson - failures are something you take advantage of.

Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site?

I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?).

But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis should be doable in 2 weeks, but what do I know?

Any test that could cause a fatal destructive error should be risk assesed and a suitable protocol approved with four eyes approval on a final checklist before going hot on the electrics. The issue here is poor project governance not human error.

$500M? Pocket change...

THE LITTLE VAX THAT COULD https://userpages.umbc.edu/~rostamia/misc/vax.html

I cost my company about 5 times my yearly salary once, long ago. I sampled an enormous amount of seismic data at 2ms instead of the proper 4ms. This was back when we rented IBM our mainframes for a pretty penny. The job ran for the entire weekend and Monday morning I was summoned by management, informed of my error, and asked, "You won't ever do that again will you?" and sent back to work.

Knowing that you are allowed to fail, but are very much expected/required to learn from your failure, makes for rather a good employee, in my experience.

> I was into my unofficial second shift having already logged 12 hours that Wednesday. Long workdays are a nominal scenario for the assembly and test phase.

Although the time pressure coming with the upcoming deadline is understandable, perhaps the bigger lessons here is that when you are possibly sleep-deprived, and have already pulled too long a shift, you are bound to make avoidable mistakes. And that is the last thing you want on a $500M mission with at limited flight window.

Reminds me of a quote attributed to Thomas J Watson:

Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?

I used to work in a rural manufacturing plant. I once totaled my car on the way to town. Our shop foreman was also a volunteer firefighter and was the first one on the scene. I survived with a bloody nose and shattered confidence. The next day at work we needed some parts from town. I didn't have a car anymore, but that same shop foreman lent me his suped up pickup truck. I was totally confused. He wisely said:

"Today is the safest day to let you drive my truck, cause I know you'll be extra careful"

Ohhhh I hate things that you “just have to not screw up.” A fiddly manual process with so many possible ways to screw something up is almost guaranteed to see a catastrophic failure.

If this really manual fiddly process was really the only way they could test the motors, I’d say that’s a big failure on the design engineer’s part.

Off-topic but I never realized how much kapton tape is used to put these things together, until I saw these internal 'guts' photos.

From reading the developer anecdotes in here I think it’s worth mentioning that if just one person can bring down the whole enterprise, a hacker only needs a point of entry to do the same.

For our databases we have separate credentials, compartmentalized access and disallowed “dangerous” commands. This now seems like obviousness, but we only got this years in. Thankfully, no (major) incidents have occurred to this date.

It's a beautiful story. As a space fan, especially of the interplanetary type, this story was riveting to me as it unveiled details of the spacecraft testing I've never imagined. I do far less important testing in my day to day but I was able to draw some similarities with the author.

Many have posted of their failures here so I suppose I could share a couple of mine.

- Pushing gigabytes of records into a Prod table only to realize the primary key was off by a digit, rendering the data useless for a go-live. It had to be deleted by the database admins and reloaded, which took precious hours. I forget why, but an update wasn't feasible.

- A perfect storm of systems issues that lead to all servers in the pool becoming unavailable, causing an entire critical system to go dark. We got it back up within minutes, but harrowing nonetheless.

- Realizing hours before a go-live that a key data element was missing, prompting a client who was now in a code freeze to make a change (they were quite upset). Pretty sure I got an unfavorable review from that, but haven't made the same mistake since.

I’m glad they didn’t fire him.

I’m a firm believer that despite all the short comings of US, what makes it great is there are millions of engineers and scientists working to push the frontier of what is possible, and trillions of dollars in economy to fund that into reality.

NASA is truly an inspiration.

And also the private aerospace industry - SpaceX, ULA, Boeing, Lockheed Martin, Blue Origin, Planet Labs etc.

No other country has that.

The fact this can so easily happen shows a lack of safety mechanisms. All the software-related stories in this comment section go in a similar direction: They could have been prevented by simple safety nets. If you accidentally wipe a production database, then it was likely too easy to do so.

Don't blame humans for occasional mistakes, it won't stop them from happening.

Hey neat - I once destroyed a $5000 prototype disk drive (I was a Firmware engineer at Quantum in the late 1990s at my first post-college job), also with back-EMF (it flowed from the braking spindle motor into a power chip and melted the circuit board). I felt terrible and that was multiple orders of magnitude cheaper. :)

With only two weeks left to go, why was a multimeter completing the telemetry circuit?

this sure seems like a symptom of a much larger organizational failure that could mistake "it's not plugged in" for "it's broken"

This clickbait title is misleading.

Thanks for sharing your experience. It reminded me of another one I'd read a while back by Andrew Latham: https://www.linkedin.com/pulse/what-i-learned-from-my-bigges...

Unrelated to the content of the article, but is the cover image AI generated? Looks kind of like DALL-E style output to me but it's hard to be sure.

(Also, I know I'm breaking the rule "Please don't pick the most provocative thing in an article or post to complain about in the thread." My defense is this is less of a complaint and more just plain curiosity!)

In my company people aren’t allowed to change org level GitHub settings without a second person watching over them, but NASA let a 28 year old kid run electricity into spacecraft without oversight?

Really seems like anything which when done improperly could cause millions of dollars in damage, there should be a second person reviewing your setup first.

Great write-up. My two favorite quotes:

> I had learned from countless experiences in this and other projects that bad news doesn’t get better with age

That's so true! We tend to sit on bad news and hope that somehow time will blunt them; but if anything the opposite happens.

And

> I still remember the shock when Project Manager Pete delivered the decision and the follow-on news: ‘These tests will continue. And Chris [the author] will continue to lead them as we have paid for his education. He’s the last person on Earth who would make this mistake again.’

We sometimes think people who made one mistake will make another one, and it's better to go with the person who doesn't make mistakes. But that's not the correct approach. People who don't make mistakes are often people who don't do anything.

So If I understand it correctly, the electrical connector was genderless? If so, that's relatively rare (the only ones I can think of are Anderson Powerpole, which I don't think are rated for interplanetary vehicles) and extremely stupid.

Edit: I suppose he could've been using alligator leads.

My friend reports his phone started dropping contacts methodically, until only one remained: a person he didn't know.

Googling him, found it was a junior dev at a FAANG. Oops.

An hour later the contacts repopulated, and all was well. But had to be a white-knuckle time for that poor shmuck.

There is a principle here that I can't see mentioned and that is how easy it is to discard something as a cause of a problem because it seems so minor, so routine (i.e. removing the "spare" multimeter) and therefore getting blinkered on what is going on.

There are more than a few times where I am scratching my head as to "how could my change have possibly broken this" only to remember a couple of hours later that I had made another change somewhere, or rebooted, or changed a config file temporarily.

I guess it just says that we all need to log everything we do, including removing spare multi-meters, so that by looking over the list we can remember these things.

> failure is not an option — it comes pre-installed

Love this. Whenever people say “failure is not an option”, I get the sense that they don’t really understand how the universe works. It’s like saying “entropy is not an option”. Uh...

Another failure is from the person who used the multimeter to complete a circuit - and then didn't even leave a note on the multimeter. That person could reasonably have anticipated this error mode, and taken steps to prevent it.

When I was 20 and in college, I used to work at a Barnes and Noble. When I was working the registers one day, I apparently forgot to put a $300 gift card in a lady's bag at checkout.

The store manager got a complaint about it like a month later, and she tracked down the gift card number from the receipt. It wound up getting loaded with even more money a few days later, and given to someone else.

Anyway, the whole incident got me fired. And to this day, I always check that I put gift cards into bags (on the rare occasion it comes up in my job as a software engineer for NASA missions).

I worked in a video startup thing and watching the app fail after people sunk like 30 mins into using it... Oh man that was cringe.

The videos were recorded because they were legal/financial related and also autotranscripted for text search. I would watch them to trace how the problem occured (I know there are many ways to log).

I was the only developer so it was my fault ha. The app had so many parts and I just used an E2E test to make sure everything generally worked. It's cool Chrome has a fake video feed (spinning green circle/beeping).

Fantastic story! One of the fascinating things about this are the parallels to Apollo 12 - Lightning strike / Power surge lead to a total loss of telemetry which raised the specter of losing the entire mission - and a similar resolution when telemetry was restored!

I've written a bit about it myself - https://flyingbarron.medium.com/lightning-strikes-92482387ca...

I worked at JPL for two years in college and helped with flight hardware testing a few times (probably in the same clean room this story took place in, albeit several years later). I can definitely see how a mistake like this could get made. A few stories I remember hearing from those days:

1. Bending pins from trying to insert a connector incorrectly 2. Running a full day of testing but forgetting to set up data recording 3. Accidentally leaving a screwdriver next to the hardware inside a thermal vac chamber in an overnight test

Fun times!

Ernie is the hero in this story.

As Fred Rogers said, "look for the helpers".

What a great write up. The tension preceding launch must have been tremendous, and lack of sleep adding to the possibility of such a small yet critical error. Good lesson for all of us.

It's a nice story with nice message. And it's pretty normal that mistakes happen, especially under pressure and within long shifts. The "mistake" itself is understandable: what is shocking to me it the multimeter thing. Learning that a mythical "NASA guy" in charge of really serious stuff cannot realize that the multimeter is measuring current, thus it's part of the circuit, and removing it will switch off something.

So let me share mine, it goes like hey can you correct this issue directly on the database of 5 years of corporate financial data? and in midst of testing and looking for the issue I wrote a delete with incomplete where clause that would delete lots of data, fortunately it was stopped because of a constraint violation, I still remember the adrenaline rush when I was processing what just happened

I thought this was about this story. Probably more interesting.

"The First Bug on Mars: OS Scheduling, Priority Inversion, and the Mars Pathfinder" - https://kwahome.medium.com/the-first-bug-on-mars-os-scheduli...

When I saw the headline I thought it may have been a write up from this:

https://www.latimes.com/archives/la-xpm-1999-oct-01-mn-17288...

> NASA lost its $125-million Mars Climate Orbiter because spacecraft engineers failed to convert from English to metric measurements

An even worse engineering horror story: https://faroutmagazine.co.uk/the-steely-dan-song-that-was-al... (more sad trombone and far less "relieved engineer, happy ending.")

I'm really at a loss of words. There is only one lesson from this incident: nobody is supposed to touch a 500M piece of equipment after 12 hours of work. Period. The author is not getting it, and nobody in the comments. The world is run by idiots and it shows.

I like this - “Let your scars serve you; they are an invaluable learning experience and investment in your capability and resilience.”

I have had couple of scars of mine. I feel like sometimes you become risk-averse and When you are launching new things you will face the fear of failure.

That one time I deleted the files on the CEO's laptop and did the backup afterwards - wrong order. Oops. Never heard anything about it though, so he didn't fully entrust all his data to the intern. Wise decision.

I felt like the real lesson is a missing flight recorder system, if each action and behaviour is properly logged, it would have been pretty clear that the telemetry was lost before chris even began testing the RAT motor

Ctrl-F, "The Martian"

"Phrase not found"

But... how? This is exactly the same story that happened in the book. Since Spirit is older is the book, perhaps this was the real-life inspiration.

Bravo to the author. I think if I had made the same mistake, and if the disaster is permanent, I would probably be scarred for life and never recover from it.

I once legit blew a $2K FPGA in postgraduate lab by fatfingering. Horrible memory.

And I thought I had a big mistake when I crashed an expensive drone.. nice read!

It's interesting story, but not a catastrophic failure.

Likely my favorite article ever; amazing writing.

$500M is a lot to waste in private money, but unfortunately in taxpayer money it's pittance and usually swept under the carpet

So, some short sighted bean counters combined the EE & CE degrees and you are the result? Ask for a tuition refund!

At least it was just a Fluke.

Decades ago I worked as a leader of a small team of sysadmins. We worked around the clock maintaining server room and critical applications for an average size bank.

One of our responsibilities was to execute nightly checklists to run various processes, do backups at correct times, etc. These processes would be things like running calculations on loans, verifying results, etc.

We had a huge ream of checklists to accomplish this and we were supposed to follow them religiously.

We had two very similar applications, one our core and another a core from another bank we bought with the same application but older version and slightly different config. Consequently, we had two tracks of checklists with very similar steps.

One of those steps was to change the accounting date in the system. The application was a terminal app. We would telnet to the server, log in, then we would execute commands in the menu driven app. To change the date we would have to go to a special menu for super dangerous applications. It required the user to log in again.

Our core system, required logging in, selecting that we want to advance accounting date by one day, entering admin password again, pressing enter, then waiting for about 4 hours while the process ran. Then the process would exit back to the menu where the highlighted option would be to advance the day.

Our legacy system required logging in, selecting the option to advance accounting date, entering admin password, pressing enter, then waiting for two hours after which a popup showed to ask a stupid question where we would just always press enter, then wait for another hour until it exits to the menu.

We quickly figured out, that we can just press the enter key twice on our legacy system. The second enter press would just leave there in keyboard buffer and dismiss the popup. This was very useful for us, as this was the only operation that interrupted what would be the only time during night where we could have a kebab...

One night I made a mistake, and I pressed the enter twice... on the wrong system. When I figured out that I did it I realised the process would exit to the menu and then should ask for the admin password.

But, unfortunately, the application had a bug (or a feature). Once it exited to the menu, it came back in but for some reason it remembered that the admin password has already been entered and started advancing the accounting date again without asking for the password.

Unfortunately, the date was December 24. For entire December 24, the entire bank was unable to process any operations while we were restoring from last good backup (before day close) and then redoing eod operations. Then on December 25 as a penalty, I had to sit for entire day with accounting department observing how they manually entered all of the operations that would normally happen automatically on Dec 24th.

One extra key pressed.

He’s the last person on Earth who would make this mistake again. That hit HAAARD

Trust but verify!!

It's crazy at how much better boomers had it. If this had happened in 2010 then he would have been fired after that.

Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.

I am a bit disappointed that the name of the author is not Howard Wolowitz.

How is this a $500 million mistake? It seems the issues didn’t cost $500 million.

Congratulations. 102 circuits would have taught this. But some short sighted bean counters merged EE & CE so you didn't get the opportunity. I suggest you ask for a tuition refund as they failed to educate you as they promised.