Lost your luggage? That's nothing – we just lost your whole flight

Finally a supervisor figured it out. Somebody had cancelled the flight, moved everybody to other flights, and then killed the flight record. For some reason, my ticket was still booked on a now non-existent flight, and that confused the hell out of the system and made it give weird info (such as the flight was in the hanger, or that my ticket didn't exist). The supervisor had to re-create the flight, move my ticket off it, and then delete the flight again.

We had no development or test environment so development were done in production. The system lacked the necessary functions to correct errors so there were frequent updates/corrections made by writing SQL-queries directly towards the production database. The incident was when I should update a single record so the system would resend it to the customer, but ran the query before writing the WHERE-clause.

Luckily there were working backups (incremental backups that ran every 15 min, almost like someone knew they might need them!) and no one had made any changes since the last one. We did a quick restore and let out a sigh of relief.

We did eventually write a tool to do those updates (and deletes) we were doing multiple times a day and were able to hand that tool over to one of the users, saving us multiple five minute interruptions each day and boosting our productivity.

Worth noting that this was part of a business that we bought from another company and we did transfer all customers to our own product with dev environments and no need to manually updating databases.

Would have been good management...had they not decided to replace said senior with the new hire junior DBA.

https://michael.mior.ca/blog/coloured-ssh-terminals/

We ran the booking websites for lots of airlines (Which was really just a frontend that connected to Sabre or Amadeus). Airline data standards were bad. They're probably still bad. You were lucky if you could get an airline's upcoming schedule or pricing rules in some format other than "manually typed into an email by a human". We had people who's job was solely just to take these emails and put the info in the database.

We had a bunch of arbitrary safe-guards against data entry mistakes. Ticket prices above 10k or under 10 were flagged as suspicious and needed someone more senior to approve them.

So this airline, headquartered somewhere with a 7 hour time difference from us, wants to do a 9 euro flight special for some holiday. There's a big marketing campaign, apparently lots of excited customers, etc.

I don't know if you've connected the dots yet but 9 euros was less than our suspicious-data-threshold of 10. So about 95% of their available flights just disappeared. Searching for them would turn up a "no results" page. The only available flights were expensive ones excluded from the sale, usually codeshares with a larger airline for a more distant destination.

The airline never notified us. It wasn't until we waltzed into the office 7 hours later that any one of us saw the problem. One of the DBAs wrote a quick "approve all suspicious entries" query and that was the end of it. Thankfully it was a 3 day sale.

Afterwards, I remember the chosen method to prevent this from happening again was to change the low threshold for a suspicious price to 8.99 or below.

That doesn't sound like they deleted a flight, how often do you hit a flight that's currently interesting to airports around the world? Of course there are exceptions, e.g. Emirates would route you via Dubai, so a flight Dubai to Frankfurt could have connecting passengers from a few dozen different originating countries, but not "every airport around the world".

OK, come to think of it, maybe the inconsistency in the DB halted everything, e.g. the system saw a few hundred passengers booked on flight with ID 463523 but there's no record of a flight with that ID...

Why not make it a team effort?

    rm -rf vv\  *