Times I've Messed Up as a Developer
medium.com
medium.com
One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root.
I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine.
I recalled a piece of advice a university friend had given me. "Never admit to anything."
I walked into the server room, switched off the PC and restored it from a backup tape that was a day old. 2 Hours later the machine was back on.
No one ever noticed, and I never mentioned it either.
Moral of the story - Don't let anyone give you access to production servers.
what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup?
Would be epic fired story.
I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.
The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
Since then I’ve adopted custom prompt and color scheme for production.
I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.
I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.
These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
From man rm:
-i prompt before every removal
-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.
I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.
It forces me to be careful and deliberate. There's never a safe option or a backstop. I treat rm like a gun - always assume it's loaded and going to murder whatever I point it at without hesitation. If I'm not 100% sure about what I'm about to delete or how a wildcard will expand, then I echo it first.
I messed up once, about 15 years ago now. Lost a few personal files. Haven't screwed up since.
https://gist.github.com/woodrowbarlow/52b02ab27d0f8088891c3d...
look at `get_hash_color`.
Staging prompts are yellow.
Development prompts are green.
I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.
In my case, it was "Don't let anyone have you do testing/debugging on the production machine."
I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.
Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.
Well - at one point I cleared the wrong table - cleared the production and not the test table.
Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.
Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.
Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.
Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.
Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).
If you are root, make another user. you can delete it later.
I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal that I absolutely shouldn't be running drop, delete, truncate, rm, dd, whatever commands on them and if I do then I am extremely careful.
This is a great idea.
Good way to save your bacon.
Cool. Sorry, but I'm going to steal this idea from now on.
For UAT / Test environments that are different than dev, I use a yellow background.
And let's not talk about my own (mostly minor) disasters.
I was logged in over VPN to a client machine. I suspected the problem I was fixing was something to do with the network misbehaving. It was a Windows machine.
I thought "I know: I'll disable and reenable the network device!".
A few moments later: "Oh. Bugger.".
Shortly followed by an embarrassing call to the client's IT department to ask them to reenable the network device on that machine...
We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state. I was on my way out the door, and now I ended up being stuck there for another few hours trying to get ahold of devops.
At the company I am at now, we had a devops person whipe our multi TB CDN on their first day trying to make an improvement to our asset sync script. It took probably 2 days to get the whole thing reuploaded.
For example, I often write my 'rm' commands as 'ls' at first and then swap out the command once I know it gets the right thing.
Another tip is to write the WHERE clause and SET clause prior to filling in which table is being updated. That way you can’t accidentally run something prematurely.
If I'm typing an ad-hoc DELETE or UPDATE statement, I'll slap the WHERE keyword on the end of the main command line. For example:
UPDATE items
SET
active = 0,
deleted_at = now() WHERE
category = 'Foo'
DELETE FROM items WHERE
category = 'Foo' AND
customer = 123
I also tend to wrap destructive DML in comment blocks. So in my query editor, I'd actually have:
/*
DELETE FROM items WHERE
category = 'Foo'
*/
EDIT: I've actually never made a catastrophic mistake of this nature, but there have been some very close calls. And I've seen it happen several times, from excellent programmers who just made a mistake.
The fact that I was entering a delete -- instead of setting to inactive and doing it in production from the command line should give you an idea of how likely the backups would have been effective. This was in '97, though when everything was seat of the pants during the full internet craze.
If you want to delete all rows, you should be made to explicitly type out that intention.
As far as I can tell, there's nothing like that in Postgres.
It slowed things down, but the forced code reviewer process saved me once in a big way.
Read up on "bus factor": https://en.wikipedia.org/wiki/Bus_factor
Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main source code file was also named program.c. To my horror it had overwritten a large part of its own source.
I remember staying in the office until about 2am to re-write the lost code, but my boss never found out.
Luckily, it was Python, which compiles to a bytecode representation when you run the script, and I had done a test-run of the script recently. After a little research into the available tools, I was able to extract the original source (and comments!) from the pyc file. It was a simpler solution than any of the un-delete utilities that I found.
Nothing works.
A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as quickly as possible. We go live, no one is the wiser.
We contact the DBA and explain the folly of his decision.
He then proceeds to rename everything AGAIN on a live system. No we didn't shoot him lengthwise, but we did fire him with some prejudice.
>He then proceeds to rename everything AGAIN on a live system.
What could his reasoning possibly be? Did he not understand your explanation?
Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment.
My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had locked out the production user from the database, causing every app in the company to go offline. I was then told to read aloud back all of the projected sales losses from that day as well as write a bunch of letters for our top sales people who couldn't make sales that day.
Wasn't fired (thankfully). Made it to a senior lead, so if I had to say anything, it's that the punishment really made it really drive home and I _never ever did it again_ . :)
The fix was not painful, just restore de user table from last night's backup. but I felt a little ashamed when i had to explain the database administrator what has happened.
From that day, I keep autocommit always off.
Doing a recursive sudo chown of the /etc folder, locked myself out of that ubuntu server.
Wrote a parser that had a condition that would not end, would watch the server spike to 100% usage.
I needed to update MySQL to take advantage of some new feature that young, eager me just _needed_ to use. Well, the package repo didn't have that version, so no worries, I'll install from source. Well that required some newer version of a core utility, so no worries, I'll just `yum update` that. No dice. No worries, I'll just force remove and install the newer one.
Well, young, eager me didn't realize essentially every command relied on that core utility, so although I had a prompt, I couldn't even `ls`, much less `yum install`.
Our site technician had to take an OS disk up with him next time he made the 2+ hour drive to the top. Luckily we did have a KVM set up so it didn't require a site visit from me (or is that unlucky...it would have been a trip to Maui!).
Probably one of the stupidest, but also the most personally effective way to learn anything.
The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrupt lol. They went back day by day - all of the recent backups were corrupt, the "freshest" working backup was about a month old. Some heads rolled as a result.
The "consultant" decided to move a database to its one machine. During this transition this person started a backup and then whacked the production database while the backup was still running because they felt the backup had "probably already backed up the database since it has been running for hours".
Dead in the water. Restored from a week old backup to the dismay of a lot of people working there whom had lost critical pieces of their work.
I too have "restored" a database to the wrong server (production was now development). This was a member list for an arts organization... After about 20 minutes of panic (what is going on....) I remembered I had set daily backups. whew.
It being a very long day, I deployed the job with the cron expression * 8 * * * instead of 0 8 * * * , which meant it ran EVERY minute starting at 8:00 AM instead of just once.
The next morning I woke up to a flurry of text and emails saying customers had complained they received emails about a hundred times. Thankfully, we had a pause in the job which meant it went out only a hundred times and not thousands as one would have expected!
it turned out that our beta testers found a lot of cases that were broken on Friday when the release was next Monday. The problem was that a ton of other classes pointed to the one in which I made changes and altered the state..
Worked through the weekend with my teamlead and got it fixed somewhere on sunday afternoon.
good bonding experience though. All had a laugh about it the next week. :-)
I did a blog post about it [1].
1: http://www.drinchev.com/blog/ssh-and-terminal-background/
In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it.
Here is the fix:
``` [push] default = current ```
Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am one of those weirdos who enjoys clean, readable git repos.
I remember at least one prod incident where a script made it out into prod and wiped out 100s of servers. Something like it had rm -rf /$SOME_ENV_VAR which was set in QA but not prod. Opps....
I look at `terraform plan -destroy` and it still terrifies to destroy an internal service.
Sometimes works.
For instance, if you usually just need to view content. Don't let him truncate / delete / do other destructive operations.
I was relying on expressions like
mode == "Test"
Sadly some parts of the codebase weren't very idiomatic and had some
"Test".Equals(mode)
I'm going to steal that one.
My boss wasn't happy.
I was sitting at my desk watching service after service slowly disappear in utter confusion, then hours spent cleaning...