Hacker News Reader

Comment by eitland | Hacker News Reader

Something similar happened a place I worked somewhere between 10 and 20 years ago.

Service technician did not see the . in the command

  rm -rf ./bin

so he proceeded to run

  rm -rf /bin

When that didn't work he did what everyone who knows a little bit Linux does and added sudo in front.

I was on a terminal from the other side of the globe when the server suddenly started acting weird.

We were able to use scp or rsync (one of then was in sbin or something) to get back the bits from an identical server, which saved me from three days of tedious work :-)

In hindsight of course we should have written the docs in a way that would prevent this exact situation but in the beginning it was just the output of the history command after I had done it, dumped into a document with some explanations.

View on HN

It is not the same thing, but recently, my pfsense router started acting weird where dns stopped working. I was bust so I rebooted it and it failed to boot. I ended up bringing a monitor and keyboard to see what went wrong and it turned out that suricata’s logs had used 1.4TB on my 250GB SSD (ZFS zstd compression is awesome), causing it to run out of space a few years after I enabled the feature out of curiously. I wiped the logs, rebooted and things worked.

My lesson from them is when a machine servers acting strangely, do not reboot and instead troubleshoot right away. Had I done that, I could have found the problem and had only minimal partial downtime (I would have had to restart dns afterward). Since I was too busy to do things the right way, my internet was out for a half hour. Your story reminded me of this, since you would have had a bigger headache if you had rebooted.

This is the origin story for anyone who has ever set up alerts for 90% disk utilization on a machine.

These are especially bad with services that generally grow their logs very slowly but when something like a net split happens or a server is down they generate as much log data in an hour as they typically do all week. So you get close to full and then an incident happens, and now you have two incidents because the screaming machine goes down with a full disk right after you lose your internal DNS server or what have you.

One trick I learned is to create a big random file (must be random to ensure no funny compression filesystem tricks out smart you) called something like BIG_DUMMY_DATA_SAFE_TO_DELETE. When you find yourself in a catastrophic space situation, you can delete the file and have a less panicky recovery process as the immediate problem is gone.

Works right up until you have another incident, go to delete the file, and realize you already deleted it the first time…

The funniest consequence of full disk was that I could not log in to a server, because the login process required writing some tiny temporary file in the user's home. Did not find a way to recover it, other than plugging a keyboard in it.

My second boss was a Sun Microsystems enjoyer and he always pronounced “superuser” as “stupid user”.

After a raised eyebrow he went on to explain, “because when the machine is broken it’s always because some stupid user did something.

It took me a couple more stupiduser incidents of my own before I instituted a rule of counting to five before hitting enter on any `rm -rf` command.

> I instituted a rule of counting to five before hitting enter on any `rm -rf` command.

That's just (should be) standard practice. Another risky command is `sudo dd`/`sudo cat` for writing disk images, always chant the disk device against an fdisk -l listing like a magical spell, lest you nuke your main drive.

When using `dd` I always verbally say "`if` is input file, because I want to take data from <whatever I wrote there> and `of` is output file, because I want that data to overwrite <whatever I wrote here>". Never have I ever had a `dd` incident, but that's such a fear I have that I developed this habit.

of stands for obliterated file

This is the way!

and dd is data destroyer

But it’s surprising how often you see the wheels turn when you explain it to someone else. It’s not obvious to everyone.

I always prepend a # when writing possibly dangerous commands, mostly rm -rf and dd. Started doing it after I wanted to dd an .iso of a distro to my usb drive but accidentally my main drive.

It can help if you press enter instinctively after finishing writing the command, and also against fat fingering an incomplete but valid command.

I've done it too, but probably not as consistent as you.

I like to prefix it with `echo` so I can see what it's trying to run, especially when wildcards are involved. Or things like `find`.

Echo is also very good for chaining with xargs to make sure you didn’t fuck up.

First, you don’t let people read and type commands by hand. Second, hire a better guy than someone who blindly does sudo because the command didn’t work.

First, there is a difference between rolling out 5 such systems a year and modern server farms, second, replacing a brilliant electronics guy because he once messed up when he helped us out on site isn't something I'd want to have on my cv. Third this is well over a decade ago.

You were lucky scp uses a binary that typically lives outside /bin - in /usr/lib/openssh/ or some such.

Many years ago, I've got to recover a remote server where /usr was nuked (and /bin, /sbin, and /lib were all symlinks into now-empty /usr). Ended up writing a one-liner perl to convert /bin/busybox-static from my local machine into a series of:

  echo -ne "\x7f\x45..." >>~/busybox-static

...and copy-pasting that, chunk-by-chunk, into a single surviving ssh/bash connection, and then used that busybox binary to pull in from a backup.

I have learned through hard experience that is you ever user sudo to edit the sudoers file, create two shell windows logged in as root before doing so.

Use visudo to edit the file of course, because not doing so can blow everything up by rendering the sudoers file unparseable and then everyone is gonna have a bad time.

But also the temptation when altering sudo is to immediately log out as super user and try using sudo to do the new command. If you’ve fucked up the file you might not be able to sudo anymore. So now use your second shell window to undo whatever you just did in the first window.

I have setup ssh remote forwards to a jump host in the past to allow remote access through a firewall. daemontools executes scripts exec’ing ssh. ExitOnForwardFailure and ServerAliveInterval are set client side with ClientAliveInterval and ClientAliveCountMax set server side to enable rapid recovery if something goes wrong.

Whenever one of the daemon tools scripts doing remote forwards needs to be modified, a second reverse forward script is added and the reverse forward from that is used for ssh, before changing the original script. The second script is removed only after confirming the first still works after the edit. This procedure prevents fatfingering from locking out remote access, since if something goes wrong, you just need to redo the previous step(s) until you get things working.

If anyone wants to replicate that, I suggest setting -nNT as arguments to ssh and restricting what the user login can do via sshd_config.

SSH is probably where I picked up this trick. It’s the same problem. Now you have to find someone with more privs or physical access to fix your fuckup. Meanwhile people are waiting for whatever you were trying to do.

> In hindsight of course we should have written the docs in a way that would prevent this exact situation

I would say the bigger failure is relying on a human typing things into a terminal rather than automating the tasks or changing the system so the task is no longer needed.

As if there haven’t been outages caused by incorrect directory interpolation in scripts.

Sure, bug happen. They (usually) happen reliably, and tests can help prevent/detect them.

It is impossible to test for human errors in advance, though.

I often run rm -rf as part of operations (for reasons), and my habit is to first run "sudo du -csh" on the paths to be deleted, check that the total size makes sense, and then up-arrow and replace "du -csh" with "rm -rf".

Use trash-cli and additionally git commit the target if you’re nervous before deletion.

It's always a good idea to allow sudo to untrained people on critical systems...

What would be safer alternative?

    pwd # Then check something?
    pushd bin #
    rm -rf .

Probably still with pitfalls

`rm -rf bin`

I think the misguided belief that `./` means 'execute script' or something (program that isn't 'installed'?) is single-handedly to blame for so much script spaghetti.

And not just for bin. There's probably an edge case where you would need to give the ./ prefix to rm, but I've never come across it. The vast, vast majority of the time just entering the name of the thing is easier and less error-prone.

Use the full path of the bin dir in your rm rf

rm -rf ~/bin could have some nasty consequences if you fat finger a return key anywhere in the middle.

Run enough commands enough times and you will find Murphy is waiting for you.

If you’re just removing a bin directory one time, odds are low but not zero. If you’re writing a run book for people to use, odds are 100% that you will have to help someone rebuild at least once.

Write a script that does the steps required?

If the problem is defined enough to create an exact series of commands for an operator to execute, it is defined enough to create a script to do it for you

Reply on news.ycombinator.com