Hacker News Reader

Comment by vijucat | Hacker News Reader

I once shut down an algorithmic trading server by hastily typing (in bash):

- Ctrl-r for reverse-search through history

- typing 'ps' to find the process status utility (of course)

- pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)

I got a call from the head Sales Trader within 5 seconds asking why his GUI showed that all Orders were paused. Luckily, our recovery code was robust and I could restart the server and resume trading in half a minute or so.

That's $250 million to $400 million of orders on pause for half a minute. Not to mention my heartbeat.

Renamed stopserver.sh to stop_server.sh after that incident :|

P.S. typing speed is not merely overrated, but dangerous in some contexts. Haste makes waste.

View on HN

Might be better having something like the following at the top of your script:

    echo "This will STOP THE SERVER. Are you sure you want to  do this?"
    echo "Type 'yes' to continue:"
    read response

    if [[ $response != "yes" ]]
    then
      echo "You must type 'yes' to continue. Aborting."
      exit 1
    fi

    echo "Stopping server ..."

It barely takes any time to type 'yes' but it makes you stop and think.

That'll make it impossible to use in scripts. This remember me the time I tried to fix my "rm -r * .o" with a CLI trash system, instead of doing proper backups.

Might be better to start reviewing EVERY command one sends to important servers, and testing them if viable. What probably is the line that vijucat took... that's the line that everybody ends up taking, the only thing that changes is the number of accidents needed.

Then you add an optional CLI argument that makes it skip the prompt, and use that version in scripts.

As 'euid' said, you can just add a check. If you don't want to add proper option checking (because this is the only option) you can do something like:

    response=$1

    echo "This will STOP THE SERVER. Are you sure you want to do this?"
    echo "Type 'yes' to continue:"
    [[ $response ]] || read response

    if [[ $response != "yes" ]]
    then
      echo "You must type 'yes' to continue. Aborting."
      exit 1
    fi

    echo "Stopping server ..."

Which will allow you to run the script with "./scriptname.sh yes" to bypass the check.

EDIT: Though, as you say, it is much better not to be running anything like this on a live server anyway. Any change should be part of a deployment procedure that is carefully checked and tested, as well as having a rollback procedure in the event of something horrible happening.

Of course, in the real world, you sometimes don't have that luxury and you just have to hope and pray :-)

Yes, this solution would have worked for our situation.

Am I wrong to assume that you weren't using an init script of some sort because this stopped multiple services, perhaps in a specific order?

Repercussions?

Luckily, and just that one time, there was nothing but disapproving glares.

Reply on news.ycombinator.com