Hacker News Reader

Comment by mprovost | Hacker News Reader

The hard part is determining with 100% accuracy what is a "critical change". Many of these incidents start with someone making a change that "should be" safe. So you either have to make all your changes with a crash cart nearby, or trust in your ability to recover from a bad change remotely.

View on HN

For my own learning, what do you mean by a 'crash cart' here?

A computer on a movable cart that is correctly provisioned to get direct access to a server in a rack in a datacenter. Nowadays just an appropriate laptop on a desk with casters.

A crash cart used to be a literal cart in the datacenter with a monitor and keyboard to plug into a rack-mounted server to recover access. Not sure if there is something more modern now.

Still is that where I work. Doesn't help if you need serial console access, for that you'll need a laptop and a USB/serial dongle, or an actual old school serial terminal such as a VT100, VT220, etc.

In a previous life, I and a biz partner ran a web company on about 5 servers. We cabled the serial ports together in a daisy chain, used serial port consoles, and had software running on every system to access the ports. This way, if a system died, you just had to ssh into the system connected to it and you had access to the failed system's console. These were Supermicro servers, so the pre-boot screens and all that were also available. It came in very handy and saved us lots of trips to Hurricane Electric.

Still useful

Presumably what's meant is the tools necessary to directly access the affected systems to effect repairs. I suppose it's by analogy to this:

https://en.wikipedia.org/wiki/Crash_cart

Reply on news.ycombinator.com