Hacker News Reader

Comment by MaxBarraclough | Hacker News Reader

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals).

Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.

View on HN

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter.

I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells.

[0] Technically he had no beard and if he had, it wouldn't have been gray.

Different-colored prompts for different machines is a great thing to do (I've been doing it for years), and very easy to implement

Solid tip. For GUI-enabled servers, use distinctively coloured wallpapers. I recommend bright red for production machines. The image itself can be just about anything, provided the colour is clear.

Doesn't hurt to use an image that's related to the server's purpose, and to put the name of the server right there in the wallpaper somewhere.

That's nifty, but sounds like more effort than changing a single color in one's .bash_profile

That's true, but depending on the configuration it may benefit all future users of the server.

Using iterm2, you can set a "badge" (large text overlay) on a terminal tab. I have a short shell function (`ib foo`) that sets the badge to arbitrary text. It's NOT as good as setting the terminal theme, but it's still very helpful to use it like this:

    ib production && ssh production-machine
    ib demo && ssh demo-machine

It's definitely helped me when testing a fix on a demo or staging instance, and has helped me avoid doing it on production accidentally.

Abel prompt that shows host name, username, and git branch

Re: Beards, color of:

Mine started turning grey in my mid 20s.

Could be related to me doing the electricians equivalent of deleting production DBs. I've drilled through the comms cable to payment terminals during opening hours. I've run over a copper gas line with a scissor lift. And yes, I've cut live 230V cables with hand tools.

That sinking feeling in your stomach you get immediately after doing something bad - it's universal across professions.

Thankfully, I've never fucked anything major up, and I've had my hands in hospitals, power plants, ISP fiber backbones, police stations and whatnot.

This reminds of a quote from, I think, Discworld.

> You're a survivor.

> But I've nearly died, dozens of times.

> Exactly.

Sounds like Rincewind. See also the octogenarian barbarians who are so deadly precisely because they've had a lifetime of experience of not dying.

That would be Cohen the Barbarian (aka Genghis Cohen) and his cohorts, collectively known as the Silver Horde.

> I've drilled through the comms cable to payment terminals during opening hours.

A friend of mine who does fire alarm systems was tasked to install one at a bank branch. He found out the hard way that one of the cables for the safes safety system wasn’t in the place where it should have been according to the plans. Safe’s safety system hosed, bank branch closed for repair.

A similar tip I picked up long ago: If you're typing a dangerous command, first type a `#` (or `--` if it's SQL, etc.), then the command. Then read it. Then go back to the start of the line and remove the comment and run it.

A related Bash command: alt+# to prepend a hash symbol and submit the line, so you can return to it later (through Bash's history) to run it.

I always do destructive SQL commands in two steps: first run a select using the WHERE clause you intend to use and verify which records will be affected, then hit the up arrow and edit the beginning of the query leaving the WHERE intact.

I also like adding redundant conditions to the WHERE so a typo in any single one of them won't sink me.

For the rare but critical manual SQL mod our common safety measure is to wrap every DELETE or UPDATE in BEGIN TRAN...ROLLBACK TRAN first. Run on test systems or snapshots multiple times, checking the result inside the transaction.

Finally, change ROLLBACK to COMMIT only when you are positive all is well.

IIRC (without checking the manuals) data-definition commands might not be covered by such transactions: such as altering, dropping tables and possibly truncates.

PostgreSQL is quite good about DDL being transactional. So I was surprised (tbf, I shouldn't have been) when Redshift autocommitted after a TRUNCATE. But DROP TABLE is transactional, go figure.

DDL is transactional in Microsoft SQL Server as well.

I do the same thing. I also keep auto commit off and make sure the rows updated looks correct before committing the change.

For SQL, always BEGIN first. If you’re unsure, run it as an EXPLAIN first. Then fire. Then commit it or roll back.

I use an alternate version on SQL: when running any modification on any kind of sensible database (which is a bad practice in itself, obviously, but sometimes you don't have a choice), always type in the WHERE clause before the table name (added bonus: do a SELECT first with that clause to see what you are modifying).

That way, if you accidentally send it, the command fails and nothing happens.

Especially useful if you're remoted in over a laggy connection.

I've done this for several years (also after seeing a video about Japanese railway operations). It doesn't seem to catch on.

It's also not perfect; it does not catch mistakes concerning "non-local" state, e.g. configuration files in /etc merging with one in . merging with some command line options. (Personally I try to avoid writing tools with defaults of this sort, but especially Java developers seem have different opinions.)

Unfortunately if you do P&C and still make the mistake due to the aforementioned tooling, you look even stupider.

Around industrial machines, I've long held and promoted the view that the machine is _trying_ to kill you, _trying_ to damage itself, _trying_ to ruin the workpiece. Only by outsmarting it at every turn, and having safeguards against every mishap, can you go home at the end of the day.

When something happens despite all that, just step back and realize how much worse it could've been, and how successful your safeguards have been up 'til that point.

Then look carefully at the procedure. Is there something about the naming or structure that could be more clear? Can you think of near-misses that resemble the failure you just experienced? Are you using boobytraps in production? Symlinks and overlay filesystems seem clever in the moment but they're bound to subvert our intuition someday. Perhaps you should get in the habit of always using full absolute paths, for instance.

There's always another gotcha, but if your workflow doesn't look as over-the-top safety-silly as aerospace, you're not doing as much as you could be. (Hint: It's not silly.)

Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.

And pilots will even callout that their action had the desired effect:

"Flaps up selected"

"Flaps are indicating up"

There's a lot to learn from the way airplanes are engineered and operated.

Prior Navy Nuke here. We called it PRO (Point, Read, Operate)- we’d point at the thing we were going to manipulate, state what we were manipulating, and announce the completed action.

For certain procedures we had a second party (“reader”) observing and acknowledging each part of each step.

Operator (Gesturing anti-clockwise while pointing at valve XYZ) Operator: Opening valve XYZ. Reader: Opening valve XYZ, aye. Operator: Valve XYZ is open. Reader: Valve XYZ is open, aye. Operator: Indications of flow Reader: Indications of flow, aye.

People can still get complacent, and things can still get missed but the deliberate mentality goes a long way. Now when GitHub makes me type out the repository name before I can delete it, I sometimes copy/paste... YOLO.

I've noticed from pair programming that the person navigating with a mouse is far less able to read and interpret their surroundings or pick up typos while typing, than an observer that simply has to watch what the other person is doing.

Like when clicking on a file in a directory you just entered and looking for the file, the observer can literally locate and point to the file for the mouse user 5-10x faster than the mouse operator.

The observer seems to interpret the information that results from the directory listing faster than the person who just did the double-click to enter the directory because they don't have the muscle coordination context switch and can immediately move to interpreting the results.

It's probably because mouse manipulation uses brain infrastructure that is more recently evolved, but observe-react is a lot earlier in the brain processing pipeline evolutionarily, and a lot more refined/involved.

I think this is because writing software is so much more than operating switches and controls. I really hate pair programming for this reason, but I love industrial-style controls and protocols involving multiple people.

Back in the '80s I worked on a financial system (SWIFT interface) for an Italian bank. It went operational and we observed 2 operators effectively doing "pair operating". We just thought it was weird Italian style socialising - one had the keyboard and the other was chattering away with a commentary. But they were surprisingly effective!

I accidentally learned when teaching a course at a site with too many people for the available machines, that pair exercises was very effective - I got lots more questions and overall learning went way up. If the pair discussed it and couldn't find an answer they would have the confidence to ask. On their own, neither would probably bother and just wait for me to go through things.

Since I have a vision impairment, I'm sure the effect is amplified very much for me, but using the mouse is such a massive break in flow:

- First you have to lift one hand up off the keyboard and put it down on the mouse. This may or may not mean taking your eyes off the screen.

- Then you need to find the mouse pointer on the screen

- Then you need to aim for what is usually a relatively small target and move the pointer there.

- If you're right-clicking, the right-click menu usually presents more small targets you need to aim for.

- If you need to use the keyboard, again you have to move your hand over to the keyboard from the mouse.

For finding the pointer, I developed this unconscious habit of slamming the mouse pointer to the very top-left of the screen. It's difficult though when on someone else's machine, where your brain isn't used to the pointer velocity or where multi-monitor means that slamming the mouse to the top-left actually puts the pointer on another monitor.

People look at me in awe when I'm using a two-pane file manager but honestly not having to take your hands off the keyboard and not having to move your eyes off the screen gives so much better flow. It's also why I like the UI of Blender - one hand on the keyboard and one hand on the mouse at most times.

And then there's the interactions with the Air Traffic Control system: flight plans, Standard Approaches, charts, etc. It's very impressive.

And it should be kept in mind that almost none of those procedures were intuitively obvious things to do. As the saying goes, safety standards are written in blood.

Came here for this.

A: “Passing control”

B: “Taking control”

A: “You have control”

B: “I have control”

This is how I remember it (6174, UH-1Y).

The TCP handshake IRL

As a fun fact, I was taught a shorter version (during private pilot instruction):

"You have the control."

"I have the control."

IDK if it changes between aircraft types, commercial/private/military cultures, or if it's just coincidence.

Yep, I had this version in a military trainer.

Or as we learned from Sully, when you're the captain, and you take charge and responsibility quickly, a simple "My plane" suffices

Rock climbing is remarkably similar. When a climber begins up a route the standard exchange with their belayer (the person managing the rope and keeping them alive in a fall) goes something like

A: "Belay on?"

B: "Belay on"

A: "Climbing"

B: "Climb on"

Then the climber begins.

It's interesting to me that highly regulated and totally unregulated activities have evolved extremely similar processes. I suppose having your life on the line is a good motivator to follow best practices.

For the ultimate low stakes version of this, when I played World of Warcraft in my younger days, tank swaps would communicate the same way.

"On belay?"

"Belay on"

Swapping the order of the words helps further.

This kind of positive assertion handover of control is still standard for very good reasons.

I searched Youtube for examples of this. This is a little bit staged, but it seems to be a real checklist they're going through: https://www.youtube.com/watch?v=JG7SkOQDDt0

Though they're not perfect. They said that one pilot is supposed to read the item, the other pilot say the answer, and the first pilot visually confirm it; but at 1:42, I noticed the first pilot say "emergency exit lights", hear the confirmation, and move to the next item without her eyes moving away from the list.

I'm not sure which of several possible conclusions to draw from that. ("Humans suck", "it is indeed staged", "the procedure has enough redundancy that the chance they're both careless on a given step is small", "the pilots feel that the emergency exit lights aren't particularly important", ...)

Routine is the killer. Have a look at the fatal maglev train accident in Germany. Service car was on the track. Presence of service car in service bay (and not on track) can be seen visually by operator (driver) in the control centre when turning head. (If I remember correctly)

Reply on news.ycombinator.com