Hacker News Reader

Comment by dkersten | Hacker News Reader

I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection.

This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don't actually do it. Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue, for the first few files/bits of data. Only after that was ok'd too did it run the remainder.

In other things I've worked on, I've taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.

View on HN

Once we get used to doing same thing multiple times a day, it doesn't matter if the log shows that we're about to take a destructive action, we'll still do it. Only thing that is foolproof is to not take the destructive action because people make mistake, it's human nature. I don't know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed).

Multiple reviewers here didn't catch the mistake

https://www.bloombergquint.com/markets/citi-s-900-million-mi...

That form had a couple weird checkboxes with odd wording. It is a famous mistake, but also rather understandable just because the form was cryptic.

Because everyone assumes that everyone else is looking at it more closely than they are. “I’ll just do a cursory look since I’m sure everyone else is doing a in-depth look.” Narrator: nobody did an in-depth search.

While this is a huge issue, a solution (well, a partial mitigation) I've seen and used is the "Pointing and Calling" technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they're doing and what they expect to happen.

It's used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&A, etc. do this even when alone.

[1] https://www.atlasobscura.com/articles/pointing-and-calling-j...

[2] https://en.wikipedia.org/wiki/Pointing_and_calling

“I’m removing that semicolon!” (Pointing)

Parent meant this sort of pointing.

https://t.co/TjfX5K54H7

Hell, GitHub does that to an extent, with the "type the name of this repository to delete it" prompts. Typing the name of the repository isn't exactly perfect, but it's an interesting direction.

There was a thread recently about a repo that accidentally went private and lost all of its stars because of confusion with GH teams vs GH profile readme repo naming. I think this type of prompt is very useful for explicitly preventing the rare worst case scenarios but the problem is making any type of prompt "routine" so that our brains fail to process it.

This is it I think. https://news.ycombinator.com/item?id=31033758

The suggestion in that post about how to fix it is good, and mirrors one I read in the Rachael by the Bay blog - type the number of machines to continue:

https://rachelbythebay.com/w/2020/10/26/num/

The take away by both is there is actually something to do which can wake people up when the stakes are high, and they might not be doing what they expect.

And most importantly, don't let yourself get into the habit of copy pasting the value

I wonder if your could print some non visible characters in there to taint the copied value in some detectable way.

yeah, that would possibly stop the copy and paste problem. to make it robust they would need to use a string of a few non-visible characters but that would fail if the browser's clipboard system doesn't copy them over for some kind of privacy initiative. might be another way it fails that I can't think of right now.

Prompt in words, but expect the value in numbers, eg: "Twenty-five" and the box requires you to type "25"? At least this specific case, it would require you to type it.

I always copy-paste into that box as well, they should probably make at least an attempt at disabling pasting into it

Azure has the same when deleting a database just a verify this is the correct one by typing the db name

I heard of this technique, but unfortunately I don't see how it can be easily applied in software engineering/devops.

Also, I now realized that aviation checklists seem to tend to be done similarly with gestures - at least from what I saw on YouTube, not sure if that's representative or only used during education (?)

Spelling out loudly the command you are about to execute and explaining the reasoning behind it can help a lot too.

Ok, but am I to do it on every single command I do on my terminal? Or on which ones specifically? If the problem we're trying to solve is that I can sometimes overlook the "dangerous commands" among "safe ones", by definition of overlooking it won't work if I tell myself to "spell out the command only in case of the dangerous ones", no?

I'm honestly trying to think of the way how I could approach this for myself, just I don't see a clear solution yet that wouldn't require me to spell out everything I type in my terminal window.

> Multiple reviewers here didn't catch the mistake

Sure, but we can only do so much. I find its good bang for buck and alternatives that might prevent that are not always available, so we do the best we can. You gotta make a call on whether its enough or not.

I'm a fan of doing things temporally so data is very rarely actually deleted from the database. Most of the time, you just update the "valid_to" field to the current time. Sometimes real deleted are required such as with privacy requests, but I think that sort of thing is pretty rare.

If your application has space concerns, you can modify this approach to be like a recycle bin where you delete records which are no longer valid and have been invalid for over a month (or whatever time frame is appropriate for your application). However, I think this is unnecessary in most cases except for blob/file storage.

mv then rm is another idiom. So long as you have the space.

For database entries, flag for deletion, then delete.

In the files case, the move or rename also accomplishes the result of breaking any functionality which still relies on those file ... whilst you can still recover.

Way back in the day I was doing filesystem surgery on a Linux system, shuffling partitions around. I meant to issue the 'rf -rm .' in a specific directory, I happened to be in root.

However ...

- I'd booted a live-Linux version. (This was back when those still ran from floppy).

- I'd mounted all partitions other than the one I was performing surgery on '-ro' (read-only).

So what I bought was a reboot, and an opportunity to see what a Linux system with an active shell, but no executables, looks like.

Plan ahead. Make big changes in stages. Measure twice (or 3, or 10, or 20 times), cut once. Sit on your hands for a minute before running as root. Paste into an editor session (C-x C-e Readline command, as noted elsewhere in this thread).

Have backups.

You mean cp then rm?

And yes, copy, verify, delete. And make sure by the code structure that you either do the three on the same files, or their fail.

Also, do it slowly, with just a bit of data on each iteration. That will make the verification step more reliable.

Anyway, for a huge majority of cases, only having backups is enough already. Just make sure to test them.

I think mv then rm is probably meant as 'windows trash bin' style.

No, mv.

Example:

  cd datadir
  mkdir delete
  mv <list of files to be deleted> ./delete
  # test to see if anything looks broken.  
  # This might take a few seconds, or months, though it's usually reasonably brief.
  rm -rf ./delete

The reasons for mv:

- It's atomic (on a single filesystem). There's no risk of ending up with a partial operation or an incomplete operation.

- It doesn't copy the data, it renames the file. (mv and rename are largely synonyms.)

- There's no duplication of space usage. Where you're dealing with large files, this is helpful.

The process is similar to the staged deletion most desktop OS users are familiar with, of "drag to trash, then empty trash". Used in the manner I'm deploying it, it's a bit more like a staged warehouse purge or ordering a dumpster bin --- more structured / controlled staged deletion than a household or small office might use.

  > ... Then have multiple people inspect the logs. Once ok'd, run again, with manual prompts after each log item asking to continue...

This sort-of reminds me of some "critical" work I had to do a couple of decades ago. I was in a shop that used this horrifically tedious tool for designing masks for special kinds of photonic devices-- basically it was tracing out optical waveguides that would be placed on a crystal that was processed much like a silicon IC.

The process was for TWO of us to sit in front of computer and review the curves in this crazy old EDA layout tool called "L-edit" before it got sent to have the actual masks made (which were very expensive). It took HOURS to check everything.

The first hour was tolerable but then boredom started to creep in and we got sloppy. The whole reason TWO people got tasked with this was because it was thought that we would keep each other focused-- 2 pairs of eyes are better than one, right?. Instead, it just underscored the tedium of it all. One day someone walked in and found us BOTH in DEEP SLEEP in front of the monitor. Having two people didn't decrease the waste caused by mistakes, it just bored the hell out of more people.

How many mistakes did you catch?

From his story I can tell he found one big mistake. The tedious work itself.

ONE real one and some occasional nitpicks to show that we were busy (after being caught asleep).

Was it worth it? No, I don't think so from an opportunity cost perspective-- even though we were the most junior folks there. A mind is a terrible thing to waste!

Another good approach is do deletions slowly. Put sleeps between each operation, and log everything. That way if you realize something is broken, you have a chance of catching it before it's too late.

> Then have multiple people inspect the logs.

I think that this is the most important part of any check. Your parent refers to checking the log five times, but, at least in my experience, I won't catch any more errors on the fifth time than the first—if I once saw what I expected rather than what was there, I'll keep doing so. Of course everyone has their blind spots, but, as in the famous Swiss-cheese approach, we just hope that they don't line up!

Yes, I love the idea of the Plan Apply.

It never hurts to ask for another set of eyes to review. At the least if something goes awry, the blame isn't solely on you.

Make a plan, check the plan, [fix the plan, check the plan (loop)], do the plan

See PDCA for more a more time critical decision loop. https://en.wikipedia.org/wiki/PDCA

Reply on news.ycombinator.com