Hacker News Reader

Two days ago I restored a customer's data from backup because they typed "ALL" and hit OK on a dialog that said "This will change ALL of your things, type ALL to confirm". This dialog only pops up if N > 100, and normally makes you type the number (ALL is a special case). I just do not know how to make this more idiotproof short of making them fax in the request.

No matter what you think is sane, some insane person will prove you wrong. In this case, tiredness + ESL = bad reading comprehension. It's going to happen.

Reverse survivorship bias...

It sounds like you found the solution, you just made it really cumbersome: undo.

All confirmation dialogs should be replaced with undo. The happy path has lower friction, and in case of a mistake they'll heave a huge sigh of relief. When possible, it's better in all cases for all users, whether novices or power users.

And many things that at first blush seem like undo isn't possible, are actually easy to make undoable with a simple tweak: deleting data? Don't actually delete it until 24 hours later. Sending an email? Wait 10 seconds to actually send it, similar to Gmail's Undo Send.

Implementing soft-delete is much easier than "soft-update", which is what this would have been.

You could still wait 10 seconds, and have a 'fake' undo button that aborts. (You can even put up a progress bar to pretend you are doing work during those 10 seconds.)

That's purely a UI element and is completely independent of how the actual destructive operation is implemented in the backend nor how hard it would be to reverse.

I don't think that would be much better than a confirmation dialog that makes you wait 10 seconds before you can click OK. It's often only after clicking around and seeing the resulting changes that it sinks in that a mistake was made, and they reach for Undo. And that would add just as much friction to the happy path.

It can be much better than the confirmation dialog, because it's meant to be implemented in such a way that you can get on with the rest of your work while the undo-countdown is ticking.

From personal experience with gmail's fake undo, in terms of things sinking in, it works almost as well as regular undo for me; and not like a confirmation dialog (which doesn't work at all).

So there's less friction, there's no extra click you need to make after ten seconds. And, also from personal experience, the force-delayed confirmation dialog I've used (I think in Chrome and Firefox for certain actions), don't seem to lead me to thinking at all. At least not any better than a regular confirmation dialog.

But in any case, all these are empirical questions, and it would be interesting to run a little user study with the different options, instead of endless speculation.

Gmail is a pretty specific case - email is fundamentally asynchronous and "delay send" for something that's already scheduled is straightforward.

Imagine trying to apply this undo to a bulk add/remove labels operation. Once you've committed the transaction, there is no simple 'undo'. It's possible to build a system capable of undo, sure, but you're talking about a lot of upfront work and complexity. Plus a fairly exotic database schema.

I don't see the problem?

I would imagine you would stick all your UI actions in something like a log, and then only apply that log to your actual data with a delay?

But not sure whether you call that 'a lot of upfront work and complexity'?

Perhaps I'm a bit blind, because I come from a part of the programming world that's very keen on persistent datastructures, where undos are trivial to implement. (https://en.wikipedia.org/wiki/Persistent_data_structure)

There's nothing exotic about it, you just need an OLAP rather than OLTP database schema: https://en.wikipedia.org/wiki/OLAP_cube

Yeah, unfortunately in some cases if you didn't plan for it from the beginning it's not easy to tack on later.

In my opinion, it's usually worth it though. You only hear from the folks asking you to restore things from backup—you won't hear from the folks who experience unnecessary friction and tell their friends or coworkers "it's okay, it works, it's kind of annoying to use though, I can't put my finger on anything specific".

Unless you're actually trying to do a huge synthetic operation where info would choke, e.g. creating an archive while removing existing files.

That one is still possible to undo, just slower and more expensive...

Then there are fun ones like Windows update holding 20 GB of insufficient undo, mechanical or hardware failures induced by extra load, and how to decide where an operation ends.

Require a different user's authentication or admin code to approve an "ALL" transaction.

While there might be emergency conditions that would make this cumbersome, in general that sorry of two person control makes sense. That's why the military uses it for especially dangerous actions or conditions. (Weapons loading on the sub i served on, for example).

Yes, it works.

But when the sample is large enough, it still happens once in a while. In 2004, a CSB investigation showed that an entire chemical plant exploded after the interlock was bypassed by the supervisor password [0][1].

> The explosion occurred when maintenance personnel entered a password to override computer safeguards, allowing premature opening of the sterilizer door. This caused an explosive mixture of ethylene oxide (EO) to be evacuated to the open-flame catalytic oxidizer by the chamber ventilation system. The oxidizer is used to remove EO in compliance with California air quality regulations. When the EO reached the oxidizer it ignited and the flame quickly traveled back through the ducting to the sterilizer where approximately fifty pounds of EO ignited and exploded.

Apparently the supervisor who owned the password didn't receive any training on the nature of the process and the dangers of bypassing the interlock...

[0] https://www.csb.gov/assets/1/20/sterigenics_report.pdf

[1] https://www.youtube.com/watch?v=_2UnKLm2Eag

Why would you ever ever have the ability do this "allowing premature opening of the sterilizer door".

Ironic it was air quality regulations - that did for them.

There needs to be something in case the implementation forgot something. These are dangerous of course, but they can also save the day when something unexpected happens.

This is a physical chemical plant not a website - if you do need to do something like that you do it manually.

Unfortunately human error can never be completely eliminated. However, I'm not really talking about this type of problem. In my previous [1], the operator understood how the tool worked; they simply made a mistake when typing the command, and the tool without warning accepted the command to reboot the entire datacenter. Particularly telling are these comments: (sic)

    Operator-1: I ewas rebooting an rb
    Operator-1: forgot to put -n
    [...]
    Operator-5: [...] i've almost done what Operatolr-1
                just did a *number* of times.

That isn't a user understanding problem; it's a dangerous tool that doesn't fail safely. In your case, at least you detected the unusually destructive action and asked for verification. Youtube isn't even attempting simple sanity checks like your "N > 100" test.

> normally makes you type the number

> I just do not know how to make this more idiotproof

Requiring explicit typing of the number or an explicit phrase like "Yes, I want to delete everything." are can help a lot.

If possible, another good approach is to explicitly show the full list of proposed changes. Phrases like "This will change ALL of ..." might have multiple interpretations (ALL what? All of the the things in my entire account? All of the things in the current/last project/group? All of the things I think (perhaps incorrectly) were referenced in this action?). If someone is expecting to change only a few records, a confirmation popup that asks "Do you want to make these changes:" followed by a huge list has a large size/presence that should conflict with their expectations. "I only wanted to change a few things - wtf is this huge list?"

Printing a list of the fist 100 or so affected files is useful to give a real wake up call and a chanse to double check.

An old office mate had been a systems programmer back in the days of mainframes. He once put a local-site patch into the mainframe boot code where the question about wiping and restoring the on-line storage required operators to type in, correctly capitalized and punctuated, the precise string: "Yes, I really do want to spend the entirety of my shift hanging tapes."

.... and someone still did it. The struggle is real.

In electrical engineering, the saying is "Nothing is foolproof to a sufficiently talented fool."

Also, in computer folklore, there are numerous stories of how non-technical users purposefully destroy foolproof mechanisms by brute-force, e.g. cut the slot on a DDR3 socket to insert a DDR2 RAM module and fry everything... And I wonder whether "don't use brute-force, if you have difficulty getting it in, it means you are doing it wrong" should be taught as the first rule when working with hardware. Unfortunately, to add the confusion, we also have connectors that can be surprisingly hard to connect and disconnect even under normal circumstances...

I think this is a good analogy for software. To the typical user, some software is easy and sensical, while other is obtuse and requires significant jiggering just to do the thing it was ostensibly designed to do.

There is a difference, however. The connectors that require a lot of force for insertion and removal have some true advantages, they are usually the simplest, cheapest and generally reliable components. Almost nothing can go wrong with a simple wire terminal, it's just a piece of rectangular or round metal. Although it can be difficult to disconnect for servicing, but you're only expected to do that once per year. On the other hand, "easy" connectors are designed in a way that, instead of requiring the mating force necessary for a good contact , it's provided by the connector mechanism itself, and as a result, they're often more complex, expensive, or fragile, such as a USB connector or a ZIF socket.

A software analogy for an "easy" connector would be, "fancy software with good user experience often has a lot of complexity hidden behind of scene, and can be fragile". But I'm not sure what would be the analogy for a cheap connector. Perhaps, a shell script?

Yes, a shell script, but for the sake of the discussion, you must imagine that to the average user, invoking a shell script is “the same” as navigating through a few settings menus and clicking some checkboxes they don’t understand: that is to say, when the UX is sufficiently lacking, users often enter “well I don’t really know what I’m doing just push through” mode.

Atleast in this instance some blame lies with computer hardware designers. Make things simple, no one puts a three the wrong way.

In my environment the adage is "when you say foolproof you mean fooldetecting".

> And I wonder whether "don't use brute-force, if you have difficulty getting it in, it means you are doing it wrong" should be taught as the first rule when working with hardware. Unfortunately, to add the confusion, we also have connectors that can be surprisingly hard to connect and disconnect even under normal circumstances...

And everyone has likely experienced in their lives plenty of appliances, self-assembly kits and other objects where some components required application of force to put together, because there's resistance coming from the feature that prevents the object from coming apart together. My rule of thumb is now that if the force seems to be veering into "could break surrounding structure" levels, or if the thing starts making unexpected sounds, then I'm doing it wrong.

... and then I have to put a CPU on a motherboard and the correct way absolutely does involve close-to-breaking forces and squeaky sounds.

Don't allow the user type "all", but another guy (admin) to verify such commands.

The bigger question I have, why did you not delete the data when asked to?

I understand idiot users, but what about users who actually want to delete it?

GP mentioned restoring from backups. You generally don’t delete from backups outside of the normal cycling policy, because otherwise they’re not backups.

Good luck checking in code that will increase the human burden and reduce automation. Imagine how many TPS reports you would have to fill out to get permission to make a system page a human more often at “Google scale”.

When the metrics finally hit the dashboards sounds like it would be career limiting.

“But we prevented pdkl95’s channel from being deleted!” is definitely going to be a solid defense for why you hobbled their infallible AI from ruthlessly executing its system oversight responsibility.

And yet they've created a widespread perception that YouTube mistreats it's creators, and we see stories like this all the time now. Surely someone at Google is capable of taking the long view. And if the culture at Google prevents that then Google is broken.

Every time a person come close the have a long time view, they get senior enough to leave the company or the product team. All companies have a tendency to lost bearing of its mission and focus on the wellbeing of the company owners or executives.

Youtube runs the way it runs because they don't care if a bunch of creators get hosed. All they care about is keeping the copyright wolves at bay.

Under DCMA, on receiving a valid counter notification they are free to put the content back on the site, remove the strike from the content uploader and let the courts handle the matter between the two parties.

https://twitter.com/leonardjfrench/status/130794817037951385...

It’s only if Google “ignore” valid dmca takedown notifications they are at risk of losing their safe harbour exceptions. How they handle the counter notifications can be pretty hit and miss. (but tbf, most of the time it’s because the person filing the counter notification didn’t fill it out exactly how Google’s bots like) I’ve seen cases where YT just automatically removed the strike from the account and restored the video on receipt of the counter notice. I’ve seen cases where they have locked the account even though the person filing the DMCA notifications has publicly admitted to taking down the video because it was critical of the work and the video only contained small pieces of the original content (exactly what you would expect to be covered under the criticism and comment portions of fair use exception).

They're not afraid of DMCA, but big money pulling out if they're insufficiently aggressive about claims...

Which is why they have Content ID. Big money can use the system to claim the ad rev / remove the content (without giving the user a strike) then putting the onus on the uploader of the video to then take the fight to the claimant for breaching their rights.

I don't know, but I've read that the counter here is that YouTube's system isn't actually DCMA - they are pursuing copyright over and above that system, and therefore the "protections" in DCMA don't apply - you broke a Google policy, not a law, and therefore you have no recourse.

There is "copyright take down requests" which are DMCA takedowns and gives the uploader strikes against their account and there is Googles own copyright system (Content ID) which Google would rather have right holders use (as they don't have to issue strikes to the users) that allow the claimant leave the video up but get the analyics from the video, claim the ad revunue from the video, mute the video (in cases where its an audio claim) or remove the video.

When you get a claim on the latter system (Content ID) you don't get a strike on your account but if you appeal the claim its (normally) upto the claimant to decide if your appeal is valid or not (sometimes YT does step in an say "yeah its fair use, have your ad rev back" but that is not the norm).

These takedowns are actual DMCA takedown requests. (even TeamYouTube are telling him to issue (valid) counter notifications to these claims - https://twitter.com/TeamYouTube/status/1306733040211824645)

They have ways to detect the "unusual". If I fill a form really fast, or something is weird in the state of my browser I get prompted to select crosswalks, road-signs or hydrants. It will be great if they would apply them on their algorithms too, when an "unusual" result occurs.

I'm now imagining a newly emergent dark pattern, where when you try to delete some expensive cloud resource, but before it'll stop charging you you get into one of those insane recaptcha loops that keeps saying "Click on all the Lovecraftian Gods that drive you insane just by looking at them!" or "type the text" where the displayed text is all in Druidic runes or something...

Sounds like the Land of Azure.

This is actually a pretty effective control. I almost deleted the wrong storage account in Azure the the other day.

We do not generally allow deletions of stateful resources to be done via automation after being bitten, and use the Azure lock mechanism.

We are now also changing our naming standards from “component-environment” to “environment-component”. In this example I almost deleted “app-prd-sa” instead of “app-dev-sa”. Much harder to do when you lead the name with “production”.

Anybody have “safer” naming conventions in use out there that I should be aware of? Didn’t find much authoritative out there via search; naming things is hard.

Reminds me of when s3 went down in 2017, and took a chunk of the internet with it (to include their own status dashboard) due to someone entering a command wrong.

https://aws.amazon.com/message/41926/

Other than websites and maybe some web-ish apps not too much else would be affected by S3 outage, a DNS outage would affect more of the Internet vs the web.

All of Google was gone from my country due to bungled router configuration for half a day some time ago. Now that was fun to watch. Everything from 8.8.8.8 to all their backend APIs for android, to all their web properties, went poof.

Yikes. Time to use a few different providers?

Websites and web-ish apps are not a chunk of the internet? Also there are tons of non-web-ish apps that depend on S3.

i work on multiple projects that are 100% data crunching with content stored on S3 and accessed solely by ec2 instances. there is no UI or public access. yet if S3 went down, we'd be up a certain creek without a certain self-propelling device and a big whole in the boat without a bucket (physically and in the cloud)

Maybe "the internet" was a bit too generic. But it destroyed a lot of tech worker productivity for the day at least.

Slack, GitHub, GitLab, BitBucket, Docker, etc all had issues.

https://venturebeat.com/2017/02/28/aws-is-investigating-s3-i...

There's actually quite a few enterprise client of S3. Lots in finance, too.

I was just coming to see if someone mentioned this! I work for AWS, and it's both surprising and heartening to see how often I see this anecdote get repeated, and many times by us to our own customers. We had teams looking for weeks at all of our other services for other instances of this.

It was a good operational lesson and one I'm happy to see shared still. If a command will let you perform a self-inflicted wound like that without checks, then it's time to review that command. Err on the side of caution and bias towards minimizing blast radius even if it means sacrificing some speed. "Move fast and break things" may be true at times, but not when the "*" character is ever involved. =)

Yeah its an interesting one. I tend to think of it in the context of, well you screwed up and deleted the wrong instance, but at least it wasn't that bad.

I also think about the response to it. Sure it sucked that it happened, and someone was probably feeling really bad about it. But it was a learning opportunity as well. Not just for them, or AWS, but for everyone in tech. Put those safety nets in place.

It is my personal opinion that you don't need "*" character ever there are other alternatives available always. And if avoiding that character makes it far safer I'm all for it.

rm -rf /+(?)

If anyone not understanding the joke, + character matches 1 or more instances, and (?) matches any one character. And / is start of entire filesystem. rm is remove. -rf (recursive, never ask verification). So it removes every file that you have rights to remove.

You can also do same thing accidentally by proxy. When my first computing teacher asked entire class to write anything to terminal to show that you had to write specific commands and computers don't understand normal language. Then she told to press enter. I asked her if she was sure. She asked why. I said that I wrote "format c:" . I have never seen teacher walk so fast in a classroom. I think that was last time she used that line in the classroom.

Because building it won’t get you a promotion at google.

That's what we called 'promotion oriented programming'.

Yeah. If the channel owner has spent so much effort it would be fair to expect actual human to spend literally a minute to verify it is legit removal case.

Because that would be sprints not spent making more money for Alphabet shareholders

But that’s not true though. Creators create content and make money with ads. No creators = no ad platform.

The number of creators harmed is less than the number of other creators, new creators and other factors. Loss of 0.00001% of the content on YT is, literally, no big deal. You, me, all other creators just don't matter.

That is BigG just don't give a f---.

Not sure your logic holds.

If Google is so big that 0.00001% loss of revenue doesn't register for them, they are surely big enough to spare 0.00001% of total engineering time to fix that loss?

In fact, I hear that as a common complaint that when working as an engineer at Google most of the time you are just making some system fractions of a percent more efficient.

Just to be sure, I am not saying that either your premise or the statement you are inferring is wrong. No opinion on that. I'm just saying that your premises don't lead to the conclusion. (But your conclusion might be right for other reasons.)

I'm saying that a loss of content does NOT create a loss of revenue.

OK, then it makes sense.

Weigh the cost of false positive against false negative. In the current landscape YT has little incentive to change.

and if you start implementing these sanity checks, they better be everywhere. Operators will start to rely on them and might become careless - they believe the system will catch any egregious typo. The one spot where the sanity check isn’t implemented will become more vulnerable.

How could some sanity checks be worse than NO sanity checks at all? In the YouTube case, I mean. This is happening ALL the time.

For the same reason human babysitters of autonomous vehicles don't succeed in preventing pedestrians from getting run over.

Right, but a human babysitter is better than NO babysitter.

An inconsistently applied sanity check creates a false sense of security. Like how a safety barrier that isn't strong enough to take someone's weight might be worse than no barrier.

Yeah, some people recommend something like

    alias rm=‘rm -i’

Unfortunately, as soon as you’re on another computer or at someone else’s shell, you’ll be more comfortable thinking less about rm-ing things and the sanity check won’t be there.

The better recommendation is to alias rm -i to del:

    alias del='rm -i'

And then get in the habit of using 'del' unless you really want a no-confirmation delete.

Then when you are on another computer or at someone else's shell you get:

    del: command not found

Instead of having a bunch of files silently deleted.

Another trick is to run

    touch —- -i

In important directories. Then rm -rf * expands to rm -rf .... -i ... and you get the “are you sure” prompt.

This strategy will fail pretty badly if you're on someone else's computer using Windows or DOS, where 'del' is the delete command.

Does `del -rf /` do a lot on Windows?

I doubt it, but `del *` should do the same thing on both systems.

Except that on windows, "del" defaults to prompting "are you sure" when one types "del *".

Right, I understand the concept. What I'm saying is that a safety barrier that sometimes doesn't work is better than rails that always guide you off a cliff; which is currently the YouTube model.

That's the same mentality of people stating because one form of pollution reduction or green energy generation isn't solving the problem that it's not worth attempting to do what you can when you can where you can. Those small moves are what gets the thing moving.

Many of the cases we hear about don't seem to be mistakes, including this one. Sometimes nobody can figure out any possible reason for a ban and sometimes it gets reinstated (that's the undo feature in action). But often there are obvious copyright issues or offensive material and the channel owner only goes clutching at fair use or freedom of speech after the ship has sailed.

Where is Bryan anyway he's been unusually quiet and I know he lurks on HN.

bcantrill where art thou

Have you heard about Oxide Computer Company? https://oxide.computer/team/

Because writing good software is haaaarddd. /s