Hacker News Reader

Comment by Aeolun | Hacker News Reader

I think the problem with this logic is it hinders a hundred people on a daily basis to prevent one person from doing something stupid once a year.

That’s just not great ROI.

View on HN

I've seen individual engineers make a mistake that caused 100 engineer's salary worth of damage.

I suspect we all have. The question is rather: how much would it have cost to prevent that mistake?

The flip side is the insurance argument: given a big enough organisation, a mistake like that will happen, even if the probability of each individual mistake is very low. So assuming one of these will happen, what can you do to reduce the impact or to reduce the latency of fixing such a mistake?

I would love to know more about the details of this example :-)

So the company I work for made the front page of HN last year for a security breach. I won't go into details, but I will say one factor was a particular instance where security was overlooked for many years due to convenience.

In response, the company has now sort of swung the other way: lots of things that have a perfectly valid reason and are reasonably safe have been forbidden or suddenly have needed to be justified in the face of extreme skepticism.

That wasn't a case of a single engineer causing 100x engineers worth of damage, but it was the case where something made hundreds of people's jobs a tiny bit easier for two decades, then caused a massive public loss of face and who knows how much reputational damage.

And I'd be willing to bet that a fair number of HN readers work at a company which has at least one system with similar properties (a bit more convenient, a bit less secure) to the one implicated in our security breach; but failure is a once-a-decade event, and their company isn't nearly that old yet. Would those engineers defend that setup using the argument from this article?

Was it the Twitter internal account management tool?

Nope. :-)

Here's an example: https://news.ycombinator.com/item?id=24196131

Here's another famous example where the damage was thousands of person-years of salary: https://www.henricodolfing.com/2019/06/project-failure-case-...

"Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems." [1]

1. https://aws.amazon.com/message/41926/?ascsubtag=e7c5af5933bd...

Indeed, but did those 100 engineers increase the company bottom line by 10% more because they weren’t hindered by too much red tape, and is that value a multiple of their salary?

Also, do you expect mistakes of that magnitude to keep being made?

The Knight Capital error really messed them up. https://www.reuters.com/article/us-knightcapital-results/kni...

Anyone who has spent a bit of time trying to calculate the lost productivity caused just the meetings needed to discuss breaches (internal, accidental), knows it's great ROI.

Everyone greatly overestimates ROI in their own area of expertise and greatly underestimates costs in the rest.

You might be right in some special instance, but generally you are not. One example: Sending files via Email is a security problem. So you prohibit this via settings, virus scanners, appliances, etc. Now you've solved the emailed worm problem. However, you've just created a "how do we move files/data/screenshots/..." problem for the whole company. Now everyone will need their own special solution for everything, support will need a screenshot upload tool because you cannot just email them screenshots anymore, marketing won't be able to do images in mails anymore, external people won't have access to internal file shares, etc.

I appreciate you're giving an example, but this is pretty much solved at most big companies currently with OneDrive (or any other flavor of cloud storage). For external sharing, you just select if the link can be seen external to the org.

Doesn't SharePoint trivially solve all this?

For external people likely not, usually Sharepoint would be behind some VPN and Firewall. Some cloud service like Dropbox might work if allowed (which it often isn't). And then there will often be the resulting sprawl of various services: this for secret internal stuff, that for external people, yet another thing for marketing, etc.

I don’t think I’ve ever heard Sharepoint and ‘trivially solved’ in the same sentence before.

The problem with using sharepoint is that now you have sharepoint, and people will start using it, with all attendant horrors.

SharePoint is fairly good at sharing files: it allows upload by drag and drop, it supports versioning and approval of documents with a decent user interface, it offers a reasonable organization of users and permissions. Running SharePoint is anything but trivial, but with some discipline it can actually solve problems for end users.

And you end up creating far more security holes as people work around it.

Shadow IT is great, it makes companies actually work rather than go bust.

Please tell this to my companies IT department. They've finally moved to Azure and Onedrive with it, but are still forbidding any files downloaded to devices or email attachments being send to external addresses because otherwise it's unsecure.

So everyone is still using WeTransfer/Dropbox/whatever like they were in the years before.

We keep a low profile. Separate network, very limited interaction with "corporate IT" (which is outsourced). We keep infosec onside by making our liason feel we are across things. Allows us to sidestep a lot of problems, because ultimately in a large company people don't have to

Took me 15 years to realise that incompetence is at worst ignored, and often leads to promotions.

I've spent 3 years telling corporate IT they have a problem with a router config that limits throughput, we got 3mbit rather than 1Gbit. They spent 3 years insisting it wasn't them, and it was the upstream ISP. I even managed to get read only snmp access, and generated cacti graphs of their router showing 400M (iperf2 in udp mode, so 400kbit/ms) going in on port channel 1, but not emerging on the ISP interface.

My shadow-IT deparement spent 3 years paying for a completely separate network connection to bypass the corporate IT one and meet our requirements (easy to do when it's a remote branch office in another country), other departments just suffered it. It was the backup link so only used about 1 day in 10.

Eventually a senior member of (non-tech) staff resigned over the issue and it started being taken more seriously.

The way this system is designed (giving the front door to corporate IT) was done over my objections, and the objections of many others, but on paper it was good. Corporate IT provide shiny SLAs (which mean squat).

Last week, 12 months after the resignation and the beginning of taking it seriously, it had been escalated through 4 different layers of corporate IT, and eventually they came back and said "we've found an errant access list and removed it, and it's now fixed".

That's it. 3 years telling them what the problem was, 3 years of being ignored, and what happens? Certainly no blame for the idiots that made the decision to use this, no comeback on corporate IT provider, but if people find out about shadow IT they kick up a fuss (so the trick is to keep quiet and keep good personal relations with potential pain points).

Oh yes, we outsourced our corporate IT. Obviously there's no money coming back, I suspect we'll get a bill.

This just tells us that we shouldn’t discuss breaches so much? My one experience with a breach has been that the whole process around it was a collosal waste of everyone’s time.

It's a great ROI if that one stupid thing a year is someone pushing your company's codebase and all your trade secrets to a public repo.

By the same logic it’s good to have your company located in an armored bunker underground on the off chance that someone nukes the city you are in.

I realize the chances are a bit different, but it illustrates the point.

Never underestimate the damage one person doing something stupid once can do.

I actually think this whole one person doing one stupid thing per year argument is grossly underestimating the amount of stupid available.

Stupidity is one of humanity's greatest renewable resources.

Depends on the impact of a single erroneous action. I’ve seen cases when folks would push to GitHub passwords (that should be kept in git to start with).

right, pushing github passwords leads to GDPR breach in end - anything preventing GDPR breach in the end is probably a great ROI.

Depends a lot on the company and what it's mission is, not every company needs to be able to move quickly on every front.

Reply on news.ycombinator.com