> Instead, most software is made by people with the philosophy of “if it looks like it works, ship it”.
I work in secure systems and it’s shocking how many people believe this - the incentives from management are all about it too.
I work in secure systems and it’s shocking how many people believe this - the incentives from management are all about it too.
- nothing is, can be, or even should be 100% secure; the optimal rate of security incidents in society is not 0 (with apologies to 'patio11)
- security is a simultaneous trade-off against costs and usability, and those two other factors are more important:
-- security is achieved primarily through raising costs for attackers to beyond profitability, and reducing impact of such attacks (due to "not in isolation from the world" below, this also mostly translates to costs)
-- if "properly secured" (in the current cybersecurity sense) product/service cannot fulfill its function anymore, then you may just as well not make it; either way, no point in paying you for security work
- security isn't done in isolation from other systems and the world at large; "if this happens we'll go straight to filing crime report with the police" is perfectly legitimate security measure (even if it works somewhat less well on the Internet); similarly, "this is secured by us having insured against it" is also a valid solution to some security problems
Those running projects with these beliefs are sputtering and producing impressive PoCs that struggle to make it into production - either through underestimating the amount of detail needed to scale, or often throwing away good practices in favor of letting LLMs handle tradeoffs that later make changes slow to a crawl.
Theres plenty of good ways to utilize LLMs to speed things up, but so many teams got so incentivized by management to move fast at any cost that they’ve thrown out “load-bearing” good practices for software. That bet hasn't been paying off the way they’d hoped. It’s now clear that they thought they’d be able to massively downsize the engineering orgs. Massive token spend is giving very little RoI and now like other companies they’re trying to rein in the biggest spenders who are often not producing value.
Not all hacks are caused by pure negligence, laziness or stupidity, but most of them are. Even a little effort goes a long way.
My grandfather spent a couple decades as a builder, ran a construction crew. Whatever the project was, he wanted to know everyone he hired personally was going to reinforce and report to him anything they had the slightest doubt about. "Always hammer in an extra nail" was basically his motto.
What we do ain't that different. The difference is that when an apartment building collapses, it's bigger news than when a govenrment database does.
Also when a building collapses, people blame the builders. When software leaks user data, the engineers and companies face no repercussions.
We are headed for scary waters.
Only just when we started to have a resemblance of security we got agile and startups breaking things (making rubbish software to capture a few bucks faster) and now vibe coding and llm assisted hacking.
The point of my, arguably rant, is that there is nothing new under the sun.
It's not a guarantee this time will be the same - but it should temper the worry somewhat.
Previously no one was dumb enough to put that in one electronic database - it was on paper.
This is going to get orders of magnitude worse.
PS. Likewise, apologies, perhaps I was at fault for style: There was an underlying more serious point ...
... that, it is indeed a serious problem, that folks shouldn't be having their souls (or anything else for that matter) crushed, and that it indeed would appear to be an issue of conflicting incentives vs. management.-
After the DOGE debacle, I suspect that all the previously really secure stuff, is now out there, too. In fact, I wouldn’t be surprised if some of these leaks, came from that.
FBI employee data is very bad.
This might be part of it...
> Why would anyone with the expertise to make these calls bang their head against the wall trying to educate bureaucrats about these things
But I suspect this might be most of it: good engineering is boring (to the recipient). Preemptively solving problems gets no credit.
The only solution I can come up with is some form of certification or paid code review from a third party. I know that at least for Windows prior to 7 Microsoft actually allowed some parties to come in and check the code/checksum on an air-gaped computer. We somehow moved to "trust more" in the last decade, and now we can trust nobody
LLMs have been a huge force multiplier. Here.
If that data got out (which probably happened within hours of the data being dumped to insecure storage), then it’s probably already been analyzed and used to leverage access.
Why are you so interested in defending DOGE?