Hacker News Reader

Comment by josephg | Hacker News Reader

> It is unthinkable to me that anyone believes there is such a thing as computer security after so many years of nonstop hacks and leaks.

Of course there is. For example, SeL4’s security and reliability proofs still hold in the world of LLMs. The problem is that most software isn’t written on that firm foundation. Instead, most software is made by people with the philosophy of “if it looks like it works, ship it”. You don’t get secure software by working like that, because security vulnerabilities aren’t visible.

We - humans - know how to write secure software. Just like we know how to make safe aeroplanes. The problem isn’t that we lack the capability to make secure computers. The problem is we don’t have a culture of security. Secure software is - somehow - niche. And as such, it’s much more expensive. And nobody wants to pay.

View on HN

> Instead, most software is made by people with the philosophy of “if it looks like it works, ship it”.

I work in secure systems and it’s shocking how many people believe this - the incentives from management are all about it too.

I like to think behind every Dev anxious to ship half baked software sits an omniscient middle manager with a vague idea of what the product was supposed to do, maybe

This will all of course end when our AGI lords lovingly manage everything /s

No because it's a problem of human communication. Taking humans out of the loop creates other social problems that have been thoroughly documented in cyberpunk mythology, not actual a solution, just trading one big problem with multiple little ones.

Please note the sarcasm.-

Noted, but I think it's important to comment seriously because this is an important topic, sorry I didn't acknowledge the sarcasm, should have opened with something like "I for one welcome our AGI overlords"

"All our base are belong to AGI" :)

PS. Likewise, apologies, perhaps I was at fault for style: There was an underlying more serious point ...

... that, it is indeed a serious problem, that folks shouldn't be having their souls (or anything else for that matter) crushed, and that it indeed would appear to be an issue of conflicting incentives vs. management.-

It's worse: it seems like almost every developer on this very site is aspiring to be that middle manager, with LLMs as their underlings.

We are headed for scary waters.

Guess you don’t remember the days of ssl on login pages, ssl strip, exfiltering data via JavaScript prototype pollution, and a million other things like that.

Only just when we started to have a resemblance of security we got agile and startups breaking things (making rubbish software to capture a few bucks faster) and now vibe coding and llm assisted hacking.

The point of my, arguably rant, is that there is nothing new under the sun.

I'm not sure my worries are assuaged by thr fact that it's been bad before, too.

It should. We went through couple of cycles of "things are bad, inmates are running the asylum" before, and nothing of consequence happened. The world still goes on.

It's not a guarantee this time will be the same - but it should temper the worry somewhat.

Uh, even the database with all the compromising information on everyone with a US security clearance got leaked

Previously no one was dumb enough to put that in one electronic database - it was on paper.

This is going to get orders of magnitude worse.

I thought security clearances are a dime a dozen, and "all FBI employees" list is full of administrative work and basically 80% mirrored on LinkedIn? (Yes, the remaining 20% - or however much - leaking is a problem.)

They are not a dime a dozen [https://en.wikipedia.org/wiki/2015_Office_of_Personnel_Manag...].

Nothing new under the sun, it's the banality of evil all over again my dude. We never left scary waters, but they do seem to be growing more agitated. Is this the storm before the storm?

We're already 20 miles off shore and in the typhoon.

I believe the technical term is “Move fast, and break things.” MVP is a huge disaster. I can see it working for applications that don’t process PID, but only an idiot ships data handling software before it’s been dragged through a lot of testing. I tested my app for two years, before finalizing, and an LLM still found a couple of holes (minor ones, but ones I missed).

After the DOGE debacle, I suspect that all the previously really secure stuff, is now out there, too. In fact, I wouldn’t be surprised if some of these leaks, came from that.

FBI employee data is very bad.

The issue is that in consumer and enterprise software, move fast-and-break-things outcompetes secure-by-default every time. Critical infrastructure needs to have a different set of priorities, but it’s very hard because the expertise is so thin on the ground. Why would anyone with the expertise to make these calls bang their head against the wall trying to educate bureaucrats about these things for $150k a year when they can easily make multiples of that in big software companies that don’t own that level of risk.

> but it’s very hard because the expertise is so thin on the ground.

This might be part of it...

> Why would anyone with the expertise to make these calls bang their head against the wall trying to educate bureaucrats about these things

But I suspect this might be most of it: good engineering is boring (to the recipient). Preemptively solving problems gets no credit.

The incentives have to change. Any breach regarding PID should have fines as a percentage of revenue of the company. Any breach intentionally covered up and found out later by a third party should mean jail time for the C level. Yes, I know it is hard to make such laws "foolproof". And yes, in the current political and economical climate it will not happen anyway.

EU has these laws

The real problem is that even for companies that wish to pay more and wait more for secure-by-default can't easily tell the difference.

The only solution I can come up with is some form of certification or paid code review from a third party. I know that at least for Windows prior to 7 Microsoft actually allowed some parties to come in and check the code/checksum on an air-gaped computer. We somehow moved to "trust more" in the last decade, and now we can trust nobody

I've yet to see any form of certification or paid code review I'd be willing to bet critical infrastructure on. And working in safety critical software, that's not for lack of trying. Good review is usually harder than building a working system and the asymmetry of offense and defense applies to anything you miss.

If DOGE is going to have an effect on network security, it's not going to be for many more years.

Not really. It’s likely that the dumped (and compromised) data might contain things like keys and URLs that could be used to pry open other sites. Blackhats have become really good at following breadcrumb trails, and using “innocuous” clues to ascertain much more dangerous access.

LLMs have been a huge force multiplier. Here.

If that data got out (which probably happened within hours of the data being dumped to insecure storage), then it’s probably already been analyzed and used to leverage access.

That's an awfully exciting narrative you've spun.

Not really. It's par for the course. I didn't say anything that isn't common knowledge.

Why are you so interested in defending DOGE?

It's not as if best practices aren't well documented, or as if CVEs don't come out every day, or as if the information is somehow unavailable to even the most junior devs to take basic security measures.

Not all hacks are caused by pure negligence, laziness or stupidity, but most of them are. Even a little effort goes a long way.

My grandfather spent a couple decades as a builder, ran a construction crew. Whatever the project was, he wanted to know everyone he hired personally was going to reinforce and report to him anything they had the slightest doubt about. "Always hammer in an extra nail" was basically his motto.

What we do ain't that different. The difference is that when an apartment building collapses, it's bigger news than when a govenrment database does.

> The difference is that when an apartment building collapses, it's bigger news than when a govenrment database does.

Also when a building collapses, people blame the builders. When software leaks user data, the engineers and companies face no repercussions.

Spain just put in place fines equaling to 30% of revenue for data breaches - that’s the only way to make companies care about it directly.

The incentives from society are all about it. What company has ever faced serious consequences for hacks or data leaks? A cheap fine is just an unlucky cost of business.

I'd argue the management has a point there; without a pressure to ship, nothing would ever get released, because computer security has not yet understood the basic concepts that every non-computer security work does:

- nothing is, can be, or even should be 100% secure; the optimal rate of security incidents in society is not 0 (with apologies to 'patio11)

- security is a simultaneous trade-off against costs and usability, and those two other factors are more important:

-- security is achieved primarily through raising costs for attackers to beyond profitability, and reducing impact of such attacks (due to "not in isolation from the world" below, this also mostly translates to costs)

-- if "properly secured" (in the current cybersecurity sense) product/service cannot fulfill its function anymore, then you may just as well not make it; either way, no point in paying you for security work

- security isn't done in isolation from other systems and the world at large; "if this happens we'll go straight to filing crime report with the police" is perfectly legitimate security measure (even if it works somewhat less well on the Internet); similarly, "this is secured by us having insured against it" is also a valid solution to some security problems

what's patio11?

https://www.kalzumeus.com/

One of the winners of the HN pareto distribution lottery

As much as you're right, the attitude in most software development is not "lets make this as secure as reasonable," it is "lets make this barely functional and then move onto the next thing."

I don't think this is always true; mature security teams consider their threat models. There's just a lot of Schneier groupies in the field as well...

This is all fine and good and the sorts of conversations we were having on tradeoffs like any other engineering org. Theres always been pressure to ship, but now its that various pockets of AI Believers have popped up, egged on by a manic management, containing such beliefs that the code no longer matters, that it’s possible to move fast and fix it up later, human review isn’t important anymore, and that lines of code is a valuable metric.

Those running projects with these beliefs are sputtering and producing impressive PoCs that struggle to make it into production - either through underestimating the amount of detail needed to scale, or often throwing away good practices in favor of letting LLMs handle tradeoffs that later make changes slow to a crawl.

Theres plenty of good ways to utilize LLMs to speed things up, but so many teams got so incentivized by management to move fast at any cost that they’ve thrown out “load-bearing” good practices for software. That bet hasn't been paying off the way they’d hoped. It’s now clear that they thought they’d be able to massively downsize the engineering orgs. Massive token spend is giving very little RoI and now like other companies they’re trying to rein in the biggest spenders who are often not producing value.

> Secure software is - somehow - niche. And as such, it’s much more expensive.

Its not expensive because its niche, its expensive because its hard. Its a lot easier to learn a bit of html and javascript and knock up some web projects then it is to become proficient at all the things necessary to be good at security. Generally it also takes consulting with maths experts who have spent their life studying cryptography and as such command a decent wage.

> Generally it also takes consulting with maths experts who have spent their life studying cryptography and as such command a decent wage.

It doesn't take a maths degree to look out for SQL injection attacks or to audit software & write up a risk assessment.

There's very little, if any, incentive to do anything more securely than absolutely necessary. It is only a problem, when it is a problem and is treated that way in nearly every org I've worked in.

> There's very little, if any, incentive to do anything more securely than absolutely necessary.

There wasn't.

I agree that security isn't bottlenecked by our ability to learn. We generally fail at the grit needed to do it day after day, especially when the benefits aren't immediately visible.

Good cryptography is written once by experts -- never by you! -- and shared widely via free libraries. It's a one-time cost, not a recurring one.

Computer security is expensive but that expense has nothing to do with cryptography.

"Its not expensive because its niche, its expensive because its hard."

I'd counter and suggest it is expensive because almost nobody gives a damn about the first 3 layers in the OSI model, and have pushed most security responsibility up to layers 4-7. That's roughly half of your attack surface still exposed.

The bank has the best doors, the best locks, and the best cameras, and it is patrolled by a guard who props the doors open to so he doesn't have to keep fooling with the locks and points the cameras the other way to extend his smoke break. SeL4 would be another system used by humans.

It's always possible to break a perfect system by moving an additional layer of abstraction outward, and attacking one of the assumptions upon which it's built. Some of our era's highest security systems - game consoles - have been broken by undervolting them until the logic failed.

> It's always possible

It's often possible. But not all systems are vulnerable to undervoltage attacks. For example, I don't think the iphone secure enclave is vulnerable to this.

And good security uses "defence in depth". Multiple layers which each individually need to be compromised to break the whole thing. To hack chrome, you need a vulnerability in the renderer or VM. Then you also need a sandbox escape, and a way to use that to attack the browser's parent process. This is much harder to do.

> not all systems are vulnerable to undervoltage attacks. For example, I don't think the iphone secure enclave is vulnerable to this.

Then there's decapping / depotting, a world of different types of microscopy - some destructive some not, directed EM attacks, etc.

> And good security uses "defence in depth"

And automation has enabled "offense in depth"

> To hack chrome, you need a vulnerability in the renderer or VM. Then you also need a sandbox escape, and a way to use that to attack the browser's parent process.

Or you just phish the user into installing your exploit. There's always another layer. Always a potential exploit. Because ultimately the same properties of the universe which permit computation within a closed system allow for predictably observing and influencing it. The expense and hassle of doing so are widely variable, of course.

> There's always another layer. Always a potential exploit.

So what? Most attackers aren't nation state adversaries. They're some kid in Wyoming messing around with deepseek. We live in a world where most exploits happen because someone was running an unpatched, 8 year old copy of wordpress. Because they put their insecure mongodb instance on the open internet. Because they used admin / "12345" as the username and password. We don't need to make hacks physically impossible for a nation state adversary. Just really, really difficult and expensive to pull off.

Honestly. If people talked about physical security like they talk about computer security, you'd have people telling you that, because walls can be physically smashed through, they don't bother locking the front door to their house.

> Honestly. If people talked about physical security like they talk about computer security, you'd have people telling you that, because walls can be physically smashed through, they don't bother locking the front door to their house.

The saying is that locks only keep honest people honest. Plenty of evidence of that: https://www.youtube.com/@lockpickinglawyer

Sure, but also if there are a dozen bikes locked to the rack all costing $1000 and a $500 bike just sitting there completely unlocked, the average thief is probably gonna take the cheaper unlocked one and ride away.

It’s not hard to get into a garage but it’s really easy to steal a lawnmower if you leave the door open all night. I wouldn’t call that thief honest but even the minor deterrent of closing the garage was enough to make you not the target.

> The saying is that locks only keep honest people honest. Plenty of evidence of that: https://www.youtube.com/@lockpickinglawyer

That youtube channel is great. But it isn't evidence of anything. Except maybe for how terrible master locks are.

How many locked doors have easily breakable glass windows right next to them, nevermind breakable walls? What we need is a red-team waiver, and an AI model, and a budget, and say something like: your site has to be unhacked by the HackerAI3000 bot after 2 days on a 5090 Nvidia GPU.

I'd stick with always. Defense vs offense in anything reasonably complex suffers from one issue that simply cannot be overcome. To defend, you need to defend against every single possible imaginable attack, from now until forever. To attack, you need to find a single attack that works. And on a practical level all systems need to be accessible by somebody, yet that somebody is himself also now a part of your security structure and is never going to be 100% reliable, both in terms of corruption and incompetence.

> your security structure and is never going to be 100% reliable

So what? Security systems don't need to be 100% provably secure to add value. It's a mistake to let perfect be the enemy of good.

> It's always possible to break a perfect system

A perfect system is either extremely limited in scope or flawed in it's assumptions.

Yes, but that’s not bad. It boils down to a threat model.

I once worked on AUD 450M banking project, the root password was kept in a kickstart file and unchanged, root SSH was allowed. The bank didn't care until I told the external security auditor who included it as part of their report.

Snitch

I sure hope you're joking, warkdarrior!

I read it as an imperative

There's absolutely no way to account for humans, who can be tricked, or pressured, or just make human sized mistakes.

Then there's no such thing as security.

By the way, there are countless ways to account for humans. There are entire branches of engineering devoted to this. If you don't want someone to leave the bank with a pen customers use for signing checks, you just chain it to the desk. If you don't want the installer to forget to put the pen-chain in, make a photo of the chain part of the checklist required to get paid. If you want to... etc.

The idea is that you determine an acceptable level of risk, then secure to that level. Maybe the acceptable level of risk chosen by companies is wrong. Maybe we need to increase that risk exposure via heavier fines and regulations. Maybe the cost of reducing that risk is too high already. Maybe we need to fund that. Maybe it's too confusing and we need to research better standard practices. I dunno. But this is not some unsolvable problem.

> Then there's no such thing as security.

There really isn't

Ask anyone seriously involved in security - whether computer science related, or in general.

A thought experiment: Think about the most important secrets a country can have - now think how they are still discovered by competing countries, enemies, etc.

As long as there are humans in the loop there is a known weakness.

I'd love to see real stats on this.

We know about many famous cases of leaks - like the USSR stealing notes from the manhatten project. But I bet there are thousands of secrets which remain secret. We just don't actually know about them, because, y'know, they're kept secret.

This simply isn’t the case. There are plenty of secrets that every military keeps just fine. See how long we’ve gone without anyone ripping off the F-22? That’s not even a new airframe anymore.

This is EXACTLY what I was talking about. The tech that makes the F-22 an unmitigated terror of the skies is of utmost importance, so it’s still kept secret. The barriers around that information are obnoxious, but effective. What blood type a subsection of your military has is of much less importance. That’s why that data was stolen (See: OPM hack) and our best weapons remain secret.

Again, of course there is.

Decades ago, I worked in a bank in an old building. The door had a card reader for access. You boop your card and the door opened. People would hold the door open for each other all the time out of politeness, even when they didn't know each other. Security told us not to do that, but it's hard to convince people to stop being polite.

I had a laptop stolen from my desk in a place like that once. (Not a bank - but similar door-card reader system). This guy came in in the middle of the day, wearing overalls. He confidently walked through the door after someone, like he belonged there. He walked up to my desk, swiped my laptop and just strolled out.

At the bank, they've replaced the door with mechanical gates and a security guard. The gates - physically - only let one person to walk through at a time. You can't hold a gate open any more. And the security guards stop anyone who tries.

Is it 100% foolproof? No. But it's way more secure. It would have stopped that laptop thief.

There's this pernicious, defeatist attitude that if you can't make a system 100% secure, so you shouldn't try. That's misguided. Most systems can be made orders of magnitude more secure than they are today. It just takes a bit of care and work.

As soon as you make something foolproof, the universe evolves a better fool

This is the key truth that we keep forgetting

I'm not really sure what point you're trying to make, or how this relates to computer security.

Funny - now you're talking about context...

You seem mad about something I've said? I'd appreciate if you come out and say it instead of making vague insults, awesome_dude.

Fine. Make the universe work for it. The whole system becomes more resilient as a result.

Look at our immune system. Incredibly complex and clever, and able to keep us alive in the face of all sorts of pathogens. It exists because of this cat and mouse game, played over millions of years.

There's people in the highlands of PNG who regularly eat each other. Of course, many are thought to have died due to prion diseases. But now these tribespeople seem to have become largely immune to prion disease. Incredible.

We still get diseases though, proving the point that millions of years of evolution are still not enough to build a perfect defence.

Your comment on PNG, seems to ignore Kuru

It explicitly mentions Kuru as "prion disease", it ignores that the mortuary practice of eating various parts of respected dead has long passed in time .. although the lingering effect on the woman and children that ate portions of the brain in the 1970s, early 1980s, is still residual in a very few.

From an evolutionary PoV a perfect defence is overkill - with two separate defences against prion diseases in that region it's only the rare variation that causes any issue - and that rarely occurs before a new generation is birthed - ie. 'perfect' from the PoV of the selfish genes.

The claim is that people in PNG are "largely immune" to these diseases, where kuru shows that they are not, and the "cure" was a stop in the practice, not some evolutionary upgrade

They are largely immune to Kuru - the number of people that could contract it (ie. the number who ate brain) was significantly larger than the number of people who actually did contract it.

The resistance came about via two separate "evolutionary upgrade"(s).

I cannot find any such claim in the literature.

It appears to me that, like BSE (aka Mad Cow disease) it really depends on exposure.

It was specifically endemic to the Fore, not so much to the Yate and Usurufa, and not particularly at all to other highland people in the general region.

There's a twofer that skittled the Fore, a ~1900 mutation that created a new form of infectious prion proteins, and a local variation that saw less uptake in the Fore of a resistant prion protein (alongside other resistant prion protein).

So, over the highlands region, there was general resistance thanks to several evolved variations, in one specific locale (the Fore) there was insufficient resistance to the mutation that hit a peak of 200 deaths / annum for about three years(?) in the late 50s.

I can't speak to "the literature", I just had a lot of conversations with the people on the ground (Mike Alpers, etc), on again / off again, since the mid 1960s.

> millions of years of evolution are still not enough to build a perfect defence.

Who said anything about a perfect defence? And since when was that the bar?

The context is very clear - but, sure you can play word games, why not.

Just, you're doing it on your own.

What seems obvious to you doesn't seem obvious to me. I'm not a malicious or incompetent enemy. But you will need to explain your perspective for me to understand it.

Acshually a protective mutation was quickly being selected for (more precisely, the lack of it was strongly selected against).

Paper about Kuru and mutations referenced in comment here: https://news.ycombinator.com/item?id=49719102

The paper relies on the fact that people who were dying did not have the gene, but those that didn't ... did

That's not really enough to say "We have found the gene" - it's just really good data to warrant further investigation

Also, the incubation period of the disease is up to 50 odd years, have there been follow up studies?

The best way to make them work for it? Put it on paper.

Put what on paper? I don’t understand your comment.

Don’t store important and sensitive data in databases or computers at all. Put it on paper.

Computers make copying, processing, manipulating, and disseminating information easy. That is a bad thing for some kinds of information.

In cryptography there are protocols that account for malicious actors

SeL4 or proof assistant are not panacea. They do not help if assumptions about the task are wrong. And correctly formulating the task in real world is very messy.

So physical security is just as important. I really like how ARINC serial bus on planes work. One can have a reader that is physically incapable of sending anything to the writer. This allows to connect entertainment systems to flight data sensors safely.

In Airbus this system is replaced with Ethernet switches that in software ensures separation of traffic. The software was proven mathematically. But I am skeptical that it is absolutely bulletproof as a client under malicious control can influence Ethernet signaling and may exploit hardware bugs.

> SeL4 or proof assistant are not panacea. They do not help if assumptions about the task are wrong. And correctly formulating the task in real world is very messy.

Nobody said sel4 was a panacea.

My claim is that doing this kind of computer security is possible. It's just expensive and inconvenient. We know how to make computers a lot more secure than they are today. The limiting factor isn't humanity's knowledge. The limit is that barely anyone wants to pay the bill.

> The limit is that barely anyone wants to pay the bill

Do you even have any experience with how most companies work? SMEs barely have the cashflow to cover their daily expenses, let alone suddenly pay thousands for regular professional security audits and overhauls of their code. This is why security is an afterthought.

> This is why security is an afterthought.

Security is always a cost center and rarely a profit center. That's the only thing that needs to be said.

Security is defence, it is never a profit center unless your business is providing security services!

Did your comment add anything to mine?

If restaurants can't make sure their food is safe to eat, they shouldn't be allowed to be in business.

If builders can't build houses to code, and the buildings fall down, they shouldn't be allowed to stay in business.

If civil engineers build bridges that fail. Or doctors hurt patients. Or police officers shoot innocent people, they shouldn't keep their jobs.

Software engineers are no different. If you collect my user data and it's at high risk of leaking on the dark web, either clean up your act or close shop.

The main issue is that market evolution will always surfaces the most cost-efficient entities within the ecosystem pressures.

That means designing the ecosystem pressures is crucial: things more meaningful than just pure capitalist private-profit logic must by enforced by thoughtful regulation or otherwise the ecosystem converges for private-profit of a small sliver of individuals (billionaires) to the detriment of all other ecosystem members (99% of the world population).

This holds for anything broader than pure private gain, may it be security, social fairness or ecological topics. Sole monetary-value optimization for private gain must be properly constrained or else it results in pure predatory capitalism that implodes society from within, may it be through leaky security, poisoned environments or social unrest.

Yeah this is an interesting argument with respect to why we keep burning carbon. For both software and systems security and rapid climate change, we know the solution. But we haven’t been able to sway the incentives and the system keeps churning out bad results.

> just expensive and inconvenient

You mean something is theoretically possible, but in practice only works at small scale and is otherwise effectively impossible. You only have so many resources for all those big topics.

And after you spent all the world's resources on the "perfect", formally bug-free software, you get hacked via social engineering or malicious insider.

> My claim is that doing this kind of computer security is possible.

But your evidence does not support that claim. SeL4 has proven that it is possible to design a secure microkernel and prove its security guarantees. It does not prove that you can build entire systems (filesystem+database+web server+browser) on top of that kernel while maintaining the same security guarantees.

I'm all for improving the state of computer security, and I'd love for capability systems like SeL4 to become more prevalent. But it's only a microkernel, and it's by no means certain that the PeopleSoft vulnerability exploited here required a kernel-level compromise.

I don’t expect every piece of software written to be proven correct like SeL4. But I don’t think we don’t need to do that to get big improvements in the security of a lot of systems. Honestly the biggest insight I take from sel4 is that we can get improvements in security by breaking up a large program into isolated pieces. Give each piece as few permissions as possible, so a compromise of one part doesn’t lead to a whole system compromise. And give them a way to talk. This has big benefits for reliability - since you can fail and restart individual processes. And it has benefits for security, since a system compromise should require an attack of multiple systems simultaneously. And it has benefits for debuggability, since you can add tracing at the comms layer or isolate modules for testing.

Wasm does this. Erlang does this. SeL4 does this. Chrome is built this way. The windows driver model is moving this way. And so on. You want a solid core to build around - which is what SeL4 and beam try to be. Then it’s up to us to use those primitives and build good software. Combine that with a memory safe language (rust, go, c#, etc) to protect against buffer overruns and use after frees. And a picture starts to form of how you can build software that is a lot more secure by default.

I don’t think perfect security is worth the cost for many companies. But so many security leaks happen because of amateur hour somewhere. Bugs happen - I get that. But a single bug in a C++ program shouldn’t immediately lead to RCE with system level privileges. This stuff isn’t rocket science.

> ...there is such a thing as computer security...

> Of course there is. (...continues to talk about software security)

A bit over-optimistic when looking at hardware vulneraribilties like Spectre/Meltdown.

Secure software isn't worth much when it runs on vulnerable hardware (at least Spectre/Meltdown could be worked around in software though, but at a cost.

There's "no such thing as computer security", because this one time computer security researchers found vulnerabilities?

You know that's their actual job right? That and fixing the problem. Which they did in both software and hardware.

One time? lscpu on my system lists two dozen CPU vulnerabilities known to the kernel. I stopped tracking them a long time ago when it felt like every week brought another information leak or bypass.

When I run that on my computer I see a big list of CPU vulnerabilities. All of them are either mitigated in software or fixed at the hardware level.

Looks to me like the security researchers have been doing their jobs.

Yes, except that they haven’t been able to fix this completely in sw and the community knows this.

Fix what?

NASA was supposedly a CMM level 5 organization and they still managed to crash a mars probe because of bad unit conversions.

Did any martians hack the probe?

Jeff Goldblum with a thinkpad, maybe

Jev Goldblum? He’s all the hype today

Artificial Life, um, finds a way

Martian packets are those with source addresses like 192.168.1.1 (on the public internet), 127.0.0.1 or 0.0.0.0.

A piece of software Lockheed made gave its outputs in US Customary units (not following their specification), and NASA expected it in SI units (as their specification expected)

That should have been checked multiple times by NASA before they launched it.

It seems like it should have been checked by Lockheed not NASA since supposedly NASA provided a specification, that specified the units, and paid for the software no?

I just had a quick scan of the incident report (https://llis.nasa.gov/llis_lib/pdf/1009464main1_0641-mr.pdf) and:

> "The output from the SM_FORCES application code as required by a MSOP Project Software Interface Specification (SIS) was to be in metric units of Newtonseconds (N-s)"

(MSOP = Mars Surveyor Operations Program). One of the recommendations was

> "Conduct software audit for specification compliance on all data transferred between JPL and Lockheed Martin Astronautics"

So yes, NASA should have checked the provided software more thoroughly, but also Lockheed should have actually followed the spec they were given. I doubt the SIS is available online to check any harder

Should have been checked by both. Lockheed should have checked because of contractual obligations; NASA should have checked because of minimal engineering practices (bugs happen in all your dependencies).

They happen and some are put there on purpose too. In this case it's not a "bug" in your dependencies, and yes NASA since it's using public funding should have been more careful, but ultimately it's Lockheed that was paid to do something no?

I work in aerospace.

Both parties fucked up.

Lockheed's job is to follow the customer's specifications.

NASA's job is to check to make sure what they paid for is what they received.

I do this every single day as a quality inspector here. I don't know why a bunch of highly-degreed engineers can't do a simple job that a person with oonly a GED does without fail.

Actually no, it's Lockheed's that should have had a test-suite and conformance-suite for it, but probably only has a C-Suite. If the program is non-trivial proving its correctness can be very expensive to prove, both money and time wise, and something that is done contrary to the spec is obviously on the one executing the spec and being paid for it, which probably wasn't cheap and probably these expenses add to the "NASA only burns money..." narrative.

"Actually no, it's Lockheed's that should have had a test-suite and conformance-suite for it"

Tell me you don't run AS9100D quality inspections without saying so directly.

Well, it seems like simple test & conformance suites would have been enough to catch this, so you wouldn't need to run AS9100D quality inspections.

Besides it looks like the first AS9100 Standard was released after the incident even happened - perhaps even as a result of this.

So it's totally irrelevant that you both audit software in the space and know the Standard, no pun intended

AS9100 is based off of ISO9001. It doesn't fucking matter, the responsibility is on both parties.

Again, tell me you don't actually handle quality without directly saying so.

No, if NASA chose not to triple check everything before sending something into space then they are as much to blame as the supplier. Blindly assuming something works like it should without testing it makes an ass out of u and me.

Very fitting for the AI age as well. "Yeah I just put the specs of the project you're paying me to do (in NASA's case, probably 100% public funds) but Claude the gimp missed the units because all previous training data use Stones and Yards as units, you should have verified it yourself! I just prompt!"

That is why the good lord invented acceptance tests.

Yeah it's about costs. I care about a lot of things, but the causes I actually give money to is a much shorter list. If personal data was radioactive, and leaking it cost companies real money, then more money would get spent on security. (and insurance, and lawyers.)

Yeah I've long said we should treat data leaks like food safety. The only way we'll see software security improve is if there were serious fines and/or jail time for leaking user data due to negligence.

PCI compliance works pretty well.

I would like to see restricted access to only us workers for us citizens data though.

Transferring us customer data outside the country should be illegal

PCI compliance is a joke. That isn't even close to good enough. I'll break all your PCI compliant stuff trivially.

What’s your main issue ? The compliance audit being lax or the compliance not being assertive enough ?

> SeL4’s security and reliability proofs still hold in the world of LLMs

Oftentimes the weakest link is the human operator who has direct access to those systems, not the computer system itself. Good old social engineering, in other words.

Even though one could use LLMs for social engineering, come to think of it, like re-enacting the movie "Her" involving a modern AI as Scarlett Johansson and an engineer working for the water utility as the romantic target.

> Oftentimes the weakest link is the human operator who has direct access to those systems, not the computer system itself.

If that is the case, the computer security team have done their jobs.

I wish that were true more often.

How the software is written isn't that important because the endpoints aren't secure. Their hardware and firmware isn't even secure enough.

<< And as such, it’s much more expensive. And nobody wants to pay.

Eh. If only it was that simple. I mean, yes, money is always a factor, but not nearly as big of a factor as 'my convenience outweighs pretty much everything ( until it causes sufficient amount of havoc.. and even then.. )'. You can see it in just about everything. It is not just the money. It is the convenience that drives most of the unsecure behavior.

Cybersecurity is a fragile system in Taleb's fragile-robust-antifragile framework. That is, as t->infinity, the probability of a hack approaches 1, because you only need to slip up once, and there is a small probability of the system's maintainers slipping up each day.

mind that just because SeL4 micro-kernel is proven secure, there's no guarantee that all leyers on top of it are. There's a presentation (https://www.youtube.com/watch?v=LhUwwsVq5E4) that's pretty much about "so we gave sel4 to students, and NONE of them managed to build utilities on top safely - now we have to provide our own Core Platform implementation to be sure"

But most important of all, the highest level - human operators - are not provable secure anyway. Any castle gate can be opened from inside - so why have the gate anyway? Security by absence is absolute

Lol, just because a tool exists that in theory can be used to potentially do part of what it takes, doesn’t mean anyone will use it that way consistently or reliably.

And one screw up, and it’s out.

Is your claim sel4 hasn’t had a ton of vulnerabilities? Because a quick google search shows hundreds

can you link some? my search results are rather subpar

the actual answer to this is accreditation. you will see safety standards rise across the board if engineers risk losing their license for shipping unsafe features.

sel4's guarantees break if you have DMA, e.g., from your NIC. It doesn't help with timing attacks. It doesn't cover your network stack. AFAIK, no one has taken up the mantle from Project Everest, so you'll need to write a verified TLS library. Once you've built all that, you can start thinking about your database/application/whatever. Then of course you'll have to verify all your dev's machines and scripts to ensure nobody is misusing a credential that can get stolen.

"So what?" you say. "Making a heavier-than-air metal tube take off and land millions of times per year without a catastrophe is also hard, and we no longer expect most or even many of those tubes to blow up or fall down."

Mother nature is not spending $$$ using AI and HI adversarially trying to find the exact combination of atoms that will cause your device to fail.

> sel4's guarantees break if you have DMA, e.g., from your NIC.

Modern CPUs support IOMMU. If you set that up, your NIC can only DMA to virtual addresses, managed by the operating system.

> It doesn't help with timing attacks. It doesn't cover your network stack

It does help with all this stuff, because your network stack and whatever else can be split off into isolated processes which talk over capabilities. Compromises in those processes are of course terrible. But they don't automatically allow kernel level takeover of the whole machine like on windows / linux.

> The problem isn’t that we lack the capability to make secure computers.

Depends on the "we". "We" have the capability to make secure computers like how "we" have the capability to make EUV lithography machines. There exists a relatively small number of people and organizations in the world who can do so. Microsoft does not have that capability. Google does not have that capability. Linux does not have that capability. Amazon does not have that capability. Apple does not have that capability. Cisco does not have that capability. IBM does not have that capability. etc. All of those organizations have tried for literal decades, thumped their chests about how they have awesome security year after year, and yet have totally and utterly failed despite their best efforts.

Acquiring the capability to do so is difficult and challenging and requires years to invent if you start right this very second and know what you need to do, which these organizations emphatically do not. We need security at scale and fast. The only way forward is to scale up working solutions rather than letting the bozos who put us in this spot fail at scale with yet another promise that this time for sure they will solve the problem they have repeatedly failed at for decades.

It's nothing like EUV.

> Microsoft does not have that capability. Google does not have that capability. Linux does not have that capability. Amazon does not have that capability. Apple does not have that capability. Cisco does not have that capability. IBM does not have that capability. etc

This is all by choice. They could easily have that capsbility, very unlike EUV.

And the reasoning behind that choice is looking at the tradeoffs and saying “shipping more features faster is more important”

Part of the argument (aiui) is that there is no way that culture could ever change to producing secure systems. There are too many weaknesses embedded in the organizational structure.

Yeah, just like a lot of organizational change, one approach would probably be to put together a small Skunkworks-type team to nail it on one specific product. Existence proofs have, in my experience, a pretty powerful effect on the naysayers.

Oh, Microsoft can just figure out how to make unhackable systems, they just choose not to. They spend all of those billions of dollars per year on security and spent all of those decades on failed attempts as a prank.

You really think that if they could have they would not have, even just for bragging rights? Or are we going with that it is some kind of task demanding enormous expenditure even though the organizations that have made secure systems are infinitesimally small in comparison?

Microsoft has spent orders of magnitude more money and time than the organizations that have succeeded and the result of their efforts is Windows. That says everything you need to know about their capabilitys.

Multiple literal trillion dollars organizations have spent literal decades failing at it. You are really underselling the capability gap.

It's not rocket science.

If windows was reimplemented as a capability based microkernel like sel4, it would be far more secure. Run drivers in their own isolated processes. Do interprocess communication between them via capabilities and shared memory. Remove all ambient authority from programs. All the programs a user launches stop automatically inheriting all of that user's permissions.

There's no secret knowledge required to do this. The SeL4 team has written extensive documentation of how they did it. They also opensourced their kernel implementation, with correctness proofs for the whole thing.

The reason windows hasn't done it is the cost. You'd have to rewrite half of the NT kernel and refactor everything else. All existing windows drivers would need to be rewritten. If you forced windows userland use a capability based system, you'd essentially be inventing a new way to write windows programs. You'd need to document that, and write a compatibility layer for legacy programs. And solve some UX problems. It would be terribly inconvenient for everyone. Oh, and some programs would run slower as a result.

They could do it if they wanted to. But microsoft just doesn't care about security as much as they care about performance and compatibility. Linux is the same.

The big irony is that security would be a lot cheaper for microsoft if they designed the NT kernel to be more like sel4. Microsoft has to spend millions on security every year because any tiny bug in the kernel (including in drivers) might result in the whole OS being compromised. In a microkernel, a buggy driver is nowhere near as dangerous.

Your knowledge is out of date: Windows has supported capabilities from the start of NT and does run a lot drivers in isolated processes, along with most OS services. It has done for years. It has the most sophisticated IPC framework of any OS (DCOM) which is integrated with the operating system kernel's security frameworks, and used pervasively both internally and by apps. And Microsoft has created a new way to write Windows programs in which apps are expected to advertise the capabilities they need (see WinRT and MSIX). This is also over 15 years old.

macOS does the same but with more developer adoption. Apps don't have the ambient capabilities of the user and must advertise what they need via entitlements embedded in the binaries, or get permission just in time.

Which is all very good, and modern platforms are much more secure than they once were. Yet "capabilities" as a silver bullet are academic overpromises. This article I wrote is more about language/runtime level capabilities but OS capabilities are not much better.

https://blog.plan99.net/why-not-capability-languages-a8e6cbd...

SeL4 isn't secure because of One Weird Trick that others would adopt if only if they could be made to care enough, it's "secure" because it hardly does anything, which is why nobody uses it and why it has no impact on real world computer security.

The hard part of desktop security is not changing the operating system. The hard part is getting app developers to care. Most security features added to operating systems are ignored by developers, which is why Apple forces you to adopt some of them as the price of admission to the app store. If they didn't nobody would use them, as can be seen for apps distributed outside of the app store. The reason is security is a market for lemons. Nobody can see the result of security investments so it's irrational to invest. SeL4 has no solution.

> Your knowledge is out of date: Windows has supported capabilities from the start of NT and does run a lot drivers in isolated processes, along with most OS services.

Thanks! I quickly googled this point before posting earlier to make sure I was still right. Gemini helpfully told me that yes indeed, drivers in windows run in the kernel's main process. Thanks, AI.

> This article I wrote is more about language/runtime level capabilities but OS capabilities are not much better.

I think I responded to this article at the time. I still find this article somewhat confusing and unconvincing. For example, you conflate Java's SecurityManager with capability systems, even though it seems more like an permission based access control system. Then you point out many of its weaknesses. To what end? What conclusion about capability systems am I supposed to draw from a criticism of this quite different security model?

A capability is not a permission flag. Unlike your example, a good capability system would generally pass all HTTP requests to a given endpoint through a single capability object. You wouldn't need different caps for each HTTP method like SecurityManager apparently requires. It's like file handles. You don't create several different file handles to interact with the same file, one for reading, one for writing and so on. We just open the file once, with whatever options are needed. Then the file descriptor can be passed into any function which needs to access that file. And whatever code receives the file descriptor doesn't know if they're talking to an actual file, or some in-memory object or something else. Just like a virtual object.

You also say this:

> File descriptors are a kind of capability provided by the kernel, but a rather odd and inflexible kind. They aren’t a great example of object capabilities.

Huh? File descriptors are often treated as the canonical example of object capabilities. This comment makes me wonder if we're even talking about the same thing. At the risk of being indelicate, are you sure you know what capabilities are? Can you give a definition of object capabilities which doesn't describe file descriptors?

The point about god objects lands. I also agree that trying to retrofit a language like java to make modules unable to share memory is difficult. But many aspects of language design work like this. Consider garbage collectors. Before GC languages existed, I could write the same article talking about the difficulties of hacking a GC into C. But that wouldn't teach me anything about how well a GC would work in a language like Java or Ruby.

Anyway, the main advantage of capabilities is the ability to split programs out into sub-modules such that a compromise or bug in one part of the system doesn't lead to the entire system failing. We can argue about whether bringing this into the language runtime is a good idea. But I feel pretty confident that this sort of separation is a good idea at the systems level, helping with security and reliability. We can look at Chrome, SeL4, Erlang and - apparently - windows for examples. Even if they don't all think of this as a capability based problem.

The SecurityManager exists in complement with Java capabilities. You could, in theory, use the Java language with a different standard library and implement a pure object capability system, as the language rules allow for that (if you disable reflection). But it would have bad ergonomics and still end up with a SecurityManager equivalent because a DSL for permissions is more convenient than doing it all in code.

Consider the most common task the SecurityManager was deployed for: stopping plugins calling System.exit() by accident. One might say, the right to exit the process should be an object capability. OK. But then where does that object come from? Java programs start at main() and it doesn't receive an object.

You'd need a new design where you pass in a god object to main(), which in turn has properties giving access to a ProcessExiter interface or something similar, and then any code that genuinely needs to exit the process would need to request it in the function arguments, threading it down the stack. You'd get an explosion of types. A simple permissions DSL is much easier to write and reason about, and it gets out of the way when you don't want sandboxing.

Why would you want all HTTP requests to flow through a single capability object? I think it's pretty common to want to let code do GETs but not POSTs. You end up wanting pretty fine grained permissions in a lot of real scenarios.

File descriptors are poor object capabilities because the interface they implement is fixed by the OS, except then there's a weird ioctl escape hatch that isn't properly typed, reflectable, wrappable or interposable. To see what can go wrong with this, consider a recent fix to the Codex sandbox on macOS:

https://github.com/openai/codex/pull/46500

The sandbox forbids writing to a file descriptor except, oops, someone at Apple forgot about the F_TRANSFEREXTENTS ioctl which is still allowed on a read only fd. It should be possible to do what is expected here and just pass in a read only fd where all you can do is call read() and maybe seek(), or perhaps pass in an fd where a specific ioctl is the only thing you can do, but POSIX has no concept of this.

A good example of an object capability system would be Mojo, which I describe in the essay. You can create objects representing capabilities and pass them between sandboxes, in an unforgeable way.

We live in a golden era of prototyping so if you wanted to make a language where everything is a capability passed into main(), you could. The code doesn't have to be executable, you could just mock out some realistic programs and see how the code feels. My guess is you'd need a lot of language features to hide the explicit object capabilities away for ergonomic reasons and it'd end up feeling a lot like a SecurityManager based system.

> But it would have bad ergonomics and still end up with a SecurityManager equivalent because a DSL for permissions is more convenient than doing it all in code.

This might be the core contention. I don't know if using actual capabilities in a language would have problematically bad ergonomics. You'd probably be passing more arguments to functions. But haskell seems to manage ok despite needing to pass IO to functions that need it. Capabilities seem similarly inconvenient. I think I'd need to see it tried. I agree - I might need to try it myself.

> Consider the most common task the SecurityManager was deployed for: stopping plugins calling System.exit() by accident. One might say, the right to exit the process should be an object capability.

I don't think this is a great example. Caps are generally for resources outside of your program or module scope. A program already has the capability to exit, so that wouldn't be something you would pass in from outside of the program.

> Java programs start at main() and it doesn't receive an object.

I agree that retrofitting caps into an existing language like java would be difficult and inconvenient. Passing a "god cap" to main() is the easy part! The hard part is just how much of the standard library implicitly depends on ambient authority. I've thought about doing this in rust, and concluded that I'd probably need to fork rust's std library.

> You'd get an explosion of types.

I've never heard of that stopping java programmers before.

The way SeL4 handles this is to have a generic call() interface for capabilities. It's very simple, and it would work fine in this example.

> Why would you want all HTTP requests to flow through a single capability object?

Capabilities are a combination of resource + access rights over that resource. If I wanted to give a module access to a REST endpoint, I'd make a cap representing that endpoint. The resource is the URL base (eg "example.com/foo/bar"). And I'd also specify access rights (eg only HEAD+GET, or HEAD+GET+POST or whatever makes sense). Then pass that object around to any modules which need access. I'd even keep the URL prefix private in the capability object. The capability object would only expose methods for http_get(), http_head(), http_post() and so on. This design would be more or less impossible to misuse. And it would be super handy for unit testing and dev environments.

It's not "one capability for everything" and it's not "a million fine-grained access rights". You want one cap per semantic resource, just like one fd per open file. If you want to refine the granted permissions, just reimplement the same interface with a different implementation of http_get() and friends. (Or, simpler: just wrap your existing RESTEndpoint class with another class which adds your extra checks).

> except then there's a weird ioctl escape hatch that isn't properly typed,

This is a flaw of the unix syscall API. In comparison, SeL4 only has 9 syscalls (plus 2 for debugging). The syscalls just let you call capabilities, and have your capabilities be called by other processes. And yield(). That's all the syscalls on sel4.

Because everything runs through that same API, it's trivial to stub out or replace capabilities provided by different components. Eg, any program can reimplement the filesystem API if it wants to. No need for FUSE, or special loopback mounting or anything like that. Because the filesystem is just a userland process which doesn't have access to the kernel's memory, there are no ioctls that you can accidentally forget to sandbox. The only special thing about the filesystem is that it holds a capability to do raw IO on the block device. (And that cap, in turn, is provided by another userland process.)

> My guess is you'd need a lot of language features to hide the explicit object capabilities away for ergonomic reasons

Yeah, I think that's our big disagreement. You seem to think that hiding object capabilities would be a necessary design choice. I think using caps directly would be much more ergonomic than a SecurityManager style design because custom caps can just be implemented in normal code. And caps are better because they encapsulate a resource, not just access control rights.

> > But it would have bad ergonomics and still end up with a SecurityManager equivalent because a DSL for permissions is more convenient than doing it all in code. > > This might be the core contention. I don't know if using actual capabilities in a language would have problematically bad ergonomics. You'd probably be passing more arguments to functions. But haskell seems to manage ok despite needing to pass IO to functions that need it. Capabilities seem similarly inconvenient. I think I'd need to see it tried. I agree - I might need to try it myself.

You can try it now yourself in Haskell! This is my effect system, based on capabilities: https://hackage.haskell.org/package/bluefin

One of the common objections I hear to Bluefin is "isn't it too inconvenient to pass around capabilities everywhere?". Perhaps surprisingly, no, I haven't found it remotely inconvenient. I find it liberating, actually.

Spot on. And it's gonna get a lot worse now that "but my slop machine's code is run through a really large number of tests, like enormously many, I don't need to understand the code!" is apparently not an embarrassing way to do development anymore.

Reply on news.ycombinator.com