Hacker News Reader

Comment by intrepidsoldier | Hacker News Reader

Just the beginning. AI is going to expose how fragile the entire computing infrastructure in our world is.

No, it will be a tsunami of new discoveries in old code bases at first, but that will settle down as those old bugs are fixed. It's been like this with every new code analysis tool (the wave may be exceptionally high this time though).

Not to mention, reachable bugs with a significant impact on security are many less.

There is a difference here. The "new coding analysis tool" that you use for the analogy here is getting better every few weeks with the release of new models.

It is plausible to assume that, for instance, a Linux kernel that was hardened for CVEs that Sonnet 3.5 could detect is not hardened for bugs that Sonnet 4.5, 5.5, Opus, Fable, and models in 2027 can and will be able to detect.

Hence, it is rather a constant catch-up game until the LLM improvements might hit a ceiling and won't get any better in this regard.

You don't even have to assume. Mythos unveiled a ton of bugs across the ecosystem back in April, so this isn't the first iteration of the cycle.

Search for "Security in the LLM age" in this HN page for a reality check, apparently 80% of the security vulnerabilities that Mythos initially flagged in the Linux kernel turned out to be false positives. Better than nothing of course, but it really doesn't look like Mythos is quite the "wonder weapon" it was marketed as (and the situation by far isn't as dire as the initial flurry of Mythos news).

IIRC, in April, Mythos found 20x the monthly average? 20% of that is still a single system doing something that takes a team of engineers 4 months.

Like, "4x as powerful as a team of engineers" is still really quite impressive

The problem isn't the 20% actual bugs found (that's great), but the 80% false positive rate which are reported with high confidence and (most likely) misleading reproduction code. An experienced programmer familiar with the code base first needs to validate all reports and throw away 4 in 5. That's a massive waste of time. If a traditional static analyzer had an 80% false positive rate nobody would take it serious.

In my hobby projects I use LLMs in my development workflow mainly for passive bug scanning, reviewing and helping to maintain tests, they are definitely useful for catching some bugs early and noticing unhandled edge cases, but they're also definitely no silver bullet (they sometimes ignore quite obvious bugs, and the fewer 'obvious' bugs remain the more one has to be careful about false positives). E.g. the funny thing is that now I'm actually slower than before due to the intense 'rubber ducking' with LLMs and cross-checking their results, but I still want to pretend that the resulting code is more robust out of the door.

Eg everything that Greg KH says in the video sounds very familiar, and it's very disappointing that Mythos still suffers from the same issues (or maybe even worse) as older models.

> An experienced programmer familiar with the code base first needs to validate all reports and throw away 4 in 5. That's a massive waste of time.

Okay, but again, even with all that extra effort, they found 4x as many bugs, so it seems like the effort is clearly worth it.

And each model gets more reliable, we get better at building proper reproduction code, etc. - this was mostly a comment about the cycle, direction, and velocity we should expect from the future, given this has already happened twice.

You assume an infinite level of brokenness is legacy code bases. Granted, working on those things one can get the impression, but I think stable code converges to a low level of bugs after sufficient scrutiny and superhuman scrutiny doesn't reveal a never ending deluge of new problems. Not to mention that the bug-chains that you need for a successful exploit keep getting longer very quickly as more problems are discovered and the code is hardened.

I expect that the incremental model improvements will become smaller and smaller until they run into the same diminishing returns effect like all other new technologies (FWIW I've not beeen seeing a lot of difference between the latest Opus and Fable models for the stuff I'm doing, so I just stick to Opus for most things. There's a noticeable difference between Sonnet and Opus though).

I'm not ready to take any bets when exactly the curve will be flat though ;) (e.g. in a just couple of months or a couple of years)

It's normal practice for companies to have a backlog of scanner tool results. Sometimes in the thousands for a larger project. Many of them are legit bugs but also highly local and so far down the stack they're hard to exploit. It takes a ton of work to triage, more than management is willing to spend. Also more than they're wiling to fix and paydown.

Well now they can just point the AI at the backlog (just kidding)

You kid but AI can absolutely make security professionals far more productive just as it does for other professionals.

I predict a different outcome: the rate of vulns identified and fixed will be more than matched by the rate of new vulns introduced by irresponsible use of LLMs on top of brittle and unwieldy tech stacks.

The result will be an overall increase in turbulence and the normalization of steadily intensifying security crises in nearly all software systems.

The only projects that will escape this fate are those which have either been already developed from ground up with rigorous and principled, verified (or verifiable) design, or those which are rewritten to gain this.

> AI is going to expose how fragile the entire computing infrastructure in our world is.

this is a good outcome. A forcing function to encourage all computing to be more secure can only be good in the long term, even if there's a lot of pain in the short term.

I, for one, am excited that we are now finally able to be "Secure™".

Wait, I guess I missed the "more", that kind of puts a damper on the whole thing.

Seriously though, security will continue to be an issue, always. Even if it was perfect, the benefits of it will not be applied uniformly. There will also be the same technology being improperly used causing new exploitables to go live.

> the benefits of it will not be applied uniformly.

why not? Any system you have permission to use and store your data should be beneficial to you if it became more secure. Unless...of course if you're the one who desires unauthorized access.

Clearly because everyone will not be able to pay for it to the same extent. Though, if you have a 1200 agent swarm to throw at a hf sized problem, I'd be excited to see your writeup.

> even if there's a lot of pain in the short term

I wonder what “a lot of pain” could mean here in a world where Crowdstrike is allowed to render half of the world unbootable without repercussions.

Not that I think you are wrong, I am sometimes just confused why we hold back on fixing security because of imaginary deployment- and business-related pains, when it is so obviously unproblematic to crash half the world for a day?

Also, these vulnerabilities may have been exploited by state actors.

I put weatherstripping on my front door and made part of my house much more comfortable and easy to heat in the winter. Then I bought an FLIR and found lots more places leaking heat. I made a list, hired a handyman, and I'm more comfortable and saving money.

Should I now expect to find even more places to seal up the next time I break out the FLIR?

Or you could categorize the current bug apocalypse under the heading "unsustainable trends will not be sustained."

The main way this analogy doesn't hold up to me is that your house is mostly static -- you're not rebuilding the walls, adding new doors or windows constantly.

But in software, especially with agents, we're constantly renovating the house. If you were renovating every 6 months I'd expect to find more places to seal up, even if you were following best practices in those remodels.

That being said, I do think we will reach an equilibrium where most vulnerabilities are found at PR time.

Some software, especially human facing products in competitive markets, will always have new security issues. But in general I agree. We are headed toward a different and generally more secure and reliable equilibrium. Hopefully it's the end of stupidly long bug lists in some products.

Does this also mean the code generated and reviewed by LLMs will not have such issues going forward?

It won't which means inevitably malicious AI will probably backdoor us. Damned if we do, damned if we don't.

Quite the opposite, particularly when the biggest vendors of coding agents insist on not allowing their models to be used to check the code they generate for security issues.

Why stop there? We should regulate who is allowed to write code in the first place!

Easily done if everyone can be convinced that learning to write code is pointless since you can just pay an AI company for access to a chatbot that will write it for you.

Fortunately there are people who write software for fun so there will always be some people who would rather do it themselves.

So, like professional orders in Europe? Thankfully everybody agreed that writing code is not engineering so this isn't mandated by law, but we were this close.

You don't need a permit or degree to draw up plans, just to write your name on the official version and get it implemented in the physical world. It's closer to deploying code than writing.

We have a lot of that in Europe. Parasite professions. Someone who does nothing but charges a lot for his signature.

“And that was how CS became a real engineering degree”

They sort of already do; the commercial American providers all require you to be 18 to sign up for a plan capable of agentic coding.

So, I guess people under 18 aren't allowed to learn to program anymore.

I've been able to use Opus 5.5 and Fable 5.1 for defensive security audits without any issues. They cannot do offensive tasks like pen-testing but in a lot of cases that's not a big shortcoming.

Any tips / online resources how to best utilize for defensive reviews?

I've repeatedly slammed into walls doing very basic tasks. It can do a dumbed-down security audit, but fails to do offensive tasks against my own codebase which is frankly how you use a model like this effectively.

It just means that there will be the equivalent of infinite man-hours of barely-functional-intelligent-man to slog through code word by word and track how it affects all other code relation by relation.

It's not magic and it's not even better or even as good as a mid human, but it's something like infinite man-hours of that drudge work per hour per user.

That will find a lot in old code, and make it a lot easier to keep on finding every little thing right as it's created in new code.

If the Western AI companies get their way, only the developers/companies that have access and paid extra for the security review will get to have a lower chance of such issues.

I'm an arms race the only winner is the arms-dealer.

You're going to get a wide range of responses on this but given the improvement in these models in just one year, and the number of bugs they're detecting which humans could not, I suspect that even median vibe-coded software is going to surpass median human coded software soon - if it hasn't already.

The important thing to remember here is there perfect isn't on the table. The benchmark is existing human-introduced bugs vs LLM-introduced bugs. Many developers have encountered odd bugs which a human would not have introduced, while forgetting about all the bugs caught which humans introduced. Or their opinion is formed by models from six months ago.

How come all the vibe coded stuff I've tried is totally bug ridden and unmaintainable then?

Mine isn’t. To point: we don’t really have a definition of vibe-coded anymore. All software has some degree of AI enhancement now. Is vibe-coded when it’s 60%? 80% 100%?

Try out Opus 5.5 on high. It’s shockingly good. Of course if you’re trying to one-shot a sprawling application with load balanced distributed DBs, you’re going to have a bad time. For small, defined features, it’s pretty fucking great.

The definition I use is shipping LLM generated code you don't understand

We use LLMs extensively on the projects I work in. We don't "vibe code", and we understand every commit.

Depends on if people are willing to pay the extra wall time to write mathematical proofs for everything to make it provably correct. Takes way way longer but modern models can do it.

It depends? LLMs are not a silver bullet for writing bug free software (IME at least, and of course it also depends on the "threshold" what actually counts as a bug). They're definitely good at not creating the trivial "mechanical" type of bugs a tired and overworked human programmer would create (but oth those are also the easiest to find with traditional debugging tools and testing).

They're definitely a useful additional tool for finding more (and more obscure) bugs, but that takes a lot of both human and compute effort too (quite a few of the reported bugs are actually false positives on close inspection, and apparently even with the latest locked down "wonder weapon" models like Mythos), and after all the reports are clean and validated you still can't be 100% sure (but at least a bit more confident) that the code is now free of bugs.

IMO, they will introduce their own class of bugs that will defy static analysis.

Because most devs don't want to do formal verified system development. I know only one OS working on that which is open source Ironclad OS hope many others follow this path. It is Ada/SPARK based but others should do with whatever language they are using. NetBSD also heard going to do something similar with C in last AGM

There is also LionsOS https://lionsos.org/ (SeL4-based).

> most devs don't want to do formal verified system development.

That may be true. Serious question though: even if most devs wanted to develop formally verified code, do you think that it is reasonable to suggest that the typical systems developer could do it with today's tools? I don't mean verified protocols (TLA+) or verified algorithms (SPIN) I mean end-to-end verified code, a-la seL4. I got the impression that this is still very specialised work. Perhaps things have advanced since I last checked.

Do you make your professional career by building formally verified systems? I ask because I don't think that the reason comes down to "because most devs don't want to do formal verified system development". It's much more complicated of course.

I could believe it. Verified system development most likely comes with metric ton of paperwork.

Want to merge the PR? I need verified sign off in ServiceNow by staff level engineer. They are on vacation for 2 weeks? Did manager fill out delegation paperwork in ServiceNow with VP sign off? Oh they did but they forgot to put in return date AND time. Form needs to be corrected and reapproved before we can go into ServiceNow and make changes.

https://sel4.systems/ is relevant in this space!

Genode as a whole isn’t formally verified, but it can use seL4 as a kernel, and it uses a robust capabilities system to sandbox basically everything, including drivers.

As it evolves I suspect there will be a push to verify more components of the stack. Once the capabilities layer can be verified, verification of most other components and drivers would become much less urgent.

in theory, it could be the best thing that ever happened to open source.

I'm not super sure about that. But if its going to exist I'm crossing my fingers it works to the OSS community benefits (eventually)

wouldn’t it result in whoever has the most money having the most secure software?

Eventually whoever has the most energy.

I don't think AI will make that true more than it is now. More investment in security should on average lead to better security, with or without AI.

I don't like AI code, but AI is a decent reviewer.

...for human code.

In the small startup I work for boss (ex-programmer) discovered fable, and ai-coded 15K lines . So much productivity! So great! He even asked multiple reviews and it was fine!

I ask it a couple of reviews and it finds only minor things. The code is a mess of duplication and different coding styles, so I start cleaning it up. After a couple of months the reviews (same ai model) start actually finding big logic bugs that were always there.

We might already be at the point where the Ai-Coder is generating stuff that ai-reviewer can't find and will automatically pass.

1M context window is what? 70-80k LOC, tops? Without comments or documentation?

That is a smallish project of a couple of components. AI will remain inherently myopic until it can keep in context whole codebases.

Exposing current problems is fine to me, but I am worried of how brittle AI code will be.

Good thing we can patch those issues up and have an ironclad system afterwards.

I am an experienced engineer and am building a product actively right now. I started pre-AI, and cautiously adopted AI. I am using AI tools a lot now, but still dedicate a lot of my attention to validating it's output and design decisions.

I recently asked it to review my code and configs from the security perspective. Wow! 90% of the things it identified were MY bad decisions dating from pre-AI development. I am honestly humbled and impressed at the same time.

AI can create slop, and it can create quality products. It depends who is using it, and how.

Ask an LLM to 'review' any text that it's generated vs something human-written and see which one it thinks is flawless.

I do it all the time, and I get feedback about tons of things that need to be fixed. I ask Claude to review ChatGPT code and vice versa. I also ask Claude to review Claude code, and it finds issues as well. I ask it to review copy I wrote and offer style, grammar, voice, and consistency suggestions. LLMs make pretty darn good reviewers.

In this case, what matters most is that the AI security review raised real issues that needed to be fixed. That is valuable.

Who said those vulnerabilities were found by an LLM?

And the aspie award goes to…

Broadcast, not expose.

We knew that for long time, there are countless meme about it, smart people protected their asses from it or exploited those.

A good start would be turning on bounds checks and stop allocating buffers on the stack.

Jep, and a lot of believes will be wiped away with it too.

While I totally agree with the statement, question is - is this challenge even solvable? Though where I come from there is a proverb - "for every malady there's a remedy", wonder what it can be this time.

And, of course, there are piles of legacy corporate spaghetti entangled in incomprehensible mess everywhere you look at. And this shit still runs, this precious hand-carved hand-weaved mess of bad decisions. I can't wait for LLMs to rewrite most of it.

As long as people use AI to review new stuff before they ship the ratio of vulnerabilities waiting should be trending downwards. Still, likely to be a bumpy ride.

> fragile the entire computing infrastructure in our world is.

It was "load bearing" just fine....

Anything is "fragile" if you put a bulldozer over it....

i mean if the building needs to be bulldozer proof..

But it doesn't need to if you lock up the bulldozer maniacs.

Which won't happen when the bulldozer machines are the proxy for the cold-war with China.

Instead of patching millions of environments, perhaps it’s better to just build a secure one from scratch?

That’s what I did with Safebox: https://safebots.ai/about/infrastructure.html

That's because lots of websites and applications are being written by low-skilled laborers in Third World nations. Many websites are rife with vulnerabilities and insecure configurations.

It's not that difficult to build secure (web) applications but it takes effort and knowledge to get it right. You can't expect a web designer who can barely code in JavaScript to build a secure back-end, configure and maintain it. That's just asking for trouble.

Even high-value sites are built by cheap laborers these days. LLMs (I refuse to call it A.I.) will expose their weaknesses within minutes.

Reply on news.ycombinator.com