Hacker News Reader

Comment by moffkalast | Hacker News Reader

Phreaking is back on the menu, boys.

We have a ton of new vulnerabilities that have emerged from widespread use of AI systems. But at the same time the frontier AI labs are releasing tools like Mythos that purport to automate/simplify the identification of vulnerabilities.

Between these two trends, I struggle to see what the future holds for the security industry.

Either way, as is always the case with the tech industry, the incumbents in this space will be getting paid the big bucks and the consumer will ultimately hold the bag. We absolutely need tougher data privacy / security laws & I wonder what catastrophic event will force law makers and voters to take this issue seriously.

My feeling is the defender wins in the long-run. There's only a finite number of bugs and vulnerabilities.

Surely there is a mathematical model here, but intuitively I do think there is an infinite number of typos and errors you could contain in the set of finite books, and similarly there would be an unlimited number of bugs and vulns in the set of Turing machines.

Not to mention new code is going to keep being produced, and if some in the industry are right, at record pace and record sloppiness too!

> There's only a finite number of bugs and vulnerabilities.

The context of an LLM is also finite.

Vulnerabilities are perpetually being created, and this will be true no matter how good LLMs become at writing code - there's simply too many factors that can contribute to something apparently benign becoming dangerous.

Lots of bugs seem to be fundamentally quite local, but potentially with global trigger conditions. Heart bleed for example could've been avoided even if you could only read small segments of the codebase at a time, but could only be triggered with more context.

I suspect that a combination of ai and memory safe languages will really shine in the next decade.

Semi agreed but I think that we are likely to see a ton of vulnerabilities found in the near term as AI's go through codebases and find all the stuff that was missed over the years. Once that period has (mostly) passed I imagine things will slowdown to somewhat similar to a normal stream of bugs and vulns and as new code is created.

something about the current trend of AI vulnerabilities reminds me of the old Asimov Robot series and the three laws of robotics:

https://en.wikipedia.org/wiki/Three_Laws_of_Robotics

I doubt you can prove that.

Do you think the attacker or defender will have been the overall beneficiary of LLMs when we look back in 5 years from now?

I don't know, I think it will mostly come down to which side has better recruiting. In other words, all things being equal, I think it's a wash. It was the second part of your claim that I don't think can be proven.

Meet me in the cereal aisle.

Reply on news.ycombinator.com