Will Cyber Security Be Replaced by AI? | Hacker News Reader

Will Cyber Security Be Replaced by AI?

blog.edned.net

Security of out all the domains of computing seems to me like it will be amongst the last fields to be automated away. The lack of precision in AI systems is pretty damning because it only takes ONE mistake to cause a breach.

As others have already mentioned, AI is already extensively used in security operations. I think we will see more and more people replaced by AI who spend their days crawling through logs since it's hard to imagine something AI is better at. This is a domain of security where sheer data crunching can make up for imprecision.

When it comes to things like LLMs and secure application development, AI is happy to suggest code that it got from a stackoverflow thread with massive and glaring security holes. Or code which is not inherently insecure, but assumes the systems it connects to can be trusted, which is a dangerous assumption. It isn't even close to comprehending the broader social, business, and security context of an organisation and achieving a balance of those concerns. It needs an IMMENSE amount of handholding from a true expert to be useful.

By the time AI can replace cybersecurity experts, it will be able to replace human IT and development experts altogether.

How about AI being an additional security layer?

Say I would look at security from a different point of view, namely behavior pattern matching. If your traceability is high enough, a trained AI might be able to sniff out potential breaches immediately.

The same is true for cheating in AI. Once we've trained models and games log enough metrics, I'm quite sure that AI can sniff out cheaters with high precision by behavior alone. Even if they don't use aimbots, AI should be able to sniff out wallhackers or other minor cheaters by simply analyzing how players are observing their surroundings.

There are plenty of tools that do ‘ai’ in security. Anomaly detection can be valuable, but for the most part they’re snake oil.

You might well be right about AI's efficacy at detecting video game cheaters, but I've never understood why this is an issue. If cheating can't be easily distinguished by a human from skilled (but legitimate) play, why would such cheating reduce the enjoyment of the game? With a well-functioning match-making lobby, the cheaters would simply class themselves out of all the normal games and end up playing other cheaters, leaving legitimate players well alone.

As someone who enjoys casually playing online games, many of which have little or no cheating detection, I genuinely don't understand why many think the problem deserves stacks of 20GB VRAM cards running AI models, let alone rootkits client-side. Please enlighten me :)

Surprisingly, no. The reason I got into security in the 90s was because I saw hackers as a check on the absolute dominion of tech enabled institutions, and a big part of that was always being the exception and finding a way to remain smarter than weaponized AI. We're going to see some truly awesome AI malware, and it will devalue some skills people and hackers have heavy sunk costs in, but we prevail.

Short term, cyber security is governance with a high technical competency bar to entry at any serious level, and the reason we're still in demand and these problems aren't solved is because competence at managing dynamics and risk is still irreducible. Every model and formalization creates a dynamic with whatever it models and formalizes, which creates an opportunity to exploit and manage it. AI mainly creates manageable dynamics and demand for governance, which means no matter what you're going to need hackers unless we're all livestock.

Long term, security as we understand it today will go the way of the Orange book, and it will evolve as a variation what it really is, a purer understanding and way of relating to tech. Fundamentally, hacking is the logic of an idea that we can apply to anything, where instead of goodguy/badguy, or oppressor/oppressed, we have true/false, where something is only true within the domain of its assumptions, and when you change those, you get understanding and leverage. It's all the things that work in practice just not in theory. AI is a figment of aggregated and nested assumptions, so all I see is opportunity. It will be as pernicious as human conceit, and I'm pretty sure it's going to kill a lot of people, but I'd say if you don't have the practice, you don't know what you don't know about our advantages over it. We're good. :)

I'm only into infosec as a hobby, but I already find LLMs extremely useful and (annoyigly!) better than me at certain things.

For instance, I can analyse some obfuscated malware code manually but it takes time and a bit of trial and error. I have used GPT4 to instantly give me the gist of what some custom JS vm is doing, or to analyse a smart contract and give me some bullet points to focus on

While I agree with you the hacker mindset and related skillsets will survive and can be applied to other domains – the pure technical skills I used to take pride on are now commoditised. It's a scary new world.

If it's anything like LLM for code review then I'd say it's not very potent.

No doubt it's good at paraphrasing code, deobfuscation sound like a good usecase. Other than that I've seen it mostly come up with bullshit :

- random opinions stated as fact (x makes code readable, y makes code simpler to understand, yada yada) - it's like a review from a mid level dev that finished reading Uncle Bob or some crap like that and suddenly there's 8 functions where there should be 2

- suggests changes that make the code subtly incorrect - like rewriting code with a different container type and messing up result ordering

- state false information like suggesting that a dictionary will be faster than array search (for fixed size/small array), or that some approach is faster because it doesn't allocate (but original didn't allocate either and the new approach actually does, it just hides it in a container), etc.

- can't catch complex logic bugs for shit, even with leading follow-up questions (once I figured out the problem and went back to see how long it would take GPT to figure it out)

respectfully disagree here - not entirely, but rather the axis upon which the split is discussed. AI at this time mostly involves natural human language (typed first) interface to knowledge, enacted in code.

The failure of Prolog and that era of AI was said succinctly by IBM's Sriram Raghavan (sp?) as "determined by rules and facts, and who is going to type in all those rules? so it went nowhere"

At this go-round, some semblance of understanding of questions, plus really a lot more data on tap, changes the equation. An "AI" freed of needing pre-built if-then-if-then could possibly work at speeds of iteration, or speeds in response times, that will startle even those prepared for this. So in the "John Henry race against the steam engine" [0] sense, no contest.

[0] https://en.wikipedia.org/wiki/John_Henry_(folklore)

What humans will retain, and build, is authority. A signature authority is a ticket to an easy life, as many in law and health care know well. "executives" of all kinds, and even the lowly salesmen, will retain that signature authority despite any competency of any other, human or machine, because it is adversarial in the first instance. The contest of will and locks will not be ceded, but the contest of erudition and good taste, will be lost by the human in most cases. That is a suggestion for the axis upon which to measure the future fates of humans in markets and governance.

AI is a surveillance state wet dream in the making. It is the humans we have to watch out for, starting yesterday IMHO.

Well, looks like "AI" nowadays is officially another name for LLMs. I wonder how long it will take for people to talk about general stuff again, because those things are getting ridiculous.

Security monitoring and classification tasks have been run by AI everywhere for a decade already. And AI will very clearly not take over any other part soon. At least from the "useful" cyber security.

There is, of course, the entire charlatan cyber security, that is maybe an even larger market. And well, LLMs excel at charlatanism.

For me I see IDS as a natural easy win; collating a shitload of sensor data, tripwires, logs... and providing carefully graded alerts and advice seems like something ANN trained on lots of good field data would excel at.

where will this good field data come from? using what context? a low risk alert for one org might be seen as something else in another.

If you look at the current major offerings like SentinelOne, they start off with a generic best practice baseline, then slowly “learn” the normal traffic on the network to be able to better define the abnormal incidents to the IDS.

You make a good point. What occurred to me is that some more standardised method for coding threat events is needed first. But that seems tractable given existing CVE taxonomies etc.

The article is decent because it debunks a lot of stuff but it is concerning it even needs to be written. What does someone even mean when they say a cyber security worker is going to replaced by AI? I fire Bob from the IT security department and step 2 is.. ? Like in concrete terms, what happens next?

It's like asking "will debuggers replace cyber security?". A tool does not replace a worker, these aren't the same category of things.

I wasn't expecting much based on the headline, but this is a great roundup of the pros and cons of applying AI to different technical facets of security. What it's missing is that the human factor is what makes the real difference to the security of a product or organization: changing or adapting to user behavior, convincing managers and developers to allocate time to security as well as features, or to alter or even forego a feature that carries more risk than it's worth.

In some cases, AI can probably help explain these issues better or analyze factors to help determine how to balance security vs. revenue, but a lot of the work is really in building relationships and trust over time to educate and influence behavior and priorities.

Hacking and exploitation is usually creative job of chaining unrelated errors into code executions. AI can currently badly repeat random learned facts but creativity - ability to come up with something new - isn't AI's strength.

AI, as it exists today, is very good at interpolating lots of data. It is far less able to do a good job of extrapolating far outside the boundary of existing data. Given the cat-and-mouse nature of security, where the parties regularly move outside the bounds of past behaviors, it is very very unlikely that AI will be useful in security at the boundary.

It will be very good at making sure known exploits are not available, but known unknowns and unknown unknowns will remain security blind spots.

Sure, after literally everything else.

Security remains top of the pile in terms of (targeted) creativity and foresight, things that separate “just AI” from “does everything imaginable for us” and indeed the threshold for what we can start to consider AGI.

Once security is replaced by AI the system is essentially untethered.

Most security jobs require almost no creativity or thought though, most of them are looking at the outputs of some commercially available AV/EDR/IDS in a commercially available SIEM/case management tool and doing the same thing hundreds of other people have done with those same tools.

If you work in research maybe you’re on, but if you work tickets you should learn how to do something else.

No, what you’re describing is what gets filtered to the people in charge of the SOAR who then automate those outcomes.

Even with the top tools and detection you still absolutely need knowledgeable people to decide whether an alert is a false or true positive and then you absolutely need humans in the mix to decide what the next steps are.

When you say “most security jobs” maybe you mean first tier SOC for a vendor? But I’ve never worked for one and as far as I know they’re universally seen as just above useless by anyone without a financial incentive to employ them.

So you think that you know a lot about security but not only do you not know the difference between SIEM and SOAR you think that most security jobs aren’t dealing with incidents? Fascinating.

It’s not an uncommon problem though I guess, I’ve worked with a lot of people who think they’re doing novel work discovering that you can use regex to find spam or inventing a DGA detector.

I’m not sure why you seem so agitated suddenly. As you know I’m sure, a SOAR is meant for automation. The simple tasks with simple decision making are piped to the SOAR.

Additionally there’s a lot of work that goes into the space between “something has alerted” and “this is an incident”.

Also, just as an aside, I’m not sure where you’re getting mixed up about SIEM/SOAR?

I’m loving the eternal September of constantly watching people totally misunderstand the industry.

Since you aren’t following the conversation, I’ll break it down for you into smaller chunks. First, most places don’t use SOAR (you can see this in the marketing or sales numbers) as they don’t have the technical maturity or budget to leverage it. Bringing it up is pointless. Most jobs (see any of the usual industry survey results or do some analysis of posted jobs) are basic SOC-monkey positions looking at a SIEM (as I previously said) and doing basic research and response.

Get alert, google signature, update ticket, interview user, remediate/slick the box, repeat. Throw in complaining about the lack of tooling to make their lives better, a good bit about how the desktop/server/dev team should really work on standardization so everything isn’t a one-off and you have most person-days at most f1000 shops.

From what you’re saying it sounds like we just have different experiences. The companies I’ve worked for have respect for security and invest heavily in it.

Before the days of SOARs we made an effort to script our own automation using Python/Bash/etc.

But to your point, these companies that you work for that don’t invest in security and instead use outdated methods: why would they suddenly pay what I assume would be top dollar for AI when they’ve gotten along fine using ticket monkeys so far? Seems like they would continue while the far more efficient ones that take advantage of things like SOARs and already have those positions at a bare minimum would be the ones to invest the money.

Security professionals that simply go through compliance routines may get replaced with AI first.

The ones doing security research are often incredibly highly creative people that think out of the box and do things that have not been seen before, unlike LLMs mostly remix what they've seen before.

Imagine LLM do chip decapping and perform fault injection on hardware and extract firmware. I think Cyber securiry is bigger than that. I think that take another few decades until we get there and once we are there it is not anymore just cyber security jobs.

AI won’t replace humans. But a human with AI will replace a human without AI.

AI will be a supplemental tool. Just a much higher level tool. That can help improve itself and the users of it.

It will be taken to a next level by the AI, both in terms of defensive and offensive capabilities.

We already have interesting startups and projects dealing with defence aspect.

I’ll note glog.ai as something I’ve seen most recently in this domain - it entails a machine learning algorithm that classifies SAST output and assists the developer in triage of potential vulnerabilities.

I always disagree when I see something like: "5) Lack of Understanding

When ChatGPT generates a human-like response, it's not a result of understanding the content in the way humans do. Instead, it's drawing upon patterns in the data it was trained on. It can mimic intelligence, but it doesn't genuinely comprehend the concepts it's discussing.". If I ask ChatGPT or GPT-4 to solve novel problems using a random library or programming language and it solves them then it is clear to me that it has understood the tasks, indifferently on how you paraphrase it if that's not understanding then we all lack understanding. But anyway, this part of the article feels more like a tangent, hopefully you're not going to use a LLM to analyze incoming traffic (maybe just for moderation).

Honestly I’m not sure I ever understand the content myself and am instead repeating patterns of what has worked in the past applied to new domains.

You should be aware that not everyone is doing this. Contrary to what you might see on infosec twitter, not everyone is a trench coat covered pile of raccoons.

The issue is apparent when you consider the kind of mistakes LLM’s make. If I ask you to guess a blue whale’s body temperature you’re not going to save 500C because you understand if that were true various other things would be happening.

An LLM is more likely to give the correct answer to the blue whale question, but when you greatly penalize wildly incorrect answers they do noticeably worse.

Yeah, I think coming up with definitions of "understanding" or "reasoning" that GPT-4 and friends supposedly don't fulfill is moving goalposts.

To continue your line of thinking, when we add salt and pepper to a dish we've cooked, are we really doing it because we have a developed a thorough understanding of the human olfactory and gustatory systems [1] system, or because it tasted good previously when applied to similar recipes?

And when it comes to understanding basic math, perhaps the patterns are still a tad to complicated for LLMs. Maybe there are too many surprising rules that appear out of nowhere and break the learnt patterns. But children struggle with those rules as well when they first come across them. Think about division through zero - when coming across the rule for the first time, you might wonder why it exists, instead of e.g. defining the result to be infinity. The answer to that question (not just "because that's the way things work!") is not obvious at all, to be honest I wouldn't be confident enough to attempt giving an explanation here.

[1] I have to admit I had to look gustatory system up: it's the biological term for the system behind the sense of taste.

> ...admit I had to look gustatory system up: it's the biological term for the system behind the sense of taste.

Ah, that must be why I consume pies with gusto. Yup, Wiktionary confirms: "Borrowed from Italian gusto, from Latin gustus (“tasting”). Doublet of cost."

I agree, I'll paste a comment I wrote a bit ago where I think I explained this reasonably well:

"> There is no "intelligence" going on here. It's not "thinking". But it can still perform calculus by just stochastically emulating how average text on the internet looks.

It always annoys me when people say this, I’ll try to explain why.

There are two possible definitions of “intelligence” you could use here; the ability to process information to get something done, and something hand-wavy about human consciousness.

GPT-4 clearly has some ability to process information to get something done. You might say that by this definition [insert trivial thing] is intelligent, but it doesn’t have to be a binary thing of intelligent or not. I think it’s fine to say maybe a calculator has very low intelligence (but not necessarily nothing), GPT-4 is more intelligent than that and humans are much more intelligent again. GPT-4 has many limitations compared to humans, but I think that just makes it less intelligent, rather than disqualifying it from having intelligence at all. Sure, it’s just predicting text, but that’s a task that requires a level of intelligence. You might say it’s not general like humans, but I’d say it has a much better ability to generalise than something like an image labelling AI, so that feels like it’s at least getting somewhere.

The second definition is useless for practical purposes because it’s not measurable or observable in any way, so it’s not useful to use that.

So I feel like this is something people say to reassure themselves that it can never get to human level, and is fundamentally different to human intelligence, whereas I think it’s somewhat similar but at a lower level."

Glad I read the article, I think he is correct.

I am sure people will try, but I think it will end being "border security" for Walled Garden Environments, which is were the Internet seems to be heading towards.

Better operating systems will greatly reduce the need for cyber security, up to 99%, by eradicating attack vectors. AI will play a role on the remaining 1% to help humans.

The last jobs on earth will be IT personel. In all different layers. Unless some skynet takes over.

Every profession thinks this. Doctors, lawyers, plumbers, chefs, tour guides, carpenters. Some jobs are actually likely to be automated away, but most jobs have a last 5-10% that’s non-obvious and very hard to automate.

Maybe so, but on the way there, 95% of IT personell will have been made redundant long before many manual labor jobs.

Existing manual labor jobs are what’s left after industrialization and robotics, if you think about it. IT is full of low hanging fruit for automation.

The ultimate post-modern irony.

What we thought was hard is easy; what we thought was easy is hard. (When, really, there is no easy and hard; there's only optimization of a neural net to a task, and what we find hard are things we were never optimized for.)

It's also karma in that the techies responsible for automating others' jobs away are the first to get hardcore automated. c':

of course it for sure maybe.

Betteridge’s Law in action ;-)

tome's law in action ;-)

"In any discussion about an article whose title is a question, Betteridge's law is mentioned with probability 1."

https://news.ycombinator.com/item?id=24545238

I hope AI brings us headlines that don't end in a question mark, because it's unoriginal and I'm sick of it?

I worked in a newspaper or two back in the day and the sub editors who wrote headlines were banned from putting a question mark in a headline.

Reply on news.ycombinator.com