Hacker News Reader

Comment by tptacek | Hacker News Reader

Then why didn't they?

I think the real point is they didn't - until it became a "marketing" thing for another company who did it for them.

A lot of these issues would be highlighted by "legacy" (pre-AI) analysis tools. The issue is that they weren't being run.

Why not? We're talking about vulnerabilities with real market value here. If it was just a tool run, why weren't the tools run?

Isn't the simpler explanation that they weren't just a tool run?

The tools are expensive. One of the major players in the market have really expensive licensing fees. Then the developers all need to be trained on how to use the tools and understand the results. It’s not something they teach effectively in schools.

Software engineering is still kind of new overall.

Which tool specifically are you thinking of that might have found this but wasn't run because of it's very high licensing fees? I work in this field, I'll be familiar with it.

Black Duck products

https://www.blackduck.com/fuzz-testing.html

OpenText products

https://www.opentext.com/products/dynamic-application-securi...

I won’t say how much they are here but they are very expensive.

Just to be clear: your claim is that the Black Duck fuzzer would have enabled the rapid discovery of kernel vulnerabilities in macOS?

Question was about high licensing fees and which tools I was referring to

I’m not claiming Defensics or OpenText DAST tools are magical “find all kernel vulns” buttons

My point is more that mature fuzzing ecosystems already existed before the recent AI-driven approaches. Protocol fuzzers, syscall fuzzers, coverage-guided fuzzers, sanitizers, dynamic analysis, etc. have all historically found serious kernel bugs

We might just be talking past each other. My question, from upthread, is this: the heyday of AFL was over a decade ago. Every major platform company fuzzes at a scale that I think is difficult for lay practitioners to get their heads around. They contract, quarterly, soup-to-nuts assessments from competing software security companies, who get full source access and are measured against each other by the quality of their findings. They run bounty programs specifically to direct public researcher attention to these exact findings.

Why didn't "mature fuzzing ecosystems" find the vulnerabilities AI is now finding? It's a pretty big gap in the "fuzzing tools already do this" logic!

> Why didn't "mature fuzzing ecosystems" find the vulnerabilities AI is now finding? It's a pretty big gap in the "fuzzing tools already do this" logic!

Because they simply aren’t ran. That’s my entire argument

You're wrong about that.

How? If the tools were ran the same findings may have occurred.

No, the point is they are very different tools that produce very different outcomes. Your syllogism doesn't work.

Claude et al have “skills” that are basically containers of tools. It’s not a huge leap to say that similar tools are in use within these containers or even same. We don’t know.

Yes, we do.

Getting alot of short line replies. Got any link to any documentation / traceability to that claim?

I work in this field, have done work for some of the vendors we're discussing, and talk daily to security engineers working on these problems at these vendors. You are wrong.

Well we aren’t going to get anywhere. I asked for evidence and didn’t receive any.

Just so we're clear you're asking for evidence that Apple and Google... run fuzzers.

All software assurance tools. Prove they run them (binary analysis, fuzzers, static analysis, etc) and how that differs from what Mythos is doing (which likely is using those tools as skills) along with objective evidence for each

Yeah I'm pretty content to rest my case here. If it's helpful to you to know this, you're wildly wrong here, like "the search bar on this website will show you to be wrong" wrong, but I'll leave it at that.

Thanks for not answering.

Apple has a massive information security organization that has pretty intense resources at their disposal.

It seems borderline impossible that there's a tool that they feel would be beneficial but that they're classed out of using by license costs or by staff proficiency.

It happens at a lot of places that the budget isn’t unlimited when it comes to information security. But even then it comes down to risk management.

We’re in a thread about an Apple vulnerability, where the claim was made that they’d have found these if they’d properly run traditional tooling.

I find this amusing as Apple were the people I had direct interactions with that didn't run stuff like fuzzers or sanitizers as a matter of course - at least not in the situations I was involved with.

Though this was ~3 years ago now, and a lot of things have changed, but these tools were very much available and well known then - they aren't new. Though perhaps as they "knew" the project was coming to a close it wasn't a priority either?

It also might have fallen through the gaps due to the Apple internal/team culture - I worked for an external vendor, and we had to work against binary built framework dumps that didn't even allow us to enable things like address sanitizer completely either, and fuzzing difficult as you'd need to trace things through their opaque binary layers before it even reached our code.

Apple did have all our code though, it was very much an asymmetrical relationship, but if they were running such things as a matter of course in CI or similar you'd see that pattern in when they reported issues it caught, and the timings from time-of-bug-caused to time-of-report. It instead suggested any such runs were piecemeal and sporadic at best.

Though, it wouldn't really surprise me if they were being run and finding issues all the time, but they never actually got back to us. This certainly wouldn't be the first time we ran into "difficulties" due to the nature of the relationship and culture.

Could be any (combination) of

- looking at components in isolation, not realizing that a component could receive untrusted input

- looking at the entire system, but not in a configuration that made the CVE possible

- having to be extremely lucky to find the issue through fuzzing, and Apple not hitting that jackpot

- having found the issue in testing, but incompletely/incorrectly fixing it

- mostly focusing testing on other components because this one’s code didn’t change and hadn’t seen issues in years

I don’t think we have enough info to know which (or something entirely different) it is.

Reply on news.ycombinator.com