Comment by mholt | Hacker News Reader

Comment by mholt | Hacker News Reader

This resonates. Unfortunately, the CVE database(s) are too noisy to be useful. It could benefit from higher standards and more thorough vetting. (Maybe take some lessons from academia.)

A "security researcher" once filed a CVE for a regular bug in Caddy [0], making claims that were totally provably false. It was assigned 7.5... the same as Heartbleed [1] -- yes, the one that leaked almost all the private encryption keys on the Internet back in 2014. When I appealed to NVD for retraction (or whatever they do in this case), I never heard back, despite several emails / form submissions. It is not well-maintained. It is poorly managed. There is virtually no oversight. NVD is a trainwreck.

More recently I inadvertently discovered a 0-day RCE in acme.sh [2]. (ACME clients are security-sensitive contexts since they typically deal with private keys and download signed credentials.) Anyway, it was assigned a CVSS 3.x score of * 9.8 * [3] -- I imagine that should be like "cyber-nuclear meltdown" territory, but no, this was actually benign as far as we can tell. Probably deserves more like a 4 or 5 or something. So, again: There is virtually no oversight. NVD is a trainwreck.

Anyway, the whole system is broken, and I'm effectively ignoring CVEs now. But if someone just tells me to patch my <whatever>, I'll probably do it, and that's good enough for me.

[0]: https://github.com/caddyserver/caddy/issues/4775

[1]: https://nvd.nist.gov/vuln/detail/cve-2014-0160

[2]: https://matt.life/writing/the-acme-protocol-in-practice-and-...

[3]: https://nvd.nist.gov/vuln/detail/CVE-2023-38198

It seems a lot of infosec folks have these shallow "X = bad" mappings in their brains. Like in that Caddy issue, "out of bounds read = bad" even though realistically you can't do anything bad with it.

I see similar thinking all the time with bug bounties at work. We had an XSS report once, but it was under an odd domain that doesn't host any authenticated resources. Yet "XSS = bad" so the report had a way higher urgency score than it should've. Sure we want to fix it - and we did - but it wasn't a credential-stealing nightmare scenario XSS.

I certainly find there are many people who take that naive view.

The right thing to do is to triage them in your context and make your own risk assessment. The CVSS numbers are a guide to which ones you should prioritise looking at, but not a risk assessment.

However, in the security and risk world, you have to take a worst case view in the absence of further reliable information. This can lead to fixing things that maybe don't really need fixing, even when a triage has been done. It's often hard to discover enough reliable information.

> "out of bounds read = bad" even though realistically you can't do anything bad with it

Schneier's law [1] is at play here. Especially in situations where the person who wrote the bad code in the first place is later tasked to fix it. If they couldn't see the problem the first time it is going to be tough for them the second time around.

I've gazed longingly at a promising bug for weeks with no idea how to weaponize it, only to finally give up and ask someone smarter who tends to point out a trick I had never seen before. Even with a career in security old enough to buy beer, I am still amazed by the clever shit I miss.

1. "Any person can invent a security system so clever that she or he can't think of how to break it."

> If they couldn't see the problem the first time it is going to be tough for them the second time around.

No. Its easy to overlook a bug like overflow or use after free, but once you see it, you understand it just fine.

This is a fair point. I actually hesitated a bit before posting my comment because I don't actually know 100% that the XSS was harmless. But the reporter didn't actually demonstrate anything other than an alert('owned').

I don't understand, is arbitrary js execution not enough?

It depends. Where is that JS executed? On the login page? The payment details form? In a restricted IFrame serving a tracking pixel? On a static page handling public document downloads with a different domain to the logged in contexts?

With a good security report, you want to include example impact. "I can run alert." - meh, but should be checked/patched just in case. "I can run arbitrary JS on a page collecting payment details, without CSP restriction." - now that's immediately bad.

You may have noticed many websites host user-uploaded content on a different domain to their main site. Github delivers some things from githubusercontent.com, Google some things from googleusercontent.com, Reddit delivers some things from redditmedia.com and so on.

The reason they do this is to give a big layer of protection against the harms of XSS - even if user-uploaded content manages to execute arbitrary javascript on googleusercontent.com the that javascript can't access cookies for google.com as it's hosted on a different domain.

Some scoring guides rate XSS as very high risk, assuming you don't have this mitigation in place. resonious had the mitigation already ("it was under an odd domain that doesn't host any authenticated resources") so the XSS wasn't a very high risk.

With that said - some people will thank you for demonstrating how a small security problem can be escalated into an account takeover, but other people will call the cops on you for hacking their website or threaten to sue you. So I would say if you're reporting XSS it's reasonable to stick with an alert box, unless you know the person receiving the reports is reasonable.

>Google some things from googleusercontent.com, Reddit delivers some things from redditmedia.com and so on.

Exactly:

>If you are injecting script in subdomains of (sandbox) domains such as: [...] ...we won't file a bug based on your report, unless you can come up with an attack scenario where the injected code gains access to sensitive user data.

https://bughunters.google.com/learn/invalid-reports/web-plat...

e.g.

https://www-tutorialrepublic-com.translate.goog/codelab.php?...

It can definitely impact the company’s reputation negatively. If any of our sites allowed alert(“owned”) it would be top priority to fix.

It's not. If it is, then why not go file a report at jsfiddle.net?

In my experience, many corporate infosec guys are basically just beancounters without a deeper understanding of information security.

Yes, unfortunately any kind of staff position that does not deliver product attracts these types who just want to hide and never be accountable for delivering value to the business. I'm not saying the positions aren't needed or valuable, but just that it is appealing to the wrong kind of people.

And unfortunately, their value is often directly proportional to the amount of workload they add to the productive segments. People wonder why security teams are the first to be cut during hard times, but this is basically why. That said, I can see both sides of it, security is obviously of great importance. But there just has to be a better way, perhaps some categorization of threat models cross referenced against the CVEs/etc.

> security is obviously of great importance.

It is. But the thing is, those corporate infosec folks I'm talking don't actually improve security. It's the same as with many audits.

Shoving out endless amounts of broken trash has negative value to society, even if it makes the company money hand over fist.

I have personally rejected a candidate that claims to know professional security tools but maintains that it is not his job to filter out false positives ("because they don't exist") before presenting the results to the developer team. The same candidate would also say "you need a firewall" but would not be able, in an interview setting, to explain how to protect the database server using a firewall - i.e., what exactly to allow.

There is a medium-to-weak argument that enforcing these minimum standards even in clearly benign places raises standards everywhere, which means it is much less likely to show up in the really bad places.

I'm just making debate however, I don't think most people are playing 3d chess when they ask for these changes, they just want the line item on their report to clear up.

OOB read allows an attacker to measure internal status such as stack status, and memory allocator status, this widget along with another flaw makes exploits much more reliable.

You -can- definitely do bad things with the data gained from it.

For the memory allocator data, read up on the "house of" attacks against glibc's allocator.

> "out of bounds read = bad" even though realistically you can't do anything bad with it.

I have absolutely seen more than enough exploits for "obviously unexploitable" vulnerabilties that I think the "out of bounds read = bad" mindset is the only reasonable mindset to have.

> It seems a lot of infosec folks have these shallow "X = bad" mappings in their brains. Like in that Caddy issue, "out of bounds read = bad" even though realistically you can't do anything bad with it.

As others have pointed out, no few "unexploitable" issues have turned out to be entirely exploitable in the hands of the right person. In a world where innocuous vulnerabilities can be chained together into very dangerous ones, this gets much worse. As a colleague of mine described to me, CVE math means 1+1+1=10.

More subtly, this interacts with one of the weirder ideas in security. Vulnerabilities exist before they're known. This means that there's likely a series of vulnerabilities lurking in every bit of software you use. It's hard to do much about those with certainty, but you can do something about the bug in front of you to prevent it from contributing to CVE math.

To put it another way - risk analysis has room for error. Don't be too certain of yours.

It seems a lot of non-infosec but technical folks have a pattern of shallow first-order thinking. Like in that XSS issue, "no authenticated resources = not a big deal" even though realistically you could redirect end-users to a phishing domain (I can assure you that a much larger percentage than you'd like would fall for this), or create and delete invisible DOM elements in such a manner to exploit a UAF vulnerability in their browser's rendering engine to perform a sandbox escape and get code execution in your users' userland, where they'll proceed to dump all of your users' saved credentials and start emptying bank accounts - all because you couldn't imagine anything bad happening from an XSS on a site with no authenticated resources and therefore chose not to prioritize fixing it. Even CPU-level speculative execution vulnerabilities can be invoked through sandboxed JS running in a browser.

Deprioritizing an XSS vuln in an end-user facing website you built because there isn't sensitive, authenticated data on that domain is like being the owner of a construction company that built a hydroelectric dam incorrectly, who noticed visible cracks on the dam that aren't supposed to be there, and decides to not tell anyone and "maybe fix it later" because the hydroelectric generator is still working fine and cracks in the dam don't cause generators to stop working.

> (Maybe take some lessons from academia.)

One thing that academia takes seriously and the CVE database doesn't is the reputation of the researcher(s). The CVE database allows one to anonymously make a serious claim against other people's work, and thus avoid consequences if the CVE turns out to be incorrect. In academia this would be unacceptable: reputation is everything.

Now I'm not saying academia does everything right, and it does have a strong tendency for credentialism which is at odds with how security research has always operated. But it makes judging people by their actions a whole lot easier...

There are still corners of infosec where doing the wrong kind of research will get you the business end of a massive lawsuit... or a pair of shiny new bracelets. A lot of us started in security in an era where that was more of the rule than the exception.

That said, reputation is quite real. For example, taviso is well known for finding weird and scary things.

CVSS is not a measure for risk. I feel like it’s pretty hard to define an objective measure for “how much should you care about this” that applies to everyone, since you’d need to know how common the affected software is among all computers and how many of them are using the specific vulnerable configuration, etc. What you can do instead is have a measure for “if this affects you, it’s pretty bad/not that bad/…”.

Heartbleed wasn’t an RCE, it just leaked sensitive data. RCE often results in a total loss of confidentiality, integrity and availability, so it receiving a higher severity rating makes sense.

Honestly if you just stick with CISA's known exploited list then you will be ahead of most operators and spend less time ghost hunting stupid shit.

I really have come to despise CVSS (and especially how tenable will put predictions in the CVSS column for officially unrated vulns) and how assessors seemingly so not have to justify their 'findings'. Not exploited, no poc, needs root to use, why is this a High? Come back to me with something good.

CISA-KEV is a good list, but updates to it can be delayed by days to months.

I personally think this is all by design to create more jobs in the booming snake oil infosec industry. Hear me out before you get the pitchforks.

I've started seeing some of the most incompetent people I used to work with are suddenly now "director of security", "senior infosec" and things like that. These were people that struggled to remember what an IP address was when I worked with them. I highly doubt they all suddenly decided to actually learn anything about how computers work.

The icing on my anecdote proof cake is my junkie brother that lives with my mom contacted me to ask about how he could get into the computer security industry about 2 months ago,.

It’s been like this. There are a lot of people that got into security without really a passion for it. Lots of bs. The way I think of it is all of this revolves around the fire/heat generated from finding vulns (on both sides). If ur not finding vulns/really understand/actively trying to understand then ur just blowing smoke and trying to stay warm. That’s just how it is.

You're conflating two systems here: CVSS' ratings and CVE's vulnerability IDing

Something receiving 9.8 when you feel like it should be a 4–5 is a common enough complaint, though typically not this extreme. If the parameters are filled in correctly (you didn't mention even checking why the result was 9.8) then that's not a flaw in the CVE system.

Conversely, CVE authorities not responding to requests for deletion aren't the same as using an inaccurate impact/exploitability calculation system called CVSS

> Unfortunately, the CVE database(s) are too noisy to be useful.

I used to hate on CVSS scores a lot more than I do these days.

Sure, the scores are rarely correct and sometimes nonsensical. To be fair, it's impossible to have a scoring system that precisely ranks every potential risk in the optimal order for everyone.

Technically, the best answer is to take every report, evaluate it internally based on exactly how this code is used, where and for what and create our own risk assessment and prioritize changes based on that. That would get us risk scores as good as we're going to get for our product.

Nobody has time for that. Even in the best funded organizations I've seen, it's not going to happen.

Turns out it's like an order of magnitude cheaper to just say we must address Critical, High and Mediums and call it a day. As much as it bothers me at a technical level, because I know we're fixing stuff that wasn't worth fixing and leaving out some important fixes that got mis-scored too low, the pragmatic reality is this is easier to do and thus can get done. On average, that will capture many of the things that really did need fixing so our security did get better. Yes, we did waste time fixing some that didn't need it, but it was less wasted time than it would've taken to do a deep evaluation so we still saved some time. So overall, not bad.

> Maybe take some lessons from academia.

In my experience, the vetting process in academia is an abject failure and mostly just political theatre.

Do you have any thoughts on CVSSv4[0]? It appears to incorporate finer-grained and organization-specific scoring to address issues many have with the one size fits all approach currently used for CVEs.

[0] https://www.first.org/cvss/v4-0/

I recently had similar thoughts about a security vulnerability in a storage component, which is sometimes even intended to be exposed to the internet as a static file host.

That thing had a security vulnerability such that you can do one curl-call and get the entire environment of the system, including the global, special cased admin user credentials for that system. So, practically speaking, an unpatched system has no authorization and is open to anonymous access.

Now I'm not sure if I'm weird, but I have a hard time imagining a worse bug in the authorization system in a data store than this kind of trivially exploitable fail open. Easily in the "Stop and drop everything and start patching". But nah, it's just a 7.5. All is good.

On the other hand, the fact that the root user can bypass some restrictions and upload files with some permissions is rated as an 8.8 because it's a possible RCE jump. Naturally.

Reply on news.ycombinator.com