Hacker News Reader

Comment by jacobyoder | Hacker News Reader

Had to deal with some automated scan tool telling a client "you have log4j - you're vulnerable!" when... we didn't. A project was including a dependency on SLF4J and that project pulled in slf4j-log4j (IIRC). But it was not configured, wasn't compiled in to the final jar, and ... even it was, it was very old log4j (1.1 or 1.2 IIRC?). The vulnerability didn't affect that older version.

We had to spend a week back and forth with

"no, we're not vulnerable".

"But we see log4j is right there in the a file in the directory".

"No, it's not vulnerable and it's not being used and we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency"

"But we see log4j is right there in the a file in the directory".

Also, there was no budget to do any sort of rewrite/refactor/etc anyway.

View on HN

> We had to spend a week back and forth with

We've had similar out of a pen test that noted we were using nginx for some of our internal dashboards/tools and the version running (1.18 IIRC) was officially EOLed upstream so a high security issue.

They initially wouldn't listen to the argument that we use stable/LTS Debian/Ubuntu release only, and those hold functionality stable (which is one of the key reasons to use them) and backport security updates where relevant. We pointed them at the package changelogs and they were convinced that there was still a vulnerability not patched but for some reason didn't want to tell us which, when we finally got that out of them it turned out to be one that was introduced in a later version so was never relevant in the first place for the one the stable repositories included.

You would think a penetration testing company would have a clue about something so common…

If it was an unauthenticated test, I'm not surprised. Raising findings off of banners is notoriously error prone, but pentesters often err on the side of raising things that are dubious rather than leaving them out.

If it was a credentialed review, you'd have hoped their scanners would plugin to the appropriate debian/ubuntu security database and give you a less false positive prone set of findings (although with debian based stuff you still have the problem of scanners reporting the "unfixed" set sometimes)

Been in similar situations, it is not fun.

"You have a vulnerable version of netty"

"No, we don't, that's a false positive"

"No see, the tool says you need netty 4.x and that's version 1.x, you need to update"

"OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8"

"You have to update it to 4.x, this is a vulnerability"

"Do you understand this isn't part of the thing you're concerned about"

"I am only concerned about shipping a product without vulnerabilities"

"Well this isn't one, because it's a false positive, here's the CVE, here's what it applies to, here's the link to this package on maven central, it's not part of netty, back off"

"But it's insecure, the tool says so"

We went round and round for about 3 hours until I told the guy (an infosec 'pro') to leave me alone until he'd figured out how to do his job... he came back to me the next morning for another 3 hour slack argument along the exact same lines at which point I told him to leave me alone permanently and communicate through management if he had anything to say. I'm not massively impressed with infosec as a profession after a few similar encounters.

Security engineer here. This is sadly common. The grim fact seems to be that we have a dearth of information security analysts with engineering experience. If they don't have "Engineer" or similar in their title, odds are they haven't had the pleasure of building or maintaining a non-trivial unit of software over more than a quarter or so.

That said, I've seen the inverse problem: engineering staff that either don't understand how their dependencies are managed (not unreasonable for less experienced teams, NPM for example has a tendency for huge dependency trees and this can be a hard problem to manage for a big project) or whom are for a plethora of reasons incentivized to push back on requests for work they don't perceive as important.

The middle ground here requires trust between both parties (InfoSec and Engineering org functions). Sadly, not all security programs are well managed and not all engineering teams have mature practices.

If you have time or capacity, you can get a lot of leverage here by educating your security analysts on how your technology works (e.g. how the build system functions, how your languages of choice share and bundle code). Trust can be built between engineering and infosec teams by educating where possible. Done well such practices can up-level both sides of these discussions and save you stress in the long run.

This is for example why we, as a company, rather took 2 hours to escalate Log4Shell internally. Which was like 2-3 hours before it went globally holy shit.

We took the time to analyze the vulnerability, and multiple competent people concluded: Yup. This is serious. Then we figured out plans, mitigations, and communication paths to update the plans and mitigations together with ops, dev and other folks. And then we shoved all of that as a large document into the larger technical organization as "Holy fuck the sky is falling, start patching, here's how to see when you have to, here's how you do. Go".

And that's also something I'm currently telling our new full-time security engineer. Be a bit mindful in the communication of criticality and vunlerability. Like, with my team of admins and SREs, you can be like "We're all doomed" and someone will be "but we don't deploy that feature, calm down. Would you like some tea? What evil thing did the CVE say to you?". Or they might escalate accordingly, if it looks bad to them, too.

But if you do that to most of our dev-teams? That'll be just an unstructured mess, which will be worse than waiting a moment to structure a proper response to the threat.

> Trust can be built between engineering and infosec teams

Yes it can, and I have worked with 'good' infosec people who were all about improving practices together. That attitude goes a long way - developers (well, the good ones) want to ship secure stuff and want to fold in best practices.

But the profession is unfortunately rife with people who don't know as much as they think they do, and who wade in heavy-handed, laying down the law to developers who might have been round the block once or twice themselves...

I wonder if it wouldn't be better to have a hybrid role that is part of engineering but has specific responsibility in this area, rather than as is often the case, an external party (or a separate team) which is perceived as adversarial.

Some organizations do do this and they call them product security engineers or similar. Sadly, this doesn’t remove the need for an informed organization at large—that function covers a lot more than securing code—but it’s absolutely something that has been done.

There is one caveat however: you’ll probably still need dedicated security analysis or security engineers outside the chain of command of your product or engineering org.

There is a degree of necessary opposing pressures and incentives for infosec departments vs product departments: product organizations are rewarded for shipping more. Infosec organizations are rewarded for preventing incidents.

The most common result of this contention are situations people often lament in the comments here—security programs that are either ineffective, heavy handed, or both.

In my mind, a good security program positions itself as an enablement function and should strive to educate and inform over all else. People close to the work will be best positioned to make decisions about effective ways to harden their systems when informed of likely risks, requirements, or other security considerations.

With any luck this minimizes the need to mandate security motivated changes and changes they dynamic when your local security professional comes by with specific concerns. It’ll be less of “the sky is falling” and more of “we saw this specific problem and would like to work with you to fix it”.

I run a security team in a large company.

What you described is usually the result of some "consulting company" (in our case big ones) that drop stuff with zero actual knowledge.

Every year we "review" these with the board and it is annoying as fuck.

Recently they got a new manager who understands we are in the same boat. They have to provide a report with some findings and I need to have a secure environment. We talked this over and suddenly the board presentation was cool.

As for internal teams, I have a few extraordinary people. They are very young and I had to coach them on how the vulnerability is actually critical or not depending on the context, data, etc.

So not all cybersecurity teams suck :)

Hi! Question from a fellow practitioner: What tools or processes do you and your team use to identify false positives in such scans and reports? How do you track out-of-band mitigations against real-but-unreachable vulnerabilities that have been identified? e.g. A WAF set to intercept requests that trigger a specific vuln, or network segmentation to limit the scope of a moderate vulnerability.

In my experience tuning out false positives and other noise is fairly straightforward when you're only looking at CVEs, but things rapidly break down when you're trying to prioritize work with limited resources (only so many security engineers and engineering teams with spare cycles on their roadmap).

I have yet to find a good solution here other than manually whitelisting specific vulns in some kind of tracker and reviewing them whenever the network topology or configuration for an application changes.

Curious to hear your experiences, though I suspect this is a common and difficult to solve problem.

Well, this is a real circus.

I have tested and deployed several scanning tool (and have contributed to nessus some years ago - their scanning capacities were really mediocre compared to the precision of the results. In out 100+k IPs on 10.x networks the time to scan was horrendous. But once an IP was scanned, the results were fine). A small disclaimer: I use several solutions but have no stakes in any of them (neither company nor personal). I have build relationships with them that helps me to provide substantiated feedback that sometimes event get taken into account :)

There are several issues, none of them has an automated way of fixing.

First there is the criticality of the finding. We found that often (especially on Linux) you will get a "raw" critical issue on a component that was not patched; And why it was not patched? Because the vendor set a low level. It is then that the discussion between the security team and the OS one starts - I have experimentally found that the vendor rating is usually better because they take into account the actual context of the deployment, especially for libraries.

OTOH crowdstrike vulnerability rating (I forgot the name of the component, it is on top of the EDR) is actually two ratings: the CVSS-reated one, and their own where they look at actual exploits and deployment. I have decided that critical+critical = actually critical that must be patched no matter what. This is the only way I found to quantitatively auto-decide on important patches. The goal is of course to patch everything but the reality is that there are so many loose ends in products that fighting for anything below the top two levels does not make sense.

The very sad thing here is that this is perceived as an issue of the OS/middleware and not the shitty products vendors deploy. They use some undocumented or weak solutions and then cry when everything breaks down. I develop myself (amateur developer) and have never had any issues for 30 years when upgrading the OS because I actually care about following the docs.

The second problem is that today products have nested dependencies. For our own products we do an awfully time taking and precise review of the actual impact on the product but this is because people actually care (this is not my team so I ma not taking any credit - I just like their approach). When a customer comes with a question we have a comprehensive response. This works or not, some people when they see "vulnerability" they stop at that and want that word to disappear.

When we are on the receiving side of such products (and log4j is a perfect example), we ask the vendor. If this is open source the answer is usually very good, there are some github/gitlab issues that address the point. Commercial vendors are much worse because they either say "no worries" or take weeks to respond. When a "no worry" comment comes from a company that cannot explain why they limit their passwords to 27 characters, it is worrisome - a bit like when my children say "no worries" (this is the moment when I start to worry).

Scanners are quite slow to respond to non obvious vulnerabilities (the ones you cannot easily tag though OVAL). For log4j for instance I coded a scanner/reflector on the night of Thursday (the information popped up about noon on Thursday western europe time). The scanners followed up after a week, but one could fine good ones open source quickly as well. So my advice would be to closely monitor GitHub for such tools when there is pressure (we switched to another tool for log4j when my team and the OS ones were successively finding better ones). This also led us to investigate a fantastic tool/framework to write scanners, which name escapes me right now but I can comment here when I am back from vacation and get hold of the gal who is looking at that.

The TL;DR version is that

- a few tools provide a way to quantify the risk (that's a huge slit but the best I can do) - at least there is a reasonable chance that you will not miss the worst ones

- GitHub activity when shit hits the fan is extraordinarily useful - the best tools emerge there, before the vendors catch up

- vendors are medium, but is is not as bad as you read

- your CISO should make the effort to communicate that there will be availability issues when the IT/security team decides to switch off a key component (say - email) if this presents a risk for the company. I have the chance to have a CEO who not only understands this but actually challenged me once to why I did not cut earlier. Having that arranged upfront talks a huge stress off the security team.

- and finally the sad reality is that the vast, vast majority of issues we get are from users who click on stuff

I am sure I forgot a lot - let me know if I can help with some details.

Thank you for the fantastic response, I always enjoy seeing how folks approach these problems. Your points about inconsistency between how vendors vs OS distributions handle things is well taken and mirrors some things I've seen.

Please do feel free to reach out via email or LinkedIn (deets in my profile), always a pleasure talking shop with folks in the industry.

Cheers!

I sometimes feel that my job description could be "arguing with people who feel passionately about something they don't understand".

Man: What is the secret to eternal happiness?

Guru: To not argue with fools.

Man: I disagree.

Guru: Yes, you are right.

Like I usually say, everyone is a doctor and a football coach, and recently a security expert.

I am coming to a point in my 25+ years career where I stopped to argue. I just tell them that they are wrong but I do not care until they put the company on danger.

Unfortunately this leaves me with fat powerpoint presentations which well to be the modern documentation of a company.

Many advisory jobs fall into this bucket. I'm of the impression that anybody who reaches senior/staff+ roles end up doing this as part of their day to day.

Isn't that just a description of life in general?

Not for the people who are arguing with me.

Over-reliance on tools is a bad problem in Infosec, often due to the range of areas they have to cover. That said a good infosec person should always have an understanding of where their tools are limited, so they can work around those deficiencies.

It sounds like some of this should be automated. If all the infosec person is doing is running a tool against a repo and reporting results, that should be automated. When there are false positives, that should be annotated in the repo with multiple people checking off on it.

I’ve been impressed by automated bug checking tools in the past and I see this as part of the same issue. I don’t see why this would need an FTE to run code against a tool. CI should be enough.

It's the "intelligent life form determines the alert is a false positive" part that's not automated.

Of course that’s not automated.

From the parent, it sounded like the their issue wasn’t figuring out if something was a false positive. It was that another “intelligent life form” ran a tool and then wouldn’t accept when another “intelligent life form” assessed that a flag was a false positive.

If the infosec person is just running a tool and reporting results, why are they part of the loop? Make running the tool mandatory for each git push back to the main repo. Then, if/when there is a false positive, allow them to pass if they’ve already been “approved” by some means (like a .infosec_ignore file).

In this case they weren't even running the tool, just looking over reports generated by the dev team.

Same experience several times. Tool picks up examples code dependencies from our dependencies is another form of this.

This is where you could probably just have renamed the file.

Once you realize this sort of shallow, automated audit exists to grease the wheels of some security theater operation, this sort of back and forth makes sense.

False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.

So somehow false positives negate all the actual positives caught and corrected? The only true solution is what? Manual audit of everything by some perfect human security practitioner? I suppose the same applies to automated development tools then.

I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh. But by in large automation and the problems inherent are required.

I was in this industry for a while (I did 5 years full time for a security company).

> I will concede there probably are some firms out there acting poorly that way.

This is a hilarious take. Here's mine: The vast majority of these firms are here for liability. They do not provide security, they provide security theater so that if/when something goes wrong, the client can claim to have followed best practices, and that it's not their fault.

This style of theater is RAMPANT in the industry. Literally most of them come in with some version of a shitty automated tool, often with false positives in the 95%+ range, and then you check off checkboxes to make legal happy, and ensure that your insurance (or your customer's insurance) will pay if you get compromised.

This is not limited to small companies, but is instead mostly how large banks, credit firms, and large enterprises work.

The entire damn show is for liability, and half of these "Security professionals" can't do anything other than read the message coming out of their tool. They are utterly incompetent when it comes to applying logic to figure out whether a specific use of a "risky" tool/language feature a genuine problem, vs a god damned fucking regex match from their tool on something completely unrelated.

I went in as a naive young dev, I came out with ZERO respect for this industry, and a healthy dose of skepticism around anything these folks are saying.

Security researchers? Generally fine (although there's a new breed of them that simply submits inane non-vulnerabilities over and over to attempt to get bug bounties from large companies)

Security advice from open source devs? Generally top notch, listen to it.

Security advice from that contractor your company paid? Expect to have 95% false positives, and they'll miss the 2 places you know actually have issues. But it's ok because you'll check all the boxes on their sheet and legal is happy.

---

I'm waiting for insurance companies to wise up and stop covering companies who are breached. Prices are already shooting way up, since it turns out security theater does a very bad job stopping real threats, and that's what this industry is right now. Might was well be the TSA of software, gonna frisk you real good, and then flunk every fucking test.

My project's client, a major local bank, mandates pen testing before we go live with the product that we're building.

Due to time and resourcing constraints, and a fixed launch date (don't ask), we had to significantly reduce scope. We are currently launching with probably 30% of the original scope. Once live we'll iterativley add features and grow our product to bring it back to the original vision.

Will we have to do pen testing again in the future? Nope. Even though a lot of our planned post launch features involve integrations with 3rd parties and the exchange of sensitive personal data.

Pen testing is 100% a box ticking exercise in my opinion. And most of the shops that offer this service are set up to provide the service such that large corporates to appease legal.

I really don't like pen testing as a practice and seriously dislike when it's a go live impediment.

What you’ve described is an issue with the bank’s procedures, not pentesting.

> I really don't like pen testing as a practice

could you elaborate on this a bit? what's wrong with it? to me it seems the only logical thing to spend external security budget on. (or internal red team if the company is so bit it can afford it.)

Thank you, that is a very interesting take. I've always played with the idea of pivoting into the security industry because playing hack the box and the like is something I do in my free time and enjoy. Maybe I'll just do some bug bounties or something for fun instead. I think it's a bit different in Europe though at least anecdotally there seems to be more code audit and the like (more product security) instead of pen testing (more network/infrastructure security).

I'd say don't let yourself be discouraged by GP. Just look into a company before you apply. Many have public reports and/or security research, both of which you could use as indicators.

Here's a repo with lots of public reports by various consultancies, you could use that as a starting point: https://github.com/juliocesarfort/public-pentesting-reports

This post is amazing. Thank you. This captures my thoughts on this problem space very well!

Security vs compliance is real. I love how you just map "compliance = security theater" because that is really the best way to describe it. The TSA bit has me in tears!

Back in the late 90s, I was working at a small web hosting company (take note). One day, a 500+ page report of a recent PCI compliancy check landed on my desk. It was nothing but "OMFG! DOMAIN1 RESPONDED TO PING! YOU WILL BE PWOWNED! OMFG! DOMAIN1 HAS DNS RECORDS! YOU WILL BE PWONED! OMFG! DOMAIN1 HAS A WEB SERVER RUNNING! YOU WILL BE PWONED!". Over and over again. For every domain we have. Complete and utter garbage report.

Better---just summarize the IPs scanned and report back which services were found running on said IPs. Then in an appendix, list why each service is (or might be) problematic. "Ping? Attackers might be able to figure out your network topology and that is bad because blah blah blah blah."

But 500 pages of this automatic breathless garbage? Utter trash.

The problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept.

In one of my past jobs I was an early-mover on doing a lot of our ops on Linux and the audit tools had no concept of the backport security model whatsoever. If you were running on some kind of LTS distro rather than a current distro, it would see "gosh you're 3 minor versions behind, you have a ton of unpatched vulnerabilities!", but in actuality you didn't, because the fixes got backported into security releases (the "31" in something like "3.1.22~ubuntu31" for example). But the tool was dumb and it just had a table that said "anything under 3.4.14 is vulnerable" even when it wasn't necessarily.

OP's "log4j 1.x is not vulnerable to this the first place" is a similar thing where either someone forgot to put in a value for "min vulnerable version" or the tool doesn't understand the concept of min-version at all. To be fair that can be genuinely tricky in java, best-case you are reading manifest files to try and pick out a version, but some legacy stuff doesn't have manifests, especially very legacy fat-jar stuff. Yes, that stuff didn't go away, it's still out there in places!

And to be clear this was a long-running issue... it wasn't a "oh our tool doesn't understand LTS, I get it" and they left me alone... this was every month they'd be back at me for a new list of utility vulnerabilities even though I repeatedly explained to them that I had set up apt-get to auto-upgrade every night and reboot, so we were running whatever the latest security backports were available, and that just like the last 10 times this was even more false-positives.

And again, this isn't just "we have to run it by you no matter what" and they leave you alone either. What security really wanted was for me to run windows and manually install the updates and get back in line with everyone else, even though that would have been a less secure outcome than my locked-down linux boxes. But this wasn't my day-job at the company so to speak, it was helping out another project with some ops and it needed to be as low-effort as possible (and to be fair their requirements weren't steep, auto-patch-and-reboot was fine by them and never caused any breakage during my tenure there).

The root problem is that these sorts of tools aren't a substitute for an actual security culture, they're often a symptom of a compliance requirement and the whole thing turns into a box-checking exercise. Someone is on their ass about this because of contractual requirements or PCI requirements, and they bought the cheapest thing off the shelf that would check the box, and they are implicitly showing you here that they don't even understand how to interpret the output of that.

> The problem isn't that audits are inherently worthless, it's that most of these tools are very low-quality implementations of the concept.

I think the problem is that the current audits are inherently worthless. I've never seen another industry that would accept a 95% false positive rate from a tool. But that's on the low end from my experiences (I've done 4 major codebase audits, and worked in the security industry for 5 years)

Tools that routinely spit out 1000 plus vulnerabilities, and 6 months later you've checked off all of them without a single valid vulnerability in the list (but 3 you found yourself during that time that were missed entirely by the shitty regex that is really the entire tool in question).

> current audits are inherently worthless

And that's not helped by the warped incentives. Security has to be, fundamentally, assessed as a holistic thing and even highly skilled technical people can't really do that with modern systems. Now you add in auditors, a profession populated by glorified accountants, working off of gargantuan checklists.

You invite and reward mediocrity. At best.

Anyone good enough to actually understand and able to work with the technology will find a job in the industry, for twice the pay. And the same applies to regulators. They can't retain technically skilled staff, so they are filled with accountants. As a result we get rules and regulations, written by technically incompetent accountants, aimed at technically incompetent accountants.

To fill the gaps we have shitty, false-positive ridden, noisy, outright useless tools that generate reports intended to satisfy accountants. Security has next to nothing to do with that.

And to top it off, we have an entire industry riddled with so-called security engineers who know how to run a tool (with some marginally helpful StackOverflow pointers) and export a report, but can't for their life actually interpret, let alone VERIFY, any of it.

That has as much to do with security engineering as burning ants with a magnifying glass has to do with biology.

You could set your build up to not auto-download questionable stuff from untrustworthy machines on the Internet.

> we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency

If you use Maven, just exclude that transitive dependency from the explicit dependency.

But yeah, dumb warnings are dumb.

  <dependency>
    <groupId>com.example</groupId>
    <artifactId>my-dependency</artifactId>
    <exclusions>
      <exclusion>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
      </exclusion>
    </exclusions>
  </dependency>

Bingo.

We (read: I) face this on an ongoing basis.

We can't upgrade the entire components that bundled up log4j for various reasons, starting from licensing rules. So we made the decision to strip out the entire JndiLookup class from every project that uses java. Clients do various scans, and rely on dumb version string matching and/or banner grabbing. We have to routinely point them to our VERY detailed and explicit log4j response document, and carefully explain to them that their scanners are relying on insufficient detection methods.

Security teams are quite content with us giving them detailed explanation about the false positive. And then they forget or deliberately choose to ignore the lesson and the next time they run their scans, get the same false positives again.

Even supposedly state-of-the-art security tools, to this day, refuse to actually verify their detections.

I recall that NVIDIA had to release an update to their CUDA Toolkit around when the Log4j vulnerability was announced. They weren't using it but the file was being distributed, presumably for a similar reason. I'm guessing they might've had similar complaints coming in.

Configure your build tool to exclude the transitive dependency.

Typically there's a way to suppress specific warnings in systems like these. In your company's situation, I would look at moving away from a scanning system if it didn't allow overrides like this.

So far this is the best approach I've found. The scanning tools rarely include that ability but if you build tooling around them you can maintain exclusion lists, for particular vulnerabilities, library/version pairs, etc.

Unfortunately it does mean there's no getting around having someone manually deal with false positives.

Yup, that's us. Same deal exactly. They already stoppered our entire team for eight months with a tools audit a few years ago, so I hung that dead fish above the conversation when this came up. "Uh, ok, don't want to do that, sooooo . . recommendations?" "Let's isolate it, toss stuff at it, and see if it lights up" "Sounds good to me. FIXED"

I thought Log4j 1.1 was vulnerable, just to a lesser extent, and there wasn't a lot of information on it because 1.x is end of life

EOL libraries are themselves vulnerabilities. It's not really good to even have this reachable on the classpath, in cases like the grandparent, you should look at removing the class from inside the JAR (or removing the JAR itself) if possible to ensure that the vulnerable code cannot get called under any circumstances.

See https://logging.apache.org/log4j/1.2/ . Plenty of vulnerabilities. Just different ones

Log4j 1.x is vulnerable, but to different things and not as badly.

> Log4j 1.x is vulnerable, but to different things

And the most common use of log4j 1.x (logging to the console or to a file, with a simple configuration) uses none of the vulnerable parts.

Someone said "hey log4j 1.2 wasn't that bad" after the log4shell vuln and came up with the idea of reload4j which is just log4j 1.2 but with its known vulnerabilities, bugs and performance issues fixed. Complete feature freeze besides that. Considering that a log tool shouldn't have that many features and that log4j 1.2 was being used by a lot of companies (just see maven central stats) there is no need to add more features to reload4j, which I find kind of cool

Log4j-api isn't vulnerable and the slf4j-log4j bridge depends on it. Log4j-core is. All of our internal security scanners failed to make this distinction so we had to post filter ourselves.

Reply on news.ycombinator.com