Hacker News Reader

Comment by scjody | Hacker News Reader

We've been shipping a Docker-based app to customers for years, and every now and then one of them runs a security scanner on our images. I have yet to see a scan that isn't a disaster of false positives (for the reasons outlined in the article and more!)

One of the craziest recent examples was a scan using a tool called Twistlock. Many of our images are built from an upstream image that may have outdated apt dependencies, so one of the first things we do is upgrade them. Twistlock flagged _every instance_ of this because "Package binaries should not be altered" (in other words, between subsequent layers in an image). I am baffled how anyone at Twistlock decided that this was a useful thing for their product to detect, or why any Twistlock customer trusts it given issues like this.

View on HN

> I am baffled how anyone at Twistlock decided that this was a useful thing for their product to detect, or why any Twistlock customer trusts it given issues like this.

If I was injecting something malicious into your containers via updates, this is exactly how I would go about doing it and exactly what would catch it.

What I'm seeing here is that Twistlock and other tools don't reliably do a good job of explaining why something is flagged in a way that's understandable and accessible to developers. Though honestly I've yet to find any approach to informing developers that actually works.

My favorite was giving them a clear link in the error message about why the build was failing and how to fix it.

It flags that because it could indicate someone got onto your system and injected their own code or changed machine instructions at the binary level, which is a pretty common way to get a remote shell.

It is annoying to have to mark false positives, but that's just the nature of the beast when it comes to being thorough about security. More annoying with this check than when you update packages in a container image instead of starting clean is that this same technique is often used to compare hashes of packages managed by an installer versus what is actually on disk, and thus flags every single package in a JIT-compiled language that caches byte code on disk as altered.

I have used Prisma Cloud / twistlock. The tampering detection is only useful for detecting changes to running containers, not for changes to binaries between layers. The latter is just dumb and causes anti-productive false positives like above.

It’s because they’re implementing the feature so they can show a CISO a big scary report and say “good thing you paid us - otherwise you wouldn’t have known!”

If they were serious about build errors they could use the built-in features of APT, YUM, etc. to only report binaries which don’t match the canonical distribution’s hashes, as has been standard sysadmin practice for aeons.

Reply on news.ycombinator.com