Hacker News Reader

Comment by jiggawatts | Hacker News Reader

The source code is not what is deployed.

There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.

If it came from an external organisation, things are exponentially harder.

Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.

IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.

Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.

View on HN

Go 1.18 onwards will solve this problem: https://utcc.utoronto.ca/~cks/space/blog/programming/GoVersi...

> The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).

My list of languages to practice were Scala and Rust. Maybe I'll do Scala and Go considering this single point.

There's also ongoing work to build a dependency vulnerability database. This is proposed to be surfaced via a built in `go audit` command.

https://go.googlesource.com/proposal/+/master/design/draft-v...

On Windows, the Microsoft compilers and linkers have inserted such toolchain/library version metadata into PE executables for decades now (the "RICH" header — https://bytepointer.com/articles/the_microsoft_rich_header.h...).

At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.

So if you make a commit to the README.md fixing a typo, now your build of the source code has a different checksum? I wonder why they don't instead hash all the source files and then embed that hash.

That doesn’t provide a link back to the source, unless you store that hash elsewhere. Giving you the problem of maintaining a database also.

You store the source packages alongside the binary packages, and then index hashes of all the source/binary package contents. Then you know which source files were used from which source/binary packages to produce a particular executable.

> So if you make a commit to the README.md fixing a typo, now your build of the source code has a different checksum?

Yes...? What's the problem?

> why they don't instead hash all the source files and then embed that hash.

This is almost exactly what the git sha is. If you're arguing that README.md files shouldn't be included in the checksum, that's subjective. Many people would argue otherwise. And it doesn't hurt to be more accurate than less.

The git commit hash is less accurate, since it includes extra files like README.md that aren't converted to any part of the executable.

This sounds like an ops problem. It shouldn't be hard to tell what's running in your production environment and tie that back to a specific commit.

It shouldn't be hard for a dev to update a dependency which should trigger an auto build and roll your pipeline. Shouldn't be hard for an upstream vendor to patch and have that pulled to kick off a new build.

IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.

> IDK where this idea that no one can know what's in a dependency tree is coming from

I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody.

How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or…

And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?

Presumably you would go ask Oracle, and then patch it when they provide a patch.

Hope you’ve got a pretty sweet support contract or you’re going to be waiting awhile.

Depending on the product, maybe forever.

My point is that patching third party binaries is generally not something companies take responsibility for. They wait for vendors to patch the binaries and ship them.

> generally not something companies take responsibility for

I have some bad news for you: viruses and hackers don't care about your support contracts and the delays they cause.

Actually, I tell a lie: the hackers love them.

That's not relevant? The original premise here is "compiled languages will make things worse because they're harder to patch" except:

a) It's very rare to patch things manually

b) It's very rare to develop patches yourself (for 3rd party software)

I'm sure some companies out there with many decades old deployments have to do stupid shit like that, but it's hardly the standard.

Having been on the receiving end of waiting a long time to get a patch for a actively being exploited zero day - the point is, even if you want to fix it yourself (which I desperately did), you can’t.

If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants.

Many enterprises are in exactly this bind. And support contracts only get you so far.

I'm losing your point. So, you wanted to patch it, and you couldn't, so how is it bad then that a compiled binary would make patching more difficult?

It was impossible to fix because it was a compiled binary with no available source. That’s the point.

Why would you have oracle without a super sweet support contract? That's the whole point.

People don’t get Oracle because they want to get stuck with a super expensive (and still only ok), or super insanely expensive (and maybe sweet) support contract. They get it because they usually don’t know about these things, are trying to do something and BLAM they get hooked.

Many people don’t even realize how long it can take to get a real issue fixed even with a super sweet support contract. For instance, I once was a wet behind the ears Oracle DBA, and when creating a schema i dumbly used a feature in the documentation without checking to see how long it had been released. Unfortunately, it had been released about 6 months before, as I found out later.

It took 6 months for them to figure out that it was the cause of our sporadic ORA600 errors (aka core dumps), by which point we’d already migrated to MySQL for many of the workloads because the database was having at best 95% reliability because of it.

Once we migrated the table off with the feature, somehow our problems ceased, but that was still well before they told us what was going on.

It is! But a lot of ops departments have problems! OPs point is that a lot of deployments out there make it really hard to remediate this kind of vuln.

Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.

Yes, some places suck. They're going to suck either way. If their system is so backwards that they need to manually patch services because everything is EOL, and compiled code will break their workflow, they have plenty of other shit to deal with.

I don't think we should optimize for garbage companies with garbage practices.

I'd say places lie on a spectrum of sucking. With a lot of places being very important and far from perfect. Those places really should get good security. That means designing for security in the common situation rather than the situation of 'best practices'.

I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).

In no way is manually writing a patch and then live patching the common situation.

The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.

> The source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.

Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build.

Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.

Yeah our org does the same thing and I'd say this is the gold standard (or at least mine) but sadly I'm assuming most companies don't do this, not everyone runs on docker nor has a build pipeline like this.

Sure, but everyone should be able to have their built process embed that info into build artifacts, whether it's by passing GCC a DEFINE flag that ends up in --version output, passing nasm a data structure embedded at X bytes into the binary, a version.txt in the output zip, an autogenerated comment inlined in each HTML/JS/CSS file, whatever... somehow, you can embed build info.

This happened last week, for real with ${dayjob} customer. Put yourself in their shoes for a second.

Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive.

Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1]

So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.

Now what?

No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.

A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server.

There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use.

Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on.

[1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir.

[2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)

> Now what?

Goodbye and thanks for all the fish. Run!

So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.

Now what?

The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.

That provides incomplete protection even for this Log4j vulnerability, and similarly trivial "self help patching" may not be available for future vulnerabilities.

This generalizes to "apply a vulnerability-specific tricky auto-fix to a binary it was never tested on". That might be a reasonable solution if there's nothing better (and since speed is of the essence, there might not be). But it's not exactly a reliable solution, and not every vulnerability will have hacks like that available, nor is it really clear to what extent they fix the problem; it's pretty easy to miss something using this kind of approach, or to have some future deploy bot restore the vulnerability in normal operation. And hacks like this can break things, both directly (the patch breaks actually used functionality unexpectedly), and indirectly (the fact that the software was patched at all breaks sanity checks or other cleverness like binary diffing or whatever).

I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.

> No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.

Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it.

If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host.

Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.

Do another release. You do have a way to track releases right?

Reply on news.ycombinator.com