Hacker News Reader

> We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.

I find it quite ironic to praise the forward-thinking of the company that has been instrumental to bring us into this mess via their vision of loading dependencies at runtime from an online repository. Sun imagined that the code and its configuration does not have to worry about how to fulfill its dependencies, but instead JNDI [1] could magically fetch the appropriate objects from wherever, don't worry.

I'd say I would even find it ironic to praise the Java ecosystem, which, due to its excessive over-engineering, has produced best practice frameworks where fully documented behavior has the latent potential to just be catastrophically exploitable by accident, as nobody is able to reasonably understand how all of it plays together. I have a hard time to imagine how you could enable loading of external code in Rust or Go by accident using run-of-the-mill logging frameworks.

[1] https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Inte...

I'm writing this from memory, with checking any sources. IIRC:

Designs like JNDI were widespread in the IT software industry at the time.

The academic prototype for all this was CORBA, the Common Object Request Broker Architecture. An object-oriented was to transmit data between systems running on separate processes, CPUs, remote servers.

CORBA was a next-gen RPC (Remote Procedure Call) framework, the data format on the wire closely follows the format for C structs using Sun-RPC, but was way better because these CORBA things were discoverable and described at run-time, rather than compile-time libraries.

That's right: RPC was initially a Sun UNIX thing.

Microsoft needed a way to do this, so they had COM, the Component Object Model, for run-time discovery and linking (late binding).. Microsoft got COM to reach across network connections, "DCOM", for Distributed systems.

Meanwhile, corporate IT was still pulling text out of databases and shoving it into C structs. It sucked.

Microsoft's object model was better, and Java was better than better, because you didn't have to license it to use it. Well, you did; it was like an MIT License.

Reason why IBM writes huge Java systems. And why Oracle owns Java. For some value of ownership.

Yay objects!

In CORBA, RPC, etc., the R stands for Remote and it will execute something on the other end of the wire.

In these Java attacks the code is downloaded and run locally, with all secrets and data access of the attacked program.

I think you could take this further. There's another "log4j" vulnerability out there right now, I guarantee it. It might be exploited, it might not be. But there's no patch.

What are you doing about it?

With log4j you have a brutal combination of:

1. RCE

2. Exposure

RCE happens frequently but often attackers don't have an easy time getting to the exploitable code. With log4j it's trivial - every app can be owned.

But here are some questions:

1. Why do those apps have the ability to make network requests?

2. For the apps that need to make those requests internally, why aren't those over mTLS?

3. For the apps that need to make them externally, why isn't that going through egress proxying?

4. Why are there credentials spewed all over your environment variables across your services? Are they short lived?

It's actually not super hard to have a worst-case scenario vulnerability like log4j be not that bad for your organization. A bit of hardening and even if something like this happens you're in a good position to wait, monitor, and patch.

it's definitely time for a language with a capabilities based security model that allows scope to the function/method level, at minimum

not just for security either. i can't count the number of times i've traced a performance problem or bug to a piece of code that was doing some io when it didn't need to

Agreed. In a capabilities system you'd see that, for some reason, your logger requires all sorts of absurd capabilities for no reason.

Haskell comes to mind?

Regarding your questions, they make sense in the context of modern software built in the last 5 years. However, most large enterprises have a long tail of legacy applications built years ago. These may be inherited from a combination of acquisitions, from vendors that have since gone under, some from. The common factor is that the original developers are long gone, and they may be supported using a skeleton crew or not at all. Documentation if it ever existed is likely lost. What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises

Yes, some organizations have so much technical debt that any security work is going to take extraordinarily large amounts of work.

That's a bummer, but I suspect many on HN can in fact build software that isn't total garbage.

Doing it right sometimes pays off in the long run, but cutting corners always pays off now.

Exactly. Managers and execs love pointing out the cost savings they made this year or this quarter. By the time the chicks start to hatch "in the long run", they most likely would have already moved to another enterprise so someone else has to clean up their mess. And by clean up, I mean repeat exactly the same mistakes since nobody promotes you for implementing stuff that might show its benefits in the long run.

Also one of the root problems with politics. 2035 emissions targets are the next government's problem. Rehabilitative policing pays off in 20 years, retributive buys voters now.

Egress is a very fair question IMO. I've worked at more than one place where Infosec would start integration talks at 'What are the static IPs and how do we rotate certs' Every layer you go lower (longer certs, domain whitelisting vs static IPs, etc.) is a tradeoff of maintenance and security.

> What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises

Only if they fail to consolidate vendor integrations and/or have already insufficient staff. (Which is often indeed a problem.)

Simple, if brutal solution; Infosec audits logs and works with application teams to confirm existing traffic and pre-document new traffic out. With properly staffed teams you should run out of surprises within a year.

I suspect that a manager that suggested / insisted on such properly staffed teams, would get fired or no longer promoted, at most big companies. Since it'd cost a lot, and not show any results the nearest time / years. And also, they might hire the wrong people, and just burn money for nothing

> Simple, if brutal solution ... properly staffed teams

The recruitment part is not simple, maybe bordering to impossible? for some larger companies. I say, based on my past experiences in how clueless big companies can be concerning what technical people they hire.

> and not show any results the nearest time / years.

Depends on how you game OKRs. Implement auditing Q1. Contact all teams Q2. Have 75% of teams Q3. 90% of teams Q4. Just an example but its all in how you sell it. Big IT understands icebergs, you just have to give a good/accurate roadmap and execute if they agree on it.

You might be correct that a lack of insight on this sort of thing is an organizational problem. My experience indicates that if you can't get a list of dependencies based on a series of e-mails down an org chart, something is wrong. Either someone is managing too many services for the level of tooling you have, or there's a complete breakdown in org comms (which happens more in larger enterprises.)

Sounds interesting, but what does "Have 75% of teams Q3" mean?

With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.

ELF binaries are just collections of another sort. You can scan for symbol names or assembled instructions common to a dependency, and you might be able to update with an LD_PRELOAD library that patches the symbol table.

Just look at what game modders have accomplished without access to source code.

This isn't how the real world works. Symbol patching is only effective if there's a symbol for it, which is often not the case (and probably wouldn't be the case here) if the binary has been stripped[0].

> Just look at what game modders have accomplished without access to source code.

Game modding is usually done on Windows given the target market (up until recently of course) and thus usually means Windows PE's, which do not have debug symbols attached - ever. Windows debug symbol databases are emitted as PDB files and are almost always omitted from game releases.

This means modding has to do a sigscan[1] in most cases in order to find something interesting - especially in the case of ASLR[2]. Then whatever modding framework (or hack) can set itself up and hook into the game.

These techniques are NOT what we should be encouraging, and are certainly not common, even for vulnerability mitigation. Re-compilation and re-deployment should be the defacto mode of operation in production, especially for systems that have sensitive information. Relying on bin-patching is not something many security specialists would regard as a "good" mitigation.

[0] https://en.wikipedia.org/wiki/Strip_%28Unix%29

[1] https://wiki.alliedmods.net/Signature_Scanning

[2] https://en.wikipedia.org/wiki/Address_space_layout_randomiza...

Besides this, who actually scans the deployed binaries? Just check which version of the binary was deployed, backtrack to the source, and get the dependency tree from there. Problem solved. (Except for shaded dependencies…, but scanning for that should happen at build time anyway.)

Good luck doing that on the custom software some vendor provided you 5 years ago and since has gone out of business.. and that’s a GOOD scenario for a lot of enterprise software. Sometimes it’s 20 years.

Lesson learned. Never buy closed source.

Imagine the vulnerability would have been in the vendor code instead of in the log4j jar. You’d. still need a way to fix and rebuild it.

Note that this doesn’t have to mean a public MIT repo on GitHub. You can have more restrictive gray box licenses between your b2b partners only.

If it's a 20 year old closed source application, there probably aren't any exploits in the wild. You can bet there are exploits for log4j.

We need some sort of escrow for source code of closed-source product. Whenever there is practical need to have access to source code, it should be available.

Okay. Just like any other software that should be firewalled in to only connect to what it should connect to. Moreover, I think meta sploit already has this vulnerability built in so it should be trivial to test/fuzz the application.

If you still have the source code for your rust code, it contains the lock file. Look in there and you'll see all dependencies and their version. Why do you want to look at the binary instead?

In commercial software the customers don’t have the source files…

Well, we're in a thread that started with

> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime.

If there is the possibility to plan ahead from a sufficiently strong bargaining position, you will not end up in the situation you described. The seller should either continue to support it with a good SLA, or you get the source code with a turn-key build system if the seller discontinues its support. Or you switch out the software if the SLA ends. Should you not be able to agree on such terms, I don't think you would have enough influence to tell the seller to write their software in such a way that dependencies are easy to hotfix/upgrade.

ROFL.

Ok, after putting the unicorns herd to bed, an event happens and congratulations, you now own a license to PeopleSoft v.whatever source code. Oh yeah, you also got the management platform for your network provider too.

Now what?

It's called code escrow and it's a real thing even if it's news to you. Although yes, it's usually an insurance plan you want to never ever need because using it will suck.

The source code is not what is deployed.

There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.

If it came from an external organisation, things are exponentially harder.

Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.

IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.

Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.

This sounds like an ops problem. It shouldn't be hard to tell what's running in your production environment and tie that back to a specific commit.

It shouldn't be hard for a dev to update a dependency which should trigger an auto build and roll your pipeline. Shouldn't be hard for an upstream vendor to patch and have that pulled to kick off a new build.

IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.

> IDK where this idea that no one can know what's in a dependency tree is coming from

I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody.

How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or…

And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?

Presumably you would go ask Oracle, and then patch it when they provide a patch.

Hope you’ve got a pretty sweet support contract or you’re going to be waiting awhile.

Depending on the product, maybe forever.

My point is that patching third party binaries is generally not something companies take responsibility for. They wait for vendors to patch the binaries and ship them.

> generally not something companies take responsibility for

I have some bad news for you: viruses and hackers don't care about your support contracts and the delays they cause.

Actually, I tell a lie: the hackers love them.

That's not relevant? The original premise here is "compiled languages will make things worse because they're harder to patch" except:

a) It's very rare to patch things manually

b) It's very rare to develop patches yourself (for 3rd party software)

I'm sure some companies out there with many decades old deployments have to do stupid shit like that, but it's hardly the standard.

Having been on the receiving end of waiting a long time to get a patch for a actively being exploited zero day - the point is, even if you want to fix it yourself (which I desperately did), you can’t.

If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants.

Many enterprises are in exactly this bind. And support contracts only get you so far.

I'm losing your point. So, you wanted to patch it, and you couldn't, so how is it bad then that a compiled binary would make patching more difficult?

It was impossible to fix because it was a compiled binary with no available source. That’s the point.

Why would you have oracle without a super sweet support contract? That's the whole point.

People don’t get Oracle because they want to get stuck with a super expensive (and still only ok), or super insanely expensive (and maybe sweet) support contract. They get it because they usually don’t know about these things, are trying to do something and BLAM they get hooked.

Many people don’t even realize how long it can take to get a real issue fixed even with a super sweet support contract. For instance, I once was a wet behind the ears Oracle DBA, and when creating a schema i dumbly used a feature in the documentation without checking to see how long it had been released. Unfortunately, it had been released about 6 months before, as I found out later.

It took 6 months for them to figure out that it was the cause of our sporadic ORA600 errors (aka core dumps), by which point we’d already migrated to MySQL for many of the workloads because the database was having at best 95% reliability because of it.

Once we migrated the table off with the feature, somehow our problems ceased, but that was still well before they told us what was going on.

It is! But a lot of ops departments have problems! OPs point is that a lot of deployments out there make it really hard to remediate this kind of vuln.

Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.

Yes, some places suck. They're going to suck either way. If their system is so backwards that they need to manually patch services because everything is EOL, and compiled code will break their workflow, they have plenty of other shit to deal with.

I don't think we should optimize for garbage companies with garbage practices.

I'd say places lie on a spectrum of sucking. With a lot of places being very important and far from perfect. Those places really should get good security. That means designing for security in the common situation rather than the situation of 'best practices'.

I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).

In no way is manually writing a patch and then live patching the common situation.

The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.

Go 1.18 onwards will solve this problem: https://utcc.utoronto.ca/~cks/space/blog/programming/GoVersi...

> The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).

My list of languages to practice were Scala and Rust. Maybe I'll do Scala and Go considering this single point.

There's also ongoing work to build a dependency vulnerability database. This is proposed to be surfaced via a built in `go audit` command.

https://go.googlesource.com/proposal/+/master/design/draft-v...

On Windows, the Microsoft compilers and linkers have inserted such toolchain/library version metadata into PE executables for decades now (the "RICH" header — https://bytepointer.com/articles/the_microsoft_rich_header.h...).

At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.

So if you make a commit to the README.md fixing a typo, now your build of the source code has a different checksum? I wonder why they don't instead hash all the source files and then embed that hash.

That doesn’t provide a link back to the source, unless you store that hash elsewhere. Giving you the problem of maintaining a database also.

You store the source packages alongside the binary packages, and then index hashes of all the source/binary package contents. Then you know which source files were used from which source/binary packages to produce a particular executable.

> So if you make a commit to the README.md fixing a typo, now your build of the source code has a different checksum?

Yes...? What's the problem?

> why they don't instead hash all the source files and then embed that hash.

This is almost exactly what the git sha is. If you're arguing that README.md files shouldn't be included in the checksum, that's subjective. Many people would argue otherwise. And it doesn't hurt to be more accurate than less.

The git commit hash is less accurate, since it includes extra files like README.md that aren't converted to any part of the executable.

> The source code is not what is deployed. There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.

Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build.

Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.

Yeah our org does the same thing and I'd say this is the gold standard (or at least mine) but sadly I'm assuming most companies don't do this, not everyone runs on docker nor has a build pipeline like this.

Sure, but everyone should be able to have their built process embed that info into build artifacts, whether it's by passing GCC a DEFINE flag that ends up in --version output, passing nasm a data structure embedded at X bytes into the binary, a version.txt in the output zip, an autogenerated comment inlined in each HTML/JS/CSS file, whatever... somehow, you can embed build info.

This happened last week, for real with ${dayjob} customer. Put yourself in their shoes for a second.

Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive.

Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1]

So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.

Now what?

No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.

A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server.

There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use.

Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on.

[1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir.

[2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)

> Now what?

Goodbye and thanks for all the fish. Run!

So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.

Now what?

The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.

That provides incomplete protection even for this Log4j vulnerability, and similarly trivial "self help patching" may not be available for future vulnerabilities.

This generalizes to "apply a vulnerability-specific tricky auto-fix to a binary it was never tested on". That might be a reasonable solution if there's nothing better (and since speed is of the essence, there might not be). But it's not exactly a reliable solution, and not every vulnerability will have hacks like that available, nor is it really clear to what extent they fix the problem; it's pretty easy to miss something using this kind of approach, or to have some future deploy bot restore the vulnerability in normal operation. And hacks like this can break things, both directly (the patch breaks actually used functionality unexpectedly), and indirectly (the fact that the software was patched at all breaks sanity checks or other cleverness like binary diffing or whatever).

I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.

> No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.

Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it.

If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host.

Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.

Do another release. You do have a way to track releases right?

Perhaps everyone is stupid.

It’s funny because when I worked in .gov IT, we were all characterized as stupid donkeys by because we weren’t able to just use NPM, etc to do cool kid stuff.

Good luck getting security people with the power and brass balls to do that in any company.

You can make dynamically linked executables with both Go and Rust. The compilers have no blame in this.

Sure. But you have a slight problem with ABI stability, at least in Rust, if you ever want to compile an ABI compatible replacement library. Until that problem is resolved, if you want to make a useful replaceable object you need to make a C style foreign function interface and use that, something that isn't exactly convenient for Rust to Rust use, even if rather helpful for making Rust libraries usable from other languages. C++ has a very similar issue, and has had it for years, albeit one mostly resolved by the advent of stable C++ ABI standards.

> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime. This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide. Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.

I think this attitude is one of the main issues for this kind of incidents in larger organisations. I'm not sure why you would be willing to lay blame at your colleagues for this. Most of the time, development teams don't tend to like some governance over the code and the dependencies they are pulling in. Not just because they know of course what they are doing, but also because they are under pressure to deliver features. That is what matters for business. Also, convincing management that budget is needed for correct tooling to track all stuff deployed; is also not as straight forward as you seem to suggest.

In this case, the problem goes even beyond just the code of your own dev teams. This is embedded in countless software packages deployed all over your organisation. Same here, people want to buy and use whatever they want. And all processes to keep some form of control over it, are mostly seen as overhead.

And I'm sure they are lots of "security" people who are just producing documents and policies which are complete detached from reality. But developers who consider security completely as somebody else his responsibility, are a problem as well.

The only good think I see coming from this mess, is that security teams probably will get the means to try to get more control and insight over this. For the coming weeks/months at least. After that, everyone in management of dev will be forgotten about it. But something tells me the security people who are working on this right now, won't.

> If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?

Our software scans take 3 forms:

1. Searching for file names

2. Running test exploits that have a payload that just logs the vulnerability with a central monitoring system

3. Dependency resolution and checks against a DB (our solution _already_ supports Cargo.lock files.

Rust (and Go, C++, etc.) only prevent option 1. Thanks to bundlers it also can miss it in JS, or shaded fatjars it can also fail kn Java too. So it's not a new challenge

Bundling/embedding/vendoring/monorepos and other names for embedded code copies really need to stop being common practice in the tech world.

What should be done instead?

When HeartBleed came out, there was a single dynamically linked library to be replaced. If you had a vulnerable version of libopenssl, then you replace the .so or .dll with a patched version, restart any applications that were using the old version, and you're set.

This works because the binary is compiled against the dynamic library, using the system-provided library. It isn't statically compiled, because even though that frees the developers from worrying about library versions, it makes the sysadmin's job tracking library versions be much harder. It isn't vendored, for the same reason.

Depend on your dependencies instead of embedding them. Specify version ranges instead of pinning dependencies. Report issues in your dependencies and contribute back fixes for them.

... hear back from upstream that your fix, developed against version X (that you pinned and use), does not apply to HEAD. Pull HEAD and realize there have been major changes which conflict with your platform, use case, or any number of your other dependencies.

Sometimes you can get away with just depending, and that’s nice, but other times there are strong reasons to embed. At that point, you own the code, and can actually do the things you need to do. (Of course, it follows that you own the code - and need to then actually do the things you need to do.)

Both approaches have their advantages, and both have potential for technical debt. Which approach offers the most advantage for the least debt will depend on the individual circumstance; I don’t feel that you can really advocate for one over the other in a general sense.

Go binaries will contain the commit hash, which contains all the information about what it was compiler with.

> With languages that output a single binary by default such as Rust and Go, we would have been screwed.

no, best practices include checking in your Cargo lock file

You can, and people do, scan native binaries for these things. It's called SCA (software composition analysis).