Hacker News Reader

Comment by BrainVirus | Hacker News Reader

If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware.

If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues.

The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.

"Use a build tool to do dependency checks!"

And then what?

"Only update libraries if they have vulnerabilities."

So, should this be a manual process where I have to dig through obscure warnings every time I build something?

"No, you can automate it!"

Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build process reliant on yet another online service.

"But it's not a hard dependency, you can still build if the 3d party service is off."

So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this?

Etc. People with clever advice don't seem to think it through. We're fighting fragile complexity of too many tools ducktaped together by ducktaping more tools to the whole setup. Again and again and again.

View on HN

> The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.

On the contrary, I believe we do. The answer is ongoing maintenance.

The basic problem is that people persist in looking at software as a fundamentally mechanistic artifact you finish and ship in a static environment. Then you move on to the next thing. This is fundamentally incorrect. Software engineering is a process that takes place in an adversarial, human-driven environment. There is fundamentally no automating this away because human decision-making work is required.

Any security expert worth their salt will tell you this. As long as you persist in trying to view software as a fixed artifact in a static environment, you will find there is no meaningful answer. Those who come to terms with the dynamic, human-driven reality will find that there's a well-understood - if inconvenient and expensive - answer.

So then what do you do with software that doesn't meet this standard? Software that works, that meets a need you have, but that is a fixed artifact, or at least will be in the foreseeable future. Do you refuse to use said software, and instead choose a less useful alternative that is actively developed?

I believe the point is that no software is a fixed artifact. Regardless of the state of its development, it's all reactive to factors outside of its control, whether that's other software, users, hardware, etc.

That is solved with standard engineering practices.

If you need to make guarantees about your product that are dependent on another piece of software, then you just have your vendor produce a spec and guarantee conformance to the spec, then you audit that the spec fulfills your fixed needs and audit that the implementation conforms to the spec.

Then, if you made a mistake, your liability is limited to your mistakes instead of your mistakes and the mistakes of your dependencies.

If, on the other hand, you do not get any guarantees from your dependencies, and you fail to achieve your guarantees then that is entirely on you as you are the one transforming the absence of guarantees into guarantees.

In fact yes. I‘d choose a maintained lesser library or an in-house implementation over an unmaintained one. This is also the main benefit of python, with its huge standard lib that effectively gives some maintenance guarantees.

To give you an example: XLSX parsing. For quite a while, the most performant library ‚xlrd‘ had a big scary unmaintained notice on GitHub, yet people kept recommending and using it, often unnoticed through pandas, because they don’t read docs. I‘d refuse to touch that and instead use openpyxl, and now newer xlrd releases have even completely removed xlsx support.

That's a risk management problem. For how long will the world around this fixed artifact remain constant? How far away is your "foreseeable future" horizon? What happens when you're wrong? I have, in fact, chosen to use actively developed and maintained systems with fewer features than unmaintained EOL'd ones.

When you choose software to solve a problem, you're not just choosing software. You're encoding expectations around the problem, the environment, and your needs. If all three of those can be assured to never change then choosing a fixed artifact might be the way forward. Should any of those ever change, then your software will either adapt or become a less suitable solution.

How often can an organization really be that certain?

That's exactly right - it's about risk. No one has "perfect" operations/ security. They have a threat model and a risk tolerance.

We see this all the time in healthcare. MRI imaging machine that cost 500k 20 years ago still works fine but only on Windows XP. Cost to rewrite the software for windows 10 is more than 500k so rather than paying for that instead they firewall the shit out of it so it can only talk to some printing / output device.

That’s just one example. Medical machines are expensive as hell and companies don’t want to rebuy or constantly invest in something that works and does one thing well.

I was visiting my parents this summer and they have a Gameboy from 1991 with a Tetris cartridge, which was authored in 1989. They use it regularly and it still works! For over 30 years this device has played Tetris without requiring any software updates.

Not familiar with such kind of software, but if it were written for, say, Linux, would it need such an expensive rewrite to work with current versions?

I still run software I wrote for Linux in 1998. It runs in a LXC container with all the old libraries and it must run in an Xvnc session with a window manager of the era because there is a Qt1 widget that opens a kind of drop down menu that just doesn't work with any modern window managers.

I probably wouldn't bother to try to port it to modern versions of Qt or whatnot even if I hadn't lost the source code 15y ago.

So yes, it happens, it's possible, it sucks

that gameboy cartridge isn't working on a modern system either...

Not to split hairs, but I think what you've described is constant investment in something that works and does one thing well. The firewalls and management and maintenance are all ongoing costs to manage the risk.

Software is never just an artifact. It's a whole system, of which the source code and binaries are some components. If you do not manage those artfacts, the next best option is generally managing the environment.

We actually are running into this exact problem with running production ML workloads and despite anyone's claims it's not a solved problem. In fact, this log4j mess is a cake walk in comparison.

The issue: ML workloads are very sensitive to the exact version of a framework/library used and these frameworks/libraries are not stable despite what their semantic version says.

So what do you do when you have a python vulnerability? Best case, an update "just works" but in many cases you either need to retrain the model (can be extremely expensive or inconvenient), rewrite it if APIs changed, or hunt down a difference in model output if some downstream dependency broke something. Stability of software in the python world is abysmal.

Is this an issue of code stability or do these libraries have leaky abstractions that cause issues with models trained on previous version, because the library implementation changed and since ML has low explainability, it's hard to tell how the change impacts the model performance? eg. making a library run faster or use a slightly different parameter internally causes some floating point imprecision that cascades into wildly different results?

Unfortunately it's all of the above. Sometimes the code isn't stable (eg; a library method changed), sometimes the behavior has changed (eg; same library method but now it has a different behavior), and sometimes it has some other unwanted effect (the computation produces a totally different result, it runs way slower, or something else).

With a service it's all much easier because you can have pretty robust integration tests and you're good to go. With an ML model it's a lot trickier because even if you have sample payloads and responses there might be other payloads you didn't think about that could cause issues. Or it might rear its ugly head when you finally retrain the model and suddenly model performance is crap.

There's also the issue that subtle bugs are a lot harder to find in ML since often the model just adapts (although potentially with worse performance). This then breaks models when the framework updates.

One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues.

I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.

I find Python is very prone to a lot of breaking changes even with minor version changes. And not just that, even minor versions have to match up specifically with another libraries minor version. It seems to be a community or ecosystem problem with Python, which is why I avoid Python as much as possible.

In every other ecosystem when minor version are updated there are very rarely issues, and there are little issues with compatibility with minor versions of libraries.

eh, people love to break stuff on minor version changes of Go libraries to

At least you find out at compile time...

My (somewhat dated) experience says that Ruby is all the way worse than Python on this.

In part due to a culture that encouraged monkeypatching. With the result that what behavior you get depends on who last monkeypatched it. So a dependency 3 deep that loads the module you expected to load last suddenly causes the monkeypatch you were depending on to disappear...

Monkey patching was the greatest thing ever. Until the community realized how bad it was. Don’t see it much these days

> So what do you do when you have a python vulnerability?

If you are willing to run other people's arbitrary (Python) code, then - vulnerability or no, you should consider the system running this code as compromised.

is this ML workload an "I trained a model, pickled it, and now we ship that around in production" type situation?

When I worked on a code signing app, which is arguably some of the highest stakes of almost anything I've worked on, we came around to an agreement that one ticket a month would be assigned to upgrade libraries, and we rotated that responsibility. We didn't stipulate what library, we didn't even stipulate which application in the suite (though it was assumed that you were likely to chose your primary application as the target), so long as something got upgraded.

This policy evolved, if memory serves, after a security issue was discovered with the XML library we were using. The fix was not back ported to our version, and the versions in between had multiple breaking changes, so it was a slog to fix it.

It wasn't even that we were having a production issue, because we were still in development and internal testing. Not a single 'real' signature had been generated yet. It was all test assets and test CA certs. But we had enough wisdom to put 2 and 2 together and get 4, so we tried to treat the situation as a dress rehearsal for some later bug that happened after we started playing for keeps. Almost everyone could see this was an untenable situation.

There were consequences of course, but we made a bit of lemonade in the process. Our integration tests were expanded to include a larger percentage of pinning tests for our dependencies. Given the gravity of the situation, we needed those anyway. We just now had a poster child for doing the work.

> we rotated that responsibility

how did that go ? I often wanted to do it but never got to submit the idea.

A couple didn't get it, a few chose things we might not have picked, but that was fine because sooner or later having 'dumb' out of date dependencies adds up to real problems.

It also took some reminders from the leads to add it in instead of letting it slip, but so far it's the least broken process I've gotten to use.

>"Use a build tool to do dependency checks!" > >And then what? > >"Only update libraries if they have vulnerabilities." > >So, should this be a manual process where I have to dig through obscure warnings every time I build something? > >"No, you can automate it!" > > Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build process reliant on yet another online service.

Sorry, this is a total strawman. Automation doesn't need to be integrated with the build in such a way that it creates noise, and it doesn't mean you need to let changes be applied without human intervention.

I'm hooked up with Snyk which scans all of my dependencies and sends me a summary email if it discovers any vulnerabilities. It creates PRs that can patch dependency vulnerabilities, and I can apply them if I approve of the change. If I don't think the change is worth it or the vulnerability is relevant, I can go to the Snyk site and mark a vulnerability to be ignored, or even tell it to ignore it for a month or whatever. I'm sure Snyk isn't unique, either.

Yes, a human should be involved in decisions. No, it doesn't need to be noisy.

I'm on Node.js and working on multiple projects with crazy numbers of dependencies, and the number of "new vulnerabilities" I see is typically a few per month at most. Some months there are no new notifications at all. It simply doesn't change that quickly, and it's not constant noise unless you ignore the notifications, and that's just bad software engineering.

This is basically what Dependabot on GitHub does too. If it finds a vulnerability it creates a PR that you can decide to apply. It's automated to the extent that it's easy but it's not making or forcing changes on you. A human is still in control.

True enough, though Snyk will sometimes offer a patch rather than a PR to a new version of the dependency.

The Snyk tools apply to the patch directly to (e.g.) `node_modules` after the latest code has been downloaded.

I think people's experiences will vary with tools like this due to company policy. For instance, your company allows you to make decisions about whether a vulnerability pertains to you, but many don't. I think that's how these tools become "noise", because the only option an engineer is given by leadership is to comply to the tool.

At this point, I think a lot of problems in software are actually misaligned or misincentivized policy gone awry. When you read about them on forums they look technical, but they're not.

You can't use a technical solution to fix a management problem, no. :)

I don't see how you have really fixed the break much. Especially if you can "push back" and say no to an update, you are just setting yourself up for when the update is even harder to accept. (That is, the further behind on an update you are, the harder to take it. Typically.)

And then there is the "I want the fix, but I don't want the added attack vector of features that comes with it."

Sure, but you're in control, and can decide to do what's best for you and your organization. Taking on the technical debt of allowing your dependencies to get further and further behind isn't a decision I usually make, but it can be a valid one depending on the individual circumstances.

I mean, you aren't wrong. But this is akin to saying you don't need a memory safe language, because you can decide what is best for your applications memory needs.

That is, I think the ask is more of "why haven't we come up with something that is a bit better at mitigating risks, here?"

Software engineering is complex. Some of the challenges are simply irreducible.

I believe this is one of them.

99% of the time it's a minor update patch that you really should just apply, or Snyk will add a patch to the build to fix the security hole directly.

Honestly it's just not worth worrying about whether a security hole is exploitable when the decision is whether to accept a minor update patch. Just accept the patch PR and move on.

The times you need to do a major update to get a patch fix it's equally important to just make that upgrade, since that means the ancient version you're using is likely out of maintenance and likely is throwing warnings that you're using a deprecated version.

Yes, it's more work to perform a major upgrade. But that doesn't mean it isn't good engineering to keep your tools up to date.

> If I don't think the change is worth it or the vulnerability is relevant, I can go to the Snyk site and mark a vulnerability to be ignored, or even tell it to ignore it for a month or whatever.

I am currently fighting with our security team because they refuse to allow developers to do this, and instead require my team to file tickets with their team, wait hours for that ticket to be picked up, explain why a thing is a false positive to a person who doesn't understand the codebase they're working on, and hope that person feels that the finding should be ignored.

I do not work at a large company.

This is greatly exacerbated by the fact that Snyk - like all package scanners - primarily flags false positives. When you start digging in to what it flags you'll find that a lot of the vulnerabilities aren't reasonably exploitable, or the impact is questionable, or you can't repro the issue. Those that do legitimately affect the underlying library often don't affect your usage of that library. I'd estimate our false positive rate is >90%. So when we get a finding our options are to go through that costly and disruptive process I mentioned above, or just upgrade the dependency (which carries associated risk of breaking shit in production).

These tools emphasize the quantity of findings they produce, and quietly ignore the quality. As soon as you have any divergence in motivations or incentives between the people running the scanner and the people being blocked by the scanner those low quality findings become incredibly costly.

This is compliance vs security. Finding vulns checks a box for SOC2, but in reality detection is the easy part. Figuring out what to fix, based on real-world usage and risk, requires much more work and is often ignored.

I'm sorry you're on the receiving end of this problem!

Shill notice: I'm working on an Open Source tool[0] that makes this problem less horrible. My colleague wrote a post about our hypothesis[1] about how we can avoid this false positive trap.

I'd love to chat with anybody feeling this pain (even just as therapy lol).

0: https://github.com/lunasec-io/lunasec

1: https://www.lunasec.io/docs/blog/the-issue-with-vuln-scanner...

Lets not forget that promotions are for those who build new stuff and those who maintain old stuff don't get promoted.

To add further insult to injury, colleges send SWE/SDEs to the work force to learn to 'actually code', while companies have no time to train jr developers so they are just thrown in the fire and hope for the best.

At most companies I've worked for, mentoring and training junior engineers is a requirement for promotion. Also, you might not get promoted for maintenance but

1. keeping it running with a healthy userbase and demonstrating high impact does, and

2. if you know how to spin it, then maintenance work can look like "building new stuff".

> At most companies I've worked for, mentoring and training junior engineers is a requirement for promotion.

You've worked for some great orgs, then! My experience is mostly the opposite. One company I worked for did value mentorship, at least somewhat, and it was a factor in promotion decisions, but not mentoring others didn't really stop people from getting promoted.

I think this is more of a "do what I say, and not what I do" thing. Nominally most companies want you to mentor junior devs, but the worker who spends less time on that, and more time on higher visibility work gets the promotion priority.

I have a very simple solution, in three parts:

1) We use very few dependencies. We err on the side of writing our own function, library, or UI component rather than installing a new dependency, even if a "better" open-source version might theoretically be available.

2) We use a monorepo. We have two Go backends sharing the same go.mod, and three Typescript/React frontends sharing the same package.json, all in the same repo. An upgrade for one is an upgrade for all.

3) Every first Friday, I update all dependencies to the latest version. Including major versions. Yes this is a pain sometimes (like when I realized we were on Webpack 4 and the latest was Webpack 5), but if you do it consistently, it's usually not that much trouble. 1-2 hours per month on average.

In terms of security, I think this is a good middle ground. The approach has other benefits than security, too.

>We use very few dependencies.

While this may solve the "exploit from 3rd party dependencies" this says nothing about the security quality of your functions. Now instead of the vuln being found in the 3rd party package and fixed, the issue remains in your program forever, probably getting exploited by nation state level actors without your knowledge.

A benefit of writing own functions - you'll not implement something you don't need, which reduces attack surface.

Until your developer decides to write their own encryption routines.

> 1) We use very few dependencies. We err on the side of writing our own function, library, or UI component rather than installing a new dependency, even if a "better" open-source version might theoretically be available.

This just means you likely have a lower quality version with the same problems but without the benefit of an ecosystem to find them.

It doesn't mean that at all. When we need one feature we write it, rather than exposing the 1000 features of an open-source library to the internet when we only need one feature.

We won't write it ourselves if we aren't sure we can do it well, i.e. we're not going to roll our own encryption. Those are the types of dependencies we do use, and upgrade once a month like clockwork.

The point is not "don't use someone else's encryption library," the point is "don't use someone else's LeftPad() function that takes 3 lines to write". That gives us fewer dependencies, which makes it easier to upgrade all dependencies regularly, which is good for security.

Yup, you’re outsourcing your decisions in exchange for not doing the work. You have to understand your risks and needs.

Your boring business systems that will live for 50 years need to have boring system level libraries or stuff you maintain. Your fast time to market or non-critical systems are optimized by speed and cost.

If your service depends on a bunch of downstream stuff, your processes need to support upgrade cycles. If you can’t keep up with that, you need to refactor.

There is pressure in our org to find ways to have fewer dependencies.

Some of this is trivial, and a very good thing - stop using bullshit exercises in fragmentation like isEven.

Some of it is very much not, and will apply pressure to find vendors who are willing to vet their own bundles of commonly used libs.

There will be numerous second order effects of that, including the ghettoization of open source that isn't "QualStrike Approved(tm)".

System level libraries are a huge risk. It makes updating much harder leading to many developers just foregoing updates unless absolutely necessary. Libraries your software uses should be decoupled as much as possible from system level libs.

Big-tech has solved this internally. Everything down to the linux kernel they're running in prod is versioned and has people maintaining it and interacting with upstream (if any). This is obviously a full time job for at least a person per dependency on average.

>The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.

Capability-based security is one possible answer. The Austral language is trying to make this a first-class language feature:

>The problem is that code is overwhelmingly permissionless. Or, rather: all code has uniform root permissions. The size of today’s software ecosystems has introduced a new category of security vulnerability: the supply chain attack. An attacker adds malware to an innocent library used transitively by millions. It is downloaded and run, with the user’s permissions, on the computers of hundreds of thousands of programmers, and afterwards, on application servers.

>The solution is capability-based security. Code should be permissioned. To access the console, or the filesystem, or the network, libraries should require the capability to do so. Then it is evident, from function signatures, what each library is able to do, and what level of auditing is required.

https://austral.github.io/spec/rationale-capabilities

What? Are you crazy? Talking about security to developers! How dare you?

There are a few issues here...

1. Depending upon context, *any* change to state (network I/O, console I/O, file I/O, environment variables, etc.) is a security vulnerability you can drive a truck through. For example: Foo is a very-locked down package that only appends text to a file the caller owns. It's formally verified to only change the file specified, verified to only append in all possible situtations, verified to not have cross-file-system identity problems, not subject to file renaming race conditions, yada yada yada. Safe? No. "foo('evil_command' '~/.bash.rc')".

2. A useful definition of a "capability" depends upon context. A *dynamic* context. Say Foo is intended to be used for generating log files. There's no way for a third-party library to determine whether the target is actually such a thing. It requires manual tuning which fits the operational context. Perhaps the file must be named "/var/log/${x}.log". Or maybe its "${home}/local/log/${x}.log". Great. Of course, now you've got to verify the context-specific defintions. And verify the tool that does this verification. And the verify the configuration of the tool that does the verification... Safe now? No. Context is *dynamic*. "foo('var/log/evil-hack-attempts.log' '0.0.0.0/0 tried-to-crash-us')", and your other context, the automated blacklisting, locks out the Internet.

3. This gets very complex very quickly. IMHO, configuring Selinux, in a real production environment with real personell, is akin to rolling your own cryptography. Can people learn to do it? Of course. But the same argument applies to writing cryptographic functions. You. Will. Make. Huge. Mistakes.

To be clear, this does not mean that capabilities are useless. It's obviously critical that they exist. However, they ain't magic. You can't "solve security with capabilities." There is a point where throwing more of them at the problem makes things *worse*.

because our industry is obsessed with abstractions - the moto is to never fix anything, just pave over it with another abstraction.

for example we didnt fix how we build applocations and manage dependancies, we invented docker.

we didn't create a secure runtime for the cloud where applications can run, instead we virtualised whole operating systems

> because our industry is obsessed with abstractions

"we do not break userland, period" -- Linus Torvalds

not breaking existing stuff is why many abstractions exist, and even why many are put in place early on, to allow for some wiggle room without having to break all the things. people tend to prefer software that continues to work, above all else.

but then someone builds a bank on top of this abstraction, and sorts of shit hits the fan.

except, it's turtles all the way down. is POSIX not an abstraction? TCP? C/gcc/llvm? glsl?

Working in the industry I do, there are a number of banks I will never used based on the terrifying lack of quality in the software they write. The SBOM on these projects could be printed out and bound as a dictionary there are so many packages included.

I think we should acknowledge, though, that not all abstractions are created equal. Abstractions that hide implementation details and allow people to change those implementation details without creating churn and extra work for their downstream consumers are good.

Abstractions that paper over problems that you are unwilling or undisciplined enough to tackle properly are usually bad. I think the grandparent was talking about this kind.

> Abstractions that paper over problems that you are unwilling or undisciplined enough to tackle properly are usually bad

some abstractions carry decades of legacy (e.g. linux file-system hierarchy), and cannot simply be replaced. things that did not connect to the non-existent internet did not have to paper over problems that simply could not exist at that time (VMs, multi-user, multi-tenant hardware). there is plenty of deep software dependencies that are no longer maintained which still rely on these abstractions and cannot be cheaply/reasonably replaced in the field. sure, we have new abstractions since then, but only new and actively maintained software can ever make use of them, and that will only end up in systems that are also actively maintained instead of simply continuing to work with exactly zero effort.

You are correct, but its not just abstractions, its specifically abstractions on a shitty platform.

The whole log4j thing happened is because java is very poorly designed from the get go, and one of those points is the insistence of making everything an object, including data. And of course, you often need to output the string representation of data.

From a theory perspective, if you want to rely on language features like strong typing to ensure correctness, then you have to take it all the way to full provability, where you explicitly guarantee bounded input, bounded output, and any side effect (where a network call is a side effect, and if your code doesn't explicitly state that this side effect will happen, it won't run).

There is likely never going to be an approach that yields 0 vulnerabilities, so instead we should be more focused on minimizing the risk as much as possible. There is a middle ground and nuance between never update and always update, just as there is a nuance in how much and what you duck tape together.

Not sure if it's the best, but my current approach is to target major revisions/versions with ABI/API compatibility as viable candidates, and update always, but at a lagged rate, ideally allowing enough time to pass for vulnerabilities to be surfaced in those versions. Log4j is a tough case due to how far back it goes version-wise, but you simply can't win them all.

Another important way to minimize risk is to have a strong ability to provide a corrective action. If replacing the version or the entire dependency of a library you use takes a significant engineering effort, then you are arguably at a higher risk due to that than you are in your decision to never vs. always update.

I still think that betteer ABI/API stability & control _is_ the ideally better solution, but even if I'm a fan of plain old C / sonames dependency management, I'll readily admit that very few people actually do any type of API promises these days, much less API.

ABI/API stability != behavioral stability, though. I can give you a dependency with exactly the same interface, that will still break your software. It's not a solution (even if it helps with one piece of the puzzle)

The only complete solution would be static analysis (read: type systems) that can guarantee everything about some code's contract/behavior relative to the caller. Short of that, it's just going to continue being a bunch of manual half-solutions like API checks, existing type-level checks, manual changelog reading, manual upgrades, testing, etc.

I think a mindset that could work is to accept software as done. This is hard but for foundational stuff it could work well. Only fix bugs in such software. Try to set a scope what the software should do, build it, then do maintenance on it. Supersede it entirely to add new functionality.

That's my armchair take on it.

Adding features to existing software is much cheaper than writing entirely new software (not to mention the cost to switch!) At the same time, new features in low level software can unlock substantial value. Say a new API that speeds up your app by 1% or cuts down on further development time by a small fraction.

> Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party.

Aren't we already doing that, to some degree, by watching for notifications of vulnerabilities, and then manually patching or updating? Sure, with this method, we have the ability to decide to ignore a vulnerability, but I'd guess most people aren't great at assessing risk (and may not fully understand the severity of a vuln, or if their use of the software makes them exploitable), and should probably just update whenever a fix for a vulnerability comes out. So you might as well automate this.

> So the exact software you compile will depend on whether or not you can connect to a 3d party service?

Anyone who builds against public package registries (Maven Central, npm, pypi, crates.io, etc.) already has this problem. Some people will go to the effort of putting their own proxy cache in front of these registries to insulate themselves from downtime, and I expect these sorts of people would do the same for a 3rd-party vulnerability updater service.

Regardless, I wouldn't expect this to be a "pull" model, wherein you wait until the next time you'd build before getting security updates. More likely you would get a notification or even an automatically generated pull request (like GitHub's dependabot does) when a security update is available. Then it's up to you whether you want something else to automatically apply those updates, or you can review them yourself and apply manually.

I think you're just making a mountain out of a molehill. There are several existing options to automate or partially automate this, depending on your trust of the relevant third party. And you can still do it manually, if you so desire.

And what would you have anyone do?

You could formally spec software and implement it, congrats you're now 1000x slower to market and any change you want to make is another 1000x investment to spec out and prove.

And let's not even mention the ridiculous idea that open source developers be expected to do this. Moreover, this doesn't even handle hardware problems!

Okay, but we could just build all the software internally, reinvent all the wheels. But how is this any better? Now you have even less manpower to fix bugs and you're probably 100x slower to market because you're building all these half baked, bug ridden libraries.

If anything, the best idea is formally verified sandboxing, so that you have strong assurances about certain apps not doing bad stuff, but this doesn't solve all problems either.

It's an unsolvable problem in general, which is why no one has solved it.

It takes a lot longer to build houses that are up to building codes, and bridges that have 10x safety factors rather than 2x. There's a reason we have building codes and large safety factors. Software engineers are discovering this today.

Software engineers are mostly aware of this. There is just little market demand for this level of robustness.

there was no market demand for building codes, either. that's why they're laws

bbut but mah free market gave us fire escapes attached to every building in new york, and they were made of wood!

I know you're being sarcastic, but this reminded me of Tanya Reilly's _great_ talk "The history of fire escapes: How to fail". There's a recording up at https://www.infoq.com/presentations/history-fire-escapes-res...

Countries that pass laws making it 10x more difficult to develop software will not survive the next few decades.

Engineering Code of Ethics dictate that safety of public and environment is paramount. I know that's true for my P.Eng governing body; it is written into our bylaws. I'm sure its the same for most others as well.

I think until Software Engineering catches up in this regard with traditional Engineering disciplines we'll continue to see issues arise from the software realm. As much as the self-titled "Engineers" proclaim it, there's more to engineering than design and production.

I've always been uncomfortable calling myself a "software engineer" for this reason, and usually refer to myself as a "software developer". Most of the job listings I see at various companies use "engineer" though, and so my official title ends up being "engineer", so that's what I end up putting on my resume. (Though I guess I don't really need to do that; I could put whatever I want.)

I am a Software Engineer. I'm licensed by our provincial body and have my P.Eng. and yeah, I still agree with you.

I've been asked why I got my P.Eng as it does nothing for me career wise, but I always have the hope that we move towards a profession that does utilize engineers with proper accreditation and responsibilities where appropriate. I work in Industrial Controls and what my job has me doing does have safety concerns. We fired a student at one point working for us because his attitude towards the work was lackluster and lazy. I see it often enough in programs I'm asked to pick up and fix too.

If Software Engineering wants respect, the profession needs to accept that we're going to piss off a lot of Developers & Architects using the title incorrectly. We still need to develop the proper rigor to take Engineering as a discipline seriously, but we also don't need to apply "Engineers" to every job.

> We're fighting fragile complexity of too many tools ducktaped together by ducktaping more tools to the whole setup. Again and again and again.

You’ve brought up three relevant points: 1. The tools are software, which strength and weakness is being easy to change. 2. The tools rely on abstraction, which strength and weakness is summarizing/hiding complexity. 3. Communication between publishers/consumers of these tools is hard.

Armchair-coaching I now come up with at least three solutions: 1. Re the weakness of #3: ensure when an issue is found at one layer, communication happens to maintainers of all other layers up and down. 2. Re the weakness of #2: collapse the layers – analyze dependencies and don’t go past more than x number of abstractions. 3. Re the weakness of #1: don’t use software.

There is no free lunch - each line of code is a liability even if it comes form a dependency. You wrote a few lines of code which depends on 100k LoC library and now you are exposed to bugs and vulnerabilities which may exists in these 100k LoC, not only in your few lines.

If you using a library you have to spend time on keeping it up to date (which includes testing and fixing whatever when wrong during an update).

And there's the tangling of dependencies. One fix in one lib may cause a bug in another that isn't maintained.

Or you can end up with conflicting versions of transitive dependencies.

Or even worse you may have a legacy "fat jar" that you can't even manage the transitive dependencies on - they're compiled in, and that's that. Something else has a version conflict? Tough shit, one or the other is going to break, and depending on how bad your build is, you may not even get an explicit choice, it may be thrust upon you by the classloader.

Sometimes you can mitigate it somewhat by encapsulating them in their own service and just firewalling the hell out of it, and modules should help the dependency conflicts somewhat, but the problem is these are basically "UXO in the backyard garden", they are a passive business risk and sooner or later you may get lucky and they go boom even though you did nothing wrong.

Despite your best intentions, sometimes you gotta update dependencies, and if you allow it to fester, then you risk that the time you have to solve the problem is during an urgent security crisis. Generally it is better to update them on your own timetable than to have it forced upon you like that. And the problems and limitations and workarounds only become worse and worse over time if you choose to actively ignore it.

it's an interesting issue technically.. if only we didn't have to suffer its reality :)

Just patch on a cadence and assume you've got some vulnerabilities lying around. When there's something serious like log4j, pull the lever to update asap. I think you're overcomplicating this.

It'd be cool if devs didn't suck ass at writing code but that's the world we live in.

The ease with which software remains mutable after deployment is both a blessing and a curse. This is particular and intrinsic to software and other digital artifacts. Things based in matter do not benefit from this blessing or suffer from this curse.

"If you automatically update your dependencies all the time, you will constantly get new bugs, issues and sometimes even malware.

If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues."

>has no meaningful answer to this...a sane industry this realization would lead to a change of the paradigm

This sounds like gene mutation and evolutionary selection pressure. It's sort of a fundamental aspect of the universe no? There may not really be a meaningful "answer" to this in principle just various half-measures to muddle along and adapt to this reality, survival of the fittest in a sense.

>no matter what "security experts" tell you

Why the scare quotes?

>In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.

Maybe because half the time people with extensive training/experience in security start to even say something they are completely disregarded (like here), or are hamstrung by budget because security is viewed as a garbage disposal that profits get shoveled into, or because they are hamstrung by pushback over every little change because people seem to think the main goal of someone doing security is pissing off users.

I was wondering when it would become the norm to call it "ducktape". It has been 3 hours and nobody has corrected it. Last week I found a 3-pack of "ducktape" in the builders market.

I'm not making fun of you or trying to be funny - it is simply an observation that I find interesting.

Where I live there's a brand of duct tape called Duck Tape. It's terrible compared to the 3M brand tape, but it's cheap so it's popular.

Also nobody that knows what they're doing uses "duct tape" on ducts, so the name is pretty meaningless.

> So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this?

This is already true with the vast majority of software using any an online software repository, e.g. Go, NodeJS, Java, Rust, etc.

Not Go

Still Go. I mean there's no central repository in the way there's one for npm, and yes you can point it to any git repo as the source for your dependencies, but the reality is most are on GitHub. So your central repository is GitHub.

Can you give me an example of a popular Go project that doesn't have any dependency on GitHub?

Yeah but you can cache the packages or store them in some fashion. I've seen JAR files in git repos far too many times...

With security, you need some online service so you can find the new CVEs every day.

> With security, you need some online service so you can find the new CVEs every day.

Or you just do it manually if that service is down. It's weird to me that the argument for not having an automated, 3rd-party service is "if it goes down then you'll have to do things manually", when the alternative is "you always have to do it manually".

If you are comfortable trusting a third-party service to tell you when to upgrade, then that is absolutely an improvement over doing security updates manually. This is why I have unattended-upgrades set up on my Debian systems to automatically install updates from Debian Security every day. Sure, it may fail for whatever reason, but I am certainly not going to take the time to (or even remember to) update every day.

Yeah, honestly there's probably just a few libraries you're going to have to care about re: keeping up to date. Everything else can get updated opportunistically/ on some cadence. Your exposed attack surface for most software is pretty much your TLS library and network stack. The more mature you become the more of the attack surface you can try to track.

But basically if you just subscribe to a few projects' releases you can pretty easily get things pushed to you when it matters.

Oh I completely agree. I was referring to the parent saying that a third party is dictating what to build -- for security this is inevitable. For dependencies this can be solved by caching your .jars or whatever, but at some point you still always have a third party dictating what you're building.

Isn't that just software though? If you fix bugs in your code, you'll introduce new bugs. If you don't fix bugs, you'll have bugs.

Ok, so what is the solution then? To me there isn't one. It's like, how do you prevent all car accidents? The only way would be to ban driving.

>So, should this be a manual process where I have to dig through obscure warnings every time I build something?

Also known as doing your job.

Better idea: stop logging.

the issue is that this can pop up in any library, not just logging. It's about keeping your deps up to date (or not) by a "3d party" (assuming he mean 3rd party)

Yes, this can pop up in any library. But only because developers aren't taught "don't put remote code execution into your code". You'd think that would be something that someone would teach, but it doesn't really come up. Remember that log4j was vulnerable because of a feature - it all worked as designed.

It's not just logging. L4J is so extensible that people have used it for all kinds of things, way waaaaaayyyyy away from just logs. So disabling logs won't necessarily cut it.

I am no expert. I ended up indexing all the open source kruft I use to hold this ship of fools together, then verified that the Log4J pieces were definitely disabled with a bunch of monitoring while I tossed stuff at it. I did mention I am not an expert, right? I am sure there is a more Pro way to do this.

F around and find out basically. If you choose to build on the house of cards, thou repeath whaet thou soweth or something.

problem: food contaminated

solution: stop eating

solution: find uncontaminated food, or make it yourself

option 1 - we and farmers, as professionals are responsible for supplying stuff that is not dangerous and works as it should

Option 2 - which you are advocating - it's every man for himself and we decent into socity of warring tribes of hunter hatherer or subsitence farming at best

option 2 is the only possible state of reality, because it is never possible to guarantee that everything work in every instance

let the buyer be aware

I mean, how do you know your webserver doesn't have a bug? How do you know your OS doesn't? Just write those yourself? How do you know yours doesn't? One must still verify the key functionality regardless.

But, you're not wrong, some people reach for every library and framework they can, and others focus on building the most minimal and understandable thing that will work. I'm not saying writing your own server in assembly, but, avoid unnecessary junk and magic and libraries.

It's the classic dilemma: in this world of ours, one can either return to monke or progress to crab.

Ironically, logging is one of the ways to help mitigate/detect security vulnerabilities.

> The current software engineering paradigm has no meaningful answer to this

The answer is to stop using garbage software written in garbage languages. Of course the current environment is too high-time-preference for this to be economical in many industries.

We have lightweight-to-heavyweight formal systems which can almost completely or literally completely eliminate issues of the type seen with log4j, but due to the marginally higher up-front cost of using such systems, only firms with some combination of high stakes, low time preference, and foresight end up using them.

What counts as a garbage language? Java guys will tell you it's rust or go, go guys will tell you it's java or rust, rust guys will tell you it's java and go.

> What counts as a garbage language?

I assume it's one with GC.

Wouldn't it be one without gc? Gc collects the garbage and disposes of it, whereas in other languages it just piles up :D

I'm not any of those guys, and of those 3 rust is by far the least garbage. Garbagicity is a spectrum.

I was sure reading your first post that you will propose rust. Rust users never cease to amaze me.

To be fair there was also a significant chance he would think Rust was garbage and only assembly (or if you are lucky, C) are acceptable languages.

You still don't know how to spot them :). The borrow checker is a great invention, but for them it's the holy grail that will save us from bugs. Like in the past people thought that GC was the solution to every problems. Youngsters must learn, we have to be patient.

I'm not a "rust user" any more than I'm a java user (less, in fact). Your heuristic is probably picking up on the fact that people who pay attention to things like this tend to have correlated opinions.

Java person here. Yeah, Go looks a bit crufty, but they're gradually undoing the earlier opinionated choices that make me think that (Who needs generics? Who needs to tune the GC? Oh, our users do), but nah, we mainly hate on Scala when we have to maintain anything with Scala deps, just that damn ABI that's never compatible.

'software written in garbage languages'

well, back to the mainframe it is then

Quite the opposite, we have very safe languages now in 2022. Any software that's not written in Rust or Haskell has to explain the trade-off of being written in an unsafe language. Of course the industry will still use unsafe languages, but we lack the awareness that we're intentionally writing garbage software in garbage languages that should be treated as radioactive material in any context where safety matters. If you pick a language without understanding this point, you're part of the problem. If you pick an unsafe language understanding that X matters more than safety, then you saw all of this coming and you made an educated gamble.

We've had safe lnaguages for decades - for example Ada

https://www.adacore.com/about-ada

These Log4j issues aren't real hardcore exploits. They wrote a logger that could run arbitrary code, then were surprised when attackers figured out how to run arbitrary code.

It isn't a language problem, it is a developer practices problem. Developers code like security isn't a thing then are confused when their programs aren't secure.

  It isn't a language problem

Exactly. This flamy sub-thread has no point here.

> It isn't a language problem, it is a developer practices problem. Developers code like security isn't a thing then are confused when their programs aren't secure.

I'd prefer that the "fix" for this isn't "require that millions of developers around the world all change their behavior, and never screw up in the future".

Languages can still protect against these sorts of bugs. Effect systems can describe as a part of the type system or function signature what kinds of things a function can do. For example, you could have a logging function where you know that the only side-effect is that it can write to stdout or stderr, or a file. Running arbitrary commands would not be allowed, and would cause the program to fail to compile.

As a real-world example, I've used the Slick SQL library in Scala, and the query type has a type parameter for whether the query is an read or a write. It's easy to build a simple abstraction on top of your data layer to never allow a write to go through, for example, if that's something that's useful to prevent bugs. I think one thing I did with that was set things up so callers could not accidentally write to MySQL replicas; only reads could go to them, and writes had to go to the primaries.

Granted, it takes discipline to build and use such a system, especially one that has such fine-grained effects. (I would expect most are more like a binary "does not do I/O" or "does I/O".)

The issue is like SQL Injection. Much like SQL Injection protection the logger needs to separate "active" strings vs attacker controlled "data" strings. It is pretty easy to design boring Java types that solve the issue. The only problem is they didn't. Unless this effects system somehow forces them to not do something dumb, we will end up with a logger that has effects set to "does everything under the sun", and a similar bug.

In what fashion does Rust prevent you from calling valid library code that has unexpected behavior?

Reply on news.ycombinator.com