Hacker News Reader

Comment by dsff3f3f3f | Hacker News Reader

> These kinds of dependencies are everywhere and nobody would even think that they could be harmful.

Tons of people think these kind of micro dependencies are harmful and many of them have been saying it for years.

I'm rather convinced that the next major language-feature wave will be permissions for libraries. It's painfully clear that we're well past the point where it's needed.

I didn't think it'll make things perfect, not by a long shot. But it can make the exploits a lot harder to pull off.

Java went down that road with the applet sandboxing. They thought that this would go well because the JVM can be a perfect gatekeeper on the code that gets to run and can see and stop all calls to forbidden methods.

It didn't go well. The JVM did it's part well, but they couldn't harden the library APIs. They ended up playing whack-a-mole with a steady stream of library bugs in privileged parts of the system libraries that allowed for sandbox escapes.

It was too complex. Just making system calls require white listing libraries goes a long way of preventing a whole class of exploits.

There’s no reason a color parser, or a date library should require network or file system access.

The simplest approach to whitelisting libraries won't work, since the malicious color parser can just call the whitelisted library.

A different idea: Special stack frames such that while that frame is on the stack, certain syscalls are prohibited. These "sandbox frames" could be enabled by default for most library calls, or even used by developers to handle untrusted user input.

Yes, but that was with a very ambitious sandbox that included full GUI access. Sandboxing a pure data transformation utility like something that strips ANSI escape codes would have been much easier for it.

Totally agreed, and I'm surprised this idea hasn't become more mainstream yet.

If a package wants to access the filesystem, shell, OS API's, sockets, etc., those should be permissions you have to explicitly grant in your code.

It's harder than it looks. I wrote an essay exploring why here:

https://blog.plan99.net/why-not-capability-languages-a8e6cbd...

Thanks, it's great to see all the issues you raise.

On the other hand, it seems about as hard as I was imagining. I take for granted that it has to be a new language -- you obviously can't add it on top of Python, for example. And obviously it isn't compatible with things like global monkeypatching.

But if a language's built-in functions are built around the idea from the ground up, it seems entirely feasible. Particularly if you make the limits entirely around permissions around data communication -- with disk, sockets, APIs, hardware like webcams and microphones, and "god" permissions like shell or exec commands -- and not about trying to merely constrain resource usage around things like CPU, memory, etc.

If a package is blowing up your memory or CPU, you'll catch it quickly and usually the worst it can do is make your service unavailable. The risk to focus on should be exclusively data access+exfiltration and external data modification, as far as I can tell. A package shouldn't be able to wipe your user folder or post program data to a URL at all unless you give it permission. Which means no filesystem or network calls, no shell access, no linked programs in other languages, etc.

tbh none of that sounds particularly bad, nor do I think capabilities are necessary (but obviously useful).

we could literally just take Go and categorize on "imports risky package" and we'd have a better situation than we have now, and it would encourage library design that isolates those risky accesses so people don't worry about them being used. even that much should have been table stakes over a decade ago.

and like:

>No language has such an object or such interfaces in its standard library, and in fact “god objects” are viewed as violating good object oriented design.

sure they do. that's dependency injection, and you'd probably delegate it to a dependency injector (your god object) that resolves permissions. plus go already has an object for it that's passed almost everywhere: context.

perfect isn't necessary. what we have now very nearly everywhere is the most extreme example of "yolo", almost anything would be an improvement.

Yes, dependency injection can help although injectors don't have any understanding of whether an object really needs a dependency. But that's not a god object in the sense it's normally meant. For one, it's injecting different objects :)

to be clear, I mean that the DI container/whatever is "the god object" - it holds essentially every dependency and every piece of your own code, knows how to construct every single one, and knows what everything needs. it's the biggest and most complicatedly-intertwined thing in pretty much any application, and it works so well that people forget it exists or how it works, and carrying permission-objects through that on a library level would be literally trivial because all of them already do everything needed.

hence: doesn't sound too bad

"truly needs": currently, yes. but that seems like a fairly easy thing to address with library packaging systems and a language that supports that. static analysis and language design to support it can cover a lot (e.g. go is limited enough that you can handle some just from scanning imports), and "you can ask for something you don't use, it just means people are less likely to use your library" for the exceptions is hardly a problem compared to our current "you already have every permission and nobody knows it".

Yes, I do agree that integration with DI is one way to make progress on this problem that hasn't been tried before.

Thanks, this was a good overview of some of the challenges involved with designing a capability language.

I think I need to read up more on how to deal with (avoiding) changes to your public APIs when doing dependency injection, because that seems like basically what you're doing in a capability-based module system. I feel like there has to be some way to make such a system more ergonomic and make the common case of e.g. "I just want to give this thing the ability to make any HTTP request" easy, while still allowing for flexibility if you want to lock that down more.

In Java DI you can add dependencies without changing your public API using field injection. But really there needs to be a language with integrated DI. A lot of the pain of using DI comes from the way it's been strapped on the side.

This was one of Doug Crockford's big bugaboos since The Good Parts and JSLint and Yahoo days—the idea that lexical scope aka closures give you an unprecedented ability to actually control I/O because you can say

    function main(io) {
        const result = somethingThatRequiresHttp(io.fetch);
        // ...
    }

and as long as you don't put I/O in global scope (i.e. window.fetch) but do an injection into the main entrypoint, that entrypoint gets to control what everyone else can do. I could for example do

    function main(io) {
      const result = something(readonlyFetch(onlyOurAPI(io.fetch))
    }
    function onlyOurAPI(fetch) {
      return (...args) => {
        const test = /^https:\/\/api.mydomain.example\//.exec(args[0]);
        if (test == null) {
          throw new ValueError("must only communicate with our API");
        }
        return fetch(..args);
      }
    }
    function readonlyFetch(fetch) { /* similar but allowlist only GET/HEAD methods */ }

I vaguely remember him being really passionate about "JavaScript lets you do this, we should all program in JavaScript" at the time... these days he's much more likely to say "JavaScript doesn't have any way to force you to do this and close off all the exploits from the now-leaked global scope, we should never program in JavaScript."

Shoutout to Ryan Dahl and Deno, where you write `#!/usr/bin/env deno --allow-net=api.mydomain.example` at the start of your shell script to accomplish something similar.

In my amateur programming-conlang hobby that will probably never produce anything joyful to anyone other than me, one of those programming languages has a notion of sending messages to "message-spaces" and I shamelessly steal Doug's idea -- message-spaces have handles that you can use to communicate with them, your I/O is a message sent to your main m-space containing a bunch of handles, you can then pattern-match on that message and make a new handle for a new m-space, provisioned with a pattern-matcher that only listens for, say, HTTP GET/HEAD events directed at the API, and forwards only those to the I/O handle. So then when I give this new handle to someone, they have no way of knowing that it's not fully I/O capable, requests they make to the not-API just sit there blackholed until you get an alert "there are too many unread messages in this m-space" and peek in to see why.

This exact idea has already been mainstream. Both Java and .NET used to have mechanisms like that, e.g.: https://en.wikipedia.org/wiki/Code_Access_Security

"it exists as a niche feature that few use and fewer understand" isn't exactly "mainstream" IMO (it's significantly less common from what I've seen than manual classloader shenanigans, for example). But yes, it's nice that it exists, and I wish it were used more - it'd catch low-effort stuff like this one was.

No, C# had it: past tense. CAS was neutered in .NET Framework 4.0 then removed in dotnet core.

alas. don't suppose you know of any good articles on why it's removed? I'd be curious about the reasoning / challenges.

there are some rather obvious challenges, but a huge amount of the ones I've run across end up looking mostly like "it's hard to add to an existing language" which is extremely understandable, but hardly a blocker for new ones.

I don't know if there were any articles specifically detailing it, but from blog posts at the time the clear message was that they didn't consider the intended security guarantees to be possible to uphold in practice, so much so that "CAS and appdomains shouldn't be considered a security boundary".

Alternatively, I've long been wondering if automatic package management may have been a mistake. Its primary purpose seems to be to enable this kind of proliferation of micro-dependencies by effectively sweeping the management of these sprawling dependency graphs under the carpet. But the upshot of that is, most changes to your dependency graph, and by extension your primary vector for supply chain attacks, becomes something you're no longer really looking at.

Versus, when I've worked at places that eschew automatic dependency management, yes, there is some extra work associated with manually managing them. But it's honestly not that much. And in some ways it becomes a boon for maintainability because it encourages keeping your dependency graph pruned. That, in turn, reduces exposure to third-party software vulnerabilities and toil associated with responding to them.

Manual dependency management without a package manager does not lead people to do more auditing.

And at least with a standardized package manager, the packages are in a standard format that makes them easier to analyze, audit, etc.

yea, just look at the state of many C projects. it's rather clearly worse in practice in aggregate.

should it be higher friction than npm? probably yes. a permissions system would inherently add a bit (leftpad includes 27 libraries which require permissions "internet" and "sudo", add? [y/N]) which would help a bit I think.

but I'm personally more optimistic about structured code and review signing, e.g. like cargo-crev: https://web.crev.dev/rust-reviews/ . there could be a market around "X group reviewed it and said it's fine", instead of the absolute chaos we have now outside of conservative linux distro packagers. there's practically no sharing of "lgtm" / "omfg no" knowledge at the moment, everyone has to do it themselves all the time and not miss anything or suffer the pain, and/or hope they can get the package manager hosts' attention fast enough.

C has a lot of characteristics beyond simple lack of a standard automatic package manager that complicate the situation.

The more interesting comparison to me is, for example, my experience on C# projects that do and do not use NuGet. Or even the overall C# ecosystem before and after NuGet got popular. Because then you're getting closer to just comparing life with and without a package manager, without all the extra confounding variables from differing language capabilities, business domains, development cultures, etc.

when I was doing C# pre-nuget we had an utterly absurd amount of libraries that nobody had checked and nobody ever upgraded. so... yeah I think it applies there too, at least from my experience.

I do agree that C is an especially-bad case for additional reasons though, yeah.

Gotcha. When I was, we actively curated our dependencies and maintaining them was a regularly scheduled task that one team member in particular was in charge of making sure got done.

most teams I've been around have zero or one person who handles that (because they're passionate) (this is usually me) - tbh I think that's probably the majority case.

exceptions totally exist, I've seen them too. I just don't think they're enough to move the median away from "total chaotic garbage" regardless of the system

This is why I secretly hate the term software engineer. "Software tinker" would be more appropriate.

ha, I like that one - it evokes the right mental image.

Well, consider that a lot of these functions that were exploited are simple things. We use a library to spare ourselves the drugdery of rewriting them, but now that we have AI, what's it to me if I end up with my own string-colouring functions for output in some file under my own control, vs. bringing in an external dependency that puts me on a permanent upgrade treadmill and opens the risk to supply chain attacks?

Leftpad as a library? Let it all burn down; but then, it's Javascript, it's always been on fire.

> but now that we have AI, what's it to me if I end up with my own string-colouring functions for output in some file under my own control

Before AI code generation, we would have called that copy-and-paste, and a code smell compared to proper reuse of a library. It's not any better with AI. That's still code you'd have to maintain, and debug. And duplicated effort from all the other code doing the same thing, and not de-duplicated across the numerous libraries in a dependency tree or on a system, and not benefiting from multiple people collaborating on a common API, and not benefiting from skill transfer across projects...

> a code smell

Smells are changing, friend. Now, when I see a program with 20000 library dependencies that I have to feed into a SAST and SCA system and continually point-version-bump and rebuild, it smells a hell of a lot worse to me than something self-contained.

At this point, I feel like I can protect the latter from being exploited better than the former.

> At this point, I feel like I can protect the latter from being exploited better than the former.

I expect that your future CVEs will say otherwise. People outside your organization have seen those library dependencies, and can update them when they discover bugs or security issues, and you can automatically audit a codebase to make sure it's using a secure version of each dependency.

Bespoke AI-generated code will have bespoke bugs and bespoke security issues.

Unpopular opinion these days, but: It should be painful to pull in a dependency. It should require work. It should require scrutiny, and deep understanding of the code you're pulling in. Adding a dependency is such an important decision that can have far reaching effects over your code: performance, security, privacy, quality/defects. You shouldn't be able to casually do it with a single command line.

For better or worse it is often less work to create a dependency than to maintain it over its lifetime. Improvements in maintenance also ease creation of new dependencies.

I wouldn’t go for painful that much. The main issue is transitive dependencies. The tree can be several layer deep.

In the C world, anything that is not direct is often a very stable library and can be brought in as a peer deps. Breaking changes happen less and you can resolve the tree manually.

In NPM, there are so many little packages that even renowned packages choose to rely one for no obvious reason. It’s a severe lack of discipline.

yup, here is node's docs for it (WIP): https://nodejs.org/api/permissions.html

Yeah, there's an entire community dedicated to cleaning up the js ecosystem.

https://e18e.dev/

Micro-dependencies are not the only thing that went wrong here, but hopefully this is a wakeup call to do some cleaning.

Discord server? Is it that much work to create a forum or a mailing list with anonymous access. Especially with a community you can vet that easily?

It wouldn't be a problem if there wasn't a culture of "just upgrade everything all the time" in the javascript ecosystem. We generally don't have this problem with Java libraries, because people pick versions and don't upgrade unless there's good reason.

From maintenance perspective both never and always seem like extremes though.

Upgrading when falling off the train is serious drawback on moving fast..

Maybe we need two upgrade paths: An expedited auto-upgrade path which requires multi-key signoff from various trusted developers, and a standard upgrade path which is low-pressure.

and then you get Log4Shell

Yes. It is a bit painful this is not rather obvious by now. But I do have, every code review, whine about people who just include trivial outdated one function npms :(

Working for a bank did make me think much more about all the vulnerabilities that can go into certain tools. The company has a lot of bureaucracy to prevent installing anything or adding external dependencies.

Working for a fintech and being responsible for the software made me very wary of dependencies and weeding out the deprecated and EOL'd stuff that had somehow already found its way into what was a young project when I joined. Left unrestrained, developers will add anything if it resolves their immediate needs like you could probably spread malware very well just by writing a fake-blog advocating a malicious module to solve certain scenarios.

> Left unrestrained, developers will add anything if it resolves their immediate needs

Absolutely. A lot of developers work on a large Enterprise app for years and then scoot off to a different project or company.

What's not fun is being the poor Ops staff that have to deal with supporting the library dependencies, JVM upgrades, etc for decades after.

I've nixed javascript in the backend in several places, partly because of the weird culture around dependencies. Having to audit that for compliance, or keeping it actually secure, is a nightmare.

Nixing javascript in the frontend is a harder sell, sadly

What did you switch to instead? I used to be a C# dev, and have done my fair share of Go. Both of those have decent enough standard libraries that I never found myself with a large 3rd party dependency tree.

Ruby, Python, and Clojure, though? They weren’t any better than my npm projects, being roughly the same order of magnitude. Same seems to be true for Rust.

You can get pretty far in python without a lot of dependencies, and the dependencies you do need tend to be more substantial blocks of functionality. Much easier to keep the tree small than npm.

Same with Java, if you avoid springboot and similar everything frameworks, which admittedly is a bit of an uphill battle given the state of java developers.

You can of course also keep dependencies small in javascript, but it's a very uphill fight where you'll have just a few options and most people you hire are used to including a library (that includes 10 libraries) to not have to so something like `if (x % 2 == 1)`

Just started with golang... the language is a bit annoying but the dependency culture seems OK

Throwback to leftpad!

Hey that was also on NPM iirc!

What I'd like to know is why anyone thinks it's a good idea to have this level of granularity in libraries? Seriously? A library that only contains "a utility function that determines if its argument can be used like an array"? That's a lot of overhead in dependency management, which translates into a lot of cognitive load. Sooner or later, something's going to snap...and something did, here.

Reply on news.ycombinator.com