There are no real solutions to the problem, except for reducing exposure somewhat by limiting yourself to a mostly frozen subset of packages that are hopefully vetted more stringently by more people.
There are no real solutions to the problem, except for reducing exposure somewhat by limiting yourself to a mostly frozen subset of packages that are hopefully vetted more stringently by more people.
THEN use artifactory on top of that.
That's boring and slow though. Whatever I want my packages and I want them now. Apart of the issue is the whole industry is built upon goodwill and hope.
Some 19 year old hacked together a new front end framework last week, better use it in prod because why not.
Occasionally I want to turn off my brain and just buy some shoes. The Timberland website made that nearly impossible last week. When I gave up on logging in for free shipping and just paid full price, I get an email a few days later saying they ran out of shoes.
Alright. I guess Amazon is dominant for a reason.
Standard libraries should include everything needed to interact with modern systems. This means HTTP parsing, HTTP requests, and JSON parsing. Some laguages are excellent (like python), while some are half way there (like go), and some are just broken (Rust).
External libraries are for niche or specialized functionality. External libraries are not for functionality that is used by most modern software. To put your head in the ground and insist otherwise is madness and will lead to ridiculous outcomes like this.
This is great when the stdlib is well-designed and kept current when new standards and so on become available, but often "batteries included" approaches fail to cover all needs adequately, are slow to adopt new standards or introduce poorly designed modules that then cannot be easily changed, and/or fail to keep up-to-date with the evolution of the language.
I think the best approach is to have a stdlib of a size that can be adequately maintained/improved, then bless a number of externally developed libraries (maybe even making them available in some official "community" module or something with weaker stability guarantees than the stdlib).
I find it a bit funny that you specifically say HTTP handling and JSON are the elements required when that's only a small subset of things needed for modern systems. For instance, cryptography is something that's frequently required, and built-in modules for it often suck and are just ignored in favor of external libraries.
EDIT: actually, I think my biggest issue with what you've said is that you're comparing Python, Go, and Rust. These languages all have vastly different design considerations. In a language like Python, you basically want to be able to just bash together some code quickly that can get things working. While I might dislike it, a "batteries included" approach makes sense here. Go is somewhat similar since it's designed to take someone from no knowledge of the language to productive quickly. Including a lot in the stdlib makes sense here since it's easier to find stuff that way. While Rust can be used like Python and Go, that's not really its main purpose. It's really meant as an alternative to C++ and the various niches C/C++ have dominated for years. In a language like that, where performance is often key, I'd rather have a higher quality external library than just something shoved into the stdlib.
Really, good stdlib still allows you to use better suited 3rd party libraries. Lack of good stdlib doesn't add anything.
> Standard libraries should include everything needed to interact with modern systems. This means HTTP parsing, HTTP requests, and JSON parsing.
There is another way. Why not make the standard library itself pluggable? Rust has a standard library and a core library. The standard library is optional, especially for bare-metal targets.
Make the core library as light as possible, with just enough functionality to implement other libraries, including the interfaces/shims for absolutely necessary modules like allocators and basic data structures like vectors, hashmaps, etc. Then move all other stuff into the standard library. The official standard library can be minimal like the Rust standard library is now. However, we should be able to replace the official standard library with a 3rd party standard library of choice. (What I mean by standard library here is the 'base library', not the official library.) Third party standard library can be as light or as comprehensive as you might want. That also will make auditing the default codebase possible.
I don't know how realistic this is, but something similar is already there in Rust. While Rust has language features that support async programming, the actual implementation is in an external runtime like Tokio or smol. The clever bit here is that the other third party async libraries don't enforce or restrict your choice of the async runtime. The application developer can still choose whatever async runtime they want. Similarly, the 3rd party standard library must not restrict the choice of standard libraries. That means adding some interfaces in the core, as mentioned earlier.
It works well, but the downside of that approach is people complaining about how abstract things are.
Meanwhile, the public API of the selected standard library need not be abstract at all. Let's say that the bare minimum functionality expected from a 3rd party standard library is the same as the official standard library. They can just reimplement the official standard library at the minimum.
Where do you draw the line though? It seems like you mostly spend your time writing HTTP servers reading/writing JSON, but is that what everyone else also spends their time doing? You'll end up with a standard library weighing GBs, just because "most developers write HTTP servers", which doesn't sound like a better solution.
I'm willing to stick my head the other way, and say I think the languages today are too large. Instead, they should have a smaller core, and the language designed in a way that you can extend the language via libraries. Basically more languages should be inspired by Lisps and everything should be a library.
That's exactly npm's problem, though. What everybody is avoiding to say is that you need a concept of "trusted vendors". And, for the "OSS accelerates me" business crowd, that means paying for the stuff you use.
But who would want that when you're busy chasing "market fit".
I don't think that's the problem with npm. The problem with npm is that no packages are signed, at all, so it ends up trivial for hackers to push new package versions, which they obviously shouldn't be able to do.
That is, how does signing prevent publishing of malware, exactly?
Yeah, of course. Also if they hosted their private key for the signature on their public blog, anyone could use it for publishing.
But for the sake of the argument, why don't we assume people are correctly using the thing we're talking about?
If you have a well vetted base library, that is frequently reviewed, under goes regular security and quality checks, then you should be minimally concerned about the quality of code that goes on top.
In a well designed language, you can still export just what you need, or even replace parts of that standard library if you so choose.
This approach even handles your question: as use cases become more common, an active, invested* community (either paying or actively contributing) can add and vet modules, or remove old ones that no longer serve an active purpose.
But as soon as you find yourself "downloading the web" to get stuff done, something has probably gone horribly wrong.
Doing it the right way would create friction, developers might need to actually understand what the code is doing rather than pulling in random libraries.
Try explaining to your CTO that development will slow down to verify the entire dependency chain.
I'm more thinking C# or Java. If Microsoft or Oracle is providing a library you can hope it's safe.
You *could* have a development ecosystem called Safe C# which only comes with vetted libraries and doesn't allow anything else.
I'm sure other solutions already exist though.
This is a standard practice in most places I have worked, CI/CD only allowed to use internal repos, and libraries are only added after clearance.
There are also tools that break CI/CD based on CVE reports from existing dependencies.
Then let's add friction. Developers understanding code is what they should be doing.
CTOs understand the high cost of ransomware and disruption of service.
If you want an actual solution, look for differences. If you somehow end up figuring out its about type of people using those, then there is no easy technical solution.
So, databases? Which then begs the question, which - Postgres, MySQL, SQLite, MS SQL, etc.? And some NoSQL, because modern systems might need it.
That basically means you need to pull in everything and the kitchen sink. And freeze it in time (because of backwards compatibility). HTML, HTTP parsing, and SHA1024 are perfectly reasonable now; wait two decades, and they might be as antiquated as XML.
So what your language designers end up, is having to work on XML parsing, HTTP, JSON libraries rather than designing a language.
If JS way is madness, having everything available is another form of madness.
Granted some (JDBC) more useful than the others. Although JDBC is more of an API and less of a library.
As for XML, JAXP was a common way to deal with it. Yes, there’s Xstream etc, but it doesn’t mean any of standard XML APIs are obsolete.
Is java.util.logging.Logger not that great?
Sure, yet everyone that used it had a good night rest when Log4J exploit came to be.
The thing is, you don't have to be this undiscerning to end up with tons of packages.
Let's init a default next.js project. How many dependencies are there?
react, react-dom, next, typescript, @types/node, @types/react, @types/react-dom.
OK so 7... seems like a lot in some sense but its still missing many reasonable dependencies. Some sort of styling solution (tailwind, styled components, etc). Some sort of http client or graphql. And more. But lets just use the base dependencies as an example. Is 7 so bad? Maybe, maybe not, but you need to go deeper. How many packages are there?
55. What are they? I have no idea, go read the lock file I guess.
All of this while being pretty reasonable.
I could just be projecting my own bad experiences with "security" folks (in quotes as I can't speak to their qualifications). My other big gripe is when they don't recongnize UX as a vital part of security (if their solution is unsuable, it won't be used).
Security: we put the ‘no’ in ‘innovation’.
I was shocked when I found out that at some of the most profitable shops, most of their code is just a bunch of different third-party libraries badly cobbled together, with only a superficial understanding of how those libraries work.
Essentials tools such as Jest add 300 packages on their own.
You already have hundreds to thousands of packages installed, fretting over a few more for that DatePicker or something is pretty much a waste of time.
Even more weird in the EU where things like Cyber Resilience Act mandate patching publicly known vulnerabilities. Cool, so let's just stay up2date? Supply-chain vuln goes Brrrrrr
This explains a lot. Really, this is the great reason of why the society is collapsing as we speak.
"There should be no DRM in phones" - "You Are An Obstruction To Making Profit".
"People should own their devices, we must not disallow custom software on it" - "YAAOTMP"
"sir, the application will weigh 2G and do almost nothing yet, should we minify it or use different framework?" - "YAAOTMP".
"Madame, this product will cost too much and require unnecessary payments" - "YAAOTMP"
Etc. etc. Like in this "Silicon Valley" comedy series. But for real, and affecting us greatly.