Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here?
I'm all for limiting bloat, but your rant here seems completely inappropriate for the issue bring described.
No, but downloading random code is just insane.
As a comparison, Debian has ~3000 packages, and every single package had an identified maintainer, with its own GPG key, validated in face-to-face meeting with an ID card by three people, an identified upstream, etc. Each maintainer is physically identified, and has passed a number of technical validation steps, explained his motivation etc.
There is also a dedicated security team that can be contacted 24/7.
The system is not perfect, but it provides a good level of security. And this is a project only made by volunteers.
As for Debian making some policy decisions that you disagree with, it's very different for Debian to make decisions than for NPM to make decisions. There's just no comparison.
As of yesterday npmjs had 516,132 packages, which was an increase of 373 since the day before. Debian's 3000 packages is a couple of weeks of npmjs activity. Even if you stripped out the unnecessary, abandoned, or duplicate packages you're still looking at a something significantly different to Debian.
For what it's worth I think a well-maintained, verified, and known secure subset of npmjs would be a great idea, but the logistics of providing that would be effectively impossible without some serious cash behind it.
Python projects I have recently worked on might max out at 50 packages. The last project I worked on with nodejs as front end had 3000+ packages just for the UI.
[1] https://github.com/sindresorhus/ama/issues/10#issuecomment-1... [2] https://github.com/kevva/is-positive
How do things like left-pad even come to be widespread dependencies? Does the node development process involve a lot of "gee I wonder if someone made a package for (simple thing I need to do with a few lines of code)"?
is a nice article on the subject of left-pad. If you can't implement a left-pad in less than 5 minutes yourself, you don't know how to code.
Not to mention the code reuse factor.
If npm took the route of debian's verification/certification, you might end up with no libs at all!
Nicely said, but you omit the elephant in the room: that the dependencies have their cost, which is quite high and is rarely matched, much less offset, by the value of these "simple libraries".
Only in volume. In code quality and man hours they're an order of magnitude ahead of all of npm.
(Just consider something like Linux the kernel, GCC, Apache, OpenJDK, core utils, Python, Postgres, etc, compared to the trite that even the best npm packages are).
It's irrelevant to your point but Debian 9 included >51,000 packages [0].
No, they mean that we need a better method than trusting some random popular GitHub / PyPy / npm whatever storage and delivery mechanism.
Strong core language libraries ("batteries included"), that come with your distribution of the language/compiler and are well maintained, used by millions, and signed would be a good start to at least have 80% of dependencies being actually dependable.
The trick is avoiding another urllib: shipped as a stdlib, and passed over for libraries with better interfaces.
perl has the idea of core and dual-life modules: when a new version is released, it ships with some additional modules. Some of these modules live in the perl repository & are called 'core modules'. Upgrading them requires a new version of perl to be release. Some of these modules are forked & published to CPAN & are called 'dual-life' modules. Upgrading them requires installing them from CPAN, and new versions of perl can provide a well-tested newer version.
All the modules used to be core-only, and over the years any which weren't heavily tied to the internals of the perl interpreter (like opcode deparsers etc.) have moved to be dual-life instead. This avoids problems like interface issues in the standard library by allowing a new version to be immediately released (with some overhead for developers who need to upgrade it for deployments), be tested in the real world, updated, and then included in the next language release.
I've never done it, but it at least sounds like there's a process where you need to convince 3 or more people that adding your package is a good idea and will not hurt security. That's very different from PyPI or npm.
It's not a free-for-all like npm, but may be npm can learn a thing or two.
However, you have to sign everything with pgp including updates and that is verified. You also have to own the domain with same path as your packages - meaning name space is larger and name clashes less likely. They actually check this and won't release unless you host project. Which explains why java open source tend to use packages like com.github.my_account.my_project
There's a running joke in python that stdlib is where modules go to die. Stdlib by definition needs to be more dependable and stable than other places. But that reduces the innovation. There's a reason urllib is terrible to use and we're on urllib3 now, yet the nicest solution (requests) is not included.
See http://www.leancrew.com/all-this/2012/04/where-modules-go-to... This applies to some extent to all languages / runtimes.
1. Similar way of identifying who is responsible for a package to Debian style repo - would allow those affected to identify a named person if malicious code added.
2. Fuzzy matching of package names (so taking the library name urllib also takes urilib, urlib, urllib2 and other names with a short document distance).
3. Built in mechanism to allow third party reviews within the repositories infrastructure.
They made it pretty clear that they get them from a distro repository. Most are much better about managing breaking changes and offering years of security patches (at least to core packages).
> Packages taken over include misspelling of urllib. What exactly do you propose as an alternative here?
Stricter guidelines and auditing should have caught this.
Even if you can work with that, you need to sort out differences between your destination system, test system, and all of your developers machines (who most likely run Mac, not your server Linux flavour). Sure, there's still docker, vagrant, etc. But this means slapping more and more layers just to make the same packages from the same source available.
And unless you want a "works on my machine" environment, you can't just leave developers to do whatever and use a different system in production. A home brew, limited system will just result in shadow it, which will use pypi unless your solution is much better and easier to use (and allows fresh versions to be imported)
I've been doing this stuff for years in big projects. It's not easy.