The Everything NPM Package
socket.dev
socket.dev
So not asking rhetorically, if we had all the insight and knowledge we have now, how would you make it different?
That said, you can still have a core set of “blessed” packages that serve the common needs.
You see it in Go, Python, .NET, etc.
Rust has a good standard library and also a large community of libraries. Sometimes those community libraries get promoted to std because they're strictly better. Sometimes the std version of hashmap is slow because std insists on using a crytographically secure hash when 99.99% of use cases would be better served with a less secure but faster hashing algorithm.
Like many things in life the ideal scenario is a benevolent dictator that only makes good choices. In practice the best way to get something good is to allow for multiple choices.
<insert parable of pottery class graded on quality vs quantity>
Meanwhile, the horrible C++ and Python API designs at least offer the needed functionality, even if the code looks ugly.
While I agree here, you also have to remember that additions to the JavaScript standard also increase the amount of time / effort for new browsers to enter the space.
The JavaScript standard (the web APIs, mainly) are already very complex, with Web Workers, Push Notifications, Media Streams, etc. that additions to it should be made cautiously -- once an API is implemented, it's there forever, so the bar for quality is much greater than that of some NPM library.
Yes it should be done carefully. There are also plenty of examples of how this can be done well, done by experienced engineers. For example, the Dart starndard library (https://dart.dev/libraries - core [1], collection [2] and async [3] in particular) is a very good model that would fit JS fairly well too (with some tweaks and removals)
[1]: https://api.dart.dev/stable/3.2.4/dart-core/dart-core-librar...
[2]: https://api.dart.dev/stable/3.2.4/dart-collection/dart-colle...
[3]: https://api.dart.dev/stable/3.2.4/dart-async/dart-async-libr...
It's still a nonzero amount of complexity. I see a lot of "v8 is really hard to compete with" comments on here so this feels very pertinent to mention. You can't have it both ways.
> Yes it should be done carefully. There are also plenty of examples of how this can be done well, done by experienced engineers. For example, the Dart starndard library (https://dart.dev/libraries - core [1], collection [2] and async [3] in particular) is a very good model that would fit JS fairly well too (with some tweaks and removals) > > [1]: https://api.dart.dev/stable/3.2.4/dart-core/dart-core-librar...
This one, at least, looks somewhat inspired by JavaScript.
That sounds great, but I'm doubtful of the simplicity behind this approach.
If my understanding is correct, v8 has transitioned to C++[0] and Torque[1] code to implement the standard library, as opposed to running hard-coded JavaScript on setting up a new context.
I suspect this decision was made as a performance optimization, as there would obviously be a non-zero cost to parsing arbitrary JavaScript. Therefore, I doubt a JavaScript-based standard library would be an acceptable solution here.
[0]: https://github.com/v8/v8/tree/main/src/runtime [1]: https://v8.dev/docs/torque-builtins -- As I understand it, Torque compiles to C++ at compile-time, which is then linked and compiled into the rest of v8[2]. [2]: https://github.com/v8/v8/blob/main/tools/torque/format-torqu...
(Or are they still trying to make Servo viable?)
There's plenty of competition, even if the current projects are in a beta (or even alpha) state. Consider the LadyBird browser developed by SerenityOS, or Servo.
Go has a big standard library too and it's mostly very well designed, useful and avoids fragmentation.
I think a similar thing happened with compiler warnings and C/C++. The language is error prone so people want warnings but a lot of the warnings don't have good solutions (e.g. signedness mismatches) so people tend to ignore them. Also they aren't easy to control, e.g. from third party dependencies.
So some people got the idea that warnings are fundamentally wrong and e.g. Go doesn't have them. But my experience of Rust warnings is that they are totally fine if done right.
This feels like a bit of a strawman, since sorting is already in the standard library and there aren’t in fact popular sorting packages for each framework (that would in fact be ridiculous).
If you want to start a real debate though, bring up date/time pickers.
There are multiple date picker, time picker and datetime picker packages for each framework, and there are debates with good points on all sides about whether the browser-provided pickers are sufficient, or whether this is an area where a level of customization is needed and what that level is keeps changing as people discover new ways of designing date/time pickers and new use cases arise that require different tradeoffs. It’s both really frustrating but also kind of understandable.
Of course, Records and Tuples would greatly simplify the process.
What I meant was that it's possible to emulate a custom key comparison predicate.
You just have to have a function that returns value for each input that behaves the way you want under strict equality comparison.
Implementing an `equals` function that returns a boolean is more convenient, sure.
Serializing (for plain objects with only JSON-serializable values that would be JSON.stringify) to strings or other primitives would of course be possible with object keys, too. But that's probably what you want for "record"-like objects, right?
And if you want better performance or compare non-primitive values, you'd have to do something more complex, that's what I meant by the lookup table.
But I imagine if you deep-compare large Record objects a lot, the performance wouldn't be any better, because the engine still has to do a deep comparison.
If I am not mistaken, Records/Tuples are in fact strictly limited to this case:
https://github.com/tc39/proposal-record-tuple#jsonstringify
So basically there is no difference to having a function serialize() that just stringifies your plain object, maybe with a caching layer in between (WeakMap?)
OK, thinking about it, the proposal really would help to avoid footgun code where performance optimizations are lacking, and too many deep comparisons or too many serializations are performed.
const idx = [1,2] const m = new Map m.set(idx,"hi!") console.log(m.get(idx)) // outputs "hi!"
console.log(m.get([1,2]) // outputs undefined
That last line has created a new array object, and Map is made to be fast so checks equality by reference. Ah, which is what you to be able to change. I guess you would want to pass a new map a comparator function, so that it does a deep equal. That would be faster than what you would have to do now:
const idx2 = String([1,2]) m.set(idx2, "yo") console.log(m.get(idx2)) // yo console.log(m.get(String([1,2])) // yo
Stringification is a very crude hack, and it doesn't even work in all cases - e.g. not if a part of your key still needs to be compared by reference. Say, it's a tuple of two object references, describing the association between those two objects.
Either way, the point is that this is really basic stuff that has been around in mainstream PLs for literally many decades now (you could do this in Smalltalk in 1980!).
Safari (iOS and MacOS) still doesn't have full support for the date time picker, which is why there are so many alternatives.
Right. For example, to sort a list of numbers:
[5, 14, 1, 2].sort()
[5, 14, 1, 2].sort((a, b) => a - b)When you combine those together you end up with a situation where "normal" js code not from a library can't be trusted on the front end because it won't work for x% of your users, and offers a clumsy API on the backend that you'd prefer be wrapped in a helper. Developers learnt that they should reach for a library to e.g. deal with localstorage (because on Safari in private mode the built-in calls throw an error instead of recording your data and discarding it after the tab closes) or make a HTTP request (because node doesn't support fetch yet and you don't want to use the complicated streams and callbacks API from the standard lib) and they propelled that culture forward until everyone was doing it.
We’ve been blessed in recent years that either languages are fully open source and come with a reference implementation, or a standards body governs the implementation detail. Sometimes even both.
Whereas JavaScript is really more a group of languages, each with their own implementation quirks.
ECMA was intended to bring some sanity to all of this. And it’s definitely better than it was in the JScript days of IE vs Netscape. But there isn’t anything (that I’m aware of) that defines What should be a part of the JavaScript standard library.
Wouldn’t it be great if there were a libc in the JS world. Something portable and standardised.
I mean, it’s not necessarily “in the JS world”, but WASM is basically that.
You may be thinking of WASI?
While it's ridiculous to expect that people will audit every single dependency and sub-dependency, it's not ridiculous to expect tooling to do the same.
Packages should be given an overall quality rating (and honestly it might be great for an ecosystem as large, diverse, and welcoming-to-beginners as JS/TS), part of the score comes from the number of different dependencies/sub-dependencies -- a social package score if you will. If a package causes the dependency graph to explode, give a warning before installing it.
Then, if you're NPM, you don't need all of these convoluted and exploitable policies around un-publishing.
You can import a list of audits from trusted auditors, which should cover all popular packages. Now you have to audit dependencies that aren’t well-known in the community, which really is the set of dependencies that you should take an extra look at. The big popular JSON libraries can be audited by either Microsoft or some of the other large projects that are using them.
You’d explicitly share your trust list in your audit file, and anything (updates or new packages) that isn’t trusted by you or one of your listed auditors is flagged for auditing.
Conflating these two not-unrelated-but-still-distinct concepts is a big contributor to why the current state of the art is so fraught.
It's not ridiculous at all. Professional programmers should answer for the dependencies they bring into their projects.
Not having a registry is neat, but I'm also unsure of what is going to happen over time as dependencies may be moved or removed. You can see that with old Maven pom.xml where some dependencies do not resolve anymore.
That’s it. That’s all you need to do.
That's what the Go module proxy is for. The authors can move or remove their repositories as much as they want, I as a dependent am not bothered by it. They would have to go through an official vetted process to get it removed from the proxy.
the community shouldn’t need to write a bunch of tiny utility packages to do common things.
in other words, make it easier to avoid the deeply nested dependency mess that js encourages.
Not to say that npm shouldn't have had namespaces by default, but I think there's good reason not to blindly do everything the way the Java community did.
I'm not saying that they should have blindly copied Java, but they should have had something.
Our devops guys scream from the seething pain whenever the have to debug some pile of shit that decides it won't build unless all the runes are aligned precisely and all the RAM in the universe is available on the build runners. And pushing this to the developers results in importing more packages thus adding to the burning tyre fire.
And after several hours of builds and 9000 layers of packages you wake up one morning and in that 50 meg chunk of javascript that is excreted from the process, someone managed to inject a "Slava Ukraini!" banner into your web app.
Over-reliance on third party dependencies is a choice. One could argue that it's unreasonable not to do it if you want to stay competitive but good luck changing human nature then. If there are shortcuts, they will be taken.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
If analyzing the dependencies for showing in the NPM web UI, while analyzing, as you exceed 40 direct or transitive dependencies, abort and highlight this package in red, for having excessive dependencies.
If installing locally, you get what you get, don't install random or crazy packages, stick to well known high-quality minimal-dependencies packages. nodejs does include file reading and writing, http server, http client, json ... that will take you pretty far. Master the basics before getting too fancy. And remember, you don't need some company's client package just to make some http requests to their API.
Have smarter users. If your package breaks because it depends on trivial code which got deleted, you shouldn't have depended on that in the first place.
Preventing people from deleting their code -- always, or even just sometimes -- was never the right solution.
Basically, you would need to start accepting that you are responsible for any dependencies you choose to include. Any upstream changes you would need to evaluate and bring in or patch yourself.
Definitely an impossible task given how broad and deep modern package dependencies are, but at least you’d start feeling the insanity of having all if them in the first place :P.
What problem does this solve?
If you want answers, state clearly what _specific_ problem you're trying to solve. Whatever the solution to it might be, vague and fuzzy questions—while magnets for chatter since they can stand in for whatever someone wants to read out of them—are not the way to get there.
(You could say that this is needlessly tedious because everyone already knows what we're talking about, but that this isn't true is exactly my position. It's certain that something like half the people reading, thinking, and writing are have in mind one thing, while the other half are thinking of another—and the third half are thinking about something different from either of those. We're also programmers, so dealing with tedium and the constraints of having to be explicit should be second nature.)
html, a crappy defective xml implementation refuses to grow up, js, while great for little html tweaks is not adopting any of the useful features found in popular npm packages. It was actively developed for 2 weeks. Ripping off it's head (nodejs) gave us a poor sailor jargon ~ but without the boats!
Therefore there is nothing wrong with npm, she is a fine ship. The harbor doesn't want to take it's much desired cargo, it must sail the 7 seas forever mon capitaine!
Does this mean it is perfect? No. Is it "crappy"? Nope.
Also, while JavaScript had a rushed development cycle, it has grown over the past 20-30 years and you can clearly write some great programs in it. Also, it has some very good features. My favorite is you can pass functions as variable arguments. It got this before a lot of other mainstream languages.
That does not mean it's good.
Lead paint was widely used in the Europe and Americas for a very long time, doesn't mean it was good
This is a great line. If HN had a quote of the month or something, this should be nominated for that.
(If you don't want Google to see what packages you're fetching, you can also turn this off with an environment variable.)
Once the referencing packages are updated are deleted or modified the shadow versions can be dropped.
When you wonder whether to add a dependency, you should ask yourself: What are the upsides and downsides of adding this dependency. One downside is always that by adding a dependency, you add a potential security problem, a potential point of breakage, and more complexity.
There are situations where these are well justified. If your dependency is stable, from a trustworthy source, and if it is a functionality that you cannot quickly implement yourself. But if you include a dependency that is effectively one line of code, the question answers itself: The costs of adding a dependency is completely unreasonable. It your list of dependencies grows into the 100s, you're doing something wrong.
Make the cost of reusing software non-zero again.
It doesn't have to be as painful as C++ without package managers, but should make every developer spend about 5~10 minutes labor work for adding each direct dependency, or one minute for each new dependency in the dependency closure.
> It was removed, but then reemerged under a different scope with over 33,000 sub-packages. It's like playing whack-a-mole with npm packages!
> This whole saga is more than just a digital prank. It highlights the ongoing challenges in package management within the npm ecosystem. For developers, it's a reminder of the cascading effects of dependencies and the importance of mindful package creation, maintenance, and consumption.
> As we navigate the open source world, incidents like the everything package remind us of the delicate balance between freedom and responsibility in open-source software.
Source: have done a bunch of AI-assisted writing to develop my own skills and the tics and specific turns of phrases really pop out to me.
Ironically, the most common place I read the tic of ending a piece of persuasive with a deliberate, unconnected conclusion that doesn't persuade and instead equivocates or states a trivialism ... is in student papers or similarly graded-like-assignments rote work.
Could be that there's a lot of that out there such that it's heavily represented in training data. Could just be a person doing a not-great writing job.
"accidentally broke NPM and all I got was this sweet permanent banner all over my Github (thats impossible to remove since they probably had to code it up last minute before removing the org/repo)"
When I was consulting for an R&D lab at eBay, we open sourced a bunch of our work in a GitHub org. It was sanctioned by eBay's OSPO; they even linked to it from their main open source directory.
7 years later, long after the team disbanded, someone in eBay's current legal team decided that the (now archival) org violated eBay's trademarks. For the last year+, every time I've opened GitHub, I've been met with the same undismissable banner.
Since the only choice they give you is to contact support, I did. Unfortunately, their support team is not responsive, and has a completely separate notifications system. It took an inordinately long time for them to respond. (I have poor reception here so I can't check, but I think it was months.) Since I'm not in the habit of checking GitHub Support for new messages, when they eventually replied, I missed it. I had to start a whole new ticket. That too was months ago, and I still haven't heard back.
So because I did some work for a skunkworks eBay team in 2015, the top 150px of my GitHub are unusable, and there's apparently nothing I can do about it until some call center decides to write me back.
You're right that a package that depends on literally everything would absolutely have a score of 0 in our system.
What I'm surprised by is the apparent cultural norm that this is just a regular everyday occurrence which entirely erodes any faith in the meaning of SemVer. Sure, we cannot 100% trust SemVer (because humans are fallible) - but there is a world of difference between trusting it ~99.9% and 0%. The JavaScript community (from the outside! I could be wrong!) seems to have simply accepted the 0% situation, and all the extra toil that goes along with it, rather than trying to raise the bar of its contributors to be better.
Biggest issues are authors that keep their libraries at 0.x forever (every minor chance can be a breaking one) and the ones that release a new major version every other week.
The times I do a minor update and something breaks are generally regarded as a bug by authors too.
I'm saying that the fact that it is (apparently) the norm in JavaScript-world that authors will regularly publish breaking changes that are not advertised as such, and that that is just an acceptable everyday uncommentworthy inconvenience, is surprising to me. How do y'all get anything done if you can't even trust SemVer enough to automatically pull in minor/patch updates to dependencies!?
In my experience the most common cause of breaking changes is accidentally breaking on older versions of the runtime, because the project is only running tests on the last version or two. Aside from that, the only notable example I can think of in the last year was a pretty subtle bug in what was supposed to be a pure performance optimization in a query language [1]. I think these are pretty representative, and not meaningfully worse than the experience in other languages.
The problem with the `*` bug is that it means you can stop anyone from unpublishing future versions of their package by simply creating a package that depends on it with a `*` identifier and publishing that to the registry.
It does if your project is also in the npm public registry and the package you're dependent on is more than 72 hours old.
It's removed from the index, and cargo will only download it using a pre-existing lock file
'everything' blocks devs from removing their own NPM packages - https://news.ycombinator.com/item?id=38873944 - Jan 2024 (102 comments)
Has no one thought of that? It seems like it should have been obvious that such an absolute rule could be easily abused to troll the system at scale.
Not sure if it's a problem though, perhaps all unpublishing requests should be reviewed by someone at the registry (and granted only when it makes sense).
Is npm specifically vulnerable to this kind of thing? Or is it just a cultural elelemnt of npm that there are more micro-packages?
At some point, Russ Cox got the Fear about this, and now https://proxy.golang.org/ is an on by default, caching proxy in the middle. You can still delete your packages whenever you want to though.
"Just install the everything package, then you will be sure to have the right package"
> First, just want to apologize about any difficulties this package has caused.
No rationale. No shame. Just the word “apologize” in a sentence.
Who downloaded it though? Surely as a dev if you download such a package it’s on you?
Upon rereading the article I can see that the word “unintended” is actually not Patrick’s but the author of the recap’s word.
Beyond that you seem to be ascribing benign intent. Reading it from the horses mouth [1] it doesn’t seems like they had any intent other than trying to find out if it could be done. In a world of worse is better creating the largest possible area of effect for your experiment seems to be a pretty easy way to amp up the consequences of your actions regardless of the risk.
It’s the world of worse is better and they’re going for the widest possible area of effect. Should we crucify these guys? 100% not. Part of this is on npm’s design and implementation. Part of it is cultural.
But these guys owe the people who were needlessly “inconvenienced” a little more than just the word “apologize”. Not their first born but some rationale which justifies or reveals that they realise it was a bit pointless or stupid.
The wildcard "any version of dependency" preventing unpublish is clearly flawed. The "everything" package folks had no malicious intentions, and nobody would benefit from a long-winded, ashamed apology. If not for NPM's flawed unpublish policy the everything team would've unpublished to resolve the issue.
I just think it would have been good to give the “I was hoping to investigate X, I did not expect Y, I can see now that it was irresponsible to do X.”
I don’t think that’s particularly long winded.