Whenever I see NPM I think of the left-pad debacle and the more recent is-odd issue:
https://www.reddit.com/r/programming/comments/886zji/why_has...
https://www.reddit.com/r/programming/comments/886zji/why_has...
So in case you weren't aware: If you're using webpack, you're using is-odd. Which itself relies on the excellent package is-number.
So, how and why did the developers let things get this bad?
If you care about the size of the code you are delivering then it makes sense to skip some larger math/numerics library and instead opt for is-odd, which itself includes is-number.
Unfortunately, when the authors of modules also do this and not just end users, it becomes hard to track down how much a dependency actually adds, and the chain of dependencies can get very deep. Once that chain is deep we lose the ability to easily determine how a change the complex system we now rely on.
And that's all really at a surface level. Once we start talking about trust and security, the problem gets much worse.
"My package has 2 million downloads, I want x% more than market rate" attitude.
It's like the equivalent of influencer marketing, but for code packages. Of course I could just be a crazy old man. Who knows?
They specifically target noob questions/discussions, post links to their blogs that talk about certain dependencies. The articles themselves have very little substance, they're full of buzzwords and spend all their words making ambiguous claims, for example they'll say that the dependency makes your code "scale", but won't give any examples from a large code-base, especially not any comparing to the alt case (without the dependency). They make claims about performance, but don't elaborate, don't show benchmarks (or the'll show benchmarks that don't apply to 99% of projects, kind of like how immutableJS is obviously not faster unless you're working with big data structures, which you almost never are in your standard CRUD app where everything is paginated server side).
This shit is everywhere, and there's a lot of people with a vested interest in making sure these tools propagate as much as possible.
But no matter the joke, in no other language you 'd see packages like "is-true", "is-false", "ifAnd" (because && is too difficult /s) and "ifOr" (because... what's that "ll".... wait, is it "II" ? Um... How do I type it that vertical || line thing again?? /s).
The problem is not with users creating "useless" packages, the problem is with the community embracing them and having completely redundant one-liners gather million of downloads every week.
It just feels... icky. As if the whole's thing is put together with duct tape and bubble gum. We didn't resort to these kind of things when we had to support IE5.5 and 6. What went wrong from 2010 till today?
Seeing "is-true" rely to "is-object"... Why? Seriously, why? Maybe I am being completely narrow-minded and bitter.
If that is so please teach me and I 'll try to be better. I am not trying to provoke only to learn.
I think it's easier to answer in reverse order. DISCLAIMER: This is part rant, part "projected-constructed worldview". I absolutely welcome balloon-popping and "that's absolutely not true" and pointers to things that debunk/disprove points, so I can have a proper idea of history, and a better personal idea of the way things are.
First of all: the Web is "new media". Yep, fussy annoying buzzword; but one that has some truth to be discovered in it.
The position/power/operations/general relevancy of companies like Google, Mozilla, Facebook, Amazon, etc, more closely resembles that of CNN, Fox, CBS, Disney, etc around the 1995-2002 timeframe, than AT&T, AOL, CompuServe, Verizon, Comcast, etc, in the same time period. These modern companies have taken up the slack of the old TV/cable media empires of recent yesterdecades.
The Web has taken the place of TV, not by replacing it, but by becoming the new media focus.
It's a bit of a good side-illustration to mention that the Web is the perfect example of something with "Shiny Actually-New Thing Which Seriously Truly Isn't Anything Like What Came Before It" syndrome, for exactly the reason I mentioned in italics - the Web, for most people, resembles nothing like TV.
I think one reason for this is the technological generation gap: older people don't fluidly relate to computers, while the younger generation don't have quite the same perception/appreciation of non-interactive TV that their elders do, so while there is some collective conversation about media agendas, it doesn't bridge the generational divide that coincidentally happened with TV>Internet, and so people don't realize the _exact same_ control agendas that went into TV now go into the Web.
But technical folks who've seen the Web grow, and change, wonder: what's happening? My working theory is that the technical changes we're seeing stem from various upstream political machinations within media.
So, to the 2nd half/2nd question. As a "new media" company, Google wants to retain captive control of as much as possible, in exactly the same way $oldmedia would. So what do they do?
Well, back in the 80s and 90s you'd run cable to the entire country to control distribution, cultivate deals and negotiations with advertisers and producers, etc etc. See Also: CNN/MSN/MSNBC/CNBC (I never did figure out what acronyms went where with those companies), Comcast (which I understand doesn't just provide garbage cable/DSL internet, but is also a TV and media network), Time Warner, etc.
So, nowadays, Google's pretty much nailed the advertising thing, undoubtedly to the chagrin of everybody else. On top of that they handle the world's searches - and you never lie to a search engine (!), and that's the world's biggest non-perceived captive audience ever right there.
But they're GOOG; they legally have to vacuum up as much of everything that everyone's perception of them supposes is within their purview, in order for the balance sheets to look right. That most definitely includes Web standards. So, how do you control Web standards?
Well, you apparently approach your cofounder and say you need to build a browser.
I wonder what the friction breakdown was - how much political, how much technical - when Google split with Apple working on WebKit, and forked/created Blink. Heh, I'll never know. But the commit rate sadly plummeted after the fact (http://mo.github.io/2015/11/04/browser-engines-active-develo...) so code quality certainly wasn't on the decision table; something else was the priority. My guess is that it was straightforward market dominance, and competing against WebKit (and iOS!).
So, (having won) one Internet (marketshare) later, now what? Well, now you have a browser, you can play in the "democratic browser vendor" sandpit as a first-class citizen! The Standards® Committees™ are full of people who absolutely love discussing Web minutae all day; all you have to do is fund conferences and shows for them to attend, and they'll go ahead and build their own candles to fly around. They're like the self-organizing equivalent of https://xkcd.com/356/, it's wonderful.
They do need feeding though; specifically, you need to feed them just slightly too much in terms of upstream changes, so they can almost keep up but not quite. Just on the edge of being able to manage everything though. This is easy to do by, instead of adjusting the workload slider, adjusting the paid employees slider instead; and you can indirectly do this by doing nothing to democratise/vindicate/humanize the standards managlement process at the W3/WHATWG level so it seems obscure and academic and boring; and then it won't seem intuitive for upper management to prioritize funding for it, and voilà.
Now you just need to find a good source of chaos to keep everybody on their toes. You do this by investing heavily in coming up with new ideas for the standards committees to implement, keeping careful track of what everyone's discussing, and prioritizing what to carefully drop quietly on the floor ("oh yes we do need to get back to that sometime sorry about that") and what to react to as an interested browser vendor. Be polite enough about other vendors' technical interests, of course; your market share will typically dictate what'll actually get maintained at the end of the day. Pick things to cooperate on to keep everything looking democratic, so you don't get cornered into implementing things you don't want to (if this even happens?).
Oh, regarding the democratic bent that seems to underpin everything related to the Web (and open source for that matter): I consider it all a psyop (I was recently reminded of the existence of this sort of thing by https://news.ycombinator.com/item?id=16918752). Focusing on social organization (and justice and related fields, FWIW) are excellent ways to sink/drain massive amounts of energy; get humans into that loop and they don't seem to be able to disentagle themselves thanks to peer pressure and the precedence that gets created. (This is why "minority groups", typically the gender-focused ones, are in the permanent media spotlight - it allows control.) Democracy is a foundational part of the US Constitution, so of course democratic leadership is a very politically correct way to do "high-level" "important" things like directing the future of the Web; on top of that, democratic "leadership" is the kind of thing most people see straight through unless someone pokes the hologram the right way (especially when everyone's preoccupied with getting their social dance moves right), so I can easily see it being used as a tool in situations like this.
In the case of Web standards groups it was particularly easy to deploy "democratic leadership" - the Web groups seem to have been engaged in a most fascinating set of caricatured circling/courting "everyone's voice is important" dances since time immemorial. I was enlightened about the historical state of the Web a couple years ago: https://news.ycombinator.com/item?id=10684426 (I accidentally locked "i336_" by setting "showprocrast" too high, woops). Besides the antitrust bits I commented on, I've also since learned that IE6 implemented an early draft version of the DOM (or it could have been the CSSOM) before it got redesigned from scratch, then the necessary reimplementation work was never approved, and this was the source of all the rendering bugs. (I need to go re-find that link, sorry!)
Reading through those mailinglist posts showed me that the various Web working groups have always been... I'll say it, somewhat stuffy, and this fact is not a new development. It's sadly so easy to psyop-weaponize though (or, in any case, the baseline likelihood that this has happened is so incredibly far away from absolute zero...), as per the xkcd I mentioned before. Meanwhile, you carefully build EME behind the scenes (the implementation of which is kinda depressing; see zb3's reply to https://news.ycombinator.com/item?id=15796420).
Unfortunately some people run off and actually go and make interesting and amazing things in the face of great odds, for example https://codepen.io/AmeliaBR/post/me-and-svg (<> https://news.ycombinator.com/item?id=14155393). It seems those who stubbornly want to succeed are handled by ignoring them and hoping they'll go away. Or - https://en.wikipedia.org/wiki/Hanlon%27s_razor - maybe it's that whoever it is is calling the shots is so laser-focused all they have is perfect indifference since these cool things not within the scope of their media/other agenda. An intersection of explanations is valid. But the features these amazing people come up of course get used in any case, IMO without due acknowledgement too.
Most Web developers seem to get caught up in the entropy and chaos of continual change and development, get blind{sid}ed by it, and don't really [get the chance to] form a foundation strong enough to get them out of the slipstream, looking far afield, and doing something different and unique. Commercialism and existential "must have a job to eat" probably has a big part to play in folding in to this, along with mental health impacting society's ability to think clearly, along with all the other issues the various in-vogue generations apparently focus on.
To do one of those zoom-from-outer-space-into-a-tiny-backyard things and zoom/focus from this wide view into NPM packaging, I actually argue that the state of the JS ecosystem is easily explainable by saying that the Web doesn't have a standard way to do libraries, so everyone solves that problem by vendoring (including/bundling) everything they use. This solves everything for Node.js and the browser.
> Continued >
Fundamentally, on the ground, you're looking at economy of scale, both in terms of implementational scale and in terms of humanity collectively responding to a problem and scale and slowly annealing (if I can borrow that from "simulated annealing") toward a solution that just happens to work. The macro level works out; the micro level looks terrible. Such is how this kind of thing works out; social scale/crowd dynamics are _really_ fascinating to watch at work.
Node.JS happened, so suddenly JavaScript needs to run on servers and in browsers, and handle backend (Databases! Legacy stuff! FoxPro! How To Port Your Visual Basic Excel Macro To JavaScript! I just found two libraries for RS485!), along with frontend (Win32 event loop inspired Rube Goldberg machines!), and everything in between (......uhhhh...... all of GitHub?). The ecosystem and infrastructure - all the library code - has to be compatible with everything and everything else, across more domains than I even realized existed until I thought this through 5 minutes ago. The easiest way to work in the browser and also work on the backend apparently is to fragment everything out, because nobody's come up with a better way to stdlib JavaScript. The jQuery-esque approach of "this system works unbelievably well within its problem domain but doesn't generalize" effectively died when jQuery aged beyond its point of relevance - and because the plugin system was so locked-in to jQuery, all the plugins died with it. That type of approach is very very top-heavy and simply didn't scale. It made sense when all we had was browser [delivery] though, backend types hadn't gotten their mitts over everything and made compilation pipelines for everything, etc. Now that's happened, it makes more sense to `cat * > out.js` (well, in more LOC, anyway) because it just scales better.
JS feels clunky, but once you look all the way down, I feel like the vertigo actually causes things to kinda make a tiny bit more sense and fit into place a bit better. Maybe.
I find it not unreasonable to use the explanations above to justify the higher-level "why???" of the way things are.
The duct tape and bubble gum isn't JS-specific. It's endemic to all of technology. Like with media, there are vested interests in things working out certain ways with security and whatnot. (My current favorite meme of this is http://web.archive.org/web/20160409181051/http://article.gma..., and I also found https://news.ycombinator.com/item?id=17160109 the other day and that was kind of very fun to read, and I also read the other day that the NSA discovered Heartbleed "very shortly" after it "appeared" in 2012 or whenever it was, which was a TIL of its own)
And so it is that we use systems that operate at cross-purposes with how we think and incorporate high-friction impedance mismatches, and which raise the chances we'll make mistakes, which affect everyone: https://twitter.com/x0rz/status/865196993215442944, https://sites.google.com/site/testsitehacking/
The above being said, I do completely understand your frustration with using Node. I have the same gripes with PHP, which I learned ages ago and Really Really need to move off of, but it just runs so incredibly well on older hardware and has zero warmup time... like 0.2ms on my 12 year old laptop... sigh. One day I'll find something, or maybe I'll make it myself.
I'll leave you with three last links which I couldn't fit into the narrative, but which I think are relevant/related.
Firstly this an interesting article that helped me properly understand Google: https://medium.com/p/e836451a959e
Next, I wanted to call out this NNTP post from 1995 (linked from "sent out via" in the above) showing storage investments: https://groups.google.com/forum/#!topic/mail.cypherpunks/4CD...
For context about cost in the above link: I found this very cute little document while randomly manual trawling (I love finding things not in Google's index) whose HTTP headers say it's from 1996: http://users.monash.edu.au/%7e%6a%6f%68%6e%6d/%77%65%62%73%6... (#1 I don't want its filename to be indexed just yet :P and #2 I want to do an experiment to see if the crawler resolves the escapes and finds the URL anyway)
NB. If you have a better term than "new media", I'm curious. It's such an easy term to overload. The symbol picked for biohazard was designed from scratch to be meaningless so a meaning could be cultivated for/associated with it over time, and that cultivated meaning would be what "stuck". (I wonder how many other things this has happened with.) https://en.wikipedia.org/wiki/Hazard_symbol#Biohazard_symbol
NB2. If you - say - accidentally hit F5 while writing a post... and you're on Linux, suddenly get really really interested in opening Chrome's task manager, identifying the renderer PID and SIGSTOPping it in as few seconds as you can. The first ~15% of the above text was recovered via gdb generate-core-file \o/
NB2a: (For completeness, if you accidentally close the tab, have working hibernation set up, and either go fishing in the swap partition on resume, or if you don't want to risk stepping on anything in swap on reboot, carefully boot something that won't automatically swapon.)
NB3: $lists_i_am_on += 5000 :P