Node 8.3.0 released | Hacker News Reader

Node 8.3.0 released

github.com

172 pointsbinarray200082 comments

Shame about npm5 being significantly broken, taking the wind of the node 8 sails.

Luckily there's still Yarn: https://yarnpkg.com/en/docs/install

Honestly it's great that npm 5 seemingly caught up with the speed of yarn but that only addresses one of many reasons I'm happy to have made the switch. Npm has been broken in exciting ways for years (and often in the same way for years -- see the infamous race condition in their tar implementation that broke npm publish at random).

So, based on the history of NPM and its issues, is it too harsh to conclude that they don't really know what they're doing, and ditching it for Yarn is probably advisable on general?

I mean, I consider myself a relatively novice programmer, so perhaps I'm underestimating the challenge involved, but I'm absolutely baffled that we ended up with 'the standard Node package manager' not being able to produce consistent results when setting things up in different environments. I mean, I've used Gemfile.lock for what feels like ages.

Saying NPM is badly designed is a bit simplistic. As much as I prefer yarn over NPM for all kinds of reasons, NPM's problems have more to do with being born out of a solo project that pre-dates most things we take for granted in Node today.

There was no CLI argument parser, so Isaac wrote his own. There was no cross-platform tar bundler, so Isaac wrote his own. Not to mention that there just weren't any best practices in Node or JS for any of what NPM was trying to do yet.

Yarn has the classic second mover advantage. The yarn devs learned from NPM's mistakes (there were more people involved than you see in the commit logs -- they even talked to some NPM people during development) and built on the vast ecosystem that exists now.

I don't like npm Inc and I think there's a serious conflict of interest in having NPM bundled in the Node Foundation's official releases (people using the NPM client is in npm Inc's best interests, whereas Yarn isn't controlled by a single commercial entity). But blaming NPM's problems entirely on developer incompetence is unfair.

It's not too harsh, vote for better tools with your usage.

NPM 5.x in Node v8.1 consistently only installs 481 out of 493 packages in our node_modules folder. No errors, everything looks absolutely fine, except 12 packages are straight up missing. That was fun to debug! Both NPM 3 and yarn produce the expected result.

That was the last straw for me and NPM is now banned in favor of yarn.

I actually had the opposite happening: a project was having some real issues with not installing proper dependencies until we started using npm 5.3+. Things work fine now and are a bit more deterministic it seems.

It's an odd project with a lot of super strange peer dependencies, many of them in beta or as pre-releases, so I'm sure it's nothing that happens all the time (the joys of using React Native and Relay where everything is at an imaginary release state). But still glad npm 5 made it work consistently.

Same. One team is running npm@3 installs twice in CI to pick up half a dozen modules that don't install the first time. On another project I managed to massage the package.json to stop it from happening.

Do you any dependencies or subdependencies with git urls? I've previously run into https://github.com/npm/npm/issues/17189, and I solved it by ... updating the dependency to a newer version that switched away from having subdependency pulled from git.

Good news, I just looked at that bug again and it turns out it's no longer reproducible in the latest version of npm.

Same thing happened to me in multiple projects on multiple machines. Weird how that seems so common and still made it into v5.

I find less issues when I `rm -rf node_modules`, `rm package-lock.json` and `npm i` again... usually when I manually edit my package.json packages.

Lol mine has been oddly adding 1000 packages only sometimes when I npm install after a pull when we are using maybe 100 shits weird.

I was a bit disappointed about how npm5's new feature to install from a local cache works. You must initially populate the cache from a registry (npmjs.com or whatever) because it caches by HTTP ETags (as I understood by experimenting); merely installing from local .tgz files results in an empty cache. But the reason I wanted the ability to install from a prepopulated cache in the first place was that I had to work at a customer behind corporate firewalls and a private registry (Artifactory), where each and every external package package has to be vetted individually for license and security issues. Which kind of defeats the purpose given the excessive transitive package dependencies in npm-land (300+ packages for a basic webpack/babel/react setup, 500+ packages for building Bootstrap from source; and these figures are still comparatively low).

In exchange, you get a new cache directory layout with directories named after autogenerated hashes, and "npm cache list" isn't supported yet :(

But I'm not complaining; npm has certainly made progress, though it's still relatively slow.

I'm a big fan of Yarn precisely because it supports an "offline mirror" feature right now. In addition to caching downloaded packages globally, it can be configured to cache them in a local folder that can be committed directly to version control. That way, the local packages will be used when someone else clones the repo and installs things.

I gave an example of using Yarn's offline mirror option in one of my "Practical Redux" tutorial blog posts: http://blog.isquaredsoftware.com/2017/07/practical-redux-par... .

While I wish them success, npm-the-company has been a big disappointment. npm-the-tool is vital to NodeJS, and it's telling that big players in the industry had to bypass npm by releasing yarn.

who is backing yarn?

Led by Facebook with support from Google, Tilde, and the Expo team.

facebook

> npm link is weird right now

This has been broken for almost 2 years now.

https://github.com/npm/npm/issues/10343

It's unacceptable that the NPM team is moving so fast without looking back considering so many projects depend on it.

I understand their negligence, the fix for almost all npm link issues is to get a paid corporate npm account with private repositories.

I have simply given up on NPM, after it broke my project multiple times.

One of the bugs currently open references NPM deleting your dependencies.

Another bug had NPM delete itself just by running `NPM install`.

How does npm5 compare to yarn?

I didn't notice anything, but I'm using yarn everywhere, so...

Speed wise npm@5 caught up a lot of ground. Iirc it was the same a while ago. However, I always felt like the release was a bit rushed. I sticked to Yarn too tho.

This is so completely and utterly wrong, unfortunately...

Yarn takes 12 seconds to install all of our node_modules, npm@5 (latest) takes over 12 minutes.

This made it completely and utterly untenable for our CI system, so we switched to yarn and shaved 12 minutes off our CI builds. It's been fantastic.

That's got to be down to caching. What's the difference if the yarn cache is cleared?

25 seconds when the yarn cache is cleared... downloading is not the slow part here.

Old results from about a month ago:

https://twitter.com/bertjwregeer/status/887450964420055043

  npm 5.3.0: added 1991 packages in 538.289s  
  yarn v0.27.5: Done in 58.42s.

npm 5, prior version 5.3.0 was buggy, that's true. But now, it's working great for me and all my projects.

yes, npm5 is the first version I've had to rollback.

its strange that the lockfile was supposed to fix problems with un-repeatable installs, but I had several issues fixed by simply deleting the lock file and re-installing.

Yeah I've had that too.

My concern with the lock file is that it doesn't seem to be consistent across platforms (OSX and Windows) because of optional dependencies. I have a project with those (freaking fsevents) and every time I do npm install I it changes the lock file back and forth.

Yeah, at one point I was doing this many times every day.

  rm -rf node_modules && rm package-lock.json && npm install

I haven't run into any issues with the latest npm. I did run into issues with getting yarn to do a basic install.

Still can't get a debugger; statement to break... https://github.com/nodejs/node/issues/7593

It's amazing to me how blazé the Node community is about breaking the debugger. In other language communities that would be considered sacrilege, but in Node the response is often "meh... I just console.log everything". Transpiling, source maps, so many hacks that barely work.

Still disappointed that Node has backed out of optimized tailcalls (edit: should have written proper tail calls here, since that's what the past 3 now specs have called for, ES5, ES6, and ES2017). I think it's the only part they haven't implemented.

I think a lot of the pushback I'm seeing against TCO comes from traditional programmers who don't like functional programming and who think that recursion shouldn't play any role in production software. I get that there are also some who have legitimate implementation concerns, but the reflexive pushback TCO gets from a lot of the V8 maintainers compared to any other new ES6 feature seems telling to me (correct me if I'm wrong here).

What they don't understand is that JavaScript programmers are rapidly moving toward functional programming. Already, there's a heavy functional influence on the React community, probably the dominant frontend framework/library at the moment. Clojurescript and Elm have had an outsize influence on the JavaScript community, and I see more and more programmers using functional paradigms in their everyday code (particularly maps, filters, and reductions). I know we'd see much more recursion if it were technically feasible.

Edit: Absolutely right, my critics. I brain farted on this one. It's not Node, it's V8. I did refer to this in the second paragraph, but stupidly blamed Node for it in the first paragraph.

Edit 2: As I said, I shouldn't have singled out Node, particularly since it's V8 that's making this call. But my criticism was more directed toward most of the JavaScript implementers who seemed to have single-handedly decided that the es6 spec was all cool except for the one part they don't like. And I could accept the criticism that implicit tailcalls are problematic, except they seem opposed to labeled tailcalls as well. I think there are lots of implementers who simply don't believe that recursion has a place in "real" software. I've not heard these exact words from an implementers before, but I've known quite a few senior developers of my generation (40s) who have said this exact thing to me (they also can't seem to get the difference between structural and generative recursion).

>Still disappointed that Node has backed out of optimized tailcalls, which is part of the ES6 spec. I think it's the only part they haven't implemented.

Node doesn't implement JS features. V8 does and Node gets updated to newer V8 releases every while. Node doesn't have much say in whats gets to v8.

>I think a lot of the pushback I'm seeing against TCO comes from traditional programmers who don't like functional programming and who think that recursion shouldn't play any role in production software. I get that there are also some who have legitimate implementation concerns, but the reflexive pushback TCO gets from a lot of the V8 maintainers compared to any other new ES6 feature seems telling to me (correct me if I'm wrong here).

You're overthinking this. There is no masterplan and no pushback. Just some issues with the old standard -- v8 has even implemented the feature in initial form (but not enabled):

https://v8project.blogspot.gr/2016/04/es6-es7-and-beyond.htm...

> There is no masterplan and no pushback.

I don't think there's a masterplan or some anti-functional conspiracy, but I do think there's pushback. They implemented the feature behind a flag but from what I've read (can't find the reference now), they are not planning to ever release it.

Probably looking for this then: https://github.com/tc39/proposal-ptc-syntax/issues/22

But their pushback is for pragmatic reasons, not because they don't like recursion/tail calls.

> But their pushback is for pragmatic reasons

I saw zero reasons given for the failure to implement in that thread (which I had not seen before), other than a brief mention of stack traces (which is quite solvable with a shadow stack). All I saw were several people (including someone involved with the spec) who were bitterly disappointed about the failure to implement. And I forgot that tail call elimination was introduced back in ES5, and yet only JavaScriptCore has implemented it.

Which again leads me to believe that most implementers believe it's just not worth the trouble to them to implement a key feature for functional programming, regardless of how much the users want it (the feedback in this thread is similar to what i've seen elsewhere).

You can read the minutes here https://github.com/tc39/tc39-notes/blob/master/es7/2016-05/m...

Could you elaborate? Based on my experience with functional(-style) programming in Clojure and Elixir, tail calls are important enough that, worst case scenario, they might even warrant their own syntax addition (like with Clojure's recur?).

I might be way out of my depth here though; I couldn't quite grok the discussion in your link. Is there's a way perhaps you could 'ELI5' why it's pragmatic to ignore this issue?

It's pragmatic because adding proper tail calls in the JS engines is difficult and raises several issues (lost stack frames, memory guarantees, etc.) but we were able to survive 20+ years without having them in JS, so it's not like they are essential.

> we were able to survive 20+ years without having them in JS, so it's not like they are essential.

And voila the attitude I was referring to, it comes out. Recursion is pretty damn essential to functional programming, and like it or not that's the way JS is headed.

>and like it or not that's the way JS is headed.

Well, like it or not, recursion did not make it to ES6/ES7.

It did make it to ES6, but the engines failed to implement the spec.

Huh. No tailcall optimization? Then I don't understand something and have to recheck.

I just wrote a Redis REPL (wire protocol) parser using recursion. Being so simple, I wasn't worried about stack depth. But while stepping thru with WebStorm's debugger, I noticed the tailcalls were (apparently) being eliminated. An unexpected surprise.

Using nodejs 6.11.1, npm 3.10.10.

It's behind a flag now, but they're eliminating it, so it will not be released.

Yeah, tail call optimization has been in since node 6, I believe. _Maybe_ behind a v8 flag.

https://bugs.chromium.org/p/v8/issues/detail?id=4698

all engines except Safari's JavaScriptCore seem to have decided that implicit tailcalls optimization wasn't a good idea after all and as far I can see are not planning to enable it.

Yeah, they say that, and yet they don't want explicit tailcalls either.

iirc JSC folks are blocking explicit tailcalls on TC39

Well, not surprised, because they actually did the work and implemented the spec calling for implicit tailcalls, and then it turned out that the other JS engines were to lazy or biased to implement it themselves (mind, all the other parts of the spec were fine, but the one thing they disagreed with they just didn't implement).

So yeah, I'm sure I'd be pretty pissed if I were the JSC devs, and not so happy about the community turning around and saying now we want explicit tailcalls. And by the way, they're also probably smart enough to see that the V8, SM, and ChakraCore are probably not really going to follow through with explicit tailcalls anymore than they would with implicit ones.

Being explicit won't make the implementation any easier, it will just firewall off any performance issues (note that JSC achieved this with implict tailcalls). And since the other engines having been complaining against how hard tailcalls would be to implement, it's unlikely they'll do it regardless of whether they're implicit or explicit.

Does V8 handle them? If not how would we expect Node to do so?

You're correct, I updated my post accordingly.

I've been waiting for this DNS behavior change for years. Great release!

From the Medium post about speed improvements in this version:

> The size of a function, including it’s signature, the white space and even comments can affect whether the function can be inlined by V8 or not. Yes: adding a comment to your function may cause a performance hit somewhere in the range of a 10% speed reduction.

Uhh... byeee

I can't even think of a reason why a comment would cause a 10% speed reduction. Something has seriously gone very wrong there.

Agreed. What the absolute hell? This entirely contradicts how I understand programming languages work. Whitespace, tokens, formatting, should be irrelevant once it's all in an AST.

This is just a simple heuristic that's good enough for real code on the web. Usually whitespace and comments are stripped out during build step. I imagine this heuristic was implemented as something simple and working and that's that.

Remember that they are not writing a monument of perfect software but something pragmatic that should work good in most cases.

Nice.

Any noticeable performance gains in everyday use?

They link https://medium.com/the-node-js-collection/get-ready-a-new-v8...

I'm mostly glad it'll come with an updated WebAssembly, hopefully fixing this bug: https://bugs.chromium.org/p/v8/issues/detail?id=6204 which blocks Luwa from working in node (handwritten wasm has a tendency to run into things emscripten would not.. I expect the same vice versa)

Will also enjoy not having to use a TextEncoder/TextDecoder shim. Kind of weird that they went full out with Unicode there, last I recall browsers were moving towards only supporting utf8

> write high performance JavaScript

Does anyone with high performance requirements writes JS? Isn't that setting yourself up for failure on the first step?

Depends on your application and style of development. In my experience Node.js is almost always much faster than Ruby/Python and in some cases it's much faster than Java Spring type apps for certain types of network applications (mainly io bound web apps serving a lot of small requests, the event loop async nature of node really shines there). It's not C/C++ or pure Java or Go type of speed but for a lot of web application development? Node.js is great performant choice.

>Does anyone with high performance requirements writes JS? Isn't that setting yourself up for failure on the first step?

You assume that "high performance" is the only requirement, so that it dictates everything.

People also have other requirements like "it has to run on a browser/on the web" etc AND high performance.

It's not like HPC people suddenly rush to use JS. But there are people targeting the web/node etc that DO want as high performance as they can get away with (e.g. web games).

> Does anyone with high performance requirements writes JS?

It's possible to call into C from Node. So there's nothing stopping you from implementing performance-critical parts of a system with native code, and then invoking it from JavaScript.

Nothing except of it being messy and you ending up with two problems (to paraphrase JWZ).

Not when JS is fast enough, and if you want standards-compliant web games (like webgl), it's your only option.

There's a big number of startups that used Node for their back-end, if it becomes faster without them having to rewrite their codebase, that's a win too.

Yes, there are use cases for high performance JS.

We have JS that we use client side that we also use server side for Hadoop processing of data. So we didn't want to rewrite and maintain this large codebase in both Scala/Python and JS, so we ran Node on Hadoop.

No, it really isn't setting yourself up for failure if you do it correctly. Node really is quite fast; for our use case it rivaled Java 8 runtime in MR CPU time, the only downside is it had double the memory footprint. With enough hardware, memory is not too much of an issue.

>Does anyone with high performance requirements writes JS?

It bothers me that people still think JavaScript is slow. I think JavaScript(engines) is the only languages which is getting significant performance improvement on monthly basis since V8 was introduced.

It is the abstraction people in JS/Frontend world people are used to, make it seem slow.

It depends on the context of "high performance". For example, would a 400ms load time be considered "high performance"?

I always say use the best tool for the job. And this usually means getting the requirements then laying out the solutions and picking the best solution out of the bunch.

"Does anyone with high performance requirements writes [JS, Python, PHP, Ruby, Perl, Tcl, Shell, Scheme, Lua]? Isn't that setting yourself up for failure on the first step?"

And yet there are a plethora of cases where all of those languages have been used with high performance requirements.

Yeah, that's my point. I'm not hating on JS or dynamic languages but to me it seems a stupid decision to go with that kind of language if you really need performance.

That's assuming you ONLY need performance.

How about you needing performance AND web compatibility?

Walmart, PayPal etc. are doing fine running large scale apps on node.

Scalability is not performance.

Yaasss! New pipeline of Ignition and Turbofan provides huge performance increases.

Read this article -> https://v8project.blogspot.com/2017/05/launching-ignition-an...

From the first chart in the link, I can't help but notice reddit is now slower with Turbofan + Ignition. It probably has something to do with a JS Dev who deeply understands how AutoCodegen works and driven the project accordingly.

I might say that some parts of code can perform differently depending on the style of source code.

I have run some very simple tests and performance is very close for real world complex apps, ~10% faster for an express hello world and ~10% slower for http stack only.

Basically TF is at parity on average but slower for code like the node core that was highly optimized to the particularities of crankshaft.

Hopefully npm 5.3.0 actually works since 5.0.0 was completely broken for me.

v7.10.0

Test suite for real world app

real 0m29.669s user 0m5.376s sys 0m0.356s

v8.3.0

Test suite for real world app

real 0m30.180s user 0m5.420s sys 0m0.220s

v7.10.0

Express Hello World

Requests per second: 6751.08 [#/sec] (mean)

HTTP Hello World

Requests per second: 13407.31 [#/sec] (mean)

v8.3.0

Express Hello World

Requests per second: 7734.33 [#/sec] (mean)

HTTP Hello World

Requests per second: 12601.42 [#/sec] (mean)

I wonder if this will lead to Electron using less RAM... nah.

It's the Chromium part of Electron that uses that much RAM.

Yea? But only because the old V8 had Crankshaft running in parallel. New version has this disabled IIRC.

Reply on news.ycombinator.com