Node.js Compiler – Compiling your Node.js application into a single executable | Hacker News Reader

Node.js Compiler – Compiling your Node.js application into a single executable

github.com

115 pointspmq2082 comments

52 MB and 32 MB for macOS and Windows, respectively, for anyone looking for the important numbers.

Not too bad; with Qt, you can get to about 10 MB - 20 MB for a dynamically linked application.

how does that compare to the numbers for pkg, that Zeit just released and was on HN a few days ago? https://github.com/zeit/pkg

About the same 30MB for a hello world application.

Hmmm... why the heck did Mozilla kill xulrunner? It's a few MB less ;-)

Just wanted to mention a similar project, which was recently announced by Zeit, called "pkg": https://github.com/zeit/pkg

I'm really happy with the effort of "compiling" NodeJS to native binary.

What I'm looking forward to is lightweight electron that will allow us to use the native browser-engine for each platform.

QT/GTK for Linux, WebkitView for macOS, Edge for Windows.

Hope the effort of node-compiler, zeit/pkg will be beneficial for the future of desktop apps.

Here's proof of concept from yesterday: https://news.ycombinator.com/item?id=14260755

One major advantage of Electron is not having to deal with browser compatibility issues - you write your code against exactly one runtime (sure, you may need platform specific flags for things - but no risk of the runtime changing/breaking after you deploy)

Leveraging existing OS engine will reduce file size but at the cost of not having a stable and known platform that your code will run on.

These days fixing those issues should not be a big deal.

Having all those polyfills / babel / webpack tools you can be quite safe about your app.

Nevertheless having a nice test suite also makes a benefit.

I can't help but think that electron could eventually become similar to Adobe Air (except actually open), where the main runtime is/can be installed separately, so multiple apps can share a runtime.

Please don't ever do this. No one likes installing separate system library of some sort, even multiple versions of it besides downloading the app.

Besides what's the point when most people would only use just a few Electron apps anyway.

App developers also have to ask users to download versions between this and that system library only causing confusions for users.

The comparison table fails to mention one huge advantage zeit/pkg has: it is able to generate executables for all three platforms from any one of them, no need to setup three dev or build environments.

Cool tidbit found via this: there's a FUSE driver for SquashFS!

https://github.com/pmq20/squashfuse/

50 MB executable?

There are quite a few minimalize Blink efforts. Like this one:

https://github.com/bloomberg/chromium.bb

https://github.com/weolar/miniblink49

The latter is around 9MB. The author is working on a electron wrapper.

Does this project include Blink? Looks like not.

It contains a stripped version of Blink with most of functions.

Blink is a rendering engine, V8 is a JavaScript engine. Node.js contains V8, but not Blink.

It's a naming thing, it does include v8 to execute js.

Blink is a fork of webkit, by your logic, most softwares based on webkit can not execute javascript???

No, Blink doesn't include V8, it's a separate project. WebKit consists of WebCore — layout engine, and JavaScriptCore — JavaScript engine. Google forked WebCore part of WebKit to create Blink. Before the fork it used a combination of WebCore and V8.

This information is readily available on, e.g. Wikipedia, so I'm not sure why you even try to argue about it.

Node.js doesn't include Blink, it only includes V8. Node.js Compiler doesn't include Blink into the build result — it includes Node.js.

Is that honestly a problem? Most Java applications I encounter is much larger than that.

50MB is nothing and if it enables you do have a single standalone binary that you're easily able to deploy I'd say it's worth the extra disk use.

I thought that was a totally reasonable file size too, until I looked up Docker image sizes for node:

https://hub.docker.com/r/library/node/tags/

Regular images are ~260MB and "slim" images are 85MB, but Alpine-based images are a paltry 18MB.

So it seems something in the Node.js compiler is ballooning up the size of the code, by a factor of about three. I wonder what.

It really depends on version of node, and the modules used. I'm more inclined to want to see what can be done as a minimal electron... or seeing electron go the adobe air route, and have a shared (continuous updating) base, but that could prove problematic as binary modules (namely sqlite) are common.

AFAIK Alpine Linux is using musl which would explain the low binary size.

Depends, really. If the target is a common OS then having people install Node.js itself plus grabbing your application as a zip file that has the code and a few different files to act as soft executable (myapp.bat, myapp.app, etc) can be significantly smaller. 50MB is not "nothing" when it comes to asking people to download it (because someone still has to download that application, and way fewer people than you think have as nice an internet connection as you). It also means people have to download a _new_ 50MB executable when the app updates, so from an "on disk size", it's nothing only if this is the only node application someone will ever run. Otherwise this becomes bloat very fast. Like bundling a full JVM with every individual application instead of going "or download this much, much smaller thing when you already have java installed". I don't need eight copies of the JRE, in the same way I don't need eight compiled apps if I already have node.js -- but that's a consideration for people who make applications and are thinking of packaged distribution. Just because you can, doesn't mean you should. Or even "should only".

You need to download the runtime at least once. If that 50MB is a problem, you can't use it in the first place.

Who likes to download Java on desktop anyway? It's absolutely the wrong way to distribute Electron apps.

It's so annoying for App developers that they have to ask what version of shared Electron the user has and install that for themselves and debug instead of just be done with it with the exact version developers tested and distributed.

I am wondering if using some of the front-end techniques for back-end would be beneficial. Tree-shaking, minimization and gzip would probably help with the binary file size. But as we go there, maybe some more advanced things like the closure[1] compiler or prepack[2] would make it faster.

[1] http://www.syntaxsuccess.com/viewarticle/closure-compiler-vs...

[2] https://prepack.io/

> But as we go there, maybe some more advanced things like the closure[1] compiler or prepack[2] would make it faster.

And then one could probably go even deeper in search of performance and instead of compiling JS to JS, compile it to C or even straight to native CPU assembly.

AOT compilation of js is probably not worthwhile.

Yeah. Like all those compilers and runtimes for the back-end languages we were already using before JavaScript ate everyone's brains, that served as the inspiration for all of this madness on the front-end side.

Back end is about stability (= don't do fancy). So unless it increases performance like maybe prepack does, I don't see much point.

How is this different than zeit's recently released pkg?

https://github.com/zeit/pkg

The readme says this:

> Pkg hacked fs.* API's dynamically in order to access in-package files, whereas Node.js Compiler leaves them alone and instead works on a deeper level via libsquash. Pkg uses JSON to store in-package files while Node.js Compiler uses the more sophisticated and widely used SquashFS as its data structure.

More sophisticated is not a great way to suggest something ks better. That said, I could image this probably does give better performance if you have a lot of I/O with files in the package.

Maybe they weren't saying that is necessarily better?

Yeah I didn't read it that way either, just a matter of factly "here are the differences" – that section probably could be expanded and clarified.

it would be awesome if someone creates a compiler for javaScript.

I mean an actual compiler rather which is much more optimized than node environment rather than just making things into executables

V8 includes JIT compiler. Could it really be "much more" optimized?

I agree V8 does an awesome job buts it compilation phase is not done separate before running. Lot could done to gain the speed in those areas.

Perhaps. JIT compiler has to care deeply for compilation time. For AOT compiler compilation time is a little bit less pressing problem which leaves some space for optimization passes that take too long for JIT.

If you look at Hiphop vs. HHVM there is certainly a significant performance upside in taking advantage of clangs or gccs ability to produce efficient code.

I'm not sure why Facebook decided to stop maintaining hiphop. Perhaps manageability wasn't great.

there is always something more that can be done

This would be pretty useful. Something like hiphop which could transpile into C or C++ would mean we (IncludeOS) could build tooling to run a node application on directly in a vm without a underlying OS.

What kind of thread support does IncludeOS offer?

Edit: IncludeOS does not offer thread support, so it's unclear how the GC of the JS VM is supposed to run.

Arguably there are already couple of popular compilers [0] for JavaScript (Babel, Closure Compiler, minifiers etc.). They just don't target native CPU assembly.

[0] Compiler is just a program that transforms code from one representation to another.

Why, thank you, Technically Correct Man!

http://www.damnlol.com/the-adventures-of-technically-correct...

This is a fantastic comic. Technically correct -- the game of the learner. Pragmatics -- the game of the master.

Nice quote!

Feel free to.

But a compiler for what? Plain JS without any APIs? JS that uses the Node.js API? JS that uses the Web APIs? The problem with base JS is that it's a general purpose scripting language that lacks some of the basics that you need before you can compile the code as something that can "do a thing" on its own, without still needing an interpreter for the third party APIs the code tries to make use of that rely on out-of-language hooks (like Node.js)

Is there any reason to compile your JavaScript except obfuscating the code !? It will most likely run slower, because of no runtime optimizations.

There is no real compilation going on here, V8 is bundled in the executable.

squashfs ? maybe docker is a better alternative - especially with the new 1.13 experimental feature "--squash"

https://blog.docker.com/2017/01/whats-new-in-docker-1-13/

  First install the prerequisites:

  SquashFS Tools 4.3
  Python 2.6 or 2.7
  Visual Studio 2015 Update 3, all editions including the 
  Community edition (remember to select "Common Tools for 
  Visual C++ 2015" feature during installation).

Why not just Node.js and Git for Windows as requirements? Python 2 and a specific VS version, really? Is using npm command to get further packages too uncool?

Because the project is based on SquashFS, and the standard Node build tool is GYP, which requires Python 2?

This is risky to say but... NPM is a bad package manager. I could blab about why forever but tons of other people already have. I've never used a popular language with a package manager as bad as npm.

It's not risky to say. It's risky to do a shallow critique with no solid analysis. I would personally enjoy hearing a critique of why npm is bad in comparison to other package managers.

As someone who has spent a lot of time with npm (not even as a webdev), it appears to solve all of my common use cases while also dealing impressively well with various versions of specific dependencies. (Like, A depends on B@0.0.1, C depends on B@0.0.2, etc.)

There's a ton of annoying little stuff (like doing nothing to prevent path names too long for your filesystem) that could be forgiven for being young, but the big problems are in its architecture.

Node module syntax is non standard and can't be tree shaken, with a little more forethought it could have been designed to support static analysis and they wouldn't have needed to reinvent the module syntax for ES6. You can't reliably tell if a piece of code is using a given package until you run it.

The dependency resolution sucks. It's better now that they finally made it so nested dependencies aren't duplicated, but why the hell do I have to run a manual dedupe command to do so? What that tells me is that npm doesn't actually track dependencies, its hardly more than a fancy file downloading tool.

The repository curation sucks. I'm not sure if you've used Python's package manager, or Java's, or C#'s, or Perls, but npm has zero package curation and is filled to the brim with garbage compared to the others. I have published packages for C# and Java and the semantics are well defined and reasonable, you even cryptographcally sign your code and the system checks to make sure you upload documentation etc... In the case of java, which has the best system I've seen, they even use something similar to DNS validation to make sure you control package namespace.

The above point is the absolute worst thing about npm and drawfs the other problems. Let me give an example by pasting the package publishing guidelines for Maven(Java) vs npm:

Publishing an npm package: >Use npm publish to publish the package. >Note that everything in the directory will be included unless it is ignored by a local .gitignore or .npmignore file as described in npm-developers. >Also make sure there isn't already a package with the same name, owned by somebody else. >Test: Go to https://npmjs.com/package/<package>. You should see the information for your new package.

I had to condense the Java guidelines but it's roughly this: create JIRA account apply for access review requirements verify namespace ownership generate crypto keys package coordinates/versioning project description URL and license info developer info SCM info package deployment generate binaries revolve package dependencies crypto sign package generate documentation online verification

Sounds like a lot of work right? It takes a few hours to setup and verify your domain/keys the first time but after that all of these steps are automated. The signing/building/doc generation/ deployment is totally automated. Npm is a toy in comparison.

from their docs: Why Do We Have Requirements? In order to ensure a minimum level of quality of the components available in the Central Repository, we have established a number of requirements your deployment components have to meet. This allows your users to find all the relevant details about the components from the metadata provided in the Central Repository. The following sections will detail these requirements.

Left-pad is a perfect example of why npm sucks. Nuget, Maven, Phython, and CPAN would have never let something that stupid happen.

> Left-pad is a perfect example of why npm sucks. Nuget, Maven, Phython, and CPAN would have never let something that stupid happen.

O RLY?

* https://mvnrepository.com/artifact/coldnew/left-pad/1.0.0

* https://www.nuget.org/packages/left-pad/

* https://pypi.python.org/pypi/left-pad/

* http://search.cpan.org/~dagolden/LeftPad-0.003/lib/LeftPad.p... (LeftPad - Why should Node.js have all the fun? - comments on the package)

-- UPDATE

About package removal...

* http://stackoverflow.com/questions/20403387/how-to-remove-a-...

I assume they're referring to the prior ability that NPM packages could be removed at will.

At least first one is a parody: "This library is clojure/clojurescript port of left-pad, it's NOT recommand to use this library actually :)."

And by date of project creation you can assume that others are parodies too: "LeftPad - Why should Node.js have all the fun?"

All of those left-pad libraries are a jest, it's trivial to do padding in those languages. I was wrong about Python and removing packages but on Nuget and Maven you can't, its not allowed.

I think it's worth noting that with Java at least, besides not allowing package removal, the namespace is tied to a unique identifier with ownership verification, so the cause of the left-pad fiasco (kik wanting his namespace) is extremely unlikely to happen in the first place.

If the curation process allows joke packages to pass through, it is completely useless. Looks like the only difference from npm is extra bureaucracy then.

Awfully vague/FUDdy criticisms there:

- dynamic require()'s can't be tree-shaken: same is true of other langs, e.g. Python. Perfectly understandable IMO as Node was designed for the server, in a time tree-shaking (largely desirable for front-end code) wasn't a priority. Also it is a standard (CommonJS).

- long pathnames is a Windows only issue and afaik fixed by npm 3's "flattening".

- npm is worse than Maven because 'npm publish' is too easy?

- you don't need to run dedupe.

- you say npm isn't curated but I'm not aware those other registries are curated either?

- 'left-pad is a perfect example of why npm sucks' ... except the ability to delete packages has been fixed, so should that be 'sucked'?

(Edit: how on earth do you format lists correctly on HN?)

> Left-pad is a perfect example of why npm sucks.

Perfect in that it was fixed over a year ago?

One of the basic features expected of a package manager is reproducibility (i.e. if two people build the same code with the same dependencies, it should produce the same result). On that count, npm is literally broken by design:

https://github.com/npm/npm/issues/10999

Well no, installing 2 packages, one after the other from the command-line (as in that issue) is not the way you 'share builds' with npm. You do that via a package.json + 'npm shrinkwrap' "lockfile".

Shrinkwrap does have known issues (mainly w.r.t. platform-specific modules and verbosity), though I've never had any problems with it, but if you want a better lockfile I suggest the Yarn client.

(Believe Python's pip behaves similarly to npm in that issue).

I prefer Yarn's lockfile handling, but saying npm is 'broken by design' is wrong.

There are evident security issues with this, as spelled out in the bug. Yeah, I'd consider that broken. Of course the repro is contrived, but it's not at all unusual to install some packages manually. The point is that once you do so, you might not get the same result later from package.json, either.

And yes, Yarn is the sane option for package management for Node. Why isn't it the default one yet?

> but it's not at all unusual to install some packages manually

If you're taking about deployments, that's definitely unusual. Outside of deployments, what's a scenario where deterministic builds are important, but it would be considered normal to manually install them anyways?

I don't see why deterministic builds would only be important for deployments. Even ignoring the security issues here (which don't necessarily apply only to deployments), there's this whole classic "works for me, not a bug" situation.

> I don't see why deterministic builds would only be important for deployments.

Great, then you should be able to mention at least one solid normal usecase as an example. I don't think there is one.

Most of those critiques boil down to "it works differently to how I'm used to" or "I don't know how to use it so I'm going to criticize it poorly".

NPM has issues but most criticism of it is really complaints about bad dev practices (like not pinning versions correctly).

One other one is that it has hilariously poor performance, and bizarre failure cases. For some forms of network failures it'll just say "callback called too many times" and crash.

There are literally no worse major package managers.

A good package manager can be misused and still work reasonably well; a bad one doesn't even work all that well when used correctly.

Or how about this for error reporting:

https://github.com/npm/npm/issues/9633

I especially like the part where it says "This is most likely not a problem with npm itself", even though it clearly is a problem with npm itself (as evidenced by non-deterministic nature of the bug).

That message is their generic message because ANY error ended up being reported in github as an issue with NPM.

Again, bad dev practice.

More like it doesn't work as well as any other package manage I've used :)

It works better than Pip and the whole setup.py mess. (Pip doesn't even have a dependency resolver: https://github.com/pypa/pip/issues/988)

It works better than most I've used - especially most of the built in OS-level ones

You clearly haven't played with PIP then :)

How many projects have you seen that are agnostic about VS versions? VS doesn't maintain a stable C++ ABI between versions, not to mention the fact that VS2017 has broken a fair number of projects. It's not like pinning a clang/gcc version.

Reply on news.ycombinator.com