Show HN: Test your JavaScript modules simultaneously in 32 different versions of Node.js | Hacker News Reader

Show HN: Test your JavaScript modules simultaneously in 32 different versions of Node.js

victorbjelkholm.github.io

142 pointsdiggan61 comments

This is a utility I build to be able to make sure my JS CLI's actually work across the version I want to support.

Right now, the tool also works across any language where there is docker images with versions, not just for JavaScript/NodeJS. It's enough with having a Dockerfile with $VERSION (that will be replaced by the tool) and changing the test command. I will make this easier in the future.

Thanks for taking a look and please let me know what you think about it.

Just out of curiosity, can't this be implemented with `nvm` and some sand boxing? Why did you choose to use docker?

Yes, it can indeed. I have three reasons for not doing it though.

The first is that nvm is nodejs specific and since I knew I wanted it to eventually work with multiple languages, nvm is out.

Second is that docker has a API I can work with (via Dockerode in this case), makes it easier to use/debug and such.

Third is the easy of isolation. It's basically what docker is about while nvm is more for just managing versions.

I still use nvm locally, when developing. But when I want to rapidly test in multiple versions, I use docker with autochecker.

This is great! Nice work.

Dumb question, from someone who doesn't work with Node.js: Why are there 32 versions of it?

There are actually 303 versions of node (including iojs, according to the Node Version Manager):

    $ nvm ls-remote | wc -l
         303

Most of those are patch releases. There are 47 minor releases and 6 major releases:

    $ nvm ls-remote | grep -oE 'v\d+\.\d+' | uniq | wc -l
          47
    $ nvm ls-remote | grep -oE 'v\d+' | uniq | wc -l
           6

Most language implementations probably have similar number of releases.

yeah -- perhaps there's a lot of negative things to be said about node, but the fact that they've released multiple versions? really?

https://clojure.org/community/downloads_older http://www.scala-lang.org/download/all.html https://www.python.org/downloads/

etc...

I don't think the issue is that they've released multiple versions, it's more that they've had so many backward-incompatible releases.

Python and OCaml have had 3 each. Libc has had 6. 32 for something that's been around less than a decade is a bit excessive.

Are all these 32 actually backwards-incompatible?

There have only been 3 major releases with known backwards-incompatible changes. The reason you would want to test your code in so many versions of Node would be to check that you aren't depending on newer features that aren't in older versions that you want to still support.

Testing across 32 versions is probably very overkill for most people though.

(Oh, I meant to say 3 major releases since 0.10, which I arbitrarily picked as what seemed to be the earliest version still in any significant use today.)

They're not strictly backward incompatible: one should ideally only have to test across the major versions (nodejs is currently on major version 5, though that's not strictly reflective given the fork to iojs, and the change in versioning system on merge back)

Testing across all 32 (or 303) versions is still going to be useful for catching bugs and quirks between minor versions (as opposed to deliberate, documented breaking API changes)

actually for the sake of correctness, you should also test version 4 which is lts (if you are a library author) (https://github.com/nodejs/LTS) I mean v0.10 and v0.12 are also LTS but I doubt that if you have a bigger library that its nearly impossible to support all four

>Are all these 32 actually backwards-incompatible?

Node uses semantic versioning [0] and the current release is 5.10.z

The breaking changes can be found here [1] By searching for `SEMVER-MAJOR`. Some seem rather unavoidable for progress (updating dependencies...) and some are for things that have been deprecated. I'm not an expert but it looks like there are several security related changes as well?

In my eyes a backwards incompatible change that breaks a feature that has been deprecated for years or is insecure is a change worth making. Stability is bad if it means remaining insecure.

[0] http://semver.org/

[1] https://github.com/nodejs/node/blob/v5.10.1/CHANGELOG.md

If you fix a bug that I'm relying on for the correct behavior of my code then it's not strictly backwards compatible.

These versions include a lot of minor and patch releases. Chances are most code will run just fine on most of these versions, but if you can automate testing them all then why not do it?

That is absolutely true, 32 versions is not all versions of node, but it's the number of tags from the mhart/alpine-node[1] docker images that autochecker is using for the testing.

[1] - https://hub.docker.com/r/mhart/alpine-node/tags/

Semantic versioning -- basically each version here is x.y.z for some x, y, z. Node patch releases (.z) happen reasonably fast and constitute a "version of node" as used here.

Even though Node isn't a framework (it's a cross-platform, runtime environment), it is used a base for application development. However, over time, things are changed and improved, but that breaks some of the API functionality upon which some existing applications are already built around. Since these applications still run fine with a previous version, they are not likely to be constantly updated to work with later versions. Therefore, previous versions are still offered to maintain compatibility with applications that have already been developed. However, if you're starting work on a new Node application, then you are better off using the latest version.

Node.js has that 60's free love nudist colony experiment feel.

Except that everyone is crazy and high off the fumes of DIY

...actually that does sound pretty 60s

I'm more intereted in why would you need to test against 32 versions of node.js? Doesn't it maintain backwards compatibility? Shouldn't testing against one version cover it?

I would think the more fundamental question is -- are these 32 (or 303) different versions so non-backward-compatible that the need to test on all of them is just an accepted thing out in Node.js-land? That everyone just kind of shrugs and puts up with (the way MS Windows used to crash on a near daily-basis for nearly everyone, not too many years ago)?

I know that other languages also have issues in this regard -- but for more mature languages it seems to be more an exception than than the rule (restricted to certain platforms of subsystems, say).

Very nice project. I built something similar for PHP: https://github.com/vectorface/dunit

Oh, that's very nice. Thanks for sharing this here, always nice to see how different people implement the same thing basically :)

I've implemented something similar to be able to run tests with different MongoDB versions: https://github.com/variety/variety/blob/master/test.sh

This is definitely something that's needed, by myself, and many other developers. Determining module compatibility across versions is always a pain. Great tool!

Thanks a lot for those kind words!

Yeah, it was needed by myself as well, so I had to scratch my own itch for a while.

Thanks everyone for the feedback and discussions, been very helpful and also motivating to see that people like/dislike the project in different ways!

As a small update, I've now made autochecker completely language agnostic, so you can test basically anything you can put in a docker container. There is some examples on how to use this here: https://github.com/VictorBjelkholm/autochecker/tree/master/e...

Again, thank you all for taking the time to give feedback, I'm forever grateful to the HN community.

Have you thought of running this as a service? Would be nice to just through an NPM module at it and make sure it will run across multiple versions. Without having to install docker, etc. Not sure I'd pay for it, but maybe somebody would...

If I could do it open-source and still earn money on it to cover cost of hosting, I'll try. Still haven't figured out a way though.

Someone else is working on something like that: https://twitter.com/bahmutov/status/720316267173810176

Other languages have this sort of test service available for open-source modules. One example is cpantesters.org, which is integrated into Perl's package repository. Not only is it easy to check what passes in your environment, it also makes it easy for module authors to fix bugs in environments that the author has no (other) access to.

Funny you should say so, the origin of the idea behind autochecker, is actually CPAN's and CRAN's way of automatically test packages.

Awesome job! Can't wait to try it.

An aside - over the last few months I began running my containers on two popular CI services and have had some serious pain. I have a slight feeling you will regret the words: "works well with CI as well!"

Hah, we'll see! Did some initial testing on Circle, Travis and Jenkins but still have a lot to test left.

Maybe I should be more careful with my words in the future...

A few months ago, I wrote something similar with the goal to test on multiple Linux distributions: https://github.com/jgillich/chimera

It however doesn't execute the containers simultaneously, that is really cool!

Would you implement binary search? A feature like git-bisect could be usefull to find breaking changes.

Haven't thought about that but it makes a lot of sense to have that. If you don't mind, could you open up a issue on Github, explaining how you can imagine it to work to help you the most? https://github.com/VictorBjelkholm/autochecker/issues/new

Would appreciate that a lot and thanks for the feedback.

I might give it a go. That was one of the first things I thought of!

By all means, please do :) Would be very interesting in having this as a feature.

I can't help it - the first thing I thought of was doing a mashup of this and leftpad as a service :)

Finally... a place to make sure my left pad module works cross-version.

Nice one Victor :)

Frankly, I feel like this is an indictment of all software development, everywhere. When you need to do this kind of a thing, you are doing it wrong. I do not mean I am doing it any better, I have the same issues e.g. working on Android, but there are probably some fundamental things we could be doing to avoid all these horrible symptoms. Things like NixOS come to mind, where people are actively researching and experimenting with possibly better alternatives. I pray somehow somebody smart and motivated can find some fundamental solutions instead of this rube goldberg overkill type answer.

This tool is the solution. I don't think you're going to live in a static would anytime soon, where programming languages and packages do not change over time.

Wouldnt the only solution to "see if you code worked with changing apis" to either

1) never change apis even if you were wrong or you have more opportunities or 2) know everything beforehand and implement the api perfect

also 3) have enough time to implement all the features after having this magic insight. A lot of people take for granted that successful software makes tough choices to ship

I see where you are coming from, for sure. I wish I was smart enough and motivated enough to try to come up with a solution to this, but my motivation goes elsewhere.

Sometimes, for you to focus on other (what for you is important), you need to throw together some rube goldberg thing to be able to move forward.

What do you recommend for how we could do it better?

Take a look outside of the node.js micro cosmos.

Python has a great tool called Tox for testing on multiple python versions: https://tox.readthedocs.org/en/latest/

The same situation applies in all language ecosystems, there is nothing exceptional about Node in this regard.

That is exactly what I thought when I saw what had been posted. I have a feeling that we have a generation of developers whose work makes sense on a small scale or in the short term, but becomes ridiculous with growth and/or age.

This is why I hate ES6. JavaScript used to be a write once, run anywhere language.

We sacrificed that for syntactic sugar.

JavaScript was never 'write once, run anywhere'. There were quite a few differences between the Netscape and MSIE implementations during the first few years of its existence and things didn't stabilise until the IE5 era.

Sure there are always obscure bugs, but if you stick to core JavaScript they are rare. And you can work through them.

If a module is heavily using ES6 and you want to run it on an older runtime you're just toast.

Yeah this comment literally made me LOL; life was hard in the late 90s

You could write vanilla JS and your code still could fail in some versions of node because of bugs or breaking changes. And in browsers, the JS language is the same, but the implementation of the various APIs has varied greatly.

So ES6 is not the problem.

Then just write es5. It'll still run everywhere forever.

Except <= IE8, of course.

The problem with nostalgia is that the past we so fondly remember never existed.

That's my plan. But it means I can't use other people's modules anymore.

Why? Put a transpiler inside your build chain.

This almost always leads to broken debugging, which I think is a bad tradeoff.

Most node developers seem to not use a debugger, but I think that's a huge productivity sacrifice.

Reply on news.ycombinator.com