From Node.js to Go | Hacker News Reader

From Node.js to Go

blog.scaledrone.com

168 pointskiyanwang83 comments

This blog post feels very empty. No code snippets? No actuall before after benchmarks/graphs? How can you state anything without showing numbers?

Yep. I was looking for same.

I can confirm some of the statements in that Blog as someone using both also in production. Memory footprint is a lot lower than node like 5-10x times what I've observed on similar code. Go has also very efficient GC, that can be used on large heaps with low latency.

I think would be cool to know at least app type. For something that has a ton of ws connections node will generally have smaller memory footprint.

We are using Go for a lot of internal tooling, text processing, also for websockets and web API. Can you provide some more data about memory footprint in node with websockets? How much memory per connection you saw? From my experience it's the other way around. But the most important point is that while both can have a lot of connections active, when you start sending messages thru that websocket connections you will see a LOT higher latency in node.

after looking at it you are right it is higher for node

Always expecting something like https://aphyr.com/posts/329-jepsen-rethinkdb-2-1-5

Hoping for? Yeah. Expecting? Rarely.

Agreed. I posted a comparison between Node and Go UDP socket performance a few months back with some code snippets. My submission was completely ignored. HN can be weird sometimes, huh.

Edit (and shameless plug): http://assil.me/2015/11/07/roadomatic-node-vs-go.html

You might need to read the previous post (and only other one!) to understand how the GeoJSON geospatial query works.

There was an analysis done recently that showed a few votes very early on (I think 4 may have been the number quoted) is often enough to bump you to the front page. Have ten developers on staff, each with an HN account? Some might see that as a free ticket.

At best it's only a very temporary traffic boost, though, unless the content is legitimately interesting enough to warrant return visits.

> Have ten developers on staff, each with an HN account?

I think that's what HN calls a "voting ring" and attempts to detect and counteract.

I don't think it can be detected the first time its attempted with any particular group of users.

Yep, that's true. I got around 5000 page views from Reddit over 2 days, but then the traffic died off.

I've noticed that posts to the same link can be made just hours apart or with a slightly more interesting headline -- while the former gets just a couple votes, the latter makes it to the front page. It's a bit strange.

What's strange about group A reacting to title A differently than a group B reacting to (potentially very different) title B?

may you share your url here ? :)

It's in his submission history.

http://assil.me/2015/11/07/roadomatic-node-vs-go.html

In particular I wished they would have elaborated on the claimed instability in x/net/websocket.

I can't speak to instability, but I had to rewrite a massive stack because gorilla gave me callbacks to execute on control packets and x/net/websocket didn't.

If you have any sort of long lived mobile connection, you will encounter half closed sockets due to mobile network quirks. At scale you really want to clean these up. To do so, you need an application level timeout and you want to listen to the PING/PONG control frames to reset that timeout.

> If you have any sort of long lived mobile connection, you will encounter half closed sockets due to mobile network quirks. At scale you really want to clean these up.

Could golang.org/x/net/context.WithTimeout be used to handle this?

https://github.com/gorilla/websocket#gorilla-websocket-compa...

Welcome to the carGo cult :^)

I use Go and Node in production. Node is used for cold paths and Go for one of hot paths (where GC latency is still not a problem). Go for sure is more readable than Node callback hell and i really like more writing synchronous like code inside gorutines than do everything around callbacks/promises. Static typing and compiler is also big + for Go.

correct me if I'm wrong but in Node you could use something like a Generator + Promise combination to achieve synchronous looking code with no callbacks

https://eladnava.com/write-synchronous-node-js-code-with-es6...

In practice a lot of libraries don't support promises, so in my experience you'll just end up with a codebase with callbacks and promises mixed. Not very pretty.

pretty sure they're part of ES6 spec[0]. If you're using a 3rd part library that supports callbacks only that sounds more like a choice rather than a limitation with Node.

[0] http://www.ecma-international.org/ecma-262/6.0/#sec-promise-...

Any library that accepts a standard node callback can be easily promisified.

See:

http://bluebirdjs.com/docs/api/promise.promisify.html

http://bluebirdjs.com/docs/api/promise.promisifyall.html

Now compare that kind of patching with writing Go code where non blocking I/O doesn't require any special syntax. I'm not fan of Go by a long shot, but it is undeniable the blue/red function problem is a pain [1] when dealing with JS.

[1] : http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...

I love that blog post, because then this makes sense: Promises are basically an inner-platform [1] that replicate all the control flow constructs of the given language, except as a different "color". Go simply is that inner platform. Go does not use promises because all of its code is already running in the promise color. You don't need anything special to "deal with" a for loop that may end up waiting for something to be read from a socket, because the for loop is already a "promise", and so forth. You can rearrange your code as much as you like, and you are not constantly juggling with stuff the compiler ought to deal with.

Manual promise transpilation is better than manual callback transpilation, but it's still manual transpilation. Why are you working so hard?

Sometimes I get mistaken as an advocate for Go. I'm not really. What I am an advocate for is using languages where you don't need an inner-platform of promises because the language is already that color. Go just happens to be the most familiar Algol-esque of the bunch right now. Don't want to learn Go? Go learn Erlang or Elixir. Or maybe if you're feeling saucy, Haskell. Or some of the other choices popping up lately. If you're writing network servers you will not be able to understood how you ever functioned without this.

[1]: https://en.wikipedia.org/wiki/Inner-platform_effect

Promises and callbacks are equivalent, it should be pretty trivial to convert all libraries of one style to the other just by wrapping calls to it in a utility function of some sort (and there are probably existing libraries to do exactly that).

The only excuse for mixed-style codebases is a codebase in the process of being converted from one style to the other.

Generators are sweet, but you can use just Promises or the async.js library to write nice flat code. No callback hell / play button shaped code.

While they allow limiting the rightsward drift of code, explicit promises are still event emitters and still require a significant amount of callbacks if you don't have generators or async functions.

The nice thing about the generator is that (with a nontrivial amount of boilerplate!) you can pretend that the body of the function* is a synchronous zone even though it makes several calls to asynchronous promises.

The basic idea is that you wrap the generator in a sort of trampoline that turns it into a promise. This 'trampoline' detects whenever the generator falls down to it (via some `yield` of some Promise) and then waits on that promise before resuming the generator-trampoline combo. The promise then returns its result directly into generator.next(), where it becomes the value of the `yield` expression, thus within the generator `x = yield myPromise()` creates a new Promise by calling myPromise, suspends execution until that Promise completes (by dropping down to the trampoline, which bounces us up into the myPromise().then()), and finally gets the value and assigns it to x.

You have just one scope, the scope of the function* of the generator, in which all of the results of all of your Promises live.

Sure but you need to know that you will pay for this abstraction with some performance penalty (depends what your expectations are). Another thing is this workaround is still more complex because you need to create generator function and then you need to write reentry points, you don't need in Go. Anyway this is for sure good solution for Node but in context of comparing Go is still in my opinion easier to reason about.

this is interesting. could you show (or point to a good example) of how GO handles this w/out reentry points? I mean you have to tell it where to return the results somehow

It really depends on your app logic. If you need to process data in waterfall like model and you have multiple tasks working in same time you still need to use asynchronous calls or you will lock the execution of code in node. In go you just lock the gorutine in which you make I/O call.

Goroutines communicate data via channels, so you can just pass a channel a message (which can be anything). The channel will sit and wait for a message, processing the message when it's received.

That said, for anything like a website or an API you probably won't need to worry much about channels or concurrency at all. The stdlib handles each web request on a different goroutine, so you write synchronous code, yet your server is concurrent.

ah yes! This behavior is also supported in Javascript but not as natively as in GO. But there are CSP libraries out there :)

http://jlongster.com/Taming-the-Asynchronous-Beast-with-CSP-...

You need a wrapper to make it work, but yes, you can do this. The first thing you need is a wrapper to make a Node-style call into a Promise:

    function node(fn, ...args) {
        return new Promise((accept, reject) => {
            args.push((err, ...vals) => err? reject(err) : accept.apply(this, vals));
            fn.apply(this, args);
        });
    }

What does this do? The argument of `new Promise` is a function which takes two functions, `accept` and `reject`, as its arguments. The function will be immediately run and is meant to do some stuff, possibly asynchronously, and then eventually call the first one of these on success or the second on failure. (One thing that takes a while to get used to about Promises is that generally the asynchronous thing starts when the Promise is created, so execution is not deferred ever.) We just take the args that it was called with, and append our own usual-Node-style callback to those arguments, asking it to pipe the Node error to the reject() function or the successful values to the accept() function.

Then you need a sort of "trampoline" to convert a generator-yielding-promises into a single promise. That looks like this:

    function go(gen_maker) {
        var gen, goNext;
        gen = gen_maker();
        goNext = val => {
            var item = gen.next(val);
            return item.done? item.value : item.value.then(goNext);
        };
        return Promise.resolve(undefined).then(goNext);
    }

Now what does this do? It defines a semirecursive function goNext which will hand itself to any new promises. The goNext function just runs the generator for one step with its argument, and looks at what it got out of that generator. If the generator has completed, it says "I'm done, great, return that last value!" and if it hasn't completed it says "okay, I have a Promise here, I need to tell it that when it's done computing what it's computing, it should goNext() with that value, to hand that value and control back to the generator." Finally after defining this magical function we begin the first gen.next(undefined) run by creating a Promise which immediately resolves to undefined, and telling it to goNext() with it. Of course, `gen` is the output `GeneratorFunction` that is generated by running a function* generator-maker on some arguments.

With those two basics, you do eventually de-nest everything to just one layer of asynchronicity. You of course cannot do better than this to get truly synchronous code, but you might not need to if your last expression would have been undefined anyway.

For example, here is how you then write something which accepts two arguments and appends the first to the second, using a 10MB buffer in memory. (Yes, I know this is just `fs.createReadStream(process.argv[1]).pipe(fs.createWriteStream(process.argv[2], {flags: 'a'})`, with some messing around with the high-water mark; it's just to prove a point.)

    // In this comment, we pretend node and go are defined as above, that we have
    // checked for exactly two args on process.argv and logged some usage 
    // instructions if not, etc.
    var fs = require('fs');
    const buff_size = 10 * 1024 * 1024;
    go(function* () {
        var in_file, out_file, buff, count;
        buff = new Buffer(buff_size);
        in_file = yield node(fs.openFile, process.argv[1], 'r');
        do {
            out_file = yield node(fs.openFile, process.argv[2], 'a');
            count = yield node(fs.read, in_file, buff, 0, buff_size, null);
            yield node(fs.write, out_file, buff, 0, count);
        } while (count === buff_size);
        yield node(fs.close, in_file);
        yield node(fs.close, out_file);
    });

In terms of larger libraries that do the same, so that you don't have to roll your own `node` and `go` functions, see e.g. http://taskjs.org/ and some other similar projects.

Callback hell shouldn't exist. It just means you haven't abstracted out your logic properly.

Sure, i fully agree in point that callback hell shouldn't exist but i didn't wrote that it exists in our code base, that's why i wrote about promises in my comment. But i still read callback hell in a lot of packages that were created before promises started to be popular. Even some of new packages that i saw are still written in pure callback way because authors prefer that old school way of doing things.

Seriously. When someone complains about callback hell its a good sign that their application is a poorly written mess.

I see it a lot from backend developers who never really knew JS that well and then moved on to the next shiny thing (right now Go, but Haskell, Rust, etc) without ever having a good grasp of WHY they has issues - instead they blame node/JS.

In a year or so I fully expect all these articles to switch to "why Go sucks" as the cycle repeats.

I don't know what HN you read but on mine the "Go sucks" trend has fully started.

I'm seeing the beginnings of it, yeah. It'll take a while tho - still plenty of positive Go articles show up on the front page.

Good language abstractions should allow elegant logical abstractions in code without trapping a programmer in a nasty web.

Since Go is compiled its going to be faster.

V8 has a JIT compiler that is extraordinarily, sometimes surprisingly fast for certain tasks.

This "oh it's AOT compiled so it's faster" mantra is not accurate causation.

While this is true most of the time, it's not entirely correct.

Python for example is also compiled into *.pyc files, but it's still slower than go.

The real reason why Go would be faster than JS, is not as much about compiled/not compiled but statically typed vs dynamically typed.

In statically typed language the types are already predetermined early on (during compilation) so the code no longer has to perform type checking at runtime.

From developer productivity standpoint how does Go (with or without framework assistance) compare to Node or Rails/Python?

Go seems to be still a bit of a black box in terms of performance/memory profiling.

On one recent project, the go memory profiler was showing about 35MB allocated while top showed memory usage at 500MB to 1GB. In another instance the performance profiler would show microseconds elapsed while the user experienced tens of minutes with no response from the program.

I really think they should consider including a function level profiler instead of just a sampler, as well as provide more insight/control over all the goroutines that are generated.

Just saying golang is better than node.js may not be saying all that much.

Can you provide more details? Because i didn't encounter such problems. Was this 500 MB in top was RSS ? What you describe should not happen, did you report a bug?

They were more like things the go runtime was doing but which I had no clue or information about, rather than bugs. The top info was indeed rss.

For me it seems like you probably leaked gorutines/not closed os resources or you encapsulated some data structures in themselves which then can't be garbage collected. It really looks like memory leak caused by app logic. But as i said you should report this if you are certain that is Go bug.

Are you looking at virtual memory / memory-mapped files?

I've seen similar things happen when using very large maps in Go, memory being eaten and never released.

Go processes usually have that large amount of in virtual memory, but RSS has always aligned with what Go says in my experience.

TL;DR: Using a lower-level, compiled programming language results in lower memory consumption and faster software. Shocker.

Besides the lack of data to illustrate their point, their point is well know and has been well documented for decades. The reason people use these higher level, higher overhead languages is generally time-to-market, maintainability and development cost.

I don't so much care about the performance difference data, I can measure that myself easily and cheaply.

Measuring frustration, development time and bug-hunting time is something that would be expensive to measure (you have to build a real world product in both languages with users and maintain it, which they seem to have done). Yet this part is glossed over and never quantified in the article.

> their point is well know and has been well documented for decades.

Node.js and go are 7 years old, so no it has not been documented for decades :p But your comment is still right.

Those are just examples of a high-level language and a systems programming language though, and the expected result is the same as comparing Perl/Tcl/Python with C/C++, or whichever decades old variation of this topic you want to pick :)

the point that has been known for decades is not, "Switching from Node.js to Go gives you better performance," it is "switching from a high-level language to a low-level language gives you better performance." This point _has_ been known for decades, but new generations of programmers keep re-discovering it. (note: I am not critizing new programmers, I am one of them. in fact, having recently switched from python to rust, you could say I am one of the many re-discovering the performance joys of lower-level languages)

Higher level languages generally being slower than lower level languages has been documented for decades.

Time-to-market is probably more directly influenced by the familiarity of the developers on the project with the language in question and the ubiquity of stable libraries and frameworks and online answers to the typical (and atypical) problems that arise. It just so happens that higher level languages have strengths in these areas because they attract more developers in general (they are easier to learn).

Maintainability cost is probably more directly related to how clearly one can express the high level concepts the program must deal with in the language in question. If the concepts are things like manipulating huge arrays of numbers, bit operations, interfacing with the system APIs, or other tasks suited to native langauges, then using a native language may be better. If the concepts are things like manipulating large dynamic data structures like strings and maps, doing a lot of asynchronous operations, and communicating with external services via JSON APIs, then a dynamic high-level language that easily speaks JSON may be better.

Development cost is a mix of the above.

I think this is a better characterization than "high level languages provide better time-to-market, maintainability, and development cost". For example, while this may be true for some projects, I know of many others where the better time-to-market, maintainability, and development cost comes from using a much lower level language.

It's all engineering. Measure. Analyze. Weigh the trade-offs. Make decisions. Monitor.

I love the anecdotal account -- especially since I am researching a similar change -- but would truly appreciate more numbers for transition stories like this. Memory usage before and after. Throughput, etc.

Also what was the cost in time and man hours of conversion?

It really depends on how much experience someone has with Go. It will be more hours for sure than Node, probably more code also. You need to watch out on couple of things also that you don't need in Node. For example default passing by value to functions so you need to use pointers for large structs if you want performance, you need to know that strings are immutable which means that if you do something like "some" + "ting" you are basically creating new string and both strings are copied to that string, if you have a lot of that kind of operations (like really a lot) then use bytes.buffer etc. A lot of things that are not so obvious for beginner in Go. But you got great tools at your disposal by default like pprof which is great profiler, data race checker etc. Standard tooling is really great.

Strings in javascript/nodejs are also immutable.

You will appreciate this: https://medium.com/@theflapjack103/the-way-of-the-gopher-669...

How did this make fp? 0 factual content. X is better then Y

It made first page by having popular values for X and Y.

Why 89 upvotes, there's zero technical details?

And considering the recent leftpad disaster, it makes this "point" a bit tone deaf:

> Node.js has been around the block longer it boasts more than double the modules

Well the leftpad disaster wasn't really a symptom of the # of modules. I think it's probably fair (and unsurprising) to say there are a lot more "significant" modules in the Node world than Go, even if we discount the 10-line basic function modules.

I don't understand either.

Is this really the best compromise?

What about NodeJS + Rust (for the heavy tasks) via Neon? https://github.com/rustbridge/neon

I wonder what was the WebSocket module used in Node. There is this new module https://github.com/alexhultman/uWebSockets which is very promising.

> Node.js has been around the block longer it boasts more than double the modules

Yes, Node.js has more third-party packages available, but the quality of Go's packages is usually higher. Also, you can do a lot with built-in packages in Go compared to Node.js.

It's hard to make a great analysis because there's no hard data on the memory used, the code creating that memory use or their infrastructure. But if I may, due to them being less than forthcoming, I'd like to speculate. Anger many, please a few.

If they needed something better than Node they may simply be doing it wrong and I'd lean that way first before subscribing to Go being the solution. Which has me thinking it's the most convenient band-aid nearby.

There are alternatives to the language/platform hopping. My case would be a decent example. Most of my stuff is in Python2. I have a newer Django project that I'm vetting CPython3 with, but will remain writing long running CPU intensive services in Python2 on PyPy. I see CPython + PyPy as a pretty reasonable approach and have been able to scale quite well for most situations which don't involve battling millions of concurrent users and petabytes of real-time data analysis. There's also the advantage of a single language backend and developer productivity. Node.js callbacks never matched gevent or Python3's async I/O and Go is adding yet another language to the stack where PyPy (or Node in their case) would likely handle it. Difficult to say without looking at how they've implemented everything but I have doubts whatever they're doing couldn't be done with resourceful coding. Most likely, these are people who want an excuse to try something else and they're just excited.

Given the self-professed extremity of their case, Node, Python, Go, et al would not resolve something so seriously memory-bound well enough that it wouldn't be worth calling out to a non-GC'd library. I would steer away from GC-anything and drive a stake through the heart. Writing a small limited-scope Rust extension to do what they needed and called that from Node.

Beating the living shit out of this dead horse.

> Another thing I found bothersome were the generated GoDoc pages. They were ordinarily not as easy to understand as hand written Node.js GitHub readme pages and came with very few, if any, examples.

I'll definitely agree here. It's not obvious right away how to write examples that get included in the generated documentation, and definitely unintuitive that it's a completely different process than the comment-based approach.

a.k.a. from one hotness to the next?

This page is worded like an advertisement.

If anyone is planning on building a semi-ambitious web system soon, you should evaluate Go. It's kind of scary how productive it allows me to be now. All of the following benefits, of course, may not apply to your project but I wanted to share my experience regardless.

I started my project as a collection of (micro?) services, all talking to a distributed kv store, containers all orchestrated, etc ... That's just how we build things nowadays right? Well one day I said screw it and rewrote the whole thing as a Go monolith. Including the DB layer, which now runs on BoltDB, an embedded k/v store. So now I have a web analytics system, reverse proxy, rest api, pixel api, license key server, reporting engine, emailing service, and a custom stripped down time series database (uses Bolt as a storage engine) complete with a custom SQL-like language (stripped down version of InfluxQL). It all exists as a single binary, which magically works on Windows, OS X, and linux, with zero run-time dependencies. This has allowed me to provide my customers with a self-hosted version of my software if they don't want to use my cloud service. My cloud service is just a differently configured version of the self hosted binary. Compilation takes just a few seconds.

Regarding performance. A single node handles thousands of requests per second. It's possible for me to optimize that further (I suspect by a lot), but I don't need to. And up to 1TB single node storage. Again, there have been larger BoltDB deployments but I simply don't need to push it very far since I can easily run multiple nodes when the time comes, each handling some subset of user accounts.

Another delightful thing is my dependency situation. I have 8 library dependencies in total, and have never even had to think about which version I'm using past the initial download. They're mostly lean and mean and stay out of the way. Things like CORS middleware, JWT library, request mux, Bolt, and a few others. And they all work together since they all conform to Go's built-in defined interfaces. What I appreciate about the library ecosystem is that most of the libraries you will use are _done_. Go has no dependency versioning, semver, etc... It sounds concerning at first but I was pleasantly surprised about how much of a non-issue that is in Go. I suspect that a project of similar scale would have resulted in many lost hours in Maven, Gemfile, or package.json.

Also, the concurrency makes me happy. I've never been one to multi-thread my programs. But Go makes it easy to build concurrent pipelines and services without needing a team of engineers to review it for correctness.

Anyways I'm rambling, I guess I should list some of the frustrations and trade-offs I made. For one, yes you will miss generics at first. It seems like a weird thing to not have in the language. It kind of takes getting used to, but you will figure out how to write the same functionality in terms of interfaces. It's certainly unique, a bit more typing, but in the end, things still get done. But, yes, you will have to shift your mind to a different programming style. Which some find annoying. On the plus side, at least for me, Go is one of those languages that has changed the way that I think about programming.

Also, not really speaking to the Go language, but if you're considering Bolt as I mentioned, don't expect it to handle replication or distributed fanciness for you. I'd have to say the biggest tradeoff I made is switching to a scheduled data file backup system in the place of an actual database service. With naive scheduled backups, you might lose a few minutes of data in the case of a total node failure. This can be improved on my end, but that starts to get pretty involved.

... I don't know why I just wrote an essay. Hopefully this encourages someone try out Go for their next project.

The right decision. If only I could do the same with java and Python.

Yep, leak of numbers exists in this blog post. Node is certainly not for websocket communication, until it's simple chats or message count is not so huge.

Reply on news.ycombinator.com