Hacker News Reader

Comment by CGamesPlay | Hacker News Reader

Because technologies like Q [1] and streams [2] provide a very abstractions around doing long-running tasks asynchronously, which jives better with node's entire philosophy. Combine that with the fact that running multiple processes is a perfectly viable way to do actual long-running tasks and node provides abstractions around that out of the box, and it really starts to look like fibers and webworkers in node are patchy hold-overs for technologists who haven't fully changed into thinking about things in "the node way."

[1] https://github.com/kriskowal/q [2] https://github.com/substack/stream-handbook

View on HN

How do promises help with CPU-bound tasks?

I already mentioned child processes, which model scaling up to multiple servers better than threads do, are more powerful than web workers, and are supported out-of-the-box with a powerful arbitrary message passing schema including the ability to transfer sockets between processes. It looks like there's some abstractions built around using multiple processes and work queues to help amortize the weight of the additional processes (see node-compute-cluster mentioned elsewhere in this thread), but I've not used anything like this.

Node's cluster module is handy, but it's a workaround for the fundamental weakness in Node - the entire thing's built on top of a single-threaded event loop which blocks.

Instead of just communicating across threads to do some async busy work, you now have to communicate across processes... along with heartbeat monitoring of those processes and endless debate over which IPC mechanism to use.

> I already mentioned child processes, which model scaling up to multiple servers better than threads do ...

All taking away threads does is eliminate high-value optimization strategies that are available to multi-machine scalable systems that do support threads.

Threads don't dictate shared mutable state as a programming paradigm, they simply make it possible to avoid crossing expensive network and process boundaries where appropriate.

There's nothing stopping you from architecting thread-based architectures to use low-cost in-core dispatch of message within a VM, and save the high-cost serialization, IPC/network overhead for off-machine message dispatch.

On which OS is crossing the process boundary expensive?

And how expensive is expensive?

> On which OS is crossing the process boundary expensive?

All of them, always. It's either expensive in terms of resource cost (requiring serialization and complex local IPC overhead), or expensive in terms of complexity (requiring shared memory and inter-process coordination).

There are threads and processes and both have their pros and cons. Yes it is great that Node has multi-process support out of the box, but what about when threads are a better solution? In my opinion, there is a place for a multi-threading support even if it is not the "Node way".

Short of rewriting the whole system around a threaded model, the only reason to use threads over processes in Node is to make the IPC faster. This means that ultimately supporting threads is an optimization that has to compete for cost-benefit with all the other possible optimizations for Node to support.

"which model scaling up to multiple servers better than threads do"

And with that I see you are operating in the Node bubble, where "threads" will forever be stuck in 1995.

It isn't 1995 anymore.

Surely you should do things like this in a separate task queue? Use a bunch of node processes for handling taking HTTP requests and spitting out responses, use a different bunch of processes for handling slow things.

Are there any practical examples (in the wild?) that are in-line with what you're saying here. Genuinely curious.

Browserify [1] is huge and is built on the stream principle. Streams do a bit of processing and can be combined. As far as q, I have seen it used in a couple places but don't have anything big-ticket off the top of my head. Here's the official public list: https://npmjs.org/browse/depended/q

[1] http://browserify.org/

I've f.e. used https://github.com/lloyd/node-compute-cluster which basically is a broker between various computation processes in node.

Reply on news.ycombinator.com