Go-like channels in 10 lines of JavaScript | Hacker News Reader

Go-like channels in 10 lines of JavaScript

pedrocattori.dev

41 pointspcattori59 comments

The real essence of Go channels is their ability to participate in the built-in select keyword. Of course, Go does not have access to any magic CPU instructions that make it something Go can uniquely do. But anything that wants to be "Go channels but in X" need to be implementing select, not a send operation. Send operations are easy. And a useful primitive! Way back in the day, Queue in Python was the way to communicate between green threads. But you don't have "Go channels" without select.

Whether that is possible in JavaScript, I don't know. It is possible in a heavy-runtime language to be so locked down that it is either impossible to implement, or impossible to implement with acceptable performance, and I'm not into Node enough to know.

(To be clear, such a thing is not a criticism necessarily. I'm not sure if you could implement select efficiently or correctly from within pure Go, either, if it did not already exist. There's a lot of runtime integration it has that is not exposed any other way. It is s perfectly viable design decision to build a runtime environment that does not give that level of access to the CPU without custom assembly or C or something.)

JavaScript has Promise.race, which is effectively the same thing (from a set of Promises, return as soon as the first one resolves).

> The real essence of Go channels is their ability to participate in the built-in select keyword.

Select is similar to a function in plan 9's threading library[1] (which is part of the Go lineage) called alt(). To use it you stick it in a switch like so: switch(alt(alts)){}. Select{} is syntactic sugar for some alt like functionality. Of course the programmer is responsible for setting up the alt structure and the channels it contains. Overall its a great library and I love working with thread(2).

See this wonderful article on Go' history in code for comparisons between Go, Limbo, Plan 9 C and Alef: https://seh.dev/go-legacy/

[1] http://man.postnix.pw/9front/2/thread

Might as well add a link to the source (that 9front repo is outdated, don't touch it): https://github.com/mischief/9problems/tree/master/sys/src/li...

Additionally, the approach in the article doesn't support anything like unbuffered channels, which is the default (for good reason) in Go. Even for a first approximation, the return type of `write` needs to be `Promise<void>`. So even the send operation is not quite right.

Disclaimer: I am the author of this library [1].

A few days ago, I ported ocaml/Event [2] to JavaScript, which provides concurrent ML-style synchronization operations.

It is possible to implement `Channel` and `select` in JS, but it is not easy to provide an idiomatic API and integrate it with the Promise ecosystem.

[1]: https://github.com/dhcmrlchtdj/sync-op [2]: https://ocaml.org/api/Event.html

I took a stab at this a while back using an object to represent the possible resolutions. The keys of the object become a signal upon resolution that indicates which branch fired. https://github.com/ggoodman/channels#select-key-string-chann...

Ultimately though, I don't believe that channels are an abstraction that makes sense in JavaScript's concurrency model. Go's contexts, on the other hand, would be a huge improvement over AbortController and AbortSignal.

It's a bit out of fashion, but same thing can be achieved without any extra code using a built-in EventEmitter:

    let channel = new EventEmitter()
    await Promise.all([
      compile.browser(channel),
      compile.server(channel)
    ])

    // in compile.browser
    channel.emit("manifest", assetsManifest)

    // in compile.server
    channel.once("manifest", (assetsManifest) => ... )

A new emitter is used each time, so the end result is the same. Curious to hear what others think.

EventEmitter doesn’t hold on to its values, so if the `channel.once` listener doesn’t get attached before `emit` is called, the value will be missed. Also, in order to wait on an event, you usually end up with a promise anyway (so `await` can be used).

There are a ton of other approaches that would work with NodeJS that would be even more simple though right? The easiest or first that comes to mind is just have your function that does whatever functionality is required for creating this assetsManifest (whatever both compileServer and compileBrowser depend on). Lets call this createAssetsManifest. Then you have your 2 compile async functions but you just don't await on them and let them run without using the await keyword.

        const manifest = await createAssetsManifest();
        compileBrowser(manifest);
        compileServer(manifest);

I am wondering if since the creation of async/await that the idea behind callbacks and async operations in general in Node has been forgotten.

Not referring to the author or anyone in particular. Just a thought since these keywords might used so often by engineers new to Node that they might not have ever learned what came before?

The manifest is an artifact produced by the browser compilation, but it can be shared _before_ the browser compilation writes its results to disk.

I _could_ refactor to have "browser compilation phase 1", then assets manifest, and then "browser compilation phase 2", but that's not how I model it in my head. Plus it would mean a diverging interface for `compileBrowser` and `compileServer` which doesn't fit my mental model either.

So prefer to use channels instead.

> but that's not how I model it in my head.

I think this is the central matter when it comes to primitives for asynchronous programming.

There exist many ways we can think about async tasks. The JavaScript ecosystem provides for multiple. i.e., event callbacks, async/await, generators, and more.

Programmer reach for tools which best match how we have learned to model these problems in our heads.

It's okay for people to use what works best for them.

Even better, make use of the built-in promise helpers:

  const manifest = await createAssetsManifest();
  await Promise.all([
    compileBrowser(manifest),
    compileServer(manifest)
  ]);

That makes for such nice, clear code!

But there is no concurrency in Javascript! So even after this exercise, the running function still has to finish before the next can start. This will only create actual concurrency when each of the actors are themselves doing pretty IO heavy things where they wait for external processes to finish. Otherwise, if you actually have compute heavy processes that should run in parallel you should use the node.js cluster package or worker threads, both of which come with IPC built in already.

Or use a language that supports multi threading in some sensible manner..

You’re asking for low-level features. JS is a high-level language.

High level languages can have parallelism constructs. E.g., Ruby, in which threads have limited parallelism (only when running lower-level code that releases the GVL) has Ractors, which run Ruby code in parallel.

I’m just saying if you want parallelism then don’t write things in JavaScript

> But there is no concurrency in Javascript!

Yes and no.

At the “ECMAScript”-level, JavaScript has no built-in concurrency, but every serious JavaScript implementation enables proper concurrency via the async primitives and the internal scheduling.

Put simply, if you know how to write async code properly, your JavaScript code can achieve high concurrency!

No. All the JS code is still executing serially in the same event queue. This is a good thing, lean into the guarantees it provides.

Sorry, I think you’re misunderstanding.

Highly concurrent code need not execute in parallel. Concurrency enables parallelism; concurrency does NOT mandate parallelism.

On top of that, it actually is possible in JavaScript environments like Node.js to write code that can, in fact, run in parallel.

But those parallel executions are running their own event queues.

In an async function, the only guarantee you have is that code between two await points is not interruptible. As you have more and more async operations, this guarantee gets pretty weak (if you have more than one pending async task and the current task yields, which one will run next?) and you have to start employing the usual locking mechanisms to ensure correctness.

From experience, it is a strong enough guarantee to prevent many race conditions.

JavaScript does have concurrency! Try the following:

    Promise.all([new Promise(t => setTimeout(t, 3000)), new Promise(t => setTimeout(t, 3000))]).then(() => console.log('done'))

You'll notice it prints `done` after 3 seconds, not after 6.

It just happened to be executed one by one but the VM handles the switching for us.

What you're talking about is parallelism, which JavaScript indeed does lack and you'd use cluster or workers for that. That's when things happen at the same time, outsourced to different cores. This is what JavaScript cannot do (yet?).

Is that concurrency? The CPU is not switching between tasks; rather, it's scheduling 2 callbacks to fire after a 3000ms delay.

If you could perform 2 tasks (e.g. console.log(Array(1e8).fill(0).map((a, i) => i)), and have both run to completion without a significant delay between the two tasks, that'd be impressively concurrent.

Concurrency is not parallelism:

- Concurrency is making overlapping progress on two or more tasks.

- Parallelism is making simultaneous progress on two or more tasks.

Concurrent tasks can run in parallel, but they can also run not in parallel and still make overlapping progress by either cooperative or preemptive scheduling. You can imagine that in some larger tasks you have these 3 second sleeps, but all tasks are able to make overlapping progress without blocking the other ones -- that's concurrency!

There is a great talk by Rob Pike: Concurrency is not Parallelism - https://www.youtube.com/watch?v=oV9rvDllKEg&t=23s

> But there is no concurrency in Javascript!

Javascript has concurrency (which async/await and promises are).

It doesn’t have parallelism at the language level (but see, e.g., the Web Workers API).

This is a case where a function is conceptually producing two promises: one for an intermediate result and one for its final result.

I would attack this one of two ways, both of which I feel are more idiomatic than trying to emulate Go in JS:

1) Factor out the intermediate computation and promise:

    const manifestPromise = buildManifest();
    await Promise.all([
      compileBrowser({manifestPromise});
      compileServer({manifestPromise});
    ]);

2) Just return the two promises from compilerBrowser():

    function compilerBrowser() {
      let resolveManifest;
      const manifest = new Promise((res) => resolveManifest = res);

      const result = (async () => {
        // Compute manifest and resolve it before the final result:
        resolveManifest(manifest);

        // Compute the result of the result...
        return finalResult;
      }());

      return {
        manifest,
        result;
      };
    }

IOW, decompose things into smaller pieces (1) and/or compose them into values that match what you need (2) and remember that a function can return a group of promises instead of a single one.

This is interesting.

Correct me if I don't understand is this only concurrent with respect to IO is that right? If you run a IO operation or network call (fetch api) in those promises, those shall be concurrent with the CPU execution of your Javascript?

I think async/await is a great primitive. I've been working on implementing a async/await switch statement based state machine* in Java for multithreaded async/await. I want to handle the scheduling of multiple promises eagerly, so when you call async task1(); async task2(); async task3(); it schedules them all independently on different threads.

* if you've used protothreads in C, this is what it is similar to.

To do interleaved concurrency in Javascript, you could do the same pattern (switch based state machine) or do my concurrent looping approach. You break up your long CPU task into lots of microtasks and switch between them concurrently with a switch statement. You switch between tasks with a scheduler loop, so you do a bit of work on each task independently. It's concurrent but not parallel.

here's my writeup of concurrent loops which takes arbitrarily nested loops and turns them into an iterator https://github.com/samsquire/ideas4#133-concurrent-loops---l...

here's my java state machine of multithreaded async/await in Java, that could be adapted to Javascript except it wouldn't be parallel.

https://github.com/samsquire/multiversion-concurrency-contro...

go channels aren't threads but means to pass messages between running goroutines (also not threads, but are boxed like one). Go has first-class async built-in using `go func(){ return long_running_thing(); }()` to spawn a coroutine that executes long_running_thing(). The problem with go is you can either lock with a mutex (bad for google scale) or you can do like Erlang do and pass messages. Ok, but how? Channels. Go has support for a special keyword called `chan` that when used with `make` it will allocate a channel of a defined type. Essentially a FIFO queue to grok it in your head. Where you pass in your object of defined type, and the other side reads it in a blocking select to broker the message. Select in Go can block on a number of channels so it's easy to listen for events in this way. SIGTERM, Messages, TCP Data, etc.

I actually think async/await is a terrible primitive, and if you’re on Java, the approach of Project Loom is vastly superior (i.e. threads can be cheap so everything can just be synchronous again). I’ve no problem with library-level concepts like promises and futures but it feels very short sighted to put it into an actual language, it’s just noise.

Why don't you like async/await?

8 years ago I wrote a npm package that turned code that looks like this:

    tcp.send("syn", function(syn) {
        console.log("received", syn);
        tcp.send("syn-ack", function(synack) {
          console.log("received", synack);
          tcp.send("ack", function(ack) {
            console.log("received", ack);
         });
     });
   });

into this:

  seq([tcp, console], function(tcp, console) {
    var syn = {}; var synack = {}; var ack = {};
    tcp.send("syn", this(syn));
    console.log("received %s", syn);
    tcp.send("syn-ack", this(synack));
    console.log("received %s", synack); 
    tcp.send("ack", this(ack));
    console.log("received %s", ack); 
  });

it was never ready for production use, but it showed that synchronous code illusion can be created from callbacks.

How would you prefer to write asynchronous code?

How does Loom handle resource starvation problem? If a thread puts a while (true) {} in there somewhere, can that thread be preempted away from that kernel thread?

I think async/await is just noise: if you already have a heavy runtime like JavaScript or Python does (I forgive Rust for this one because it's so tied to reducing runtime overhead), you might as well handwave the distinction between green threads and system threads away and just pretend all asynchronous calls are synchronous (e.g. like how the eventlet or the Go runtime do it): it's painful when you have to call an asynchronous function from a synchronous context!

If you put while (true) { } in an asynchronous function, wouldn't you also have a resource starvation problem? For what it's worth though, Loom threads are planned to be fully preemptible.

Sounds like we’re agreeing - nobody _wants_ to write async code cos it’s terrible. You didn’t quite perfect the syntax above but I’m not denying it’s possible - Clojure treats async as a library, not a fundamental language feature, for example. And now with Loom the implementation is even simpler. People want to write synchronous code that can run in parallel without having to do something weird. History will hopefully prove that this is best solved as an implementation detail, not as part of a language’s semantics.

Hear, hear

The worst part of Rust, too, is that async/await is tacked on— and it’s just a bad primitive to start with.

I have many more thoughts on this that I might like to blog about, but it can be improved by having a async runtime of some kind (we don’t call malloc a runtime, but it is— and you need something similar for ergonomic async) — to wit, the de facto pseudo-answer for this in Rust is tokio (which should just be promoted to the stdlib, at this point; the vast majority of async code depends on it’s particularities).

It’s also causes quote-unquote “function coloring problems” left, right, and center — which is also avoidable. Just for kicks, consider a language with “call-async”/“yield”. Tie that in with a runtime and this whole topic becomes much, much, cleaner. Async at the callsite! (This is the blog I’ve been meaning to write)

Generators are, IMO, a much better primitive: they’re a strict superset of the functionality of async/await and they allow a bunch of other sorts of asynchronous control structures.

You might as well avoid wrapping the types and just split the writer and reader: it's good practice to do this anyway, since you probably only need one half of the oneshot channel on each side and having a handle to something that can both read and write seems like a logic error. You can also augment the writer function to throw into the promise, if required!

  type Sender<T, E = unknown> = ((e: null, v: T) => void) & ((e: E) => void);

  function oneshot<T, E = unknown>(): [Promise<T>, Sender<T, E>] {
    let send: Sender<T, E>;
    const recv = new Promise<T>((resolve, reject) => {
        send = (err, v?: T) => err == null ? resolve(v!) : reject(err);
    });
    return [recv, send!];
  }

This needs a comparison with streams, as opposed to promises. Streams are what you would use to achieve this in Node.js land.

https://nodejs.org/api/stream.html

Well, the context is Node.js, and I would definitely use promises over streams for what’s described in the article (albeit more directly). Node streams are complicated with a lot of historical baggage, and massive overkill for waiting on a single value produced by another operation.

The point was that it's not a single value, it's a stream of values (e.g the compileServer can start working with partial results of the compileBrowser function). Of course if it was a single value, promises are fine. A simpler and more modern alternative would be async iterators/generators[0].

[0] https://blog.logrocket.com/comparing-the-stream-api-and-asyn...

> The point was that it's not a single value, it's a stream of values

But it wasn’t. Despite the author calling it a channel, it only supports one value.

Hmm, the way I see it is that this only makes a difference if your "more work" is async, in which case I would consider if the function needed refactoring instead - not possible to say with toy code of course. Something like:

  await makeManifest().then(manifest => Promise.all([moreWork(manifest), compileServer(manifest)]))

If, however, "more work" is synchronous then the promise would not resolve until the next tick which puts you in the same place as before + extra overhead from a pair of extra promises.

Yep, totally could do that. But I like I mentioned here (https://news.ycombinator.com/item?id=34659597) I wanted to keep the interfaces for `compileBrowser` and `compilerServer` since my mental model for it is that those two are "siblings".

I've come to the conclusion that it's simply not possible to fully emulate Go channels in JS/TS. They're too tightly integrated into the rest of the language.

See this talk for some more background (ideally, watch the video). https://go.dev/talks/2012/concurrency.slide#1

Be warned, you will never look at async/await in the same way again.

AFAICT after the first write, the inner promise is resolved and will return the same value it was resolved with even if subsequent writes are made. This is totally different than Go-like channels which can receive multiple writes, often handling buffering them until they are consumed.

That's correct. The last three paragraphs of the article cover this and include links to libraries that have "full" channel implementations.

Author refers to typescript code as “javascript,” considering the distinction no linger meaningful.

Perhaps a sign that it’s finally over, and typescript has won in the community.

I think the author might just have a problem labeling things properly. Not only the JavaScript/TypeScript confusion, but the code snippet is also 21 lines, not 10 lines.

But actually:

> Ignoring the Typescript type definitions, its only 10 lines of Javascript!

So removing the types (JavaScript), it is 10 lines, but if you wanna write it in TypeScript it's 21 lines. Proof how verbose TypeScript can make codebases/examples?

I mean, in this case there's not really a difference with regards to the actual substance of the code.

I do see your point though. Typescript has taken off in the last few years.

For multi-value support, append to a list when writing and use a Generator for reading, yield list.shift()

I don’t understand why there is a bang operator in the last code sample on line 18. Is it a typo?

Its a Typescript-ism for asserting that the value is not `undefined`. So you're telling the typechecker to trust you on this one

Thank you, didn’t know about this one!

That is (was) a good thing! It’s basically the `as any` of strict null checks, and just as unsafe.

Now that you do know about it, please use it sparingly if at all, i.e. when you’re absolutely sure you know more than the type checker, or when you’re in a context where it’ll be caught by other means. My typical lint setup disallows it in source code without an explanatory comment, and allows it in tests under the assumptions that either they’ll fail if wrong or that a reviewer will call out the test as overly complicated.

But the real question is...can chatgpt built this for you? :P (JK)

generator functions exist, which is not even needed in this case, you could just pass a promise as an argument

The main trick on display here is that you can store a reference to the `resolve` method of a promise that normally would have to be called within the promise definition itself.

Of course, you don't need a channel abstraction to do that. To me, channels are the most intuitive and self-contained way to solve the problem, so I wanted to use that model of concurrency in JS.

To be fair, my first thought was that generators solve this roughly with the same (JS-equivalent) semantics. But my second thought was that bidirectional JS generators are painful to write and use. They’re objectively better (particularly because they’re not inherently infectious when they yield), but they’re also unergonomic as heck.

Reply on news.ycombinator.com