Show HN: Axax – Async iterator extensions for JavaScript | Hacker News Reader

Show HN: Axax – Async iterator extensions for JavaScript

github.com

74 pointsscanr48 comments

This is more or less what RxJS calls an Observable (except with nicer syntax). Here's a very nice table that makes the lack of this feature in ESnext seem like a strange omission:

https://danreynolds.ca/images/tech/push-pull.png

Asynchronous iterators / observables are so much more useful that synchronous iterators that they should've added that to the standard first.

Wouldn't it be wonderful if ESnext allowed you to write event loops that maintain state with block scope variables instead of all the hacks we have to resort to now?

EDIT: RxJS is mentioned in the README. What I like about this is that it doesn't try to implement the kitchen sink and is a fairly atomic dependency. Now if we could just get to where we don't need transpilers anymore, that would be nice.

In case that link is your site, the SSL hostname is for `www.` not for the main domain, so it doesn't load in Safari. Chrome for whatever reason didn't care.

Chrome shows an interesting notice in the console.

> Redirecting navigation danreynolds.ca -> www.danreynolds.ca because the server presented a certificate valid for www.danreynolds.ca but not for danreynolds.ca. To disable such redirects launch Chrome with the following flag: --disable-features=SSLCommonNameMismatchHandling

It's not my site. Just didn't bother to hunt for that image in the RxJS docs and looked for it on GIS instead, since I was on a phone.

>Wouldn't it be wonderful if ESnext allowed you to write event loops that maintain state with block scope variables instead of all the hacks we have to resort to now?

I think this is already possible:

    let event = null
    while (event = await getNextEvent()) {
        ... // process event
    }

getNextEvent() creates a promise, registers an event handler which resolves it and returns the promise.

Have I misunderstood your question?

Suppose the events in question are network messages arriving via DOM callback. You call getNextEvent() and then what? You're not inside the event callback, so there's no event to hand back. You could push events onto an array as they arrive in the event callback and pop them off from getNextEvent(), but how? You can't await an array. Iterators won't help you either, because they are for pulling multiple results out of a function synchronously, not having multiple results pushed to the caller asynchronously. That's what asynchronous iterators / generators are for. Think of it like looping through an array, except the loop will halt to wait for elements that aren't populated yet.

Now, on top of that, add the ability to run map(), reduxe() and filter() on a series of events. Instead of having to wait for all the elements in an array to be available, you can process it as you're receiving it, yet treat it as an already complete array in terms of being able to filter and transform it. It's a very powerful concept and I suggest you read about Observables in the RXJS docs. It took a while before it clicked for me.

> run map(), reduxe() and filter() on a series of events.

Can't you do that with a btree already?

Yeah... Well, I'm pretty sure you're wrong:

    'use strict';
    
    function getNextEvent() {
        return new Promise(resolve => {
            document.getElementById('button').onclick = event => resolve(event);
        })
    }
    
    async function enterEventLoop() {
        let event = null;
        while (event = await getNextEvent()) {
            alert(event);
        }
    }
    
    enterEventLoop();

And thanks for recommending RxJS, but I've been using reactive programming on various platforms for the past 6 years, so I'm pretty sure it already had "clicked" for me.

Neat, I'd think the same example applies to Rx! Likewise, their library (seems to) do modular imports, which gives pay-as-you-go tree shaking. This is a good push to clarify that!

Reassigning the event handler? That doesn't count. What if that was a WebSocket connection? Would you keep removing and adding event handlers? What if the JS engine decided to invoke the 'message' event handler 3 times in a row before invoking the next iteration of that loop? Or alternatively, what if your event loop needed to call an async function before it could process the next message, and another event fired in the meanwhile? You'd only receive the first message.

Yes, there is a relationship, with RxJS on the "push" side and async iterators on the "pull" side.

An interesting alternative to the library mentioned here is that IxJS is the RxJS team's own implementation of the sorts of operators that Axax is trying to provide, though for both ordinary iterators and async iterators: https://github.com/ReactiveX/IxJS

IxJS gets the benefits of trying to share as much as possible the same API as RxJS, making it really easy to use bits of IxJS and RxJS in the same project, depending on use case.

So I finally took the time to learn `async await` the other day. Holy cow does it clean up my promises. One thing that burned me bad though was that without a `try` anywhere, my 100 line async function failed silently. I think because its transpiling to promises and nothing was handling a failed returned promise. Really really frustrating.

Anywho, I appreciate that there's lots of more complex things one wants to do with async iteration. But I often see these complex examples that use all kinds of neat features that I probably don't need to be bothered with. I don't want to set up pipes and such, I just want to give a collection, a function, and get the result in an async function. I want this:

  let result = await map(collection, expensiveFunction);

How would this be done with Axax?

And I guess the point of my feedback: you chums are experts with this stuff. You wrote the library. Don't forget the "step stool" examples to help newbies get into the library, too!

No need to get any extra library involved; the standard library handles your case:

  let result = await Promise.all(collection.map(expensiveFunction));

(I’m presuming expensiveFunction is an asynchronous function; if it is instead a synchronous function, then don’t bother about the Promise.all part, just do it all at once asynchronously, as the effect will be the same: `new Promise(resolve => setTimeout(() => resolve(collection.map(expensiveFunction))))`.)

Your last snippet only defers the code execution to the next iteration of the event loop. If it's synchronous, Since it's "blocking" anyways (no IO or network requests), why not just `let result = collection.map(expensiveFunction)` ?

You could toss the expensive function in a web worker to make it non blocking.

Too expensive to do in one frame. Too cheap to use a worker. Alas. :)

Depending on the data you are sending transferrable objects could be your solution.

Typed arrays can be zero-copy transferred to a worker extremely quickly, modified, then transferred back if needed.

No costly copying or cloning needed. But it won't work with just objects, just typed arrays.

Yep. In my example I want each iteration of map to be queued so they all occur whenever there isn't rendering or whatnot to be done.

I was making the assumption that it was completely synchronous, and thus that using a new setTimeout per item wouldn’t be helpful, but I neglected to consider UI responsiveness—and so on reflection such a thing might be useful. Taking that into account, it becomes rather messier:

  new Promise(resolve => {
    const out = [];
    const iterator = collection[Symbol.iterator]();
    function next() {
      const item = iterator.next();
      if (item) {
        setTimeout(() => {
          out.push(expensiveFunction(item.value));
          next();
        });
      } else {
        resolve(out);
      }
    }
    next();
  });

JavaScript’s threading model is rather hostile to expensive CPU-bound functions. Workers are just too hard to use unless you go all in with that architecture. I love being able to work with things like Rayon in Rust.

”One thing that burned me bad though was that without a `try` anywhere, my 100 line async function failed silently. I think because its transpiling to promises and nothing was handling a failed returned promise. Really really frustrating.”

This is a promise thing, it doesn’t really have to do with async/await or transpiling.

Node.js and some browsers have a warning or event you can listen for https://developer.mozilla.org/en-US/docs/Web/Events/unhandle...

It's not just certain environments, I'm fairly sure that if you use a normal promise polyfill like babel-polyfill provides, then unhandled promise rejection failures will always trigger window.onunhandledrejection, which is analogous to window.onerror (unhandled promise rejections in async code are the equivalent of unhandled exceptions in sync code). If you set up both of those to do any error handling/reporting that you need, you can avoid any issues of promise rejections getting swallowed.

"...without a `try` anywhere, my 100 line async function failed silently."

I've played with catchify a bit.

https://github.com/majgis/catchify

https://www.npmjs.com/package/catchify

Feels less awful. But I'm still ambivalent. Callbacks vs promises vs async/await vs catchify's golang style error propagation. Yay.

I intended to weave in timeouts and tracing (metrics). See if that makes nodejs dev work sufferable. Maybe a catchify & timeoutify & traceify hybrid will keep me from playing in traffic.

async/await works best when all of your async code uses it, so it’s kind of silly to pass a callback into an async function like this example from the article that inspired catchify:

    async function asyncTask(cb) {

async functions return a promise that rejects if an error is thrown (either directly or by a promise that is awaited on), so there’s no need to handle every error if you’re ok with it just propagating.

I use await/async everywhere in my projects but I still have to use try catch.

Am I doing something terribly wrong?

You're probably doing it right

> One thing that burned me bad though was that without a `try` anywhere, my 100 line async function failed silently.

You should be bubbling errors up to some top level and at least catching them there.

Wrapping entire async-fn bodies with try/catch (or outright overuse of try/catch) is probably the most ubiquitous beginner mistake I see in modern Javascript.

For example, imagine if your top level was even just run().catch(console.error).

I really need to find where to do that. It's inside a redux dispatch and I dunno... Nothing is catching them. It's just silent.

Its silent because its inside another promise and that one have catch that is swallowing it somewhere in the chain.

Especially when used with control flow like for loops and if/else. It's incredible how much cleaner it makes the code.

Great work! (And hey I'm biased; I have contributed to this project)

It's clear this project has some overlap with the sort of problems that make many people reach for rxjs. What would make you use this project in place of rxjs in those situations?

Thanks, John! And thanks for contributing.

I find that their are async problems where it’s easier to reason about control flow using async iterators vs observables.

This slideshow (link provided by @smg) does a great job of explaining when async iterators can be useful vs rx: https://docs.google.com/presentation/d/1r2V1sLG8JSSk8txiLh4w...

I’ve been looking for something like this for a while; a nice well thought out interface for async iterators that is simple to integrate into any given event system. I quite like the `fromEvent` function and `Subject` class: they should make it really easy to achieve what I’ve been after, without me writing yet another set of boilerplate interfaces around a stream!

What advantage does this have over enumerating the iterable and using the regular `map` and other such calls.

In my mind, I'm thinking this creates something like a chain of generators in python, but would love if someone could explain in JS terms.

It's nice. I do that with generators in my programming language.

  let* choice from_confirm "Yes or No?"

This code will create a loop over the generator function form_confirm(). It behaves like if it was blocking.

How much of this is already satisfied by ES2018 async iterstors? Either way it's nice to see it already exist in a pure format.

I guess fromEvent and the Subject class aren’t, and being able to know that any Subject can be operated on with this admittedly nice function set without issues is a boon above and beyond ES2018 async iterators, I feel

This is just a number of utility functions I found useful to use on top of async iterators. The great thing about async iterators is that the implementation for a lot of these is relatively simple.

Indeed. Yeah this is very nice. Great work!

Author here. Happy to answer any questions.

Nice work! Reminds me of callbags, but for async/await hehe

Thanks! Someone else showed me callbags just today. Looks great.

Great work!

From the presentation by Benjamin Gruenbaum - `Why Async Iterators Matter` there is slide where he shows how some Rxjs code can be rewritten using async iterators.

https://docs.google.com/presentation/d/1r2V1sLG8JSSk8txiLh4w...

Is there a chance we could see some more examples of complicated Rxjs patterns being replaced by code using Axax?

Would love to. Do you know where I could find some in the wild? They often help inform what functions to add to axax.

Here are some Rxjs examples from an older version of the library

https://github.com/Reactive-Extensions/RxJS/tree/master/exam...

Any chance of adding a wrapper for async iterables that allows chaining of filter, map, etc rather than needing to individually import each function?

Similarly, a forEach would be welcome for those of us who prefer to avoid for loops.

Thanks for your suggestions. Chaining would be neat. For each should be pretty simple to do.

For other stuff, anything from lodash or ramda that could reasonably be applied to async iterables would be welcome, possibly with some tweaks. For example, a `find` method that returns a Promise that resolves with the first matching result, or optionally rejects after X milliseconds timeout, or a `reduce` that returns an iterable that returns each new reducer result.

Reply on news.ycombinator.com