What if JavaScript had a synchronous async flavor? | Hacker News Reader

What if JavaScript had a synchronous async flavor?

guido.io

38 pointsprotoduction53 comments

Brilliant and horrifying - I love it.

It's a transpiler that puts "await" in front of every expression, resulting in code like:

  (async () => {
    async function fetchPageBody(url) {
      return await (await (await fetch)(await url)).text();
    }

    async function sleep(ms) {
      return await new (await Promise)(await (async (resolve) => await (await setTimeout)(await (async () => await (await resolve)()), await ms)));
    }

    const body = await (await fetchPageBody)("https://enable-cors.org/");
    const numDivs = (await ((await (await body).match(/<div/g)) || (await []))).length;

    await (await console).log("Number of divs:", await numDivs);

    await (await sleep)(2000);
    await (await console).log("This prints after two seconds.");
  })();

That's right. And it also makes every function async. Literal expressions are not awaited - a tiny optimization ;)

I actually looked into this just yesterday because I was interested in doing this in my JS-targeted language :)

As far as I can tell, the only real problem is performance. Wrapping synchronous expressions in Promise.resolve() (which is what this effectively does) is apparently not optimized out by V8, and is egregiously inefficient: https://madelinemiller.dev/blog/javascript-promise-overhead/

For my use-case, probably what I'll end up doing is having the compiler be smart and only invisibly insert those asyncs/awaits where it actually needs to

Because of how the microtask queue works, it can't optimize that out without introducing ordering issues. If you have two Promises, A and B, and each one has a callback that will then do another await, the implementation needs to interleave them, since microtasks are guaranteed to run in FIFO order. So a single await, even to resolve a constant expression, can't be omitted.

Fair enough. In practice I think it's wildly dangerous to make those kinds of assumptions about async task ordering, but I can understand that engine authors have to follow the spec and don't have the luxury of dodging the issue.

I tried this exact thing last year, and it does not work for a particular reason: the JavaScript event loop guarantees that a function will run to the end before any callbacks will run, but an await is an implicit callback (a Promise under the hood). This means that you break the guarantee that variables or other memory locations will have the same value between statements.

This is exactly the same problem that languages such as C++ have when dealing with multithreading, and why constructs such as semaphores, mutexes, etc. exist. JS does not have these, and using the technique in the article will completely break the language.

Compare these two fiddles I prepared and see for yourself:

This one uses await to block for a second, which in the meantime changes the value of the variable between statements: https://jsfiddle.net/gq1btLdu/

This one uses a for loop to do synchronous work for about a second, then prints the initial value twice. You will notice that the setInterval never had the chance to run, since callbacks are scheduled to run only once the current function ends: https://jsfiddle.net/gq1btLdu/1/

You are right that it's a terrible idea (I hope it was clear from the article that it's an experiment and should stay that way), but could you help me understand why this is fundamentally broken?

I would say that by using setInterval you are buying into this kind of behavior and would have to create your own synchronization.

If you are saying that this transpilation step changes the semantics of the code, then I completely agree!

If you have code like:

let x = 0;

doSomethingAsync(); // does not touch x

console.log(x);

There's no guarantee that x will have the value 0 by the time you get to console.log(x). The value of x might have been changed by an outside function that just happened to run between your statements. In synchronous JS this would never happen as callbacks are scheduled to run only at the end of a function, and if they change x the change will only be applied after x is printed.

The change in behavior will break a lot of existing code, since the synchronous behavior of a function is a core feature and assumption of the language.

Isn't that only really a problem with global variables? Local variables would be safe from this kind of behavior I'd think.

Thank you for taking the time to explain :)

I completely see that this will break existing code, but I don't see how this is different from the behavior of Go (where the setTimeout code would be something in a different goroutine) or Python?

That is false except for the case that x is a global variable, no outside function can change the value of x, only in-scope procedures can do that.

I guess it is true in this particular example, since 0 is a primitive and assigned by value. But what if you got it from an outside function, such as init(), and it was an object instead? Then you would have to guarantee that the return value of init() would not change between statements.

In the end this shows that JS is unequipped to handle such behavior without significant changes to the core language.

> But what if you got it from an outside function, such as init(), and it was an object instead? Then you would have to guarantee that the return value of init() would not change between statements.

Then even in a synchronous context you'd have no guarantee that it wouldn't be modified:

  function soSomethingSync() {
    window.someObj.someProp = 12;
  }

  let x = window.someObj;

  soSomethingSync();

  console.log(x)

This has nothing whatsoever to do with sync vs async.

It is one of the main reasons people like immutability/functional programming. But that's its own topic.

You are guaranteed that only code you call will be run. Sure you can call anything you want but the point is you have full control over what that is. If you yield execution with an await, you no opt out of that guarantee.

Its a fundamental concurrency pattern used in a lot of languages, often for UI threads and such. There's a lot of information on this topic and it certainly is about async/yielding.

Then this is not related to async/await but regardless even then JavaScript has always had the ability to change the writability of properties such that trying to change a property in an object will yield a no-op with Object.defineProperty, and now with newer features like Object.freeze and private properties.

Not just globals, anything passed in or passed out, right?

For a practical example, take this code I wrote recently:[1]

    const backends = {}
    
    async function setupBackend(host) { /* boot server */ }

    function getBackend(host) {
        if (!backends[host]) {
            backends[host] = setupBackend(host)
        }
        return backends[host]
    }

That if block is effectively a critical section: it relies on explicitly not awaiting the result of setupBackend(), so that the promise will be stored into backends[host] (to be reused by subsequent calls to getBackend()) before anything else can happen. Injecting `async` will break this behavior, causing the backend to be booted multiple times.

[1] https://github.com/wolfgang42/webd/blob/8cf28447468dd4745262...

Why not simply set some intermediate value to indicate that the backend is booting? Such as:

    const backends = {}
    
    async function setupBackend(host) { /* boot server */ }

    function getBackend(host) {
        if (!backends[host]) {
            backends[host] = 'FLAG' // or something
            setupBackend(host)
                .then(backend => backends[host] = backend)
        } else if (backends[host] === 'FLAG') {
            // no-op
        } else {
            return backends[host]
        }
    }

Subsequent callers of getBackend also need to be able to await on the backend boot before they can do anything useful. If the backend is currently booting, your function will return undefined in the “no-op” case, giving the caller no way to tell when it’s finished (other than retrying, I suppose).

I tried writing some code for this comment that used an explicitly constructed Promise as the intermediate value with some logic to resolve it once setup was complete, but then I realized that it had exactly the same problem with that being implicitly waited on. Maybe there’s some clever way to work around this but it’s going to be a lot more complicated.

Of course, if you do insist on a JS dialect with implicit await, the easy fix for this problem (since you’re transpiling anyway) is to just introduce a `noawait` keyword that turns the await insertion off for a block, to explicitly mark it as atomic.

[ETA: also, in the general case there’s a race condition if someone calls the function again while it’s awaiting the initial flag value. That doesn’t happen with your code because the OP library happens to not await literals, but that’s kind of fragile: I can easily see a situation where someone tries to introduce e.g. a counter into the flag and causes a non-obvious race.]

Sorry, but you've got some fundamental misconceptions here.

First off - and this is going to sound pedantic, but it's important to get our terminology straight - a "callback" is any function that gets passed to another function so that it can be "called back to" later on. It is entirely possible for this to happen synchronously. Example:

  function doACallback(theCallback) {
    theCallback()
  }

  doACallback(() => console.log('hello'))
  console.log('world')

This code runs synchronously; the event loop is not involved at all. It will print "hello" and then "world".

Now, the most common usage of callbacks is for various asynchronous things that happen on the event loop. Input events, setTimeout/setInterval, and yes, Promises.

However, async/await only concerns itself with Promises. Not with setTimeout or setInterval or events. So if you want to use it to take something asynchronous and make it appear/behave as if it were synchronous, that thing needs to be in the form of a Promise. You demonstrate this in your fiddle by wrapping setTimeout in a Promise inside the sleep() function.

But, you then go and use a setInterval, which is totally outside the domain of Promises, and so async/await has nothing to say about it. Your example only demonstrates that by "doing something async" you're putting the ordering of certain things at the mercy of the event loop. In practice, the answer to this "problem" is that if the ordering of asynchronous things matters to your logic, then that order needs to be enforced via .then() chaining or async/await or otherwise. Never just rely on the ordering of the event loop itself. If you do, then you already have a race condition, whether you've realized it or not. Both of the code samples would be problematic if you ever did this in production.

Further: this is most definitely not the same problem that C++ solves with semaphores, mutexes, etc. Both languages can have race conditions, but JavaScript cannot have data races, which is what those constructs exist to deal with. That is why JavaScript doesn't have them.

To get specific: in JavaScript, only one thread can actually be running (with the memory space these variables exist in) at a time. The async stuff may make this less than obvious, but it's a firm truth. JavaScript code can't be interrupted arbitrarily, it can only yield control of the thread at a given await. You might argue that making this invisible might make it easier for people to write those kinds of bugs without noticing, but it certainly wouldn't "completely break the language".

Edit: I should clarify that technically making everything async/await like this would change the behavior of existing code in a subtle way. It's just that realistically, the behavior that's changing is not behavior you should ever rely on in the first place. It technically isn't undefined behavior, but good code would treat it as if it were.

You're saying you write your js as if it could yield control at any moment and you guard against this now even though the language guarantees you don't have to because you consider that guarantee akin to undefined behavior?

Adding arbitrary concurrency when exclusive execution was previously guaranteed will certainly break the language.

I'm not sure what you mean about Python. In Python, if you run an async function without an await, you will simply get a coroutine object that is ready to be executed by... someone (it's essentially a generator under the hood).

In fact, unlike JavaScript, you can't even call await from a synchronous module context. You have to either schedule that couroutine object onto an existing async event loop, or create a new event loop and run it your self.

Yup, Python async functions need to be explicitly awaited / scheduled pretty much exactly the same way as JS... article might be confusing this with the fact that most of the stdlib is just plain synchronous, even for web requests etc.

Maybe they used gevent which "magically" makes things async without the keywords.

I went experimenting in a slightly different direction, there's usually a lot of concatenation going on in Javascript with e.g. array manipulation, so I decided to try to delay the `.then()` as much as possible. So instead of these:

    const value = await Promise.all(data.map(op1))
      .then(value => value.filter(op2))
      .then(value => Promise.all(value.map(op3)));
 
    // OR (a more readable version)
    let value = await Promise.all(data.map(op1));
    value = value.filter(op2);
    value = await Promise.all(value.map(op3));

Now you can do this:

    const value = await swear(data).map(op1).filter(op2).map(op3);

I integrated it with my other library `files`, where it's very useful out of the box:

    const list = await read('data.csv').split('\n').map(...);
    const readmes = await walk('readme.md').map(read);

https://www.npmjs.com/package/swear

https://www.npmjs.com/package/files

Ooo, I like that. So I'm newer to JS, but this suggests chaining from promises isn't possible, and that's effectively the change you've written?

It is possible chaining promises (as in my first example), just just need to wrap it all in a `.then(value => value.[operation]())`, but this way you can just use the `.[operation]()` straight on the data like you would do if there was no promise in the way.

I think this doesn't work for javascript because it actually matters when you await. It has implications for the UI. It also has implications about object life times.

Of course, the same problem can happen in Go if you use it to make a GUI application, but Go is built entirely around the concept of goroutines so it has obvious ways to handle the problem. Javascript does not. `await` as an explicit suspension point is a pretty good way to handle the problem.

I view being "annoyed" at having to be explicit about control flow to be not a real problem. By that I mean, whatever "solution" you come up with is likely to bring with it its own set of problems that no one wants to deal with. And for what price? Because you are "annoyed" at having to type "await"?

Exactly. What is with this fascination of removing information from code?

My first thought when reading this was, ok, OP is "annoyed" at having to "run functions to see if they return a value or a Promise", then goes on to describe a world in which you have no idea whether anything returns a value or a Promise when reading the code.

That's 100x worse. At least when I'm writing it I'm probably running the function anyway. And it's worth pointing out that you shouldn't need to run anything to know what it returns, that's what docs are for. Or you could just look at the return statement. But even if for whatever reason you don't know what data you're working with, I'd rather figure it out once and then make it explicit, than have to figure it out every single time the code is read (which could be hundreds of times, or more).

If you're interested by this, take a look at Stopify[1], which further adds first-class continuations to javascript.

[1] https://www.stopify.org/

Ooh, thank you for a fascinating reference about Stopify. I'll be reading the paper [1] it's based on with great interest.

Just recently I was reading "Exceptional Continuations in JavaScript" [2], which I see is cited in the Stopify paper - so it's like a continuing saga.

[1] Putting in All the Stops: Execution Control for JavaScript - https://arxiv.org/pdf/1802.02974.pdf (PDF)

[2] Florian Loitsch. 2007. Exceptional Continuations in JavaScript. In Workshop on Scheme and Functional Programming. http://www.schemeworkshop.org/2007/procPaper4.pdf (PDF)

I am a casual user of JS and was always wondering how to write true asynchronous routines in JS.

The kind you fire and forget and they end when they end. An example would be fetch, but all the docs/tutorials I saw are needed on async await where you have to explicitly yield in your code (as opposed, say, to threading in python or goroutines)

> With this flavor of Javascript you can no longer do two asynchronous tasks at the same time, everything will be executed synchronously.

The main pattern I use to dispatch multiple promises is resilient to this:

    const [r1, r2, r3] = await Promise.all([
      async () => { ... },
      async () => { ... },
      async () => { ... }
    ])

No. I think your code will be transformed into:

    const [r1, r2, r3] = await Promise.all(await [
      await (async () => { ... }),
      await (async () => { ... }),
      await (async () => { ... }),
    ]);

I remember debating this when async/await support was first coming out. My opinion was (and still is) that it would have been better to make "await" the default behavior and have a separate "noawait" keyword for those rare cases when you actually really want the promise instead of the value.

Somewhat related, I am working on a front end scripting language that implements this idea, but at the runtime layer:

https://hyperscript.org/docs/#async

Basically, the result of every expression is inspected and, if a Promise is returned, the promise is resolved before execution continues. It's all interpreted and really slow for CPU-intensive stuff, but if you are writing async code that isn't CPU intensive it lets you write stuff in the linear manner:

  fetch /message
  put it into my innerHTML

One interesting thing that fell out of that work is that you can have event-driven control flow, like a loop that executes until an event is received by an element:

https://hyperscript.org/docs/#event-control-flow

  <button id="pulsar"
        _="on click repeat until event stop
                      add .pulse then settle
                      remove .pulse then settle
                    end">
    Click me to Pulse...
  </button>

Or, people can just learn to deal with the nuances of Javascript.

I'm following hyperscript's development keenly :). I've considered putting together a small hyperscript plugin for Starboard Notebook. Often one simply wants to put together a quick interface with some interactivity, together with lit-html and bootstrap styles this could be really productive.

If we designed a new async language today, would it include async/await keywords? I think they don't add anything conceptually and it's just a patch for languages that didn't design for async from the start. Am I missing something?

Maybe not async as I think the await keyword is probably enough on its own. As for why it exists...

Await/explicit yielding enables cooperative multitasking where the programmer has exclusive control of some execution context, be it a thread or fiber or whatever, until they decide to yield.

Without that you only have implicit scheduling and execution could be yielded at unpredictable times. The programmer then needs to add other things like semaphores to specify the critical sections and you run the risk of deadlocks.

There might be a new language that comes along but right now await exists for a reason.

I think this is a compelling vision, but as a practitioner of functional core/imperative shell, I make use of the async/await keywords to understand where I/O is happening and to work towards moving it to the edges of my program.

I can do this without those keywords, but there's actually something quite nice about being certain that a given function does no I/O simply based on its function signature.

Great thought experiment. I also wished it would be possible to code in Javascript without thinking about asynchronous stuff. It should be possible theoretically.

Id settle for implied async whenever function contains awaits.

    In languages like Python and Go you write
    asynchronous code as if it’s synchronous.

How do you do that in Python?

My interpretation was that the author meant "you (normally) write things like web requests in a synchronous (blocking) way". There are important differences though between that and what async/await (even this invisible version of it) does. It's also different from what Go does, in a similar way.

What if this is baked into the runtime as a mode?

  'use async';

Yes, or a keyword that you use to declare that a single function works this way "asyncAll".

Or, insert 'use async' inside the target function body (just like with 'use strict' now).

There are many things to hate about JavaScript but the inbuilt forced async nature is one thing I absolutely love about it. Thank god that except a few reminants like `alert` and `prompt` almost everything is async by default.

I remember that just until a few years ago the whole tabs and everything would freeze if a site showed a blocking alert popup. Can't imagine what it would do if other things like http req, dom operations, etc were also synchronous.

DOM operations are always synchronous. When modifying DOM, there is often a full layout+paint cycle before the next line is run.

There is no inbuild async nature. It just did not have a stdlib whatsoever, so node could come along and offer a completely async one. There were async server frameworks in python before, like twisted and tornado.

For a long, long time, the language itself lacked almost all of the niceties that make async programming bearable. Node code quickly devolved into a giant mess of callbacks nested twenty levels deep.

Reply on news.ycombinator.com