The Problem of Async Programming, and a Crazy Idea for Solving It | Hacker News Reader

The Problem of Async Programming, and a Crazy Idea for Solving It

medium.com

14 pointslorean_victor24 comments

Callback hell was never about promises, but callbacks (!) which would require a third-party library or some gymnastics to implement control flow. Promises and async/await both solve it reasonably well. The problem presented can be rewritten as:

    Promise.all([
        doA().then(a => doC(a).then(c => [a, c])),
        doB().then(b => doD(b), b_; return b; })
    ]).then(([a, c], b) => doE(c, a, b))

No need for `new Promise(resolve => ...)`. Or with async/await:

    const [a, b] = Promise.all([doA(), doB()])
    doD(b)
    const c = await doC(a)
    doE(c, a, b)

Is it worth introducing the complexity of visual programming to solve this 'problem'? I have a much harder time understanding the graph than the code.

yeah I think the article actually starts with that Promise-based code snippet. Even the sequential graph is based on the promise-based code.

Look closer - the original is way more convoluted - there is no reason to use `new Promise` constructors when you're dealing with other promise-returning functions.

yes you are right the constructors aren't indeed needed.

as for the complexity of the graph, basically the way I see it is that even this promised based method is a way to solve the problem of putting the graph into the proper promise-based format (for example, chain doA and doC and return an array of their result) vs simply drawing the intended graph. I mean it does take sometime to get used to it specially if you are really used to just working with code (which I myself originally was), but since the graph is the more intuitive way to represent it compared to other methods, after going through the learning curve it seems to be less complex and easier to manage.

that said its not like I have conducted proper academic research on the matter and it might as well be a pretty subjective thing.

Doesn't seem new at all, just like functional programming with lazy values. The new part isn't the async it's the visual programming of which there are others. Does incorporation async make visual programming different?

Yeah I don't think so that on its own the visual programming makes any difference (or even adds any value). The main point here is that the "visual programming" is coincidentally a solution for async programming.

The actual problem with async functions is mixing them with non-async ones. Having all futures/promises makes it easy. Now if the visual style could represent current mixed js code, that could be awesome for display and editing I'd have to try before deciding.

Yeah we reached that point pretty early on while developing the PaaS with the tool, so had to incorporate support for mixed synchronous JS blocks. you can check out how to do that here: https://medium.com/connect-platform/inline-coding-in-connect...

> To see that for yourself, try to re-write our async code snippet again using only async/await statements and see how does that pan out.

With async/await, wouldn't the example just be:

    let aOut = doA();
    let bOut = doB();
    let cOut = doC(aOut);
    doD(bOut);
    doE(cOut, aOut, bOut);

That is, it looks the same as the sync version.

The difference is in the "do" functions themselves: they would be async (meaning they return a promise, and they accept promises as parameters).

so in your code sample, would 'aOut' be a promise or a value? in case it is a promise, then basically doC() would needed to be re-written to handle when it gets a promised value instead of an actual value, and if not, then doA() must be invoked synchronously, i.e. the `let bOut = doB()` statement must be awaiting `let aOut = doA()` statement which is not what the desired async execution flow is.

Yes, aOut would be a promise. It would look like this:

    async function doA() {
        ...
        return "the-result-of-a";
    }

(Which returns a promise. Assuming execution reaches that last line of my sample code, the promise will resolve to "the-result-of-a" when you "await" it.)

> ...then basically doC() would needed to be re-written to handle when it gets a promised value instead of an actual value

Well, wait a minute... it wasn't written in the first place! Let's write it though. It would look like this:

   async function doC( param ) {
       ...
       // we use "await param" where we want to
       // access the value of param. e.g.:
       const paramValue = await param;

       ...
    }

So not bad.

yeah really not bad at all, didn't think of that solution originally. edited the article accordingly.

I don't think the leap to visual programming is necessary considering we have tools like make that work essentially identically and remain text based. All this is really doing is moving the dependency declaration to lines instead of words and this would get very messy very quickly.

An example in make (if I'm reading in the right direction) format which I'd argue is more readable and could scale much better would be:

  doA: doC doE
  doB: doD doE
  doC: doE
  doD:
  doE

Shouldn't it be like

doE: doC, doB, doA doD: doB

etc?

anyways the problem with this approach is that doD() is assumed to be a general function (or statement or async command) that is not by definition bound to doB(), but rather in this specific case we need it to be executed after doB() is done.

> but rather in this specific case we need it to be executed after doB() is done.

Why is that? If it needs any result from doB() then it is a dependencies that must be specified. If it needs to be done only after doB is successful then it is a dependency, just an implicit one we rely on the procedural execution to handle. In either case some sort of surrogate dependency (or better syntax) would have to be introduced.

But the point is that either would necessitate or be aided by graphical programming.

ok imagine doB() is a function that loads a record from the database and doD() is for example a password hash matcher (really imperfect example, just trying to get the point across). You might want to use both doB() and doD() in other contexts as well, so basically it is not the case that you would want to execute doD() always after executing doB(), neither is it the case that executing doD() always requires executing doB() beforehand. It just happens that in a specific case you would want to execute doB() and then pass its result to doD().

Is the author aware of Dataflow programming ?

These do give the described benefits, however, the tradeoffs are massive.

to a pretty limited degree, could you elaborate more?

You basically reinvented it, along with directed acyclical graphs and called it a 'crazy idea for solving async programming'.

The "crazy" label is merely a hint to the inherent resistance of most programmers towards "visual" tools, disregarding them for reasons not-withstanding in this case. yet here it seems like actually the "visual" way is the approach with more fine-control and detailed (and maybe sometimes more complex as you embrace the async nature further). It was by no mean an indicator of any of this being a "new" or "novel" idea.

p.s. while it originally used to be acyclic graphs, I had to relax that constraint due to people's request.

- Syntax is slower to write. Copying and pasting is a non trivial operation. In order to benefit for the clean representation of the process, you need to spend a good amount of time ordering the boxes around. The representation can be misleading (wires crossing, overlapping boxes)

- Very low density of information per pixel compared to any textual language (if no zoom feature). Any long name that has to be rendered on screen become fugly. Since we tend to order the boxes in a process from left to right, only the equivalent of a single line of code can be represented on the screen at once when displaying the names of variable and functions.

- Functions / sub diagrams generally sit in their own separate window (or worse, file). This discourage making functions, to a degree depending on how good/bad the UI is. Getters and setters in OO visual code are a pain.

- Execution order is not guaranteed to be sequential, so no try/catch structure. Error handling has to be explicit. Witch mean dragging along an additional error variable all over the place.

- The graph representation doesn't translate well into source control and diff tools.

- If you need to fire an arbitrary number of parallel stuff at runtime, you still need a "call this asynchronously" structure.

Coming from a LabVIEW dev. The benefits described in the article are still absolutely valid.

first off, many thanks for the feedback.

> - Syntax is slower to write. Copying and pasting is a non trivial operation. In order to benefit for the clean representation of the process, you need to spend a good amount of time ordering the boxes around. The representation can be misleading (wires crossing, overlapping boxes)

copying can be made much easier. on these kind of features, we are acting on user feedback to prioritize. but since this one specifically is one that my co-founder has been bugging me since the day he joined, I don't think it'll be too much down the pipeline.

also, automatically re-ordering the boxes to get the flatest version of the graph (which seems to correlate to the "cleanest code") is something on our mind, which would alleviate that issue.

> - Very low density of information per pixel compared to any textual language (if no zoom feature). Any long name that has to be rendered on screen become fugly. Since we tend to order the boxes in a process from left to right, only the equivalent of a single line of code can be represented on the screen at once when displaying the names of variable and functions.

that is partly by design to encourage modularizing. on that front, we have gone for possible overshooting and then slowly tuning towards the optimal point.

> - Functions / sub diagrams generally sit in their own separate window (or worse, file). This discourage making functions, to a degree depending on how good/bad the UI is. Getters and setters in OO visual code are a pain.

is that because of the increased inefficiency of switching between sub-diagrams? if that is the case, thats also a user feedback that has been on our radar for sometime and will probably tend to it soon-ish.

> - Execution order is not guaranteed to be sequential, so no try/catch structure. Error handling has to be explicit. Witch mean dragging along an additional error variable all over the place.

actually it does not need to be sequential for proper error handling. each node in the graph is equipped with its own error-handling, i.e. if an error is somehow thrown during its execution the node will report it properly, which has in many cases led to faster detection of error-sources in the context of the projects we have done with CONNECT up until now. still, admittedly the error-handling system needs to become more robust, and all of this is basically only referring to errors that you wish you would catch and fix during development phase. for errors that should be handled on runtime, yes they need to be somewhat dragged along, which I do not necessarily see as a bad thing, as it forces you to explicitly choose the corner-cases that might occur and need to be handled in runtime. this separation actually helps completely avoid using one big 'try/catch' that would blind you to 'bugs' that should not make it to runtime in the first place.

> - The graph representation doesn't translate well into source control and diff tools.

that is true as mentioned in the article as well. we've tried to make it really human-friendly, however we have plans for further tooling specifically on that front.

> - If you need to fire an arbitrary number of parallel stuff at runtime, you still need a "call this asynchronously" structure.

that is true unfortunately. personally I am still thinking on how to represent such dynamic branching within the graphs without causing more confusion. since this is also a corner case, we have kept it on the drawing board until a proper solution is found.

Yes, because XML is too old to us.

and to be fair its a really redundant format. I mean I personally would prefer the current undesirable situation rather than doing XML-based coding all day.

Reply on news.ycombinator.com