Hacker News Reader

"It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation."

This rings so true to me.

I've recently realized how every single non trivial part of my app is in fact a workflow problem : it could be ideally written as a pipe of asynchronous steps, glued together. It's true both for the frontend part and the backend.

I believe that's the point of reactive frameworks, but somehow those frameworks are usually designed around continuous streams of incoming events. Which isn't what i've noticed is the most widespread case. One-shot instanciation of pre-designed workflows would be really ideal.

Thanks for this thoughtful and insightful comment.

Many problems can be workflow problems, sometimes even pulling in a rule engine, or require a job queue to do things that can fail.

Then you have software such as https://temporal.io/ which is really powerful for resilient workflows.

Imagine coordinating the user with a workflow with asynchronous data collection steps.

Imagine programming "reaction to user behaviour as a workflow engine". Can coordinate global user behaviour with a resilient workflow script.

   await user.login();
   if (user.showTutorial()) {
     await user.tutorial();
   }
   await user.checkout();
   await user.submitOrder();

You could have seamless weaving of code to be executed on the frontend, or on the server, the workflow is cross-machine and cross job queue.

This workflow engine of yours seems to expect a very predictable and linear sequence of actions. What happens if the user clicks "log out" after user.tutorial();

It's basically making an already easy case easier.

I assume you would have conditions for each workflow which are then pattern matched to user behaviour to see if that workflow is relevant.

You still have a globally defined happy path of coordination but your overarching application logic isn't spread everywhere but contained in one place.

The preconditions for the logout would trigger a different workflow.

I am the author of additive-gui, which is based around the idea that you provide all the rules of the GUI and the computer works it out - what applies when. Additive GUIs loosely models dataflow between components and layout.

https://github.com/samsquire/additive-guis

It's not ready for anything, it's just a proof of concept. But it doesn't really implement workflows yet.

I would rather maintain a codebase that uses this pattern of user workflows, similar to state management than sprinkled logic everywhere.

Think of workflow engines as a runtime, not a hardcoded sequence of steps that can be activated or deactivated based on events, like a state machine.

Real workflow engines (the ones you find in large companies used to model business processes) handles conditions and branching.

But my personnal experience is that linear flows work really well in the context of apps with just those two additions :

The ability to cancel a running flow. The ability for a step to « do nothing » if the conditions for its execution aren’t met.

Even linear workflows can handle conditional execution, you just lift the condition into a value, like into a Maybe/Option type, and later stages only execute if there's a value.

Or if you want to be more literal, declare a specific type:

    type Authenticated(User) = No | Yes(User)

And each stage of your workflow matches on Authenticated.Yes, and so only executes if the user is authenticated. That's basically what any system does internally, this just makes that implicit behaviour explicit.

I’m a big fan of temporal and xstate (xstate for UIs in particular) lately, but they both present a major issue in my mind. They provide excellent foundations to create very reliable software, but the tools and conventions they offer are hard to learn and to work with.

That is such a huge setback because virtually no one I work with is willing to learn these tools in order to write better software. Even people I think are very intelligent (certainly more so than I am) think they can write these kinds of tools themselves ad hoc, as needed. It simply isn’t true; it’s a bad idea almost all of the time.

If we could find some way to make these tools more intuitive and attractive to depend on, I think it could be literally transformative. I know similar tools are popular in more engineering-centred software, so it isn’t necessarily impossible. On the web and mobile software side at least, getting people to define their application states and flows with any rigour seems to be like asking someone to file their taxes and write an essay about it afterwards.

But I also get it. These tools are a lot to absorb. Sometimes it feels like they’re in the way. Though I’d argue that when they feel like they’re in the way, it’s often because you didn’t anticipate a workflow stage or application state and its absence in your mental model is making the tool hard to use because they simply won’t accommodate broken workflow or state models. That’s a good thing, and something we should want from our tools. It’s something many people love about Rust, for example. Yet again though, many feel as though Rust gets in the way as well.

Thanks for your comment and thoughts.

I think this tech is harder to get started with than not using it and that's a problem you've noticed.

I played with temporal on my workstation and thought it was really interesting but it is more things to deploy and maintain in exchange for reliability and robustness.

I had the choice between using Rust or C recently, but because the domain was new to me I chose C to get it done faster. Rust definitely has a learning curve.

Thanks for the compliment. Since you seem to like it, i’ll emphasis a little as to how i came to realize that :

It’s very common that workflows are suited for data manipulation: write the data to disk, when it’s complete send it to the network, then once it’s completed, etc. I/O are known to be asynchronous, so we’re already equiped for that.

What i noticed is that the screen of your app is also a source of asynchrony, and as such, everytime we interact with it (animations, transitions, waiting for user interactions), we’re actually dealing with problems of exactly the same nature.

Would love to talk more.

You're right, users are asynchronous event sources. I feel UI development is very hard to read and complicated to implement complicated features.

Turning desired behaviours into code is pretty difficult for me.

I have written a few Java Swing apps and an Electron app. I also tried to use Qt but I wasn't a C++ developer so that didn't go so well.

There is a good article that the mouse is a database. The idea that the mouse creates data that can be reacted to. https://queue.acm.org/detail.cfm?id=2169076

> the screen of your app is also a source of asynchrony

Exactly!

That’s why wrapping a “dialog” or “form” or “screen” into a Promise is such a powerful technique. When the user closes the dialog, the promise resolves with a result (e.g. whether the user clicked OK or Cancel), which then you can use for whatever else needs to be done, including invoking another dialog/form/screen!

This makes UI composable, and with async/await “hiding” the promise continuations, the syntax for doing that is essentially the same as when composing ordinary functions.

Have you ever worked with redux saga? It’s exactly what you’re describing.

I often wonder about some sort of unified distributed system, encompassing frontend & backend into a single whole. One where user input is just something the system can ask and wait for, no matter which frontend or client it comes from.

But I’ve only ever been able to catch glimpses of it. More of a nebulous feeling and intuition than a real understanding of how such a thing would work. Something that feels obviously right, that will make perfect sense once understood, but that I still can’t begin to grasp.

It’s frustrating.

I’m also pretty sure I must not be the first, and that it either already exists or involves some complexity, maintainability or evolution issues I have no idea of.

Your example reminded me of "it".

I like your thinking a lot.

There was a Haskell project that seamlessly transferred state between frontend and backend but I don't think it was a distributed system. I don't remember what it was called.

Writing APIs to glue together data fetching and actions and GUI state is all very siloed. If you could talk about the system as a whole including GUI interactions at the same time as system interactions that could be truly powerful.

Imagine multi omnichannel event streams that map to the users notifications, email inboxes, chat interface, post, deliveries, accounting, customer data, synchronisations, integrations, microservices and business CRM and ERP. Everything is linked together by powerful workflows. An interaction with a customer is just an extension of the system. It's a distributed system of human tasks as well as digital tasks and interactions between the customer and the company.

Exactly!

The first obvious risk that comes to mind is that nothing can be made that correctly takes into account everything it might some day be required to be able perform, so it can easily be done wrong and would probably need to be very flexible / loosely coupled.

And while the distributed aspect adds complexity, I feel that involving different devices, not all of which will on, or online, all the time, makes it a necessity. Not accounting for that would doom it.

But yes, that's a big part of it.

For the interaction / capabilities discovery part, I've lately been drawn to some sort of declarative interface, akin to Apple's "App Intents", dynamically exposed depending on the system's state. It also reminds me of how REST APIs were supposed to be discoverable.

But I'm not sure, and it could be a dead-end.

Another thing that comes to mind is Bell's Plan 9, and how someone who used to work there said that when he would come home, he'd just open his computer and everything would be there, just like he had left it at work. It's not enough, and the goal wouldn't be to have the exact same interface replicated everywhere, but this single distributed state, with each device just being a window into it, feels like a start.

Not that "it" would be an operating system. That too would be a doomed effort (many people can't change their OS nowadays, and most people who could wouldn't do it just to use a product or service). It would have to a paradigm, and perhaps a framework, or a language.

You've got me thinking again. I'm not going to be able to get anything done for days now.

https://github.com/hyperfiddle/electric ?

Oh! Thanks! It's interesting. I'm not familiar enough with closure, and I can't tell how well it handles distributed states, but I'll try to play around with it. Thank you!

I think this is true for a subset of software. Although I think a more general truth is that how you organize data pretty is what defines your software. It's the root of virtually all of your problems and successes in designing software.

It's also why it's so unfortunate data modelling is often ad-hoc by defaulting to some bucket-of-json model with no regard for the needs of the application.

I also think surprisingly often, given some ideal data modelling, it's both faster and easier to use synchronous processing because you no longer end up having data far away in weird formats.

> But the implementations are rarely extracted out for general purpose usage and rarely have a rich API.

SolidJS pitches its state management as being robust enough for use outside of itself.

IMO, most state management tools baked into ui frameworks aren’t robust enough to be worth extracting.

This sounds almost like you've just invented Haskell-like `IO`.

All operations get "queued up" in IO and only run "at the end of the world".

To write your program you `flatMap` over the `IO`: An `IO` contains the computation(s) that will run at some point, but you can map on the result of them right away, and return another `IO` value; than `flatten` the `IO[IO]` data structure (which makes `flatMap`). In Haskell you have extra syntax for `flatMap` called "do notation". In Scala, where there are library solutions for `IO`, you can use "for comprehensions" instead. In both cases the nested `flatMap` calls get sequenced. This way and you can write code almost like a consecutive chain of imperative procedure calls but it all gets "queued up" and the whole program only runs when the `IO` data structure gets evaluated by the runtime ("at the end of the world", as last call in your program).

But that's not really related to data flow.

I too loved Knockout's observable model - I've found Mobx a worth successor. As it's entirely independent of any framework it is very practical for general purpose usage. I use it for reactive state with both React and in backend Node code with no problems.

I'm not sure generalisation here is ideal - quite often these are specialised implementations for specialised cases, and using them in other situations tends to only work so far.

For example, I'm currently working on a spreadsheet-style tool where the reactivity is largely being handled by SolidJS signals. It works fairly well up to a point (and that point is probably good enough for the client's needs), but it's very clear that there are big limitations here, and a more complete solution would bundle its own reactivity system. Things like computing results that spill across several cells just don't map cleanly onto conventional signals, so we instead have lots of ways to manually trigger recomputation, rather than just setting up the perfect pipeline flow. Likewise, figuring out where data loops are happening just isn't really possible.

That's not to say that SolidJS is bad for this sort of stuff - it has been great, and it's impressive how well the underlying reactive primitives work even for this project. But I think even when the underlying theory between these tools is pretty similar, the practical tradeoffs that need to be made are very different. And as a result, the different libraries servicing these different use cases will look very different.

I suspect this is the reason why these implementations are rarely extracted out more broadly. The sort of system that works well for one situation will rarely work so well for another.

> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation.

Angular relied on RxJS.

And RxJS is itself an implementation of the language-agnostic Reactive framework.

Jane Street/Yaron Minsky/at al came to similar conclusion with their formalized incremental implementation [0] - that this type of computation reappears in different contexts (build systems, ui, optimal data processing/display on large volumes of high frequency data etc). It's very interesting approach indeed.

[0] https://blog.janestreet.com/introducing-incremental

I wonder if there is some wisdom in Rich Hickey's Transducers

There is also differential dataflow.

I feel all the ideas are related and could be combined.

What I want is a rich runtime and API that lets me fork/join, cancel, queue, schedule mutual exclusion (like a lock without a mutex), create dependency trees or graphs.

I am also reminded of dataflow programming and Esterel which a kind HN user pointed me towards for synchronous programming of signals.

May be of interest: I released a port of Incremental to Rust the other week. It has a bit of a way to go in polish and docs, but the core implementation and API should be very familiar. (To the extent that it uses GADTs for node kinds just like the OCaml!) It has Expert nodes so incremental-map is feasible and already works. I’ve been using it as the state management for some UI with much success. Credit to the authors because it’s a good design. https://github.com/cormacrelf/incremental-rs

> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation

This is exactly the reason to use state machines/data flows IMO.

Every implementation is unique enough that simply following how data flows through the different states and transitions, and where the sinks and funnels are, will tell you everything you need to know about what your system is actually doing at any point in time.

The challenge there is, things like that are a shitload of instrumentation and requires a lot of forethought to not just jam everything into a framework that puts boundaries on what you can design and implement. So for 99% of applications, it's not worth the hassle and you're better off with just basic text documentation.

I believe the consequence of this line of thinking is the old idea of writing domain-specific interpreters. In your example, the UI generate a list of commands, and you execute/reduce over at an arbitrary time to get to a new global state, instead of tangling all the UI code with dispatching commands and managing state immediately.

> [constraints] the implementations are rarely extracted out for general purpose usage and rarely have a rich API.

Yes.

One of the goals for Objective-S [1] was that it should be possible to build constraints using the mechanisms of the language (so not hardcoded into the language), but then have them work as if they were built into the language.

Part of that was defining how they should look, roughly, and figuring out the architectural structure. I did this in Constraints as Polymorphic Connects [2].

For syntax, I use the symbol |= for a one-way dataflow constraint and =|= for a two-way dataflow constraint. This combines the := that we use for assignment and the | we use for dataflow. Also relates the whole thing to the idea of a "permanent assignment", which I think was introduced in CPL. The structure is simple and general: you need stores that can notify when they are changed, and a copy-element that then copies the data over. At least for one-level constraint. If you want to have multiple levels, you can

I was very surprised and happy when I discovered that I had actually figured this out, sort of by accident, when I did Storage Combinators [3]. There is a generic store that does the notifications, which you can compose with any other store. The notifications get sent to a "copier" stream which then copies the changed element. Very easy. And general, as it works for any store.

For example, I have been using this to sync up UI with internal state, or two filesystem directories. And when I added a store for SFTP support, syncing to SFTP worked out-of-the-box without any additional work. ("Dropbox in a line of code" is a slogan a colleague came up with, and it's pretty close though of course not 100%)

[1] http://objective.st (Site currently being revamped)

[2] https://dl.acm.org/doi/10.1145/2889443.2889456?cid=813164912... / http://www.hpi.uni-potsdam.de/hirschfeld/publications/media/...

[3] https://dl.acm.org/doi/10.1145/3359591.3359729 / https://www.hpi.uni-potsdam.de/hirschfeld/publications/media...