Managing State with Signals | Hacker News Reader

Managing State with Signals

tonsky.me

208 pointsgeospeck124 comments

It looks like we are going to keep reinventing dataflow constraints[1] over and over again, always with slightly different terminology (Rx, FRP, signals, ...)

So it (a) appears to be a very useful or at least attractive concept, and (b) somehow difficult to fit into current programming languages/practice in a clean way.

[1] https://blog.metaobject.com/2014/03/the-siren-call-of-kvo-an... (HN: https://news.ycombinator.com/item?id=7404149 )

I feel like reactive programming is approaching ripeness for a mainstream programming language or cross-stack paradigm within the next few years. React, Svelte, Redux etc in the frontend world has certainly paved the way on the mainstream side, albeit in a simplified environment (singlethreaded, does not cross the network boundary and can simply share memory cheaply). I wonder if a refined version of this with will prevail, or something else like Rx? What are the primitive operations in such a model? Streams, futures, “watch” APIs?

ReactiveUI is based on Rx and is very popular in the .Net world: https://www.reactiveui.net/.

What's the difference between reactive and functional programming? Both seem to focus on expressing relationships between objects and having compiler/runtime infer the computation, instead of explicitly specifying the computation.

Perhaps if you framed your question subtly differently and asked yourself what is the difference between Erlang and OTP, the answer maybe easier to discern?

What's the exact difference between Erlang and OTP that would answer my question?

OTP is the means by which the programming language is able to accomplish tasks such as process to process comms, how to supervise actors etc. It's the enabler for the distributed computing compared to Erlang which is the programming language. They're (obviously) complementary but OTP is the magic that enable one to deliver on the reactive manifesto. Functional programming needs augmenting with mechanisms that provide the distributed computing features, it's not enough on its own.

Thanks for the explanation.

Where does reactive programming fit in that concept? Is OTP a "reactive" library? If so, why - what exactly makes it "reactive"?

Most of the time the term used is "Functional Reactive Programming". If that answers your question.

Although there exists a paper about "Reactive Imperative Programming" (with Dataflow Constraints): https://typeset.io/pdf/reactive-imperative-programming-with-...

> Both seem to focus on expressing relationships between objects and having compiler/runtime infer the computation, instead of explicitly specifying the computation.

That's true of every declarative approach. Functional, reactive, relational, regex etc. are all to some degree declarative.

To answer your question: Functional programming is really more about avoiding mutation at the language surface, while reactive programming is about how mutation propagates through code, so it's an abstraction over specific mutation. This is why they pair nicely together.

Elm was positioned to be this. Still technically could be, but adoption isn’t high. Original Elm had signals.

I'd go even one step further, diffusion/chemical/tissue-level programming. I'm sense the idea of multi agent state morphism becoming a norm in people's mind. (from css to to react to kubernetes).

The native (future) solution for Apple platforms is SwiftUI [1] + Combine [2]. I use it on a side project and for somebody who works with React 9-5 it feels very natural. In short SwiftUI is like React & Combine is like RxJs.

[1] - https://developer.apple.com/tutorials/swiftui [2] - https://developer.apple.com/documentation/combine

While I agree, I’d say I find SwiftUI easier and more intuitive than React. And Combine often much more frustrating than Rx.

The idea in and of itself is not what is being rediscovered. Nobody is reinventing the wheel. It's whether the implementation is ergonomic and powerful that determines whether it catches on.

Signals have become enough of a buzz word that they may catch on just for the hype, regardless of whether they are particularly powerful or ergonomic.

I feel like signals have been a buzzword for almost 10 years now.

May just be my memory failing me or the circles I ran in over the years, but I remember the related buzzwords always being observables and streams. I hadn't seen much interest/hype in the actual term "signal" until solidjs used it and push heavily on comparisons with react hooks.

"Signal" has been used in FRP circles for some time [1,2]. The original FRP stuff was events/signals and behaviours. But I agree that JS didn't use this terminology until more recently. S.js is maybe one of the earlier ones, but that was still over 8 years ago.

[1] https://scholarworks.rit.edu/cgi/viewcontent.cgi?article=651...

[2] https://github.com/14427/signal

[3] https://github.com/adamhaile/S/tree/e897ec1212a073bb1fe695e1...

Surprisingly, FRP doesn't have anything to do with dataflow constraints at all.

In FRP, a program is fundamentally a function of type Stream Input → Stream Output. That is, a program transforms a stream of inputs into a stream of outputs. If you think about this a bit more, you realise that any implementable function has to be one whose first k outputs are determined by at most the first k inputs -- i.e., you can't look into the future. That is, these functions have to be causal.

The causality constraint implies (with a not-entirely trivial proof) that every causal stream function is equivalent to a state machine (and vice-versa) -- i.e., a current state s, and an update function f : State × Input → State × Output. You get the stream by using the update function to produce a new state and an output in response to each input. (This is an infinite-state Mealy machine for the experts.)

Note that there is no dataflow here: it's just an ordinary state machine. As a result, the GUI paradigm that traditional FRP lends itself to the best are immediate mode GUIs. (FRP can be extended to handle asynchronous events, but doing so in a way that has the right performance model is not trivial. Think about how you'd mix immediate and retained mode to get an idea about the issues.)

When I first started working on FRP I thought it had to be dataflow -- my first papers on it are actually about signals libraries like the one in the post. However, I learned that basing it on dataflow and/or incremental computation was both unnecessary and expensive. IMO, we should save that for when we really need it, but shouldn't use it by default.

someone calls conal elliott

A couple of notes:

1. You seem to be confusing "dataflow constraints" with "dataflow". Though related, they are not the same.

2. Yes, the implementation of Rx-style "FRP" (should have used the scare quotes to indicate I am referring to the common usage, not actual FRP as defined by Conal Elliott) has deviated. And has deviated before. This also happened with Lucid.

3. However, the question is which of the two is the unnecessary bit. As far as I can tell, what people actually want from this is "it should work like a spreadsheet", so dataflow constraints (also known as spreadsheet constraints). This is also how people understand when used practically. And of course dataflow is also where all this Rx stuff came from (see Messerschmitt's synchronous dataflow)

4. Yes, the synchronous dataflow languages Lustre and Esterel apparently can be and routinely are compiled to state machines. In fact, if I understood the papers correctly the synchronous dataflow languages are seen as a convenient way to specify state machines.

5. It would probably help if you added some links to your papers.

Mathematically, RxJS and Vue/Solid Qwik like FRP are equivalent. There is interesting proof by Meijer (who invented Rx) in the famous "duality and the end of reactive". https://www.youtube.com/watch?v=SVYGmGYXLpY

> Mathematically, RxJS and Vue/Solid Qwik like FRP are equivalent.

Yes, these are basically the same thing. However, they have little to do with Conal Elliott's "FRP". Which itself is also badly named.

> Meijer (who invented Rx)

Well "invented". What's a bit surprising is that with both Rx and the Rx-style "FRP", there either is no public history of the ideas at all (Rx) or it is patently wrong (Rx-style "FRP"). Or a bit of both.

For example, the "Reactive" part of Rx-style FRP appears to come from the definition of system styles by Harel. Both the connection of synchronous dataflow with the term "Reactive" and with FP languages are made in the paper on Lustre, which is a language that integrates synchronous dataflow into an FP language. But there is nothing inherently FP-ish about synchronous dataflow, it was previously integrated into to the imperative language Esterel, and they also made a variant of C with synchronous dataflow.

Again, nobody mentions this, it is all presented as having been invented out of thin air and the principles of Functional Programming. (Or as having come from Conal Elliott's FRP, which is not true. Ask Conal Elliott).

Once I figured out the connections, I asked Erik Meijer, who has "I am the original inventor of Rx..." in his bio. He admitted that he was "inspired" by synchronous dataflow. And of course that is pretty much all it is. Except they dropped the requirement for it to be synchronous.

What do you get when you drop "synchronous" from "synchronous dataflow"? FRP, obviously ;-)

Just like Objective-C is Smalltalk + C, and Objective-C - C is ... Swift?

Controlling narratives is important.

I think the important contribution of Meijer and RX is realizing the duality of IEnumerable and IObservable which enabled them to use the same programming constructs. The meat of RX is in all the combinators so you can express complex behaviours very succinctly.

Hmm...Smalltalk had the same iteration methods over streams and collections since at least Smalltalk-80, so not sure what the important contribution here is.

(Waiting for the LISPers to chime in...)

An observable is not the same as a Smalltalk stream, directionality wise. Smalltalk streams are like unix streams or a C stream.

Whether the flow is push or pull is a fairly irrelevant implementation detail for a dataflow system.

That doesn't mean it can't have big effects, but if you're relying on those, or they do become relevant, you should probably ask yourself whether your system is really dataflow.

All this is documented in Meijer's actual paper though? (Your mouse is a database) as well as his aformentioned talk (duality and the end of reactive).

He for sure invented observables (as we know it and as the mathematically dual of enumerables) - that doesn't mean it was the first ever reactive system or the concept of data being dependent of other data.

If I remember correctly he also explicitly talks abut Conal FRP and notations in https://www.youtube.com/watch?v=pOl4E8x3fmw if interested

I have seen plenty of Erik's talks and certainly read Your mouse is a database a bunch of times.

What is "documented" there is the alternate history of all this coming out of fundamental insights into FP and dualities and all that.

But that isn't the case, it just sounds a lot better than "I found a really complicated way to map dataflow onto FP concepts".

But we as an industry just love complicated.

Also, as someone who both works on open source reactive stuff (like MobX) and literally works on a spreadsheet app (Excel) I can say what we do is entirely different from what these systems do because of different constraints and both are reactive.

I'm curious. How so?

In a gist: MobX just keeps the whole dependency graph of what data needs to update when what computed runs. Excel can't do that because it has to be able to work on large files with impartial data.

Aren't excel formulas the expression of data dependency graphs?

In which case it is kept in the document?

Isn't the Excel document still in memory?

> any implementable function has to be one whose first k outputs are determined by at most the first k inputs -- i.e., you can't look into the future. That is, these functions have to be causal.

    def f(input_stream):
        i = next(input_stream)
        j = next(input_stream)
        yield i + j
        f(input_stream)

This function produces k outputs when given 2*k inputs, so it's either acausal or impossible to execute. Right?

For GP’s stream-transformer / state-machine equivalence to work, you need to sprinkle option types throughout so each input yields some kind of output, even if empty. So more like

  def co():
      i = yield None # hurray for off-by-one streams
      j = yield None
      while True:
          i = yield i + j
          j = yield None

This won’t help if the output stream produces more than one output item from each input item. You could sprinkle lists instead, but in reality multiple simultaneous events have always been a sore point for FRP—in some libraries you can have them (and that’s awkward in some cases), in some you can’t (and that’s awkward in others).

But I am not criticizing his stream-transformer / state-machine equivalence, I am just curious why he thinks functions of Stream A -> Stream B have to produce exactly 1 output for exactly 1 input.

Now, I know that Haskell in its pre-monad days used to have main have signature [Response] -> [Request]: the lists being lazy, they're essentially streams. Each Request produced by the main would result in a Response being provided to it by the runtime. This model actually has to be strictly 1-to-1, and indeed, it was so easy to accidentally deadlock yourself that switching to IO monad was quite welcomed, according to SPJ in his "Tackling the Awkward Squad" paper.

I guess what I wanted to say was that to me (given that the comment was presumably targeted at people who already know how all of FRP, stream transformers, and state transducers work, or can at least make a good guess) it was within the limits of acceptable sloppiness to mix up (State, Input) -> (State, Output), (State, Input) -> (State, Maybe Output), and (State, Input) -> (State, [Output]), or the equivalent on the stream transformation side. The point of the comment does not really depend on what you choose here.

> Note that there is no dataflow here: it's just an ordinary state machine.

It sounds like you've converted data-flow to its state-space form. It's still data flow, just in a variant that might be easier to compute.

FWIW you probably need a pair of functions :

    next state = F(input, current state)  
    output     = G(input, current state)

Which in the signal-processing/control systems world is

    s = Ax + Bs
    y = Cx + Ds

Aka the "state space" formulation where A, B, C, and D are matrices, x is the input, s is the state, and y is the output. There are infinite ways to formulate the state space and infinite equivalent signal flow graphs that represent the same thing.

A language/framework that explored well this idea is Elm, which is used in front-end apps. It started with Signals and FRP, then 7 years ago changed course to the Model-View Update architecture [1], today known as The Elm Architecture. It is still a bit niche, but a handful of frameworks in other languages like Rust are following this architecture for GUI apps.

[1] https://elm-lang.org/news/farewell-to-frp

It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation.

But the implementations are rarely extracted out for general purpose usage and rarely have a rich API.

I've been thinking a lot about a general purpose "epoll" which be registered on objects that change. I want to be able to register a reaction to a sequence of actions on arbitrary objects with an epoll style API.

One of my ideas is GUI thunking. The idea that every interaction with the GUI raises a new type that can be interacted with, to queue up behaviours on the GUI. This is essentially Future<> that are typed and the system reacts to the new type based on what you did and presents a GUI that is as if the operation you queued up was completed. (You can interact with the future because any action on something that isn't done yet, is queued up)

It's a bit like terraform plan and apply, but applied to general purpose GUIs.

For example, you can click download file, then queue up installation and then using the application, ALL BEFORE it is installed. Because the actual computation is separate from the type information that was queued up.

Imagine using AWS Console to set up an entire infrastructure and wire everything together but not actually execute anything until the very end when you click "Run".

https://github.com/samsquire/gui-thunks

I feel we are still early days with regard how to build computer user interfaces that are easy to build, maintain and understand the code for.

I used knockout and angularjs 1 and I enjoyed Knockout especially. ko.observables and the map plugin makes creating reactive code very straightforward.

> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation.

Angular relied on RxJS.

And RxJS is itself an implementation of the language-agnostic Reactive framework.

"It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation."

This rings so true to me.

I've recently realized how every single non trivial part of my app is in fact a workflow problem : it could be ideally written as a pipe of asynchronous steps, glued together. It's true both for the frontend part and the backend.

I believe that's the point of reactive frameworks, but somehow those frameworks are usually designed around continuous streams of incoming events. Which isn't what i've noticed is the most widespread case. One-shot instanciation of pre-designed workflows would be really ideal.

Thanks for this thoughtful and insightful comment.

Many problems can be workflow problems, sometimes even pulling in a rule engine, or require a job queue to do things that can fail.

Then you have software such as https://temporal.io/ which is really powerful for resilient workflows.

Imagine coordinating the user with a workflow with asynchronous data collection steps.

Imagine programming "reaction to user behaviour as a workflow engine". Can coordinate global user behaviour with a resilient workflow script.

   await user.login();
   if (user.showTutorial()) {
     await user.tutorial();
   }
   await user.checkout();
   await user.submitOrder();

You could have seamless weaving of code to be executed on the frontend, or on the server, the workflow is cross-machine and cross job queue.

This workflow engine of yours seems to expect a very predictable and linear sequence of actions. What happens if the user clicks "log out" after user.tutorial();

It's basically making an already easy case easier.

I assume you would have conditions for each workflow which are then pattern matched to user behaviour to see if that workflow is relevant.

You still have a globally defined happy path of coordination but your overarching application logic isn't spread everywhere but contained in one place.

The preconditions for the logout would trigger a different workflow.

I am the author of additive-gui, which is based around the idea that you provide all the rules of the GUI and the computer works it out - what applies when. Additive GUIs loosely models dataflow between components and layout.

https://github.com/samsquire/additive-guis

It's not ready for anything, it's just a proof of concept. But it doesn't really implement workflows yet.

I would rather maintain a codebase that uses this pattern of user workflows, similar to state management than sprinkled logic everywhere.

Think of workflow engines as a runtime, not a hardcoded sequence of steps that can be activated or deactivated based on events, like a state machine.

Real workflow engines (the ones you find in large companies used to model business processes) handles conditions and branching.

But my personnal experience is that linear flows work really well in the context of apps with just those two additions :

The ability to cancel a running flow. The ability for a step to « do nothing » if the conditions for its execution aren’t met.

Even linear workflows can handle conditional execution, you just lift the condition into a value, like into a Maybe/Option type, and later stages only execute if there's a value.

Or if you want to be more literal, declare a specific type:

    type Authenticated(User) = No | Yes(User)

And each stage of your workflow matches on Authenticated.Yes, and so only executes if the user is authenticated. That's basically what any system does internally, this just makes that implicit behaviour explicit.

I’m a big fan of temporal and xstate (xstate for UIs in particular) lately, but they both present a major issue in my mind. They provide excellent foundations to create very reliable software, but the tools and conventions they offer are hard to learn and to work with.

That is such a huge setback because virtually no one I work with is willing to learn these tools in order to write better software. Even people I think are very intelligent (certainly more so than I am) think they can write these kinds of tools themselves ad hoc, as needed. It simply isn’t true; it’s a bad idea almost all of the time.

If we could find some way to make these tools more intuitive and attractive to depend on, I think it could be literally transformative. I know similar tools are popular in more engineering-centred software, so it isn’t necessarily impossible. On the web and mobile software side at least, getting people to define their application states and flows with any rigour seems to be like asking someone to file their taxes and write an essay about it afterwards.

But I also get it. These tools are a lot to absorb. Sometimes it feels like they’re in the way. Though I’d argue that when they feel like they’re in the way, it’s often because you didn’t anticipate a workflow stage or application state and its absence in your mental model is making the tool hard to use because they simply won’t accommodate broken workflow or state models. That’s a good thing, and something we should want from our tools. It’s something many people love about Rust, for example. Yet again though, many feel as though Rust gets in the way as well.

Thanks for your comment and thoughts.

I think this tech is harder to get started with than not using it and that's a problem you've noticed.

I played with temporal on my workstation and thought it was really interesting but it is more things to deploy and maintain in exchange for reliability and robustness.

I had the choice between using Rust or C recently, but because the domain was new to me I chose C to get it done faster. Rust definitely has a learning curve.

Thanks for the compliment. Since you seem to like it, i’ll emphasis a little as to how i came to realize that :

It’s very common that workflows are suited for data manipulation: write the data to disk, when it’s complete send it to the network, then once it’s completed, etc. I/O are known to be asynchronous, so we’re already equiped for that.

What i noticed is that the screen of your app is also a source of asynchrony, and as such, everytime we interact with it (animations, transitions, waiting for user interactions), we’re actually dealing with problems of exactly the same nature.

Would love to talk more.

You're right, users are asynchronous event sources. I feel UI development is very hard to read and complicated to implement complicated features.

Turning desired behaviours into code is pretty difficult for me.

I have written a few Java Swing apps and an Electron app. I also tried to use Qt but I wasn't a C++ developer so that didn't go so well.

There is a good article that the mouse is a database. The idea that the mouse creates data that can be reacted to. https://queue.acm.org/detail.cfm?id=2169076

> the screen of your app is also a source of asynchrony

Exactly!

That’s why wrapping a “dialog” or “form” or “screen” into a Promise is such a powerful technique. When the user closes the dialog, the promise resolves with a result (e.g. whether the user clicked OK or Cancel), which then you can use for whatever else needs to be done, including invoking another dialog/form/screen!

This makes UI composable, and with async/await “hiding” the promise continuations, the syntax for doing that is essentially the same as when composing ordinary functions.

Have you ever worked with redux saga? It’s exactly what you’re describing.

I often wonder about some sort of unified distributed system, encompassing frontend & backend into a single whole. One where user input is just something the system can ask and wait for, no matter which frontend or client it comes from.

But I’ve only ever been able to catch glimpses of it. More of a nebulous feeling and intuition than a real understanding of how such a thing would work. Something that feels obviously right, that will make perfect sense once understood, but that I still can’t begin to grasp.

It’s frustrating.

I’m also pretty sure I must not be the first, and that it either already exists or involves some complexity, maintainability or evolution issues I have no idea of.

Your example reminded me of "it".

I like your thinking a lot.

There was a Haskell project that seamlessly transferred state between frontend and backend but I don't think it was a distributed system. I don't remember what it was called.

Writing APIs to glue together data fetching and actions and GUI state is all very siloed. If you could talk about the system as a whole including GUI interactions at the same time as system interactions that could be truly powerful.

Imagine multi omnichannel event streams that map to the users notifications, email inboxes, chat interface, post, deliveries, accounting, customer data, synchronisations, integrations, microservices and business CRM and ERP. Everything is linked together by powerful workflows. An interaction with a customer is just an extension of the system. It's a distributed system of human tasks as well as digital tasks and interactions between the customer and the company.

Exactly!

The first obvious risk that comes to mind is that nothing can be made that correctly takes into account everything it might some day be required to be able perform, so it can easily be done wrong and would probably need to be very flexible / loosely coupled.

And while the distributed aspect adds complexity, I feel that involving different devices, not all of which will on, or online, all the time, makes it a necessity. Not accounting for that would doom it.

But yes, that's a big part of it.

For the interaction / capabilities discovery part, I've lately been drawn to some sort of declarative interface, akin to Apple's "App Intents", dynamically exposed depending on the system's state. It also reminds me of how REST APIs were supposed to be discoverable.

But I'm not sure, and it could be a dead-end.

Another thing that comes to mind is Bell's Plan 9, and how someone who used to work there said that when he would come home, he'd just open his computer and everything would be there, just like he had left it at work. It's not enough, and the goal wouldn't be to have the exact same interface replicated everywhere, but this single distributed state, with each device just being a window into it, feels like a start.

Not that "it" would be an operating system. That too would be a doomed effort (many people can't change their OS nowadays, and most people who could wouldn't do it just to use a product or service). It would have to a paradigm, and perhaps a framework, or a language.

You've got me thinking again. I'm not going to be able to get anything done for days now.

https://github.com/hyperfiddle/electric ?

Oh! Thanks! It's interesting. I'm not familiar enough with closure, and I can't tell how well it handles distributed states, but I'll try to play around with it. Thank you!

I think this is true for a subset of software. Although I think a more general truth is that how you organize data pretty is what defines your software. It's the root of virtually all of your problems and successes in designing software.

It's also why it's so unfortunate data modelling is often ad-hoc by defaulting to some bucket-of-json model with no regard for the needs of the application.

I also think surprisingly often, given some ideal data modelling, it's both faster and easier to use synchronous processing because you no longer end up having data far away in weird formats.

> [constraints] the implementations are rarely extracted out for general purpose usage and rarely have a rich API.

Yes.

One of the goals for Objective-S [1] was that it should be possible to build constraints using the mechanisms of the language (so not hardcoded into the language), but then have them work as if they were built into the language.

Part of that was defining how they should look, roughly, and figuring out the architectural structure. I did this in Constraints as Polymorphic Connects [2].

For syntax, I use the symbol |= for a one-way dataflow constraint and =|= for a two-way dataflow constraint. This combines the := that we use for assignment and the | we use for dataflow. Also relates the whole thing to the idea of a "permanent assignment", which I think was introduced in CPL. The structure is simple and general: you need stores that can notify when they are changed, and a copy-element that then copies the data over. At least for one-level constraint. If you want to have multiple levels, you can

I was very surprised and happy when I discovered that I had actually figured this out, sort of by accident, when I did Storage Combinators [3]. There is a generic store that does the notifications, which you can compose with any other store. The notifications get sent to a "copier" stream which then copies the changed element. Very easy. And general, as it works for any store.

For example, I have been using this to sync up UI with internal state, or two filesystem directories. And when I added a store for SFTP support, syncing to SFTP worked out-of-the-box without any additional work. ("Dropbox in a line of code" is a slogan a colleague came up with, and it's pretty close though of course not 100%)

[1] http://objective.st (Site currently being revamped)

[2] https://dl.acm.org/doi/10.1145/2889443.2889456?cid=813164912... / http://www.hpi.uni-potsdam.de/hirschfeld/publications/media/...

[3] https://dl.acm.org/doi/10.1145/3359591.3359729 / https://www.hpi.uni-potsdam.de/hirschfeld/publications/media...

> But the implementations are rarely extracted out for general purpose usage and rarely have a rich API.

SolidJS pitches its state management as being robust enough for use outside of itself.

IMO, most state management tools baked into ui frameworks aren’t robust enough to be worth extracting.

I too loved Knockout's observable model - I've found Mobx a worth successor. As it's entirely independent of any framework it is very practical for general purpose usage. I use it for reactive state with both React and in backend Node code with no problems.

Jane Street/Yaron Minsky/at al came to similar conclusion with their formalized incremental implementation [0] - that this type of computation reappears in different contexts (build systems, ui, optimal data processing/display on large volumes of high frequency data etc). It's very interesting approach indeed.

[0] https://blog.janestreet.com/introducing-incremental

I wonder if there is some wisdom in Rich Hickey's Transducers

There is also differential dataflow.

I feel all the ideas are related and could be combined.

What I want is a rich runtime and API that lets me fork/join, cancel, queue, schedule mutual exclusion (like a lock without a mutex), create dependency trees or graphs.

I am also reminded of dataflow programming and Esterel which a kind HN user pointed me towards for synchronous programming of signals.

May be of interest: I released a port of Incremental to Rust the other week. It has a bit of a way to go in polish and docs, but the core implementation and API should be very familiar. (To the extent that it uses GADTs for node kinds just like the OCaml!) It has Expert nodes so incremental-map is feasible and already works. I’ve been using it as the state management for some UI with much success. Credit to the authors because it’s a good design. https://github.com/cormacrelf/incremental-rs

> It's interesting how every build system, frontend framework, programming language implements its own promise pipeline/delayed execution/observables/event propagation

This is exactly the reason to use state machines/data flows IMO.

Every implementation is unique enough that simply following how data flows through the different states and transitions, and where the sinks and funnels are, will tell you everything you need to know about what your system is actually doing at any point in time.

The challenge there is, things like that are a shitload of instrumentation and requires a lot of forethought to not just jam everything into a framework that puts boundaries on what you can design and implement. So for 99% of applications, it's not worth the hassle and you're better off with just basic text documentation.

I believe the consequence of this line of thinking is the old idea of writing domain-specific interpreters. In your example, the UI generate a list of commands, and you execute/reduce over at an arbitrary time to get to a new global state, instead of tangling all the UI code with dispatching commands and managing state immediately.

I'm not sure generalisation here is ideal - quite often these are specialised implementations for specialised cases, and using them in other situations tends to only work so far.

For example, I'm currently working on a spreadsheet-style tool where the reactivity is largely being handled by SolidJS signals. It works fairly well up to a point (and that point is probably good enough for the client's needs), but it's very clear that there are big limitations here, and a more complete solution would bundle its own reactivity system. Things like computing results that spill across several cells just don't map cleanly onto conventional signals, so we instead have lots of ways to manually trigger recomputation, rather than just setting up the perfect pipeline flow. Likewise, figuring out where data loops are happening just isn't really possible.

That's not to say that SolidJS is bad for this sort of stuff - it has been great, and it's impressive how well the underlying reactive primitives work even for this project. But I think even when the underlying theory between these tools is pretty similar, the practical tradeoffs that need to be made are very different. And as a result, the different libraries servicing these different use cases will look very different.

I suspect this is the reason why these implementations are rarely extracted out more broadly. The sort of system that works well for one situation will rarely work so well for another.

This sounds almost like you've just invented Haskell-like `IO`.

All operations get "queued up" in IO and only run "at the end of the world".

To write your program you `flatMap` over the `IO`: An `IO` contains the computation(s) that will run at some point, but you can map on the result of them right away, and return another `IO` value; than `flatten` the `IO[IO]` data structure (which makes `flatMap`). In Haskell you have extra syntax for `flatMap` called "do notation". In Scala, where there are library solutions for `IO`, you can use "for comprehensions" instead. In both cases the nested `flatMap` calls get sequenced. This way and you can write code almost like a consecutive chain of imperative procedure calls but it all gets "queued up" and the whole program only runs when the `IO` data structure gets evaluated by the runtime ("at the end of the world", as last call in your program).

But that's not really related to data flow.

Not sure I like the mixed push/pull approach. If you're already traversing the tree to mark nodes as possibly dirty, you might as well recompute the node's value and store it while you're there. Otherwise on pull/lazy update, you're traversing the tree all over again! Terrible for cache locality, particularly for large graphs.

You might be tempted to say that the lazy approach might avoid some recomputations, but if a node isn't actually going to be accessed then that node is effectively no longer live and should be disposed of/garbage collected, and so it will no longer be in the update path anyway!

The mixed push/pull approach has only once nice property: it avoids "glitches" when updating values that have complex dependencies. The pull-based evaluation implicitly encodes the correct dependency path, but a naive push-based approach can update some nodes multiple times in non-dependency order. Thus a node can take on multiple incorrect values while a reaction is ongoing, only eventually settling on the correct value once the reaction is complete.

In other push-based reactive approaches, you have to explicitly schedule the updates in dependency order to avoid such glitches, so perhaps this push/pull approach was picked to keep things simple.

I implemented the eager recomputation model for the observable utilities in vscode [1] and it quickly fell on my feet because of these glitches.

In particular this is problematic if you have observable optional state that has inner observable/derived state and someone reactively reads the outer state and then it's inner if the outer one is defined.

Then you clear and dispose the outer state and at the same time set some other observable value that the inner derived depends on. With eager recomputation, it can now happen that the inner derived is recomputed, even though the inner state is disposed.

[1] https://github.com/microsoft/vscode/blob/fe9154e791eafb4f18d...

Yes, if you want to better tolerate glitches you have to separate internal and external reactivity, and only run the external ones after the full reaction is complete. I think this would prevent the scenario you describe, assuming your operators are well-defined.

The other option is to use FrTime's approach and only update nodes in dependency order.

With dependency order you mean dependants before dependencies? (and dependencies lazily when they are requested again by the dependant)

If you update dependencies before dependants, the dependant might not depend on all it's dependencies anymore (because a derived might depend on A only if the observable B is true) and you do too much work/run into glitches.

Dependency order = values are computed in the same order as they would be in a purely pull-based system, which is intrinsically glitch-free

Push-based systems permit better efficiency and minimal state changes, but they should endeavour to preserve the above property for external observers.

Can one not just note the timestamp of last change with a monotonic clock. Then, dependencies check the timestamp of last computation, and pessimistically determine if recomputation is needed.

Seems like the trouble here is you'll have to traverse the tree every time to check timestamps but if the dependency is dirty that needs to happen anyway.

That's effectively a pull-based system, which works fine but is inefficient. With a binary dependency tree of depth N, you have to touch 2^N nodes in such a pull-based system every time you evaluate it, no matter how many nodes you may have updated.

If you only updated 1 node, a push-based system will only update nodes that have changed, which will be considerably less (likely linear in depth). For instance, consider:

    var evenSeconds = clock.Seconds.Where(x => x % 2 == 0);
    var countEvents = evenSeconds.Count();
    var minutes = clock.Seconds.Count(x => x / 60);
    var hours = minutes.Count(x => x / 60);

Even though evenSeconds and countEvents is only updated every other second, and minutes once every 60 seconds, your pull-based approach will have to check all nodes up to the root every time clock.Seconds changes.

In a push-based system, clock.Seconds would trigger seconds+1. If that's not even, propagation stops there, if it is even then this updates evenSeconds, which would then trigger an update for countEvents. Ditto logic for minutes and hours.

You can see the push-based system permits minimal state changes via early termination if downstream dependents won't see any changes.

In a push-based approach, can you bundle state changes into a transaction and recompute derived state when the transaction ends and all inputs have finished changing?

That's effectively what I suggest in the other replies. You separate internal from external callbacks, and run the internal ones first and enqueue the set of external ones, then run those at the end. All subsequent externally-driven changes should be enqueued until the next "turn", ie. after all changes have settled. This preserves transactional semantics.

I spent 11+ years at Goldman, working in SecDb Core, and various SecDb-related infrastructure.

Goldman's Slang language has subsets of both lazily-evaluated backward-propagating dataflow graph ("The SecDb Graph") and forward-propagating strict-evaluating dataflow graph ("TSecDb"). They both have their use cases. The lazily evaluated graph is much more efficient in cases where you have DAG nodes close to the output that are only conditionally dependent upon large sub-graphs, especially in cases where you might be skipping some inputs, and so the next needed graph structure might not be known at invalidation time.

Ideally, you'd have some compile-time/load-time static strictness analysis to determine which nodes are always needed (similar to what GHC does to avoid a lot of needless thunk creation) along with some dynamic GC-like strictness analysis that works backward from output nodes to figure out which of the potentially-lazy nodes should be strictly evaluated. In the general case, the graph dependencies may depend upon the particular dynamic values of some graph nodes (the nodes whose values affected the graph structure used to be called "purple children" in SecDb, but that lead to Physics/Statistics PhDs coming to the core team confused by exceptions like "Purple children should not exist in subgraph being compiled to serializable lambda")

TSecDb already contains a similar analysis to prune dead code nodes from the dataflow DAG after the DAG structure is dynamically updated. (For instance, when a new stock order comes in, a big chunk of TSecDb subgraph is created to handle that one order, and the TSecDb garbage collector immediately runs and removes all of the graph nodes that can't possibly affect trading decisions for that order. This also means that developers new to TSecDb often get their logging code automatically pruned from the graph because they've forgotten to mark it as a GC root (TsDevNull(X))... and it's pretty bad logging code if it affects the trading decisions.)

Risk exposure calculations (basically calculating the partial derivatives of the value of everything on the books with respect to most of the inputs) are done mostly on the lazy graph, and real-time trading decisions are done mostly on the strict graph.

Sounds cool and quite intricate. Distributed reactive computations might indeed change the basic logic I described. Most reactive systems I've worked with and thought about are local.

I took a similar approach in my Racket library, gui-easy[1,2]. Though I opted to not defer any computations. Any observable (similar to a signal from the post) update propagates to observers immediately, and there's no incrementality -- observables are just boxes whose changes you can subscribe to. Regarding the disposal problem, I used weak references and regarding the where to take observables and where to take concrete values as input question, I decided that any place an observable can go in, a concrete value can as well and it's been a convenient choice so far. For fun, here's an example[3] that builds the todo UI from the post.

[1]: https://docs.racket-lang.org/gui-easy/index.html

[2]: https://www.youtube.com/watch?v=7uGJJmjcxzY

[3]: https://github.com/Bogdanp/racket-gui-easy/blob/364e8becaafa...

Elm started as a FRP language and ended up giving up FRP[1] for ease of use.

If ease of use is targeted, signals might not be the best approach. I distinctly remember things becoming easier when they went away.

[1] https://elm-lang.org/news/farewell-to-frp

Thanks for the link.

That was one of the most interesting developments in the already fascinating saga of Elm. Czaplicki gets a lot of flak, but it's almost all about how he keeps tight control of his language & ecosystem. No one duns him for brains, because he's clearly very smart. So when he says, "everything related to signals has been replaced with something simpler and nicer", it's news, eh?

Topical, because Angular 16 was just released with ... Signals.

Used in some places to simplify RxJs or not need it at all. Some, not all.

They are looking towards a future where they can get rid of Zone.js and its strategry to change detection. I see this as a step along that part.

https://blog.angular.io/angular-v16-is-here-4d7a28ec680d

The author does a lovely job of covering a number of the interesting ideas in this space. But reactive programming is such a tough sell. I know from experience.

I maintain a reactive, state management library that overlaps many of the same ideas discussed in this blog post. https://github.com/yahoo/bgjs

There are two things I know to be true:

1. Our library does an amazing job of addressing the difficulties that come with complex, interdependent state in interactive software. We use it extensively and daily. I'm absolutely convinced it would be useful for many people.

2. We have completely failed to convince others to even try it, despite a decent amount of effort.

Giving someone a quick "here's your problem and this is how it solves it" for reactive programming still eludes me. The challenge in selling this style of programming is that it addresses complexity. How do you quickly show someone that? Give them a simple example and they will reasonably wonder why not just do it the easy way they already understand. Give them a complex example and you've lost them.

I've read plenty of reactive blog posts and reactive library documentation sets and they all struggle with communicating the benefits.

Is it because of cognitive load? I briefly looked at your project, and I'm still not convinced that your solution is easier to comprehend than

function onLoginClick() { validateFields(); networkLogin(); updateUI(); }

I think you are absolutely correct. In this simple example, it's not easier to comprehend. Especially for someone who already understands how composing software out of function calls works.

My goal with that example was to point out how there are implicit dependencies between validateFields() and networkLogin() and updateUI(). They need to be called in the correct order to make the program work. Our library makes those dependencies explicit and calls things for you. It's not a big deal when we have a 3 function program. But when we have dozens or hundreds of interdependent instances of state, those implicit dependencies become a real struggle to navigate.

Now we're convinced our library works well. We use it every day. But it's also very reasonable for you to be skeptical. As you say, there's cognitive load. As a potential user, you would need to spend your time to read the docs, try a few out ideas, and understand the tradeoffs. That's a lot of work.

I'm glad you took a look at the project, though. The fact that we've failed to make a case for it is certainly on us. Which gets back to my original point. I don't know how to sell a solution to complexity.

Curious as to choice of 'demands' vs 'dependency' and 'supplies' vs 'provider'. The latter of these are fairly commonly understood and used.

As an aside, "complex, interdependent state" is possibly one of the few areas that lend themselves naturally to visual programming (which is a fail in the general programming case). Why not just draw graphs?

> I've read plenty of reactive blog posts and reactive library documentation sets and they all struggle with communicating the benefits.

I just googled 'visual programming for reactive systems' and this (interesting project) turned up:

https://openmusic-project.github.io/openmusic/doc/reactive.h...

So the approach is specifically addressing complex interaction patterns between components. To highlight the solution, just do what these guys did: here you can see the benefit of 'reactive components' just by looking.

https://openmusic-project.github.io/openmusic/doc/images/mid...

(2c - good luck w/ the project)

It's a reasonable suggestion about the naming of demands and supplies. We landed on those names because they had some of the correct connotations, and it was easy to be clear with each other what we were talking about. A "Demand" is a very specific thing when using Behavior Graph. "Dependency" is a common term that means half a dozen things in a program at any given time. "Supplies" is just a reasonable opposite to "Demands".

That being said, we do often consider renaming them and other parts to feel more mainstream reactive. But honestly, I secretly suspect that fiddling with names won't make a difference to anyone.

I do also agree that there is some appeal to visual programming paradigms. It's pretty easy to look at the examples you linked to and get some quick sense of understanding. But those typically work well when there are a handful of nodes and edges. The software we maintain at work can have 1000's of nodes at runtime with more than 10000 edges. There's no visual representation that won't look like a total mess. Whatever their faults, text based programming languages are the only approach that have stood up at scale so far.

So our library is just a normal programming library. You can use standard javascript/swift/kotlin to write it.

Thanks for your feedback! :)

That sounds very familiar to me. I'm now retired but in my last job I put together a simple C++ framework to do something similar. I used this in a small part of a financial trading application. Like you, I was convinced it would pay off to apply much more widely, but I found it difficult to "sell" the idea. As you say, the real payoff comes in complex situations, which simple examples don't really convey. However lots of people seem to be playing around with reactive programming now, so perhaps it's an idea whose time is coming.

The previous article in the series is also a very interesting read: https://tonsky.me/blog/humble-state/

Thanks for the link. I really enjoyed reading both

This is what MobX does and so does Vue and a few other frameworks like Solid or Qwik (though with a different rendering model) - hardly a novel idea.

Heck even in lisps there was reagent which was basically this and had atoms/signals :)

That site really shows the best way to do dark mode

It is a fuck-you dark mode

The yellow background is a bit abrasive, so I was happy to see a dark mode switch.. but yea, fuck me.

Also known as "real dark mode": https://twitter.com/i/status/1657755808300519426

It predates it by a few years https://github.com/tonsky/tonsky.github.io/commit/fd5e59a837...

Interesting that this is Clojure and it doesn't mention Hoplon/Javelin[0] as prior work. I've used Hoplon/UI[1] to implement a moderately complex web app ~6 years ago. The library is practically a prototype, pretty much dead by now. However, I found the ideas interesting.

I find the biggest benefit of using a fringe library like this is the ability to read and understand the whole implementation. It's really simple compared to something like React.

[0]: https://github.com/hoplon

[1]: https://github.com/hoplon/ui

Well, it’s mentioned in a sense that “Reactive UIs never became mainstream” :( Would be interesting to understand why

FYI, they recently resumed activity on Hoplon. Good to see!

This looks really interesting but the colour scheme makes it impossible to read.

Have you tried the dark mode? I thought my browser was broken when I turned it on.

Wow, truly night mode. Hilarious!

Should really call it spotlight mode!

On a side note, if you do want a normal darkmode, the Darkreader extension works very well including on this site.

Too much skeuomorphism at its finest: real "dark" mode that it simulates pointing a flashlight.

It's one of the first times I've been delighted by an unexpected UI in years. We've all settled on conventions in a world where anything is possible.

I actually found it surprisingly comfortable to read, but of course, everyone is different. You can always use the browser console to disable/change the `background-color` (in this case also `background-image`) CSS property of the body.

The fact this comes from the same author as https://grumpy.website/ makes it extra hilarious.

Yeah, toggling a checkbox should toggle the checkbox.

DarkReader is your friend.

https://darkreader.org/

I use it on every site except a select foew that have beetter Stylus UserStyles.

HN happens to be one i use a custom stylesheet for.

I'm working on a project to make bootstrap work without js. Similar state management, check at https://noojs.pages.dev/components/accordion.html https://noojs.pages.dev/components/modal.html

This article would be more compelling if the demo videos showed a UI that actually worked. I'm left wondering if their day job is building consent dialogs for websites.

What do you mean by “actually worked”?

The checkmark never disappears in the checkboxes however the standard behavior for a checkbox is that the checkmark should disappear (or appear) when the checkbox is toggled.

Furthermore, towards the end of one of the videos, the checkmark seems to turn from green to gray and back to green again with a single click.

The observable syntax is confusing. Lisp uses *earmuffs* syntax for global variables, if you only use one muff for an observable variable, how would you express a global variable that's observable? Using **lopsided muffs*?

Unicode to the rescue: *:eyes_emoji:earmuffs*

Clojure only uses earmuffs for dynamic vars

Designer of this theme shoud be stabbed with a keyboard.

Downvotes are well deserved, I didn't mention that my comment was about night part of the theme.

Reply on news.ycombinator.com