Hacker News Reader

Comment by amath | Hacker News Reader

Author of the post here. I would love to hear thoughts on folks experience using the different architectures and more pros and cons to each one. There was surprisingly not a lot of information comparing these different architectures that I could find and I would love to be able to update this post with more details.

View on HN

There's the Seven Concurrency Models in Seven Weeks book. It's a bit old, and not so much about frameworks, but more about programming languages and their available threading models or alternatives to pure threads. It still covers a lot of the same theory, but not distributed, and a bit lower level.

https://pragprog.com/titles/pb7con/seven-concurrency-models-...

I put here an aside: All this is super-complicated for most use-cases.

What I have never used, but is inferred from https://en.wikipedia.org/wiki/List_of_concurrent_and_paralle... and in special the original CSP is how all this lacks total control in build your network patterns.

In special, is my dream to have a language that allows to express better what is explained starting here:

https://zguide.zeromq.org/docs/chapter2/

There is not (AFAIK) languages that make nice to properly wire this AND keep the semantic understanding/composability of it.

For example, is hard to know if a function if is running in a thread/process, what is their priority, etc.

I imagine that a language with orchestration (like Erlang/Elixir) in a explicit way will be far easier, but is hard to figure how do it with good composability

For static composability, a "ported graph" model makes a lot of sense. This model has graph nodes representing a unit of processing with its input and output data coming through labeled "ports" and edges connecting the ports. Subgraphs can be defined which then appear as a single node in constructing a larger graph (the composability). Such a model can then be mapped to an implementation (ie, used as configuration) in an abstract way that leaves the ability to select the data transport (edges) and graph execution independently from the graph itself. That is, the same graph may execute in a single multi-thread (MT) process or as a multi-process (MP) distributed application.

Most simplistically, ports are designated as "input" or "output" (from point of view of the node). But, these designations can be expanded so that ports can carry designations matching the ZeroMQ socket patterns. The implementation can then leverage the transport agnosticity of ZeroMQ when supporting adaptation to MT vs MP but with the cost that all edges require data serialization instead of pointer passing.

besides Ray, are there other tools or platforms that embrace the agent actor architecture model?

https://www.erlang.org/ https://elixir-lang.org/

Elixir is amazing, see also stuff like https://github.com/bitwalker/libcluster . I'm always surprised there's not more excitement generated by the possibility of having language-level features that would normally require all the weight of something like k8s. I don't get the impression that many people are seriously using it for data-engineering work though, which seems like a shame

Elixir definitely!

If you need to use Go, there is also ergo: https://github.com/ergo-services/ergo

Yet another cool tool is lunatic: https://github.com/lunatic-solutions/lunatic

Estuary Flow derivations [1], I suspect (I work here)? You write a program which has messages pushed to it, which maintains and updates an internal state, and which publishes output messages.

You can build cyclic data-flows [2] - not just DAGs - which seems a key ingredient.

You can also write contract-based tests that verify the end-to-end behaviors of your "agents" [3].

Today, you're limited to SQLite or TypeScript. Looking forward, we aim to open it up to any program packaged in a docker container.

[1] https://docs.estuary.dev/concepts/derivations/

[2] https://docs.estuary.dev/concepts/derivations/#approving-tra...

[3] https://docs.estuary.dev/concepts/tests/

Akka uses the actor model. https://doc.akka.io/docs/akka/current/typed/actors.html

IIRC, Akka based their implementation on Erlang (possibly OTP), but it's been a long time since I did anything with either Erlang or Scala, and I never used Akka in Java.

Akka was indeed inspired by Erlang. For a modern take, there's Apache Pekko (incubating), a fork of Akka 2.6.x https://github.com/apache/incubator-pekko

Dask added an actor model after seeing it in Ray. I’ve used dasks in some computationally intensive applications (already had existing dask infrastructure which is why we didn’t go with rays)

Reply on news.ycombinator.com