Hacker News Reader

This could be pretty cool for a real time trading system. Go would be pretty solid for the numerical stuff and then BEAM could be used to coordinate all the trading agents. I have written some Elixir modules that call out to C binary executables to do computations that I needed to be faster than what could be achieved with the Erlang/Elixir standard libraries, but had to perform the business logic that called out to these faster implementations in the Elixir code. Will be interesting to watch where this goes.

Would NX help your use case for computations in Elixir?

I haven't looked at it much. is Nx just like the equivalent of pandas DataFrame?

I could have some use for it but I haven't really checked it out.

I'm working on a trading system separately from my initial command on this HN post, and tried to pull in all the new fancy Elixir libraries for machine learning stuff (axon, xla, exla) but had some issues with the protobuf dependency locally.

Nx is more like Numpy, Explorer is the one that provides data frames (closer to dplyr than pandas though). We provide precompiled assets for both, so hopefully it eases installation. :)

As someone without a lot of knowledge of GO and with only a trivial understanding of OTP/the gen_server impl, how would this add friction? What exactly is wrong with it and what would be a better alternative?

I go over this fairly deeply in https://www.jerf.org/iri/post/2930/ .

The Big Idea is that it's rarely a good idea to transliterate a library. You should translate a library. When I was done translating into Go, I found that what was idiomatic in Go was much simpler than Erlang. To some extent this is because Go simply can't do some things Erlang can do (in particular Erlang is one of the very short list of languages that can safely use "asynchronous exceptions", or to put it another way, can let one thread simply reach out and nuke another thread, so safely it can be incorporated into the core design of systems rather than treated as a dangerous special case)... but if that is the case, why port over the aspects of the Erlang design that are now dangling free in space, unconnected to anything?

Maybe the answer to the challenge of asynchronous exceptions is that “threads” are not well suited to robust parallelism.

Certainly, I reached this conclusion in the past. A better paradigm is the “process”. Unlike a thread a process can be killed arbitrarily (your asynchronous exception), and completely safely (all resources are released etc). Software can be composed of a collection of processes and thus be much more robust (e.g. Chrome). It’s great that Erlang built this into the language itself, but any language can leverage processes for the same effect.

As an aside, Lunatic (leveraging wasm to create “process” like sandboxes inside one process), looks very promising as a way of leveraging the power of process isolation without all the fuss.

I took a pretty good distributed computing course in college, and I loved threads when they were 'new' to most people, my way of representing information in my head was pretty amenable to thinking about concurrent code. But what I found over time is that I was seeing a lot of bugs (and not seeing a few), but I could never manage to explain to other people how to avoid the problem. I could show them the code change that fixed it, and they mostly understood, but I spent way too much time following people around doing cleanup work.

Shared state threads are simply not fit for human consumption, and if you're in the (for argument's sake) 5% who understand them well, you should consider if you really want to inflict them on your coworkers. Even if you agree to write the 2-10% of critical code that needs esoteric language constructs, your coworkers are still likely to have to step through your code and the longer they're in a cloud of unfamiliar code the harder it is for them to connect cause and effect and fix their own bugs.

You end up playing yourself by creating a class of bugs that preempt you from other work you may have been enjoying doing or is potentially blocking other people.

A functional core architecture avoids a lot of these issues because you can get away with a lot more threading when you have a block of code that is side effect free. You still have to watch PR closely for people violating that invariant, though.

I like threads. I like threads a lot. I like them a lot better than promises. But I hate shared state threads, as you say. Threads should "own" as much data as they can, as strongly as they can.

Erlang is nice in that it codifies this. I write my Go in this style as much as I can, but enforcement is good. I wish Pony was doing better.

Shouldn't the patterns transcend the language implementation? The fact that this exists in Go should be proof of that.

Go’s concurrency model and Erlangs concurrency model are fundamentally the same.

Erlangs mailbox and Go’s channels are the same primitive.

And Erlangs processes and Go’s Goroutines are also the same primitive.

The big difference between the languages (from a concurrency perspective, there are other differences) is that idomatic Go is generally built around spinning off many short lived goroutines passing objects back and forth via channels, whereas Erlang generally spins off many long lived processes that represent objects, communicating mutation requests via mailboxes.

But you can do go style concurrency in Erlang, and Erlang style concurrency in go. You’ll just be swimming upstream the entire time, because your concurrency style won’t mesh well with any of the libraries out there which will be written in the idiomatic go/Erlang concurrency style.

So the idea you can create an Erlang style concurrency framework in Go isn’t actually surprising. Go and Erlang concurrency models are just opposite sides of the same coin.

It's not about concurrency, it's about failure domains.

The go VM doesn't natively let you take out monitors/links, so if "the right thing to do" when some part of a set of concurrent tasks fails is to "just give up" (suppose task 3/5 reaches out to an external service and comes back with a 500 error)... Handling the teardown of the whole thing (and potentially rollback state) is gnarly in go. In Erlang it's often zero lines of code. In Elixir, this architecture let the anointed db engine roll back a partially completed database transaction by default (0loc) in such a scenario.

There’s a very interesting aspect to what you mention here that is often overlooked. In Erlang every process has its own garbage collection. Every process has a stack and a heap. Stopping a process, or a process dying, automatically cleans up any resources used.

I actually think it’s rather difficult to mimic Erlang message passing in go, but trivial (and not uncommon) to mimic go channels in Erlang.

Idiomatic Erlang and Go are fundamentally different. In Erlang there is one mailbox per process and multiplexing is archived via polymorphic messages. In Go there can be multiple channels and one multiplexes them using the select statement.

Erlang message queue is a priority queue, while in Go a channel is a strict FIFO. In fact implementing a priority queue on top of channels in Go is rather hard. A much better approach is to code a priority queue directly using mutexes.

In Erlang (at least the last time I checked this) the message queue is unbound while Go gives a lot of control over the channel queue size.

As was already pointed, Erlang also supports reliable cancelation of threads, while in Go a thread has to be coded explicitly to support cancelation.

All of this leads to very different style of multithreaded code in the two languages.

> In Erlang (at least the last time I checked this) the message queue is unbound while Go gives a lot of control over the channel queue size.

You can bound the queue by proxy, becsause there is a max_heap_size process flag you can set. If your process is configured to use on-heap messages, and you have a max_heap_size, that provides a bound, although maybe not in the units you'd like.

You can also do things where your process might check it's own queue length (which is very cheap) and discard messages if it's too long. This doesn't work if your process gets stuck, or if discarding is still slower than the incoming message rate.

Another option is a process that lists all processes and finds their queue length (not so cheap) and kills process above a threshold.

You could also modify BEAM to do what you want. When I was at WhatsApp, we had a patch so you could do process_flag(flush_message_queue, N) and it would drop N messages (or all of them if N == 0), and this worked with process_flag/3 if you wanted to drop messages from the queue of another process.

Of course, that sort of thing doesn't fit the BEAM messaging guarantees, so unlikely to be accepted upstream. (See also the message prepending patch, to put a message to another process at the front of its mailbox)

So, the big difference between Go and Erlang style concurrency is that go style concurrency doesn't give you a readily available reliable way to keep an eye on other goroutines, as I understand it, where there are mechanisms in Erlang (links and monitors come to mind) to help accomplish that, and, in fact, it's one of the core parts of the Erlang model of concurrency

It does not give you a way to reliably track arbitrary goroutines that "this" goroutine (for whatever that may be) wants to track, the way an Erlang process can just "link" to anything it is capable of naming the PID for.

However, you can construct a reliable mechanism where one goroutine can start another and know whether or not the one it started has failed by using the available primitives, as I did in https://github.com/thejerf/suture . It's an easier problem since there's no cluster and no network that can get in the way. I've also done the exercise for the network case: https://pkg.go.dev/github.com/thejerf/reign#Address.OnCloseN... but that only functions within the network defined by that library because, again, it just isn't arbitrarily possible.

(I suppose it's relevant to some of my other comments to point out that I've also implemented basically Erlang-style concurrency in Go, with network, but as a relatively idiomatic translation rather than a blind one.)

"Go’s concurrency model and Erlangs concurrency model are fundamentally the same."

See https://news.ycombinator.com/item?id=34564228 .

At the 30,000 foot view, yes. On the ground, no.

I cautiously agree, the only point I'd make from my own perspective is that the other reference point for this would be Akka, which was/is a behemoth of boilerplate and torturous implementation, so it begs the question of whether its a pattern that just has friction and is it worth trying to reinvent again.

We did a book reading of a Scala book with a bunch of senior Java developers, and even they thought Scala was too baroque. If Java developers think something is too complicated you've really, really gone off the deep end.

I've heard that Scala has spent a lot of time since then trying to simplify itself, so I can't speak to now, but I do know that Akka came into being when Scala was at its worst. When in Rome, do as the Romans.

> You can use this to join a Go system to your Erlang cluster directly.

That’s amazing