Comment by KallDrexx | Hacker News Reader

Comment by KallDrexx | Hacker News Reader

Parent Full thread

Preemptive scheduling is great, but there are some times when it is bad. For example, I was having problems in my Elixir code for live streaming that it was taking too many reductions to fully deserialize and process the incoming message (nothing major was in this code path, just a parsing the TCP packet and unwrapping the different layers). This ended up causing the process to get (what seems to be) overbalanced and I ended up at having delays transferring 5+mbps video through it.

I broke the deserialization up into 3 separate processes (1 for handling the I/O, 1 for packet deserialization, 1 for message deserialization) and while this fixed all the issues (presumably because it can balance the processes more effectively since no single process went over the reduction limit) it turned the effectively synchronous parsing process into a split async process and caused a lot of extra complexity around that.

I'm sure there's more to what caused the issue than just reduction limits but it also made me wary about the magic happening under the hood.

Not just scheduling — the ability to forcibly terminate processes itself is a key part of why Erlang can be resilient to failures.

It's a key area where I think Go went wrong. Without the ability to kill a goroutine, you can't implement caller-enforced timeouts and canceling — anything that should be able to be canceled has to explicitly check for some signal sent by the controlling party. This is why Go ended up with Context, which infects everything it touched.

And you also can't create Erlang-style supervisor trees; you can't create self-repairing hiearchies of control. And you can't do things like use a debugger to attach to a process and terminate runaway goroutines.

You may have been able to bump the process priority to give that process more scheduler time if it's a critical process in a hot path. Perhaps you tried that though. Definitely have to be careful with it, but always an option to keep in mind!

I haven't tried that, but I'd have to make sure that every inbound and outbound RTMP connection had high enough priority to correctly execute, and that seemed like a lot of tweaking that had to be done in order to maintain the constant flow of video throughout the app with low latency.

At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly (not just with that but other potential issues I had in the back of my mind as well).

> At least for my operation it looked very much like I would really need to become a BEAM vm expert to make sure things ran smoothly

From someone with zero experience in building concurrent systems: isn't this true for every sufficiently complex system? If your system is complex enough, then no general framework will help much, right?

At some point sure, but I can get concurrent systems up and running pretty easily in Go, C# and Rust without going too deep into the underlying details of the asynchronous framework. Erlang's VM does a lot more than any of those frameworks or runtimes do though because of how it handles per process garbage collection yet globally shared binaries, the preemptive scheduler and how it balances process load, the hot-reload capabilities etc... It does a lot of things really well but there's a lot of magic behind it, and I seemed to brush against that much faster than I have in other languages.

Do you mind me asking what kind of throughput you are seeing for a process like that where you unpack video across some processes? I'm working on something similar using Go and it'd be great to know if Elixir is a good alternative or not.

I know of some Erlang based platforms that achieve 10gbps throughput. I hadn't done exhaustive testing on my prototype as I've changed gears away from my home-grown solution to modifying some open source C++ media servers that are a bit more proven than my own stuff.

Reply on news.ycombinator.com