Lmax disruptor is different from the others because it is based on engineering (what works on actual hardware) opposed to 'science' (how you imagine it ought to work)
So yea, I don't use STM much, and I never use actors, I use queues quite a bit, but I benchmark them first and use the one that fits best with existing infrastructure and causes the least amount of friction with my client while still getting the job done. It has nothing to do with "actually working on hardware" or "science". Except perhaps that I apply the scientific method in choosing my tech, and that never ends up being LMAX for the jobs I work on.
I mainly use the reactor framework, and I have a choice of lmax and a bunch of other queues and also the choice to schedule on the same thread with low overhead.
No, abstractblockingqueue and other general purpose queue implementations are great for all sorts of things. Disruptor has a pretty specific use case - high rate of constant data flow through a system, where it's okay to spin loop threads to consume data. It also requires pinning threads to CPUs to get the benefit gain of cache locality on the ring buffers. It's pretty darned specialised for general use.
STM was intended to make it less error prone to use threads in places where concurrency was the right fit for the problem.
For example, a web server is naturally concurrent. You definitely can implement a web server in a single-threaded manner, but you'll end up just faking concurrency anyway. So there are times where you'll want to use concurrency because the problem you're solving is naturally concurrent. In these cases, having an STM is valuable, because it makes it hard to get state changes wrong in a concurrent environment. It is not necessarily a way to make your program faster.
Concurrency is somewhat orthogonal(although a nice-to-have), it's more about not doing things that run counter to your hardware.
Over in the games space we don't have the advantage of scaling wide on multiple machines so a couple things that sound heretical are there for good reason.
This is great for simple games but it gets hairy really quickly for larger games and it can easily become a mess. There are a few event/callback driven game engines out there and a lot of the most popular AAA game engines provide some kind of event subscription model (I think UE does it, at least, but I never really used it much).
The biggest problem is with figuring out who is in charge of what. Let's say I broadcast the event "player attacks enemy", then the enemy has to receive that event and respond with "enemy plays hurt animation" then "enemy plays dead animation" "enemy broadcast death" and then "player plays acquire XP animation" "player broadcasts level up" etc etc. The order can easily become a mess. And then "player attacks enemy" and "enemy evades attack" and "player plays missed animation" etc etc...
It really becomes very messy for big games, especially if you want to build an game engine entirely based on the broadcasting of events. I think Angel-Engine, now called Angel2D iirc, was mostly based on that and it worked fine for small scale stuff but easily become a mess.
It certainly is easy to understand the logic behind the game that way, it's great for small indie games or game competitions/prototypes, though.
I did read a bit on that, and tried to implement a simple game on it. What I've learned is:
a) it seems to be vapourware; there are some blog posts about it and examples floating around, but that's pretty much it
b) it seems to make sense only for MMO games, which are glorified interfaces to database tables anyway - hence the ECS model, which is about treating your game design as a database problem - fits it well
c) experienced people I talked about think this is a stupid approach, for reasons I'm probably too dumb to understand (except the efficiency ones)
d) in games that are not database-like MMOs, it's getting very hard to figure out what should be a component or a system, how many systems should you have, etc. etd.
e) it seems to seriously complicate implementing game logic - you end up spreading code that belongs together across many different "systems"
It is true that there are some degrees of limitations and implementations that different game engines go through and none of them is a 100% pure CES engine (something like http://www.chris-granger.com/2012/12/11/anatomy-of-a-knockou... ), where all systems are separated from the components and the entities containing them. Usually you'd have sets of components and systems and whatnot and different hierarchies, but the gist of it is still a Component-driven system.
One thing I vividly remember from learning about ECS is that the main sources on it repeatedly remind you that what UE or Unity do is not ECS, and that the way they understand the concept of "Entity" and "Component" is different. Entity Component System is not what UE or Unity do.
Using an ECS for game design actually has enormous advantages when you are coding a high-performance game in a language like C or C++: you can lay out the components of the entities however you want, so that systems on the hot path have access to as many components as they'll need without blowing the data cache. I'm not actually doing that for this game, but still, it has advantages outside of just MMOs.
[0] https://nullawesome.tumblr.com, https://github.com/bitwize/nullawesome
Intended pun involving threads?
On the subject of pools of threads, a few days ago there was a post about the Naughty Dog Engine using fibers (inside threads) for cooperative multiprocessing. They made it so it'd have a cleaner API and be easier to use, but at the end of the talk they said it had also improved performance (because of less context switches).
Also having very little knowledge in games programming (a few experiments in QBasic 4.5 something like 20 years ago, when I was still a child :D ), the most I can get out of this is: multithreading/multiprocessing in games is hard.
STM might make things simpler, but if many threads are contending for the same data, then they will need to be serialised and while STM might make this serialisation easier for the programmer, it won't automatically make it perform better[1].
For parallel threads to be faster than a single thread, you need the threads to be as independent from each other as possible. You can use actors, STM and such to do this, but its not a silver bullet that just works. You need to apply the correct technique given your situation and requirements.
I think where Clojure largely failed (and I say this as a big-time Clojure enthusiast) is in its promise of making concurrency easy through these magical primitives. Hell, since its release, Clojure has gained more concurrency primitives (futures, core.async, transducers) because the original ones (threads, STM, agents, atoms, vars) weren't enough.
As for a game engine, I would make sure that each entity wrote to its own independent data, so that writes never block each other. That is, they can all read global world state and they can read the previous state of the other entities, but each only ever writes its own data to its own atom (or whatever; I'd have to think about it more to figure out the most appropriate place to store the state). Then I can update all of the atoms in parallel. This is just off the top of my head, so it may need a little refinement. Agents provide a nice async update, but if you have many in flight, they still need to serialise and take turns, losing much or all of the parallelism.
The point is, I don't believe we'll have a magic "throw cores at it" concurrency technique any time soon, we will continue having to think about it and structure our code to keep the cores as busy as possible and synchronisation as low as possible.
LMAX disruptor is indeed a fantastic piece of engineering and it works so well because it eliminates the need for synchronisation (because the updates always happen in a single thread). The awesome "concurrency" from disruptor comes from the design of their ring buffer, which allows many threads to quickly write to it, and the single reader to quickly read, all without synchronisation. Basically, the processing thread can be kept busy without ever having to wait on writers. Fantastic!
My favourite means of parallelism, I think, is still a work-stealing task-based system like what Intel Threading Building Blocks provides, because its relatively easy to think about and deal with, quite flexible (allows firing off tasks for async things, "going wide" for nicely data-parallel things, allows pipelined processing for sequential steps over a collection of data).
But... there's no concurrency/parallelism silver bullet.
[1] STM can of course make sure the locks are as fine-grained as possible, but at the end of the day, if two things write to the same place, then they need to happen serially.