Making an RPG in Clojure (2010)
briancarper.net
briancarper.net
I tried the threading approach using agents (which later evolved into agents inside atoms or was it atoms inside agents? It's been a long time) and came to the same conclusion as the author. I tried to move to a more centralized CES approach, which didn't work out.
It's interesting to see how some people can come down to the same path and achieve the same results (for the good or for the bad). I think my experience with Clojure has been overall positive as far as game development goes, although definitely not empty of woes.
Currently what I'm (very on and off) working on is a Visual Novel/Interactive storytelling engine on ClojureScript, at http://novelette.moe and Clojure is great for it. Feel free to trawl through my github profile (link in my profile) if you're curious about some of my (failed? abandoned?) Clojure gamedev projects (among other things).
I finished the game :-)
To be honest I don't know how it compares, my engine is closer to something like Ren'Py but I plan to provide more freedom to the writers thanks to the power of functional programming and the DSL-like implementation of Lisp. I haven't formalized on a proper syntax for the scripts yet (although a rough example can be seen on my demo game: https://github.com/Morgawr/ld31/blob/master/src/cljs/game/sc... ), but the idea is to have something like the script of an actual novel. The author can define acts and chapters and have actors enter the scene, interact, define the state of the scene (add/remove sprites, backgrounds, music) and define the transitions from chapter to chapter within the novel.
It's been a ton of fun bringing a functional approach to games using Clojure and I gave a talk about it at ClojureWest this year. https://www.youtube.com/watch?v=TW1ie0pIO_E hope this helps others looking to build games!
Of course you get no benefit if you serialize the game through a single ref. Just because you have a fancy feature like agents doesn't mean you don't have to think about how your concurrent code is behaving.
I chose Clojure for the server side code of my MUD. It wasn't for the agents or STM, it was for immutability. Immutability gives me the ability to easily have a consistent snapshot of the world which I can use to perform some operation, such as saving, event propagation, or some AI calculations.
Speaking of event loops, I feel like they weren't discussed very much at all at my school, despite being at the core of not just most game engines, but most GUIs. I think if I'd embarked on writing a GUI system when I was fresh out of school, I'd probably have done it in a massively threaded way like the developer in the article started their game. An event loop would have somehow seemed too … primitive.
Beyond that, throwing massive concurrency at an inherently serial process tends to give you worse performance vs just recognizing that it's a serial process and not trying to throw concurrency at the problem. This isn't just about fancy concurrency primitives like agents or STM, I've seen plenty of hobby games that ran terribly because they threw concurrency at the problem without analyzing it first, but ended up completely serial because of critical sections in the form of Java synchronization blocks.
As an example of concurrency that you would probably have in every single player game: game logic feeds into render logic, but render logic shouldn't be changing the state of the world. So you can be performing the game logic of game loop #N+1 while the render logic of game loop #N is running. Also, usually, when saving the state of the game all you need is a consistent snapshot at the time. Immutability makes both of these easy.
The rest depends upon the features and design of your game though. If you aren't going to actually try to analyze where you can benefit from concurrency, you're better off not just randomly throwing concurrency at the problem.
For me though, it was a weak attempt at a simple Bomberman clone that would be playable over the network, so that my kids could play it across our different family computers. It was even kid friendly, being about planting timed water balloons, and avoiding getting wet!
It seemed like Clojure would be ideal for this, because the JVM has excellent networking libs and GUI libs for simple 2d graphics.
But I severely underestimated how much time it would take though, so it never really even got to a playable single player state.
The main part I got stuck on was trying to settle on a data format, how to represent tiles and players and items and everything, in a form of immutable data that was easy to both analyze and transform.
The author's idea of mapping a giant 2d string to actual data is genius, although I suspect it wouldn't scale well for an RPG, and for my game it wouldn't have been very helpful since the terrain would be auto-generated (since needing to be somewhat randomized).
It's surprising to see the author here mention using Agents as the original primary building block of the game engine. I've never directly used Agents or found them helpful. To this day I really don't know what they're good for.
Including procedurally generated maps, items and 3 small levels.
Far too many "making of" articles present you with a finely polished end product and pretend that its evolution was child's play. That leaves us readers disheartened and reluctant to try similar things.
Lmax disruptor is different from the others because it is based on engineering (what works on actual hardware) opposed to 'science' (how you imagine it ought to work)
Intended pun involving threads?
On the subject of pools of threads, a few days ago there was a post about the Naughty Dog Engine using fibers (inside threads) for cooperative multiprocessing. They made it so it'd have a cleaner API and be easier to use, but at the end of the talk they said it had also improved performance (because of less context switches).
Also having very little knowledge in games programming (a few experiments in QBasic 4.5 something like 20 years ago, when I was still a child :D ), the most I can get out of this is: multithreading/multiprocessing in games is hard.
Concurrency is somewhat orthogonal(although a nice-to-have), it's more about not doing things that run counter to your hardware.
Over in the games space we don't have the advantage of scaling wide on multiple machines so a couple things that sound heretical are there for good reason.
This is great for simple games but it gets hairy really quickly for larger games and it can easily become a mess. There are a few event/callback driven game engines out there and a lot of the most popular AAA game engines provide some kind of event subscription model (I think UE does it, at least, but I never really used it much).
The biggest problem is with figuring out who is in charge of what. Let's say I broadcast the event "player attacks enemy", then the enemy has to receive that event and respond with "enemy plays hurt animation" then "enemy plays dead animation" "enemy broadcast death" and then "player plays acquire XP animation" "player broadcasts level up" etc etc. The order can easily become a mess. And then "player attacks enemy" and "enemy evades attack" and "player plays missed animation" etc etc...
It really becomes very messy for big games, especially if you want to build an game engine entirely based on the broadcasting of events. I think Angel-Engine, now called Angel2D iirc, was mostly based on that and it worked fine for small scale stuff but easily become a mess.
It certainly is easy to understand the logic behind the game that way, it's great for small indie games or game competitions/prototypes, though.
I did read a bit on that, and tried to implement a simple game on it. What I've learned is:
a) it seems to be vapourware; there are some blog posts about it and examples floating around, but that's pretty much it
b) it seems to make sense only for MMO games, which are glorified interfaces to database tables anyway - hence the ECS model, which is about treating your game design as a database problem - fits it well
c) experienced people I talked about think this is a stupid approach, for reasons I'm probably too dumb to understand (except the efficiency ones)
d) in games that are not database-like MMOs, it's getting very hard to figure out what should be a component or a system, how many systems should you have, etc. etd.
e) it seems to seriously complicate implementing game logic - you end up spreading code that belongs together across many different "systems"
It is true that there are some degrees of limitations and implementations that different game engines go through and none of them is a 100% pure CES engine (something like http://www.chris-granger.com/2012/12/11/anatomy-of-a-knockou... ), where all systems are separated from the components and the entities containing them. Usually you'd have sets of components and systems and whatnot and different hierarchies, but the gist of it is still a Component-driven system.
One thing I vividly remember from learning about ECS is that the main sources on it repeatedly remind you that what UE or Unity do is not ECS, and that the way they understand the concept of "Entity" and "Component" is different. Entity Component System is not what UE or Unity do.
Using an ECS for game design actually has enormous advantages when you are coding a high-performance game in a language like C or C++: you can lay out the components of the entities however you want, so that systems on the hot path have access to as many components as they'll need without blowing the data cache. I'm not actually doing that for this game, but still, it has advantages outside of just MMOs.
[0] https://nullawesome.tumblr.com, https://github.com/bitwize/nullawesome
So yea, I don't use STM much, and I never use actors, I use queues quite a bit, but I benchmark them first and use the one that fits best with existing infrastructure and causes the least amount of friction with my client while still getting the job done. It has nothing to do with "actually working on hardware" or "science". Except perhaps that I apply the scientific method in choosing my tech, and that never ends up being LMAX for the jobs I work on.
I mainly use the reactor framework, and I have a choice of lmax and a bunch of other queues and also the choice to schedule on the same thread with low overhead.
No, abstractblockingqueue and other general purpose queue implementations are great for all sorts of things. Disruptor has a pretty specific use case - high rate of constant data flow through a system, where it's okay to spin loop threads to consume data. It also requires pinning threads to CPUs to get the benefit gain of cache locality on the ring buffers. It's pretty darned specialised for general use.
STM might make things simpler, but if many threads are contending for the same data, then they will need to be serialised and while STM might make this serialisation easier for the programmer, it won't automatically make it perform better[1].
For parallel threads to be faster than a single thread, you need the threads to be as independent from each other as possible. You can use actors, STM and such to do this, but its not a silver bullet that just works. You need to apply the correct technique given your situation and requirements.
I think where Clojure largely failed (and I say this as a big-time Clojure enthusiast) is in its promise of making concurrency easy through these magical primitives. Hell, since its release, Clojure has gained more concurrency primitives (futures, core.async, transducers) because the original ones (threads, STM, agents, atoms, vars) weren't enough.
As for a game engine, I would make sure that each entity wrote to its own independent data, so that writes never block each other. That is, they can all read global world state and they can read the previous state of the other entities, but each only ever writes its own data to its own atom (or whatever; I'd have to think about it more to figure out the most appropriate place to store the state). Then I can update all of the atoms in parallel. This is just off the top of my head, so it may need a little refinement. Agents provide a nice async update, but if you have many in flight, they still need to serialise and take turns, losing much or all of the parallelism.
The point is, I don't believe we'll have a magic "throw cores at it" concurrency technique any time soon, we will continue having to think about it and structure our code to keep the cores as busy as possible and synchronisation as low as possible.
LMAX disruptor is indeed a fantastic piece of engineering and it works so well because it eliminates the need for synchronisation (because the updates always happen in a single thread). The awesome "concurrency" from disruptor comes from the design of their ring buffer, which allows many threads to quickly write to it, and the single reader to quickly read, all without synchronisation. Basically, the processing thread can be kept busy without ever having to wait on writers. Fantastic!
My favourite means of parallelism, I think, is still a work-stealing task-based system like what Intel Threading Building Blocks provides, because its relatively easy to think about and deal with, quite flexible (allows firing off tasks for async things, "going wide" for nicely data-parallel things, allows pipelined processing for sequential steps over a collection of data).
But... there's no concurrency/parallelism silver bullet.
[1] STM can of course make sure the locks are as fine-grained as possible, but at the end of the day, if two things write to the same place, then they need to happen serially.
STM was intended to make it less error prone to use threads in places where concurrency was the right fit for the problem.
For example, a web server is naturally concurrent. You definitely can implement a web server in a single-threaded manner, but you'll end up just faking concurrency anyway. So there are times where you'll want to use concurrency because the problem you're solving is naturally concurrent. In these cases, having an STM is valuable, because it makes it hard to get state changes wrong in a concurrent environment. It is not necessarily a way to make your program faster.
――――――