A Thought Experiment: Using the ECS Pattern Outside of Game Engines
adventures.michaelfbryan.com
adventures.michaelfbryan.com
Despite "clear" definition from Wikipedia, ECS as a pattern suffers from multiple personality disorder. There are several different goals that all share the same name, and different implementations pick a different goal set, ending up looking not quite the same, and getting different kinds of benefits.
So outside games, if you pick "ECS as composition over inheritance", you get something like here, or Clojure's "let's use maps for everything". If you go for "ECS as a performance optimization" - a frequently touted benefit - you'll end up storing a lot of global arrays (perhaps arranged in structures), each packing every instance of a component's property across all entities.
If you go for "ECS as a way to have data-driven entities modifiable on the fly" or "ECS as a way to make game logic cleaner", then congratulations, you've just reinvented a relational database! Since you're not developing a game, you might as well store your data in SQLite and call it a day.
(In fact, I'm currently working on a roguelike game as a side project, whose distinct feature is that it stores all its game data in an in-memory SQLite database, precisely to experiment with a pure form of "ECS for game logic" pattern.)
In the end, the author explored one way of doing ECS. There are plenty others, worth their own thought experiments :).
This is basically what I am doing in the backend I am writing for an app that I am working on. (Also not a game btw.)
No database, no overhead :D
You really can fit a lot of data in memory, and I think a lot of people tend to forget that.
I so so wish that I get to see the day when memristors become cheap and abundant. It would be so pleasant to program for a system like that I think. No persisting to disk or loading from disk, ever – imagine that!
I love the history of ECS. Scott Bilas' goals with his work in Dugeon Siege, and before that, in Gabriel Knight 3 [1], was to have something close to what you're doing, in order to speed up development. He was storing game logic in text files, as opposite to having it hardcoded, to enable level designers and artists to iterate faster. I guess he would have considered SQLite if it was as widespread back then.
It was only later that people realised ECS could be a performance optimisation, associated it with OOP and made libraries for it.
[1] A very fun adventure game, and fun to reverse engineer. One of the "Easter Eggs" involves teaching you how to extract data from itself.
FWIW, what I'm doing is storing all the runtime state in in-memory SQLite, instead of writing my own ECS. Components are rows in tables, entity IDs are used as foreign keys, game logic just does regular SQL SELECTs to query the game state, and then UPDATEs, INSERTs or DELETEs stuff as needed.
I came up with this experiment when I was writing yet another implementation of ECS for that game, and in the middle of optimizing it for processing performance I realized that all I'm doing is essentially hand-writing database indexes, so why not use a proper database instead? And it turns out, SQLite is fast enough when used in-memory.
Unrelated, but I also guess that using SQLite for offline data could be a timesaver, since you can leverage web tech to make tooling for your artists and level designers.
Widely panned at release due to being forced to release old code after VCS failure, though I'd argue the only problem after patches were left was that movement logic change didn't play out well (it was a brave attempt though).
Had some bad performance issues but I never tracked down whether they were related to SQLite or issues in their engine.
Fortunately, existence of prior art doesn't invalidate any of the goals I had for this experiment, so I'll just carry on with it.
Before SQLite, I was working on my (yet another) own implementation, and eventually realized that I don't like the "Systems" part of the ECS, given I planned a lot of logic that required inspecting multiple components of any given entity. Thinking about it, I figured out that a better abstraction for a "System" would be a piece of code executed on a set of entities, defined by some selection logic. That is, a query. I quickly ended up with this interface:
(select-entities
:components (has-position has-health (not player))
:where ((entity-id (has-health (hit-points hp)))
(< hit-points 50))
:order-by (((lhs (has-health (lhs-hp hp)) &optional (targetable (lhs-priority priority)))
(rhs (has-health (rhs-hp hp)) &optional (targetable (rhs-priority priority))))
(if (and lhs-priority
rhs-priority
(= lhs-hp rhs-hp))
(< lhs-priority rhs-priority)
(< lhs-hp rhs-hp)))
:into my-array)
(The example selects non-player entities with position and health components which have less than 50 HP, and orders them by HP and, if both compared entities have a "targetable" component, also by targeting priority.)Which, as you can see, looks essentially like an SQL query. :components is select + join, :where is filtering condition, :order-by saves me from doing an explicit sort on the results, and the :into my-array part allows this query to not allocate new storage for results each frame.
Now each time such call showed up in my code, some extra book keeping storage would be allocated, and underlying logic ensured that most of the query doesn't have to be recomputed every frame. For instance, the :where part was handled by tracking addition and deletion of components to/from an entity. Implementing similar optimizations for :order-by part, I quickly realized that most of my "book keeping" tricks are essentially equivalent to compiled queries and indices on tables. At which point I ditched it all and switched to SQLite.
(I meant to write a blog post about this at some point. I guess I just wrote half of it here.)
From the point of view of the code operating on these entities, this is a query. That it happens to return results in constant time because it's being continuously executed in the background - that's just an implementation detail.
The example I've pasted doesn't execute a full query every frame either. The first time it's called, it ensures that there exists an array within ECS system that stores entities which match the :components part of my query. That array is then continuously updated when appropriate components are added or removed from entities. Multiple different queries using the same :components part (which gets canonicalized, so you can write them in any equivalent order) reuse that array.
The above is equivalent to a typical System in ECS, except I don't create a class or a global instance for it - I just shove a call to select-entities in the place I need it. This makes the architecture a bit more flexible conceptually, resolving some discomfort I had with Systems.
The :where and :order-by parts get executed each time, on the aforementioned array. I could've made them execute when components are added or removed, at the cost of increased memory usage (no more sharing of the filtered entities array); I chose against it in this project. But this isn't really a problem; :where and :order-by subsume the code you'd typically put in your System - processing entities matching a particular condition (e.g. hp < 50), and processing them in particular order, respectively. There should also be a :limit part, allowing you to pick first N entities for processing, but I switched to SQLite before finishing that feature.
Finally, if I pass an array as :into, the result set is stored in that array (which I've statically allocated previously), instead of allocating new memory.
So in the end, this does the same stuff you'd write in your system, but it handles book keeping for you. The interface lends itself for extra optimizations (this is Lisp, I could parse and process the code I put in :where and :order-by clauses, though I don't do that right now). And because it's not a System, not a Thing, I end up using it more often. Say I need to code a spell that picked weak NPCs and healed them automatically. Writing a System that's responsible for handling this effect is something that wouldn't even cross my mind. But a query like in the example above? Sure, why not? So I use it, and it executes almost the same code that I'd written otherwise, except I get some optimizations handled to me automatically. It's a deep interface, and pretty clean conceptually.
(There goes the other half of the blog post I wanted to write. I guess I have no excuse but to do it now.)
void autoheal_system() {
ecs.inner_join(ecs.status, ecs.inventory, ..., [](status, inventory, ...){
if(status.health < (0.2 * status.max_health) {
inventory.find("potion").use()
}
})
}
One big advantage with this system based implementation is that you are processing all entities in batches compared to the naive OOP style where executing entity.foo() may run Dog.foo or Cat.foo and thrash the instruction cache and cause branch misprediction because of the virtual function dispatch that is constantly switching between functions. I haven't done parallelism with this pattern yet but as long as you do not cross reference other entities it shouldn't be too difficult to just slap on an OpenMP pragma and have a parallel_inner_join function that runs everything on multiple cores.Small world. I'm writing a roguelike in Clojure and indeed use maps for everything. Datascript seemed promising, but it is hard to beat Clojure's builtin data selection functions.
FWIW, for the same game I described above, I also ended up cloning half of Stuart Sierra's famous Component library to use it as my half-baked Dependency Injection framework. This was motivated by the fact that - due to various life reasons - I do a good chunk of my development from a sidearm machine via SSH to my desktop, so I need an architecture where I can develop and test the game in headless mode (no graphics, mock inputs) as much as possible. I write a lot of unit tests now.
When I need a particular behavior, I just create a new class and keep everything related to that behavior in a single file. Entities and systems are little more than arrays, and components can query their entity for other components at runtime.
So there may be a MouseEventComponent or a TouchEventComponent, added by the engine depending on your device, then a PointerEventComponent to provide a layer of abstraction for OS-independent player input, and finally some PointerControlledShootingComponent etc.
Or a KeyboardEventComponent feeding a generic DirectionEventComponent, and you can swap the keyboard for a gamepad at runtime, or have different input sources for different entities.
To my understanding, doing stuff like that did not seem as intuitive in the more "traditional" ECS patterns, as opposed to just editing a list of classes to update every frame, or components holding references to other components in other entities (e.g. to share an input stream, or to implement a self-playing demo mode.)
It's meant to be for 2D games only and I haven't hit a wall with it so far, performance or otherwise, but some things like serializing the object graph for save/load or parallel execution might be a little tricky when I get to those.
My goal is to eventually be able to describe game scenes with a declarative SwiftUI-like syntax, where you'd add components like view modifiers and so on. Come to think of it, SwiftUI might be considered as an ECS for apps, or is that stretching the definition a little too far. :)
To me, doing stuff like this seems more intuitive than more "traditional" ECS approaches, but perhaps that's because I was never comfortable with the idea of Systems.
Eyeballing the code, the only slight architectural drawback I see is that the code must always belong to a particular Component, even when that code operates on multiple Components as equal partners (this is the ECS eqiuvalent of "your language doens't have multimethods" problem in OOP). If I understood correctly, you solve that by creating a Component composing other Components.
One particular thing I like about your overall architecture is that it explicitly covers state machines - both at the game (scene) level and at per-entity level. Various other ECS examples I saw on the web tend to completely ignore this topic, and it's pretty much the first thing I hit when I start using them: where do I put state machines?
Overall, OctopusKit looks like a solid piece of work. Congratulations, and do a Show HN: if you haven't done it yet!
--
[0] - https://invadingoctopus.io/octopuskit/documentation/architec....
> the ECS equivalent of "your language doesn't have multimethods" problem in OOP ... you solve that by creating a Component composing other Components.
I'm not sure about having solved that, but yes, a component may check its entity for other components to inspect and/or modify, even add/remove components from its entity, or add/remove entire entities from the scene (like a ShootingComponent asking a PhysicsComponent to simulate recoil, and spawning bullet entities.)
If a dependency is missing, a component alters or omits parts of its behavior or silently skips its frame update.
I'd really like to try the SwiftUI idiom in games, where the code is a declarative description of a scene, that an engine can take to construct the actual scene using whatever architecture it wants under the hood.
I also wonder how far you could take SwiftUI itself for making games without using any other framework.
TS: https://github.com/netgusto/ecs-typescript
Go: https://github.com/bytearena/ecs
Very interesting programming concept indeed!
---
Edit: These implementations support tagged queries, and indexed views for faster iterations (ECS uses iterations a lot).
http://www.dataorienteddesign.com/dodbook/
Prior HN discussion: https://news.ycombinator.com/item?id=20380397
The original goal was an ASCII game but since I needed to create the resources for it it morphed into Texel. I think the use of ECS here wasn't a bad decision but Specs proved to be just too cumbersome and overoptimizing.
As others have mentioned ECS is a bit "loosly defined" and each implementation seems to go over the line to add more specializing one way or the other. I want to switch to something more simple and elegant like DCES (https://crates.io/crates/dces) for my next refactor.
I think for "runtime resource management" ECS is fine if you need a fairly large, distinct pool of entities to handle. One of my main usage problems was the re-use of same components in the same entity.
E.g. imagine having an entity with a global world position, but also an internal position for something like "last cursor position". It's not possible to just "add another Position component" to the same entity, for good internal reasons, but still. It's pretty important people take these kind of limitations into account before planning out their entity/component maps.
> Components provide some basic guidance for structuring a Clojure application, with boundaries between different parts of a system. Components offer some encapsulation, in the sense of grouping together related entities.
> Each component receives references only to the things it needs, avoiding unnecessary shared state. Instead of reaching through multiple levels of nested maps, a component can have everything it needs at most one map lookup away.
https://github.com/weavejester/integrant
You can move from using component's functions/macros to using a map and cross references using reader literals.
Having used both, I find integrant to work better for me.
Stuart Sierra Components are more akin to Unity’s behaviour components prior to their introduction of an ECS. They’re a way of encapsulating state which may have dependencies and has a lifecycle (needing to be started or stopped).
ECS components are purely blobs of (usually fine grained) state, the existence and combination of which in an entity creates behaviours (that is, the systems which implement the behaviour will dynamically do so for any entity that has the prerequisite components, but the components are just data).
I always preferred naming ECS components as “traits” (they are the traits that an entity has, eg “animated”) and the systems “behaviours” (they are the behaviours exhibited by entities that have certain traits).
Most applications that aren't games don't need to support things like finding all active objects of a type or maintain many discrete actors executing their own run loops.
But Unity isn't dumping a lot of marketing into talking about the decorator pattern so everyone is jumping right to ECS now.
I'm curious because I started using ECS-like techniques exactly because traditional OOP in games felt a lot like yak shaving.
I didn't think that Java coders were especially well represented in the gaming world. It has been over 20 years since C++ programmers learned that inheritance is just a way to dress up function pointers (not that that does no good, but function pointers are pretty specialized machinery).
Seriously, you don't need to use a named style to write programs. Your language offers lots of different facilities you can put together in any way that is useful to achieve your aims. Functional, OO, ECS, MVC, procedural, data-oriented, reactive, whatever -- it is all just code.
You have only three problems: code needs to be organized well enough that you can understand it, code should run fast enough, and you should be able to change one thing without rewriting everything else. Nothing else matters.
EDIT: Actually, now when I think about it, this approach sorta makes every object (or Entity) into an independent, isolated database. Hm ...
What do you mean by that?
GameplayKit has its limitations, but I've been building an entire engine around it, and I like it so far, especially for the ability to stick with pure Swift and native APIs:
It's kind of meta, really; you can imagine ECS itself as an entity, and different meanings of it as different components that could be included in it.
ECS is, traditionally, just a way to dynamically add behaviour (components) into "blank slate" entities.
Instead of making a EvilMonster class in a game you just have it as a conceptual entity (normally it's just an integer, like a database id) associated with a bunch of components (normally just plain structs) and their parameters: Renderable, RigidBody, Collider, HasEnergy, HasTransform, AudioEmitter, etc.
You can stitch entities and components together using either code or some external data source. The "S" part of ECS is called the system, and it implements the behaviour defined by components.
The upsides of this IMO is that this is a fantastic way to structure your code. The data-oriented aspect is also great for enabling non-coders to assembly complex entities. And some other clever people (such as the sibling answer) found out this is also a great optimisation technique.
There are multiple implementations, of course, so people have different opinions.
One problem off the top of my head is run-time changing of components. An entity that inherits from multiple mixins can't inherit from new mixins (or lose existing ones) at runtime.
Nothing in the "Inheritance isn't Always the Best Tool for the Job" section talks about runtime changes in behavior.
> Most object-oriented languages are designed so that an object's underlying type will be the same for its entire lifetime. This makes things interesting when users want to scale a Circle without maintaining aspect ratio.
Mixins simply fix single inheritance so a subclass's parent class is parameterized, and so concrete classes can be composed of several non-related mixins (which will often seem like multiple super-classes).
ECS is a much different and bigger beast. It's a runtime composition pattern, and has almost basically to do with classes or objects.
If you're only complaint about OOP is with single-inheritance, ECS is probably overkill.
Is it the fact that (some implementations of?) mixins' members could collide and conflict with eachother?
Your 3rd paragraph, did you miss "nothing" or perhaps another word?
Its probably better to say it might help you write tighter loops because (ideally) you have a small amount of code looping over a large array of data, instead of a sea of actors hopping around the heap. Of course, you can make tight loops in a lot of different patterns...
Newer graphics APIs also allow you to build many command buffers at once (in parallel) and allow you to fill GPU vertex/index/texture buffers in parallel from multiple threads once you've mapped them into your address space.
GPU compute is also basically async and operates in parallel with rendering on modern GPUs. See https://www.extremetech.com/extreme/213519-asynchronous-shad...
Rendering is, in practice, parallel. You can enforce sequential ordering if you need it, but you often don't. (Z-buffer based rendering effectively makes parts of your scene parallelizable since the rendering is order-independent, and as demonstrated above tris can be rendered in parallel)
I've been doing scene rendering in parallel for something like 8 years on Direct3D 9 (XNA) and classic OpenGL. Most of my current parallelization is explicit ordering of scene elements which allows me to prepare buffers/draw commands in parallel, and filling GPU buffers in parallel. If I ever move to Vulkan or D3D11/12 I'll be able to exploit parallelism more there. Those old APIs allow mapping GPU resources into user address space which in some cases already allow you to prep future rendering while existing operations are in flight.
It's also common for modern D3D and OpenGL drivers to create hidden threads in your processes that perform rendering operations behind the scenes while you issue your sequential commands from your threads. This effectively turns those APIs into secretly-parallel APIs, and the driver threads can exploit any parallelism hidden away like performing multiple buffer uploads at once or building command buffers in parallel.
>Most of my current parallelization is explicit ordering of scene elements which allows me to prepare buffers/draw commands in parallel, and filling GPU buffers in parallel. If
That's not naturally parallel. You did all that work to know the order of the object before hand. Its not naturally parallel like, say, functional programming with no side effects is naturally parallel.
And you've hyper focused on the 3D render pipeline when many 2D frameworks have a single Layout/UI thread.
And the major point is none of that has to do with ECS!
You decorate your entities with components that get updated via a system.
Common Lisp Object system perfect and natural for this kind of programming.