Hacker News Reader

Not that different from MVC a couple decades ago, then! :D (just kidding)

I love the history of ECS. Scott Bilas' goals with his work in Dugeon Siege, and before that, in Gabriel Knight 3 [1], was to have something close to what you're doing, in order to speed up development. He was storing game logic in text files, as opposite to having it hardcoded, to enable level designers and artists to iterate faster. I guess he would have considered SQLite if it was as widespread back then.

It was only later that people realised ECS could be a performance optimisation, associated it with OOP and made libraries for it.

[1] A very fun adventure game, and fun to reverse engineer. One of the "Easter Eggs" involves teaching you how to extract data from itself.

Didn't know that origin story, thanks!

FWIW, what I'm doing is storing all the runtime state in in-memory SQLite, instead of writing my own ECS. Components are rows in tables, entity IDs are used as foreign keys, game logic just does regular SQL SELECTs to query the game state, and then UPDATEs, INSERTs or DELETEs stuff as needed.

I came up with this experiment when I was writing yet another implementation of ECS for that game, and in the middle of optimizing it for processing performance I realized that all I'm doing is essentially hand-writing database indexes, so why not use a proper database instead? And it turns out, SQLite is fast enough when used in-memory.

There's a game that already did that, Sword of the Stars 2.

Widely panned at release due to being forced to release old code after VCS failure, though I'd argue the only problem after patches were left was that movement logic change didn't play out well (it was a brave attempt though).

Had some bad performance issues but I never tracked down whether they were related to SQLite or issues in their engine.

Well, of course there is. Here I thought I'm doing something somewhat original...

Fortunately, existence of prior art doesn't invalidate any of the goals I had for this experiment, so I'll just carry on with it.

Of course it does not invalidate your experiment, but I wanted to suggest maybe looking at save files from the game.

Are you referring to the rougelike Sword Of the Stars:Pit or to the base game itself

No, I'm referring to 4X Sword of the Stars 2 itself. The Pit happened later, on a different engine (SotS2 is heavily windows-specific .NET code with DX10 dependency, iirc)

That's super cool. I think that approach is quite scalable, too. Only games with very complex logic would require something faster than SQLite for game logic.

Unrelated, but I also guess that using SQLite for offline data could be a timesaver, since you can leverage web tech to make tooling for your artists and level designers.

Last time I built an ECS it was effectively a bare bones relational database. A system just queries the information that it needs. The code basically looked like this. There was no need for explicit where or select queries because all I really needed to check was the presence of a component. Here is a contrived example. ecs has an inner join function that accepts all components (stored in an array) that you want to join and everything gets passed to a callback that receives all the components.

    void autoheal_system() {
         ecs.inner_join(ecs.status, ecs.inventory, ..., [](status, inventory, ...){
             if(status.health < (0.2 * status.max_health) {
                inventory.find("potion").use()
             }
         })
    }

One big advantage with this system based implementation is that you are processing all entities in batches compared to the naive OOP style where executing entity.foo() may run Dog.foo or Cat.foo and thrash the instruction cache and cause branch misprediction because of the virtual function dispatch that is constantly switching between functions. I haven't done parallelism with this pattern yet but as long as you do not cross reference other entities it shouldn't be too difficult to just slap on an OpenMP pragma and have a parallel_inner_join function that runs everything on multiple cores.

> If you go for "ECS as a performance optimization" - a frequently touted benefit - you'll end up storing a lot of global arrays (perhaps arranged in structures), each packing every instance of a component's property across all entities.

This is basically what I am doing in the backend I am writing for an app that I am working on. (Also not a game btw.)

No database, no overhead :D

You really can fit a lot of data in memory, and I think a lot of people tend to forget that.

I so so wish that I get to see the day when memristors become cheap and abundant. It would be so pleasant to program for a system like that I think. No persisting to disk or loading from disk, ever – imagine that!

You always need a memory hierarchy. Not even DRAM is fast enough for our CPUs so we have L3 caches and those are not fast enough either so we also have L2 and L1 caches and registers and physical wires which temporarily store the state of the processor by taking advantage of propagation delays.

New NVMe drives claim 5GB/s on PCIe 4, is memory mapping something like that not good enough for you already?

In these applications you primarily care about latency. Even these new SSD's have latencies 3 orders of magniture higher than DRAM. The 20 usecs is access time is about 100k executed instructions (scalar + single thread).

In what applications? The parent didn't even say what they were working on. Everyone wants ideally wants a perfect memory technology with no latency, large sizes and persistence, so I'm not sure what you are talking about exactly. If something needs less latency, you cache it or load it into memory. You seem to be saying that this current technology, that actually exists, is not good enough for a mystery application because it isn't as good as theoretical memresistors.

I could use some critical feedback on my system [0]. I went with what felt to be the most intuitive pattern for my own brain, but later found that it was apparently against the grain: making everything plain old classes and keeping all the logic inside components.

When I need a particular behavior, I just create a new class and keep everything related to that behavior in a single file. Entities and systems are little more than arrays, and components can query their entity for other components at runtime.

So there may be a MouseEventComponent or a TouchEventComponent, added by the engine depending on your device, then a PointerEventComponent to provide a layer of abstraction for OS-independent player input, and finally some PointerControlledShootingComponent etc.

Or a KeyboardEventComponent feeding a generic DirectionEventComponent, and you can swap the keyboard for a gamepad at runtime, or have different input sources for different entities.

To my understanding, doing stuff like that did not seem as intuitive in the more "traditional" ECS patterns, as opposed to just editing a list of classes to update every frame, or components holding references to other components in other entities (e.g. to share an input stream, or to implement a self-playing demo mode.)

It's meant to be for 2D games only and I haven't hit a wall with it so far, performance or otherwise, but some things like serializing the object graph for save/load or parallel execution might be a little tricky when I get to those.

My goal is to eventually be able to describe game scenes with a declarative SwiftUI-like syntax, where you'd add components like view modifiers and so on. Come to think of it, SwiftUI might be considered as an ECS for apps, or is that stretching the definition a little too far. :)

[0] https://github.com/InvadingOctopus/octopuskit

That's an interesting take. I read the README and the architecture document at [0]. Based on that, the way I see it, your approach is different but essentially equivalent to the typical non-performance-oriented ECS architecture. Entities are still runtime-configurable, Component updates are done in batches and in appropriate order. You've thinned out Systems, moved their code to their corresponding Components, and provided places to hook component mount/unmount code.

To me, doing stuff like this seems more intuitive than more "traditional" ECS approaches, but perhaps that's because I was never comfortable with the idea of Systems.

Eyeballing the code, the only slight architectural drawback I see is that the code must always belong to a particular Component, even when that code operates on multiple Components as equal partners (this is the ECS eqiuvalent of "your language doens't have multimethods" problem in OOP). If I understood correctly, you solve that by creating a Component composing other Components.

One particular thing I like about your overall architecture is that it explicitly covers state machines - both at the game (scene) level and at per-entity level. Various other ECS examples I saw on the web tend to completely ignore this topic, and it's pretty much the first thing I hit when I start using them: where do I put state machines?

Overall, OctopusKit looks like a solid piece of work. Congratulations, and do a Show HN: if you haven't done it yet!

--

[0] - https://invadingoctopus.io/octopuskit/documentation/architec....

Thank you!

> the ECS equivalent of "your language doesn't have multimethods" problem in OOP ... you solve that by creating a Component composing other Components.

I'm not sure about having solved that, but yes, a component may check its entity for other components to inspect and/or modify, even add/remove components from its entity, or add/remove entire entities from the scene (like a ShootingComponent asking a PhysicsComponent to simulate recoil, and spawning bullet entities.)

If a dependency is missing, a component alters or omits parts of its behavior or silently skips its frame update.

I'd really like to try the SwiftUI idiom in games, where the code is a declarative description of a scene, that an engine can take to construct the actual scene using whatever architecture it wants under the hood.

I also wonder how far you could take SwiftUI itself for making games without using any other framework.

I have gone down the exact same line of thinking. I think there is value in creating a simple relational data structure to have simple select and insert functions but not the overhead of sqlite. Many times when I think about it, the state of most programs could use a handful of the same structure and be done.

Yeah, I think so too. My goal after hopefully making a completely playable game running on SQLite is to eventually rip it out and replace with my own ECS, based on gained experience. I start with explicit SQL queries, and am slowly building an interface on top of it. Or rather, I'm moving towards the interface I abandoned prior to switching to SQL.

Before SQLite, I was working on my (yet another) own implementation, and eventually realized that I don't like the "Systems" part of the ECS, given I planned a lot of logic that required inspecting multiple components of any given entity. Thinking about it, I figured out that a better abstraction for a "System" would be a piece of code executed on a set of entities, defined by some selection logic. That is, a query. I quickly ended up with this interface:

  (select-entities 
     :components (has-position has-health (not player))
     :where ((entity-id (has-health (hit-points hp)))
             (< hit-points 50))
     :order-by (((lhs (has-health (lhs-hp hp)) &optional (targetable (lhs-priority priority)))
                 (rhs (has-health (rhs-hp hp)) &optional (targetable (rhs-priority priority))))
                (if (and lhs-priority
                         rhs-priority
                         (= lhs-hp rhs-hp))
                    (< lhs-priority rhs-priority)
                    (< lhs-hp rhs-hp)))
     :into my-array)

(The example selects non-player entities with position and health components which have less than 50 HP, and orders them by HP and, if both compared entities have a "targetable" component, also by targeting priority.)

Which, as you can see, looks essentially like an SQL query. :components is select + join, :where is filtering condition, :order-by saves me from doing an explicit sort on the results, and the :into my-array part allows this query to not allocate new storage for results each frame.

Now each time such call showed up in my code, some extra book keeping storage would be allocated, and underlying logic ensured that most of the query doesn't have to be recomputed every frame. For instance, the :where part was handled by tracking addition and deletion of components to/from an entity. Implementing similar optimizations for :order-by part, I quickly realized that most of my "book keeping" tricks are essentially equivalent to compiled queries and indices on tables. At which point I ditched it all and switched to SQLite.

(I meant to write a blog post about this at some point. I guess I just wrote half of it here.)

I've never written a game where I needed to query for entities (except spatially). Have I just not created the right type of game? If I needed to find certain types of entities they'd get added/add themselves to some list on creation. Querying things at runtime seems anathema to game dev if you want perf which most game devs obsess over

> If I needed to find certain types of entities they'd get added/add themselves to some list on creation.

From the point of view of the code operating on these entities, this is a query. That it happens to return results in constant time because it's being continuously executed in the background - that's just an implementation detail.

The example I've pasted doesn't execute a full query every frame either. The first time it's called, it ensures that there exists an array within ECS system that stores entities which match the :components part of my query. That array is then continuously updated when appropriate components are added or removed from entities. Multiple different queries using the same :components part (which gets canonicalized, so you can write them in any equivalent order) reuse that array.

The above is equivalent to a typical System in ECS, except I don't create a class or a global instance for it - I just shove a call to select-entities in the place I need it. This makes the architecture a bit more flexible conceptually, resolving some discomfort I had with Systems.

The :where and :order-by parts get executed each time, on the aforementioned array. I could've made them execute when components are added or removed, at the cost of increased memory usage (no more sharing of the filtered entities array); I chose against it in this project. But this isn't really a problem; :where and :order-by subsume the code you'd typically put in your System - processing entities matching a particular condition (e.g. hp < 50), and processing them in particular order, respectively. There should also be a :limit part, allowing you to pick first N entities for processing, but I switched to SQLite before finishing that feature.

Finally, if I pass an array as :into, the result set is stored in that array (which I've statically allocated previously), instead of allocating new memory.

So in the end, this does the same stuff you'd write in your system, but it handles book keeping for you. The interface lends itself for extra optimizations (this is Lisp, I could parse and process the code I put in :where and :order-by clauses, though I don't do that right now). And because it's not a System, not a Thing, I end up using it more often. Say I need to code a spell that picked weak NPCs and healed them automatically. Writing a System that's responsible for handling this effect is something that wouldn't even cross my mind. But a query like in the example above? Sure, why not? So I use it, and it executes almost the same code that I'd written otherwise, except I get some optimizations handled to me automatically. It's a deep interface, and pretty clean conceptually.

(There goes the other half of the blog post I wanted to write. I guess I have no excuse but to do it now.)

I’m intrigued by your approach! If you do end up writing that blog post and/or releasing the source, I’d be interested in learning more.

Drop me an e-mail (address in my profile), I'll let you know when the post is up.

> or Clojure's "let's use maps for everything".

Small world. I'm writing a roguelike in Clojure and indeed use maps for everything. Datascript seemed promising, but it is hard to beat Clojure's builtin data selection functions.

Small world indeed. I work a bit in Clojure from time to time, so I have a passing familiarity with the coding style.

FWIW, for the same game I described above, I also ended up cloning half of Stuart Sierra's famous Component library to use it as my half-baked Dependency Injection framework. This was motivated by the fact that - due to various life reasons - I do a good chunk of my development from a sidearm machine via SSH to my desktop, so I need an architecture where I can develop and test the game in headless mode (no graphics, mock inputs) as much as possible. I write a lot of unit tests now.