nikki93

I actually tried D as part of this exploration. I forget but it was confusing which of dmd or the llvm one to use, and ultimately the language ergonomics were not that much of an improvement over C++. Like "auto" is still super weird. I'm going for an actual improvement here, not another botched but slightly improved thing from the past. I think D's tooling worked the least stable-ly out of the box of everything I tried. It wasn't promising. That said, D was one of the things I explored the semantics of for ideas, and it's good that it tries to do the metaprogramming stuff. UFCS is also ok but needing to decide foo(o) and o.foo() at each callsite is actually mental overhead vs. the choice being decided in Go (I wrote the whole engine and game in Nim before so I have experience with this).

Just because something has a bunch of years in it doesn't mean it's a good idea for a specific context. The transpiler I have here is just 1500 lines of code and captures all the semantics it currently supports. It uses Go's parser and typechecker from the stdlib and feels on the whole more polished than D as a result (generics are definition-checked, Go's package / module system just work, all the existing Go editor support and godoc etc. just work, ...). It's much easier and straightforward to metaprogram by just editing this simple piece of logic than squeezing it into language features (I've also done the same engine in Nim, explored in Zig, and written it once over in C++). I can, for example, make it so if you mark a function a certain way, it's also compiled to GLSL and useable as a shader (with structs shared). Or make it so types marked a certain way have all their pointers reference counted. There's way more control in this scenario, and the point is to have control to take matters into one's own hands and actually improve things.

Yup, pretty much. I do think that "GC" usually these days refers to a mark and sweep GC (certainly in the context of a discussion about Go?) or at most a refcount kind of thing applied to all reference-like language entities, not "any lifetime management system," the way I see folks use that term.

But yeah I've found that the entity component data structure is a good lifetime management system very well suited to the game scenario, so there's no need for a different / more complex thing. And this is an exploration in how that + a simple / ergonomic language around it (along with growable arrays (slices)) pan out when making games in practice. There's no manual free calls or lifetime management anywhere in the code.

Re: "using GC but then needing to / being thoughtful to make sure it's going ok" -- that's the thing. It seems better to not have to need to think about it, by having a system that's better suited to the thing you are working on. You also don't need to "often go back and optimize some of this stuff later too" because it just already has good performance with the straightforward code, and you don't need to add complexity. "splitting references out / pure data" -- that is indeed what the language nudges you to do by only having pure data. Essentially: yes, you can achieve the desired thing with intentionality and extra cognition in a different system (the same was true with other kinds of cognitive overhead in C++) and this is an exploration in developing a language + tools that focus on and bias toward the desired thing by default. Like I'm imagining folks getting started with gamedev using this + the integrated tooling and internalizing the practices you're talking about (that's a stretch vision, the current scope is to just build and test it in the context of one specific game project).

I'll be releasing this engine + an example game with it soon, but here's what the code for this main game project (the one in the video) looks like (all the components, the top-level game loop, and then some example game logic): https://gist.github.com/nikki93/0425d9ead9eb7810075434d006f3... It's just data structures, and then functions that do gameplay stuff on them. No lifetime management.

I'll read your statement as a question and answer that.

I was using C++ for data-oriented gameplay coding, and I was interested in exploring making a language frontend that compiles to it to clean it up, as a side project (lots of dark corners to run into with C++, and I collected some experience on what those were since I'm managing a C++ game engine codebase at work). I needed a core that was basically a cleaned up C, which is what the C-ish core of Go is (the part other than goroutines, channels and GC), and Go has a good parser and typechecker library you can use. Goroutine and channel are cool for distributed server code or whatever, but not actually that useful for game programming. The main thing is having structs, procedures, some nice ergonomics over those (slices, type inference, non-escaping lambdas, occasional generics) and then metaprogramming so you can reflect over the data structures and have serialization and inspector UI. These are the elements actually relevant to game programming.

I've written my own Go (subset + extensions) -> C++ transpiler and using it on a game project: https://www.youtube.com/watch?v=8He97Sl9iy0 -- No GC, it does have slices and has access to an entity/component API and with that I think you're basically set and don't need GC for games.

Example transpiler input / output: https://github.com/nikki93/gx/blob/master/example/main.gx.go... becomes https://gist.github.com/nikki93/97ff376abb6718427387bb9cca2f... Can call to C/C++ (including templates) w/o overhead.

That said, for logic that is heavy on async and escaping closures like how a lot of Go server code tends to be, a GC is maybe a reasonable tradeoff?

"Starting Forth" and "Thinking Forth" may be good books (the latter is kinda more architecture / philosophy though).

Not saying this means you actually need to add constructors and destructors, but I think the main tricky bit is when you have eg. a grow / shrinkable array and want to call something on it that removes elements and it should be calling destructors on those. AFAICT there's no clear place to add a `defer` that makes it happen at the right time at the lexical site those elements are originally added. I think this is where the main RAII complexity comes from vs. the lexical scope local variable scenario which is definitely handled by `defer`.

All that said, not an argument for having that, just wanted to nuance it. I think not having it and keeping the language focused on its priorities can be good, for example.

Pooling is indeed what the ECS I use (entt) basically does--every component type has a contiguous pool of instances. Compiling Go to C++ lets me use entt among other things (target all of C++'s targets including Wasm, have some types of metaprogramming like statically reflecting all component types, etc.). The GC thing is just one of the results. There is no comparison for the amount of control you get vs. vanilla Go (where things can escape to the heap "whenever").

Yeah the Go -> C++ compiler I linked from my other comment is pretty much overlapping with this idea. TinyGo is still a bit early afaict and also tries to exactly implement Go semantics but I'm kind of interested in extending / adapting it to my use case.

I think the main thing is it starts to become a distraction from just writing the gameplay code. I don't have to implement the pooling stuff now that I have this compiler--naive / simple code tends to also start off with a high perf ceiling. But yeah if I did go further with the game in vanilla Go I might have to try the pool approach. Having worked on game engines with GC language runtimes (using Lua etc.) before, you always ultimately hit a perf ceiling due to lack of memory control and wish you could move out of it, but the runtimes don't give you a way to do that incrementally.

Ultimately in the game scenario the GC is actually just ... not helpful. Game logic code already explicitly handles lifetimes to some degree (eg. when this entity collides with that one, destroy it, etc.) -- emergently deciding when to free things based on references is usually not what you want. You do want it for resource management (like a texture cache), but it actually makes sense to kind of roll that on your own and adapt it to the game. So having a GC and then fighting it just sounds like an ill-fitted solution.

I've been working on a Go -> C++ compiler pretty much mainly for this use case, that skips the GC and concurrency stuff -- https://www.reddit.com/r/golang/comments/r2795t/i_wrote_a_si... -- Includes a demo video of a game I'm making with it and a built-in scene editor that uses reflection etc.

Repo for compiler itself: https://github.com/nikki93/gx (no README.md etc. yet, will be getting to that when I next have a chance (it's a side project)). It just takes around 1500 lines of Go thanks to the parser and typechecker in the standard library.

Go's perf was definitely non-trivially bad for me on WebAssembly.

Have you compared the performance and generated binary size of C vs. Go on WebAssembly?

One of the newer (newer than C, C++, Common Lisp, ...) languages I feel comfortable about regarding this is Go. Definitely feels like I can write a simple Go program to generate a static website for example and it'll probably still work in that same state for a while.

Nice! This is definitely my main usecase for reflecty things lately, along with serialization for the data (game engine scenarios). I've been doing static reflection + attributes through this: https://godbolt.org/z/enh8za4ja Which actually comes out to a not-so-bad syntax and the attributes can be read constexpr (some limitations though: simple aggregate-y structs, and all reflected fieds must appear before non-reflected ones).

Lately I've been working on a Go (pretty small subset -- no GC or concurrency) to C++ compiler and generating meta stuff as part of it. Mostly because I like how focused Go's syntax is and also it having an official parser and type checker in the standard library makes things pretty quick to get started.

I should play with Circle some time! It's definitely touching on a lot of the desired things from a metaprogramming layer.

When I've tried it out for wasm game development, the biggest issue I had was frequent GC pauses (was a few every second) causing frame drops. TinyGo gets at this issue. I've used C++ since then, focusing on a small subset that keeps things flexible and compiling fast. Been hacking on a Go -> C++ compiler lately that produces that subset.

Thank you! We also have a Discord server (link at bottom of website) -- the app is rough around the edges still and some support / inspiration usually helps.

There's also NerdSEQ: https://xor-electronics.com/nerdseq/ + actual Eurorack modules! :D I like the Ableton clip style main view in NerdSEQ which allows you to live mix and match patterns, haven't come across as good a flow for that in regular software sequencers, especially ones that let you do chains of patterns of different lengths and polyrhythm them, while having some of them be automation parameters for some part of the DSP and others notes for other parts of the same chain (Buzz, Buze, SunVox kind of let you do that).

Ah yes that would also be an issue!

This may be due to requiring WebGL 2 specifically, and not WebGL generally.

Figma uses webassembly. We also use it for our web engine at https://castle.xyz, I gave at talk about that you can watch here: https://youtu.be/0FjCaoc1uzs where I go into some details and also do some live wasm game development (not sure how "serious" since we're still working on our app though).

Much, much better performance than JS is possible because (among many other things) you can perform a lot of optimizations during AOT compilation due to eg. LLVM, avoid GC and arrange memory to help cache efficiency.

Re: JS call overhead, ultimately using instanced drawing so you have a constant number of draw calls even for a growing number of objects is what helps.

Would be nice to see a short screencast or at least a screenshot giving a sense of the workflow this enables (or maybe I'm missing it on the site -- looking at it on mobile).

FWIW, the overhead of calls across the cgo barrier is quite high, which can have API architecture implications -- https://github.com/dyu/ffi-overhead#results-500m-calls.

The tradeoff would be different in an embedded context, for sure. I don't know that cJSON being worse due to not supporting allocators is "ironic," it's just the tradeoff there. Compile time actually really matters to my usage (changing gameplay code and seeing results immediately). nlohmann json was actually increasing my compile time by nontrivial amounts (like a 50-100% increase).

Re: strong typed -- agreed. That's basically what I do with cJSON using a static reflection system in C++ (it recursively reads / writes structs and supports customization points). So it's kind of like using cJSON to give yourself a thin C++ layer. Agreed that a typed approach is more sensible than writing raw read / write code yourself (although that does make sense in some scenarios).

Yeah the support for pointer / patch etc. is a definite plus. The customization point thing I tend to do with my own customization points, but it's pretty good if a bunch of libraries settle on a standard customization point (eg. I think serde in Rust has achieved that a bit due to the ecosystem there) (definitely want it to be static).

I didn't realize that about the array complexity. Can you not just initialize a JSON array of N elements in O(N) (conceding a `.reserve(N)` style call beforehand if required)? rapidjson is pretty good about that sort of thing, cJSON's arrays are linked list so I basically think of its performance as at a different level and it's mostly about compile time for me.

The API side of this library is pretty good. I've generally avoided it due to the compile time, and preferred rapidjson due to both better performance and compile time if I wanted a C++ API, or cJSON if a C API is fine and I really just want an extremely low compile time (lately I've been playing with a live coding C++ context for a game engine, as a side project, I use cJSON there; rapidjson on main project though) (cJSON's tradeoff is it sticks with a single global allocator that you can only customize in one location, and also seems to perf slightly lower on parse).

The cJSON's C API or rapidjson's somewhat less ergonomic API has felt fine because I usually have read / write be reflective and don't write that much manual JSON code.

Specifically for the case where compile time doesn't matter and I need expressive JSON object mutation in C++ (or some tradeoff thereof), I think this library seems good.

Another big feature (which I feel like it could be louder about -- it really is a big one!) is that you can also generate binary data in a bunch of formats (BSON, CBOR, MessagePack and UJBJSON) that have a node structure similar to JSON's -- from the same in-memory instances of this library's data type. That sort of thing is something I've desired for various reasons (smaller file types with potentially better network send time for asynchronous sending / downloads (for real-time multiplayer you still basically want to encode to something that doesn't embed field string names)). I do think I may end up doing it at one layer above in the engine level and just have a different backend other than cJSON etc. too though...

I do think that in practice a hierarchical ontology like that is still best not modeled as a language-level hierarchy because the language inheritance / hierarchy concepts are often not the exact semantic you want, IME. Esp. if it then ties to static types, since you then can't change the hierarchy at runtime. I think even with a data-oriented approach to a hierarchy -- your code isn't necessarily hierarchically organized, it just handles data that happens to express a hierarchy. And you want to be in control of the semantics of said hierarchy with more freedom (and explicitness) than the language-level hierarchy gives you -- so you want your own code that interprets the hierarchy expressed in the data and performs your desired semantics. This also allows artists and narrative or gameplay / level designers to go see and edit the hierarchy and add elements to it.

An example is the prefab hierarchy you get in Unity, which is expressed through the data (prefabs and their relationships). (Note: I mean specifically the prefab inheritance hierarchy, not the transform spatial hierarchy -- the former has more overlap with the "is a" relationships). The code processing this hierarchy could've just been plain C code that parses the files and maintains an in-memory set of structures about them, even. You then get to define how properties inherit, what overriding means, etc. yourself.

I don't think ECS necessarily means a "build the world" approach. I think it's best kept at the level of a data structure with some given set of operations: create entity, destroy entity, add component to entity, remove component from entity, get component on entity, and the big one -- query / iterate through component combinations on all entities that have them.

Just like arrays and structs, it's yet another data structure to be used in the general data-oriented approach, one that becomes useful because those creation / destruction patterns come up in games and adding and removing components is a great way to express runtime behavior as well as explore gameplay.

The "focus" on ECS may just come from it being an interesting space as of late vs. arrays, structs and for loops that have been around for ever, but it's mostly just an acknowledgement of common array, struct and for loop patterns that arise. There's also a lot out there about the systems part and scheduling and event handling but I think it's almost best to start out with simple procedural code (that then has access to the aforementioned data structure) and let patterns collect pertinent to the game in question.

One big aspect I personally dig is if you establish an entity / data schema you get scene saving, undo / redo, blueprint / prefab systems that are all quite useful and basically necessary if you want to collaborate with artists and game designers on a content-based game, and empowers them to express large spaces of possibilities without editing the code.

It depends on the storage pattern: archetype storages may actually keep those components together / interspersed anyway, or hybrid things like the "groups" in entt. It does, generally speaking, just seem to give you an opportunity to see this issue re: cache misses arise in practice and rearrange your storage accordingly, by decoupling your processing logic (body of a query) from the actual layout. Esp. if the ECS provides a switch like "store A and B interspersed in a single array" that you can enable or disable at any point and profile both ways (part of the data-oriented design idea: orient your data for how it's used in practice).

The performance / cache-orientation point gets talked about a lot re: ECS, but IME there are also (and maybe moreso) other important benefits:

- Combined with an inspector UI, you can explore gameplay by adding and removing components from entities and arrive at design emergently. One way to look at this is also that you write components and systems to handle the main gameplay path you start out thinking about, but your queries encode many other codepaths than just that (a combinatorial explosion of component membership in entity is possible). This lets you get a kind of "knob crawl" that you see in eg. sound design when tweaking parameters live with synths too. It lets artists and designers using the editor explore many gameplay possibilities.

- The way I see the component data is it's kind of an interface / source of truth, but some subsystems may end up storing transient data elsewhere at runtime (eg. a spatial octree or contact graph for physics). However as components are added or removed or component properties updated, the caches should be updated accordingly. You get a single focal point for scene state. Once some state is expressed as a component you get undo and redo, saving to scene files, saving an invidual (or group of) entity as a blueprint to reuse, ...

The cache thing feels like a minor point to me, inside a larger category of allowing you to massage your data based on access patterns by decoupling the logic acting on it. With performance being one of the goals of said massaging along with many others.

I also find myself not really focusing on the "system" aspect as much as the entity / component; esp. re: embedding constructs for that into a library. I've found you can get far just having one large "updateGame()" function that does queries and then performs the relevant mutations in the query bodies, and you can then separate code into more functions (usually just simple free functions without parameters) from there that become your systems. There's a bit of a rabbit hole designing reusable scheduling and event systems and whatnot but I feel like just simple calls to top level / free functions like this on a per-game basis seems a lot clearer and ultimately more flexible (it's just regular procedural code and you're in control of what happens when). I like seeing the backtrace / callstack and being the owner of things and then being explicit all the way up vs. entering from some emergently scheduled event dispatch system.

I don't think it's "too much work" if you really need it or are really interested in doing so and the approach results in incremental benefit vs. one big blob of benefit at the end. It would be quite a bit of work and not super automatic, probably. You have to implement all the files named 'wrap_*' in the Love codebase, kind of. It's definitely possible to start working on a game project to test with and do it incrementally, which is probably strategically the best path and also fun, which can make it worth it. Something like:

- Get direct Love C++ project working without Lua

- Make some level of test cases with it to get a sense of C++ API usage

- Embed squirrel interpreter into test

- Start binding API to squirrel and testing more and more of API incrementally from squirrel

- Port more and more test cases till it's all covered (or prioritize based on need of game(s))

The nature of the thing you are increasingly covering in Squirrel could be a particular game or a set of demos (maybe both).

Love does work through wasm in the pure C++ approach, have been using it on a project for a bit now. It's definitely better that way vs. also using Lua since you can't use LuaJIT and vanilla Lua running in wasm esp. with a GC isn't geat perf-wise.

Agreed re: C API -- would definitely need a wrapper there. The C++ API is also a bit less ergonomic in some ways than the Lua one (particularly how images are initialized) but also more in others (type checking and autocomplete is great). Lovr (topic of this post) uses LuaJIT's C FFI actually with a C API internally so it's more in that direction, FWIW.

What does "library-driven approach" mean in raylib's case and how is it different from Love's approach? In both cases I just have a CMake project that builds my application to either a native executable (including mobile in Love) or web, with all dependencies vendored, calling functions and using types from either API.

I do think raylib's C API fits this well and with the least impedance mismatch, having used all of them -- Love in Lua, Love in C++ and raylib in C -- extensively. eg. you can directly manage and render vertex buffers in raylib and go down to the 'rlgl' level, in constrast I found Love's default image rendering to have perf issues in web (it uses buffer orphaning) and I had to do something more manual with Love's meshes.