Automated testing in Bevy
chadnauseam.com
chadnauseam.com
https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-62
I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)
This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.
But what's the alternative here, when your code depends on other services and other logic that might return complex data structures, or need to do dozens of database calls?
Should you go for primarily integration tests, which might not only slow everything down, but also force you to think about the possibility of either needing to mirror your current environment per test run (e.g. launch a database, run all the migrations, generate needed test data, launch your app against the database instance, run tests against it, clean everything up), or being limited to a single test run at a time with no parallelism?
I wonder if there is any books on game/software QA akin to Masters of Doom or Stay Awhile and Listen? Those pretty much skip over QA, it's usually just the devs playing their own game (which anecdotally is usually biased and of limited use).
The talk is from 2018 and references some of the trends in the industry towards more automated testing.
ECS is a paradigm where entities are the data-free IDs, components contain data associated with entities, and the systems operate on entities and components in batch, usually with database-like queries.
I have yet to find a good ECS reference, but Unity’s intro[0] gives some decent flavor for it. Note that only the DOTS subset of Unity (part of it anyways) uses an ECS. The default GameObject/MonoBehavior way of writing code in Unity isn’t part of an ECS.
[0]: https://learn.unity.com/tutorial/entity-component-system
For example, she spends a significant amount of time there explaining how to implement her component placement strategy while working around the borrow checker and at the end proclaiming how great it now is compared to a cpp implementation...
...not realizing that what she's just done is implement a custom allocator (not literally allocating memory, but the operations are the same: give me space for a component, free it, memory is latter accessed by an address instead of being some opaque thing hidden behind language semantics, etc) that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic.
> showed me that Rust is mostly recommended by people who don't know what they're talking about
You do realize that this is, like, colossally, stereotypically bad reasoning, right? It's ad hominem and improper generalization simultaneously, which is really quite impressive and indicates a singular degree of motivated reasoning.
Yes, lots of Rust evangelists are idiots (not Catherine West, u/spoiler did a decent job showing that). No, that doesn't significantly affect the technical properties of the language that make it useful or worth learning about. There are good reasons to ignore Rust, yours is not one of them.
You somehow managed to ignore all the points of the talk. Like literally every one of them, and I'm not even exaggerating. And then you somehow both managed to focused on irrelevant details and to talk out of your ass (like, at least be right about the details?), all to make a tangential point about how "sUpErIoRlY sMaRt CpP dEvs ArE" because they know this really basic shit.
But, I'll respond in case your misinformed comment manages to deter people from watching it. Let's start.
First you talk about "borrow checker issues" she's demonstrating in the OOO design. The point here was to illustrate the amount of coupling and "explosion" in size with this approach. The point was "OOO is probably not a good design paradigm for game dev". Then she continues how Rust surfaces this bad design paradigm much earlier than it would've happened in other languages/ecosystems. That's all.
More, what you call a custom allocator with tombstoning/generations are just things that exist with a lot of ECS implementations (regardless of language), and it's just become "tribal wisdom" at this point that this table-esque storage backend work really well with ECS (I dunno which one came first; older game devs might know more about ECS history/evolution than I do).[1]
Also not sure what you mean by "that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic"... Maybe I misunderstand what you mean, but it sounds to me like you're comparing working with `Option` as being the same failure mode as working with `void*`. That's just utterly ludicrous nonsense.
Her ECS implementation is... Well, it was made to fit inside a few slides for a 40-minute talk. So, yeah it's not great; there's plenty of small issues with it, but none of them are with Rust, or with ECS itself... Did you expect a production ready ECS library you can copy off of slides and use them in your next game, lol? If anything, her implementation has too much passing resemblance with what a C++ implementation would look like (which is something she's very familiar with and her starting point) than a Rust one. So, I'm just bewildered why this is the bone you decided to pick.
If you're interested in what a production-ish ECS implementation in Rust looks like, check out Amethyst's Legion or Bevy's ECS. Although, Bevy takes takes ECS a few... staircases (rather than steps) further with its own ECS. I really encourage everyone to read up on Bevy's ECS and check out the unofficial Bevy cheatsheet book; I'm certain at leat UE game devs will know how to appreciate how beautiful and ergonomic the design is, but others should as well). Aside: her ECS implementation is still safer than most C/C++ implementations used in published games, but that's a bit besides the point since that's the borrow checker doing its job.
Anyway, the irony is that she much better explains the pitfalls of her own implementation details that you do and she also explains why they're "fine" sometimes. And it was kinda implied that it's fine (from what I remember) because it's no worse than what you end up doing in C++ or Java when implementing ECS, but at least it's safe in Rust. She acknowledges that there's probably better ways to do it.
Here's a few links on some of the stuff I mentioned:
- https://github.com/amethyst/legion
- https://bevyengine.org/learn/book/getting-started/ecs/
Edit: formatting
[1] Edit 2: I kinda phrased this poorly. I'm not saying the best way to implement tables or just ECS is this approach, but it's common enough approach used to avoid memory bandwidth/allocations in games.
[0] https://www.gamedevs.org/uploads/data-driven-game-object-sys...
Entities are basically tables (just the table, not the data in the table), components are fields/data in them and systems are the SQL you write for them. Its obviously not a perfect representation, especially for components, but it gets the rough idea out there.
Maybe CSE is a better name for it. Its a longshot, but maybe entity being first in the list tricks people into thinking they play a big part, whereas actually they are, as you say, just an ID. Components are probably most important, then systems, then entities
Entity as ID; Components as Columns, but only kinda; Systems as Application code or functions
Tables in this database/ECS analogy are just tables here too, or memory rather. The data is often conceptually (and implementation wise) laid out as tables in ECS, too
In an object-based design, objects are monoliths, with fields. You get a reference to player, and each Player instance has attributes. When you work on a player, you get the Player instances as a whole.
In an ECS design, you'd create a player entity; the main differences are that the Entity is just a reference, but most importantly, you access each field (called "component") separately.
The bulk of querying is performed on components, which, importantly are all global; entities are typically retrieved starting from a component instance they own. This makes the radical difference with a traditional object-based design; example:
- in OB, you have, say, players and enemies; when you draw, you'll cycle the players, and enemies, and draw them.
- in ECS instead, you'll query something like "drawable sprite" components, and draw them
It's a very interesting design, as it forces to reason more generically. It tends to get messy very easily though, since it's harder to think in terms of location of a system (in the project).
The best book for beginners to understand ECS is (IMO) Hands-on Rust (https://pragprog.com/titles/hwrust/hands-on-rust).
> So your saying everything is a entity
Not sure about other ECSs, but Bevy has also the concept of "resources", which are globals; they're monolithic, and there's only one isntance of each type.
The storage part of ECSs could actually be considered a form of columnar db.
For example, locking is at column level, so you can have as many mutable accesses to a single entity (like a row) as there are columns (in reality, it's more complicated, due to access via iterators, but that's the principle).
Entity is Mario, the koopa troopers, the coins, the clouds, the floor tiles, the spikes, the mushrooms. All the proper nouns.
Component is the XY location in the world, ability to jump, sprite image, the height, are they affected by gravity. All the adjectives.
System is gravity - all entities with weight drop at the same rate until they hit ground. Another system may be the collision mechanic - if a mario entity lands on a koopa entity, the Koopa dies and some sound effects plays.
Functionally, an Entity is an ID. A Component is Data. A System is a function that operates on a set of Components.
In a sense "that's it". Having the hard separation between data (components) and code (functions) leads to some interesting consequences. For example the ability to more easily multi-thread System updates because each System clearly defines what Component type it needs const or mutable access to.
Most "things" don't exist in ECS. There's no singular "Mario" or "gravity". If you feel like you ought to split it into multiple identifiers for your systems (for example, treating each part of Mario like a different sprite), then you do so.
The hardest part of ECS IMO, is that dependencies between systems are super loosely defined since it's meant to be modular and the dependencies can only be tracked by components. The semantics of ECS systems are also highly technical (do you have hard sync points, how do you deal with recursive entities etc...) which means it can be quite fiddly to work with.
What ECS architecture does is offload the testing burden into data instead of code. The code itself can do the right thing in all circumstances, but as you componentize your higher-level behaviors, more business logic becomes bindings of data to other data. So after a 100% green test suite, you still have bugs and all of them are of the form of "mysterious runtime behavior". The engine code doesn't know how to automatically check for a "good data shape" or flag it as a bug; you can add checks, but it's project specific and asset-specific compiler development: a missing animation is a different kind of bug from a typo in a text prompt, or a hitbox that's slightly misplaced. Thus, for a lot of categories of assets, your cheapest test is still to periodically eyeball things and say "that's in spec" or "that's fishy".
ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.
So when maybe the first iteration 20 years ago was all fun and giggles "look a monkey could edit a few knobs and it all moves around fine", today it's a tangled mess that an ever rotating set of new people build on top of with no validation tool, no formalized syntax, no ability to test nor explain the holistic logic of.
So, as you say, we end up running the thing and looking if the output makes sense, the code working more out of luck than engineering.
New startup idea. Probably applies to any "hard to test" thing.
If you are doing everything by hand in C#, its super easy. The moment you introduce editor workflows or third party libraries, I have no idea how anybody can write sensible tests.
Using Unity is like a weird mix of no code tools and code.
Haha, great summary of my 5 years experience in Unity. Whenever I am too lazy I am switching to ECS/MonoBehaviour, then I regret of not having properly testable architecture.
I guess time to fix tests is expensive...
E.g. I had a test that hit a 3rd party sandbox that would go down occasionally. I could have mocked it to avoid this but I had way bigger fish to fry. The flakiness triggered a very specific error that we felt safe ignoring (other kinds of flakiness we would not ignore). The corresponding prod API never had issues.
Sometimes it's a select statement without an order by in which case even if it's not strictly a bug but just quickly adding an order by solves the flakiness forever so you might as well.
Sometimes it's a pretty terrifying race condition in the code that will absolutely crop up in prod.
Another big big title that makes use of automated testing is Riot’s League of Legends.
https://technology.riotgames.com/news/automated-testing-leag...
This is pretty weird blanket statement or rather it very much depends on what kind of game you are making. If you are making a casual single player experience then having glitches isn't that big of a deal. However if you want to make the next e-sports game then glitches matter a lot more especially if they can be abused to gain power - be it actually powering up your character in some way or having (near) invincible position.
Another aspect many game devs ignore is speed running. Having glitches in a single player game might not be that big of a deal for normal playthroughs, but many games have had long lives after their initial release as people want to speed run them and there glitches matter. If your game is broken it can create situations where speed running isn't fun.
This isn't true. Running more tests takes more time even with automated tests.
Game engines strike me as much the same thing for similar reasons. Yes, it would be really hard to not only have Unity as it stands today but also have all the requisite changes it would take to make it really testable, but without that you're really going uphill. It's an uphill battle at best and a sheer cliff face at worst trying to test these things. And of course when that's the case, every step your local engine programmer takes only takes you farther in the direction of being untestable because why would they bother structuring their code to be testable when the framework it is embedded in is untestable?
I have no solution. Only the observation that both of these systems (and also the browser, if you think of "the browser" as something other than a really funky large GUI) have problems from the very, very beginning, and until the foundation gets its act together w.r.t. testability it is always going to be a major uphill battle with anything built on that foundation.
(I'm not so idealistic that I imagine any world where a AAA game has 100% automated test coverage or anything, but even in my own limited experience in game dev and less limited, but still limited, experience in GUI programming, both places where I tried to leverage my extensive years of writing automated tests even in hostile legacy codebases, I found myself not merely stymied a bit, not merely inconvenienced a bit, but comprehensively blocked at every turn. An simple example: A table cell in a table widget that had a .SetColor, but no way whatsoever to fetch the color. Meaning my code to highlight rows matching a certain criterion could not be tested; in this case, for that reason alone. What a ^#&*$@'ing stupid reason to be unable to test some non-trivial code, at least from my perspective gained over years of writing a lot of other test code... and discovering plenty of errors in code of similar complexity in places where I could test it.)
Both Unreal and Unity uses ECS.
Entity-Compontity-System (ECS) uses these nouns differently; an Entity ends up being little more than an identifier for a large number of Components, which are data structures, and Systems are the code that modify groups of Components. "System" here is a noun, not a descriptor of the practice. The usual analogy is similar to a relational database (in fact, the point of the system is to achieve wide-scale performance like RDSes do): Entities are rows, identified by a primary key, Components are columns, and Systems are the SELECT/INSERT/UPDATE/DELETE queries which acts on the data.
Unity has a new-ish ECS framework under the DOTS initiative; Unreal has no parallel. Very few games are using these.
It uses different terminology such as fragments. Traits and others but its incredibly powerful and scalable.
This is a separate system then the ec of actors and components.
ECS has come to mean a very particular style of architecture. It's not a great name, but it's what the industry has converged on. Naming things is hard.
Unity's MonoBehavior framework uses GameObjects and Components. You don't have to squint hard to see that it contains Entities, Components, and arguably Systems. However ECS has come to mean an architecture that is quite different from MonoBehavior. So different infact that Unity is writing an ECS framework called DOTS!
Both DOTS and MonoBehaviors containing entities, components, and functions that operate on them. However the two Unity systems are extremely different. MonoBehaviors are not an ECS.
Then some people started talking about a single implementation style of ECS as if that was all ECS is, Unity DOTS is an example of that, but the base Unity is still ECS at its core.
Unreal's framework is a mix of inheritance and components. It's clunky AF. AActor I'm looking at you!
Neither Unreal nor Unity provide a mechanism to cleanly query such as "give me all gameobjects in the world contain this set of components". And they definitely don't provide a mechanism to perform that query quickly.
Unity will let you search from components within a GameObject, in children, or in scene. These queries are also slow as fuck and should generally be avoided.
The type of code you write in Unity and in a Bevy-style ECS is quite different. I think ECS is overrated in some ways as it's actually pretty difficult and non-obvious how to accomplish a lot of tasks.
I don't love the name ECS. It's just not a great name. However I've come to terms with it. ECS has colloquially come to mean something that is extremely distinct from MonoBehaviors. You're welcome to keep fighting the community accepted definition if you want. I don't think it's useful or helpful to do so.
But yes, everyone admits the names are confusing. But at no point does Unity or Unreal say their architecture is an ECS one.
"Component Systems" were popularized ~20 years ago. I'd consider Unity MonoBehaviors a typical Component implementation. Scott Bilas's 2002 GDC presentation "A Data-Driven Game Object System" helped evangelize the idea.
ECS is a new term that implies a handful of implementation details. The name is somewhat confusing, but it's not redefining a previously popular term.
I believe the first time ECS made an appearance at GDC was Blizzard's stellar 2017 talk "Overwatch Gameplay Architecture and Netcode". The term went mainstream in the years after this.
EC was the answer to God objects and deep, inflexible inheritance hierarchies.
Among other things, ECS is partially a performance play - it's more cache friendly. I'm not sure what all the tradeoffs are though, I've only really used EC.
A proper system, for example, would be able to add a new enemy entity every 5 seconds and switch the flag that the level is cleared once the total count of present enemies is 0. In fact, it would be quite trivial to implement it. But in Unity you would need some sort of an architectural solution to bypass inherent limitations of EC-S.
There are a few ECS approach and to this day it's still vague who does it best (depends on your goals).
One I found interesting is the "sparse entity component system".
My guess is what he meant was that the Bevy approach is cleaner to test upon.
https://docs.unrealengine.com/5.0/en-US/overview-of-mass-ent...
I realize that there's the whole "grouping" and "batching" concept in ECS paradigms, but I'm failing to see how that's much different from using the functions in `UWorld` that can query both components and actors of a certain class.
A small note, the code samples are a rather large font size on mobile. Setting the size to 0.9em for mobile widths works well for me!