> I think you're basically set and don't need GC for games.
This is kind of a semantics question, I think. The vast majority of games need some sort of lifetime management, and it's just a question of who does the lifetime management and what mechanism you use for it - refcounting, a mark/sweep GC, freeing everything at certain points of time, arenas, etc. If you're using an entity/component system to manage lifetime you have a GC - you wrote it.
In my experience shipping games in C# compared to shipping them in C/C++ - you can do everything without the GC touching your stuff if you're really dedicated, but it's often not worth the trouble considering that any modern GC can handle scattered temporary per-frame allocations for you no problem with very minimal pause times, as long as you're thoughtful about it and the set of objects it needs to walk isn't too big. For example, if your data structures mix native data with pointers to object instances, a GC will have to sweep all of that data - splitting texture references out of a big table of draw calls means that the draw calls are now pure data and they don't need to be swept by a GC.
Personally I prefer always having access to a GC because it means code that doesn't need careful lifetime management can be simpler to write and doesn't have issues like double-frees hiding inside it - things like automated tests, configuration UIs, debug consoles, and things you run once at startup or when loading a level. You can often go back and optimize some of this stuff later, too - for example LINQ is a notoriously messy feature in .NET's standard library that allocates tons of short-lived garbage, but the compiler makes it possible to replace all those LINQ data structures with non-allocating ones without having to rewrite your queries - but doing that moves costs elsewhere.
If you're getting specifically harassed by pause times you're likely going to be paying costs with other systems, like if you use refcounting any time you touch that refcount you're burning cpu cycles and pushing other stuff out of cache (and the refcounting gets much more expensive if you have to use atomics for thread safety).