That's not my experience, though game development is certainly a niche and Java may not be the top choice for that.

You might sometimes have to reach for SoA-like structures and reference them via indices, at least for the core ECS, but for the rest you can easily use bog-standard Java -- not everything has to be "ultra-fast, specially written java", just certain hot loops.