You're identifying one of the (many) underappreciated cost in the RLWE FHE stack. The evaluation key problem doesn't scale, and no amount of hardware acceleration fixes a per-user storage and loading overhead that grows with the security parameter.
I think one counterpoint here is that you can still get value in using FHE in settings that don't involve scaling the number of users (e.g. B2B, public/private collab).