IMO https://ory.sh has completely changed this calculation. With the help of Ory Kratos it makes sense to roll your own auth.
What am I missing?
Authorization is a whole other ball of wax. You can sometimes get by with RBAC, but it is far more often entangled with business logic. I've seen a set of new companies that offer outsourced authorization like permit.io and cerbos, and for an app of a certain complexity, think they are worth evaluating.
How do you prevent a user from assigning themselves roles they shouldn't? Is there some kind of cage preventing escalation?