For an MVP or a prototype, I think it's okay to use an off-the-shelf tool. For something serious that will have long-term legs, I would do it myself. I hear all of your concerns and arguments and agree there are a lot of footguns. But again, having spent the better part of my adult life using and interfacing with these tools, I have an innate understanding of how to model auth correctly (separate it from the user, separate users from an 'org' or 'team' entity, etc).
You said it though, 'it depends' is really the right answer here.