I'm on the other end of the spectrum.
I mean, I see the benefit of such "roll X quick by adding a gem" for a proof of concept, a demo or even an MVC.
But even for the latter, it breaks down rapidly. Such gems bring their own libraries. They force you to "do it their way". A gem like devise comes with its own addons (omniauth) that paint you even quicker into a corner.
I've consulted and developed at many Rails codebase where most of the problems were caused by years of development on top of such gems. A weak, wobbly base, and the entire castle built on top of that. Where 90% of the devise features were unused or worked around. Where more code was present to customise and work around devise-isms, AASM-weirdness, CanCan-troubles, than a complete "roll your own from scratch" would have ever caused.
So, my "devise advise" always is: sure, use it to get your product to market rapidly. But be sure to pull it out and replace it with a focused, fitting architecture, sooner rather than later.
Such gems fit very well in the problematic development-workflow where one "searches for an existing gem which offers some version of Foo" rather than "figuring out how we want to do Foo, then look for a gem that does it our way".