I agree, frameworks can be beneficial to productivity.
I think my feelings about them mostly have to do with a lack of control over your own creations on multiple levels. Personally I'm a bit slow and I can't hold a lot of things in my head at once, so all of these nice-to-have code quality principles are actually pretty important to me.
I can deal with a library being a bit of a black box, as long as I can make some basic assertions about it such as which parts are thread-safe, what sorts of exceptions to expect and so on. Usually it doesn't take too long to learn how to use it well enough.
With a framework, the learning part is much more convoluted. The tutorials and documentation are usually written for "task-based" learning, i.e. you have a specific goal and the documentation just tells you which buttons to push, which levers to pull in that case. And stack overflow will fill in the blanks. When your (preferably very small) app is covered by that, you're golden.
But when you're doing something more complex or have a specific issue, you need the "model-based" approach. You need to understand the architecture behind the framework, and usually that means you'll have to read most of the source code with zero guidance. And you'll probably have to wade through lots of code that is only there to avoid some boilerplate in "hello world"-level projects. Most people won't do that, and then they're stuck with an incomplete mental model based on a patchwork of experiences. It can get pretty cargo-cultish sometimes when you end up configuring something you don't understand because in your experience it makes things run better.
I admit that I'm oversimplifying/overgeneralizing and frameworks are popular because they tend to help people meet deadlines. And I use them too. But it feels wrong, like I don't really know what I'm doing.