I think the main difference between libraries and frameworks is that libraries interact with the language, whereas frameworks generally exist in their own little world, attached to but not embedded with the language itself.
That is, frameworks have an ecosystem associated with it, whereas libraries are the ecosystem.
And then the issue becomes that the ecosystem is fragmented between frameworks and the language, and things become a lot less general (suddenly you need interop code to switch between the framework’s world and a language library, akin to ffi), and you start getting locked into the framework’s world view, which tends to be quite limiting. Particularly in that when what you want, and what the framework wants, conflicts, everything goes to hell.
I think in general frameworks should be avoided when it can be composed out of libraries, in the same fashion that libraries should be avoided when it can be composed out of simple handwritten code (eg avoid 10-line libraries, perhaps accept 1k, definitely consider 5k, assuming you’re actually making use of most of it).
Its only when the benefits are significant should you reluctantly accept a framework. But most go the other way — they start with frameworks and reluctantly leave it, not realizing there are distinct and heavy constraints that make the framework so convenient in the first place.