With good care, you can create libraries suited for your problem and keep code size and complexity down. Don't forget that when adding frameworks you're also adding their complexity to your project. Imagine having to debug the internals of angular... I shiver just too think of that.
The thing is that using frameworks is cool up to the moment when you reach a framework quirk or bug and need to dive into it as well, then you're no longer working in your problems domain but on generic framework land and that can be much trickier than building your own, specially if it is a solution made for a single problem.
But yes, I agree with you that if complexity grows enough you start requiring a special type of developer with high skills to keep this bespoke code running well.