You don't NEED a framework, just like you could theoretically build a car from scratch.
If your app isn't complex enough to merit using a framework, then you aren't really building something worth talking about.
Frameworks do exactly what they say, provide a common framework for your team to work off of while they build a complex application.
I feel like most of the FUD around JS frameworks comes from simple people building simple apps and wondering why everything needs to be so complicated. Frameworks enforce structure, which becomes more and more important as your app grows in complexity.
Frankly it would be great for me if this train of though picked up steam, plenty of contract work from companies who can't maintain their "homegrown" frameworks anymore.