I have asked myself that question over the last few years and I have come to the conclusion that I really want these frameworks. In the case of jQuery, for example: you can program everything in Javascript without using jQuery. On the other hand, jQuery is not just a lot of code, it's tested code. It's not much fun having to add browser/engine-specific code to everything you do. Thankfully, the need for this specifically seems to be slowly fading away now that all browsers are to a certain extend on the same level.
In the case of such frameworks like Backbone/ember/Angular: these are implementations of MVC or MVVM or other such patterns, in Javascript. These make it easy to share data between web services you create, and, in the case of e.g. Knockout, bind data to controls on your page. You save yourself from having to write AND test all these things yourself. I have tried, for the first time, Backbone mere days ago, to see what the added benefit is. It gave me what's required to send data (in JSON format) to, for example, my controller, but what's nice about it is, for example, model binding.
It might not be entirely unlike PHP's situation: you can program your projects in plain PHP, or you can use the Zend Framework or Symphony or whatever you prefer, which adds a lot of (tested) logic that you otherwise would have to program yourself.
Do you NEED it? Not using these won't limit you, but you'll probably find yourself programming these things out yourself in a lot of cases, where many smart people have done this already.