Angular can be quite fast compared to jQuery in an app of substance. It is not constantly reading/writing to the DOM in most cases, which saves a lot on the performance front. It is also a smaller library than jQuery, and you're not naively applying global or complicated selectors. Plugins rewritten to be pure javascript + Angular also can have a much smaller footprint & be more powerful to boot, such as the Angular UI Bootstrap project (5 KB minified & gzipped as opposed to 15-35 KB? I forget Bootstrap's js size) & contains all of the power of Angular to modify your functionality with them & more.
How do you refactor out bloat from a framework without throwing away the framework?
This is my whole point about not using frameworks. They're a siren that WILL eat you in the end.