Why, frameworks exist for a reason and keeps us from reinventing the wheel. Many times a contributor to a framework has labored to make his routine as efficient as possible on the broadest array of platforms. Many times we can forgo a lot of optimization work by using the framework rather than rolling our own.
In some cases, yes, but I'd argue that in most cases probably not.
Frameworks are made to make development quicker and easier, but its at the cost of performance.
In any case, if you're concerned about performance, you should examine the source code of the framework anyways to see if it's doing anything unnecessary from what's required of your application.
$("#myid") vs document.getElementById("myid")
for example, $() supports a vast amount of functionality.
But for internal sites, storing my framework with my app lets my local web server cache it, instead of forcing users to traverse the network and go outside our firewall to Google's CDN.