http://zoompf.com/blog/2010/01/should-you-use-javascript-lib... http://statichtml.com/2011/google-ajax-libraries-caching.htm...
tl;dr: the URLs are too fragmented to get cache hits for commonly used shared libraries, so you end up having to do a DNS lookup and download them over a cold TCP/HTTP connection. This is slower than simply serving the 20-30KB of data directly from you own site.
Update even more recent stats from 2013. The Html5 boiler plate project decided against shared JavaScript CDNs: https://github.com/h5bp/html5-boilerplate/pull/1327