This is a process and we're giving up a lot of stuff like Google Analytics (however, it's debatable how useful that is with our target audience anyway). As we move forward, we're also combining (compiling) our resources into a single, cacheable SPA.
Any time you incorporate code from another site, you're:
a) leaking information about your visitors
b) increasing the probability of malicious injection
c) for required assets (JS/HTML, usually not CSS), lowering your overall site availability/reliability -- even if that third party site is often more reliable than your site
d) for less frequently used CDN assets or less frequently used sources, possibly also decreasing the speed of site delivery, especially if resources are requested less often than cache expiration (depending on type of edge caching)
e) similarly, possibly slowing down site by increased name resolution requests (this can be mitigated somewhat with pre-fetching)
f) increasing variation in load times (weakest link)
g) trading off latency for bandwidth
The biggest reasons for us, however, are (a) and (b). This is also why TLS should forced enabled with HSTS for many sites. Security trumps efficiency, especially when the differences are measured in milliseconds.