You need to be careful with your wording here. The "MAJORITY" of the top 1m websites (based on Alexa rankings) don't use Google's CDNs (i.e. load a resource from googleapis.com):

http://httparchive.org/trends.php#perGlibs

...and that's across all versions of all libraries.

Quite how that correlates with how many of your first-time visitors will already have the library cached – because they happen to have recently visited another site that uses the same version of the same library – depends on how much your visitor demographic intersects with those sites that use the CDN. You then need to offset that against the DNS lookup time requires for the rest of your visitors to work out whether loading the file from Google's CDN makes sense.

If you're talking about repeat visitors, it doesn't matter where the file was served from, so long as you apply the correct cache-controlling headers.