How big is the "full" set of polyfills anyway? Probably small enough that a single download isn't an issue on even dialup?
--
That aside, polyfill ALL the things. Hopefully making it easier makes it common, which lets devs use a nicer version of Javascript.
Google's polyfills for custom elements that they ship with Polymer weigh in at around ~151Kb of JavaScript, around 10x the size of a small library like Underscore, and 4-5x times the size of an MVC framework like Angular. And they only cover about 5-6 specs (the 4 webcomponent specs and a few more to support them) Those are admittedly hard to polyfill features that will be bigger than most, but it turns out that polyfilling "everything" is hard and would add significant page weight.
Right now it's around 75 kB without minification and without gzip. However, this doesn't include any code for doing feature detection.
So, my somewhat educated guess would be: around 30 kB (which is comparable to jQuery).
Nice guess :) that seems not too bad, tbh.