Polymer and Angular 1.x do come with a lot of bloat. I wouldn't call a MD text entry field "simple". Material Design was released ahead of web technologies really being able to manage it well, at least on mobile.
Polymer itself is 47KB. If you add the web component polyfills (16KB), that's 63KB minified and gzipped. I tend not to consider the polyfills because they're not Polymer and are a stop-gap that goes away over time.