For me, on Firefox 30, I can see the basic structure of the page immediately, then my browser freezes for ~1.5 seconds, and I see the rest. I checked out the front page in FF's devtools, and could only see two uncached network requests, both to youtube.com. So I am assuming that Polymer itself isn't hitting the network. Furthermore AJAX doesn't freeze the browser, so I can only conclude that it's the actual JS that is causing that lag.
On Chrome 35 the freeze is for a much smaller duration but I can still feel it.
I'm currently on my (fairly beefy) desktop, so I imagine it's even more noticeable on an older laptop or phone.
EDIT: indeed, from a cold cache, it's even slower, and wow there are a lot of HTTP requests.